Wikimedia Foundation Annual Plan/Quarterly check-ins/Audiences1 Notes October 2017

Present (in the office): Toby Negrin, James Forrester, Abbey Ripstra, Sarah Rodland, Heather Walls, Volker Eckl,

participating remotely: Joe Matazzoni, Lindsey-Anne Frankenfield, Sherry Snyder, Subbu Sastry, Niharika Kohli, Ed Sanders, Deborah Tankersley, Amir Aharoni, Maggie Dennis, Joady Lohr, Neil Patel Quinn, Dan Garry, Eileen Hershenov, Stephen LaPorte, Roan Kattouw, Erica Litrenta, Katherine Maher,

Notes:

Primary note taker: Dan Garry
Secondary note taker: Accepting volunteers!

James: Welcome! This is Contributors and Audience Design. Abbey will be talking about New Editors, which is towards the end of the deck. Joe will now talk about the Edit Review Improvements.

Edit Review Improvements - Program 5, Goal 1

Joe: After five months in beta, the new filters for edit review are standard on almost all wikis. A few wikis have small issues that we are dealing with. We have also extended the new tools to watchlist as a beta feature.
Joe: Now the project is leaving its more active phase, let's talk about why we started it. We were influenced by the research by Aaron Halfaker into effects of harsh edit reviews, and the machine learning platform ORES to help mitigate that. For the first time, reviewers can know when they're reviewing the contributions of good-faith newcomers, who might have problems with their contributions but are editing in good faith. We also saw the potential for improving this for all users, and we got some experience working with machine learning.
Joe: This was released a few weeks ago. Early adoption is strong. People are trying the new features that are not top 25. People are also trying the feature of saving sets of filters, which is a big benefit for power users.
Joe: This was the first project to take ORES out of the labs. There are some lessons learnt from working with AI. First lesson: AI is magic! It does things that computers really aren't supposed to be able to do! In testing, we discovered that if people don't know how something could work, they will assume that it doesn't work, and will avoid it. Providing explanation on how they work is very beneficial.
Joe: Machine learning concepts are usage patterns are unfamiliar to users. Expect to spend a lot of time iterating on design. Daisy Chen spent a lot of time performing multiple rounds of research that was very beneficial to this project. The system we ended up with makes the tradeoff with these kinds of systems, e.g. tradeoff between precision and recall. It may still present too many choices for some users. This (on the slide) is a design we haven't built, but would like to, where we have pre-sets for people who want to do certain things. Balance between providing configuration for power users, but simpler route for newer users.
Katherine: I <3 the idea of recommended variants
Joe: Way to provide help is to provide "just-in-time" help, e.g. telling people that they've chosen some conflicting filters when they do that. It helps people get unstuck. Anticipating the problems users will face with complex tools is itself complex, and creating this takes more time than anticipated.
Joe: Users are very sensitive to load-time changes. Speed is crucial. We underestimated technical differences and usage patterns between recent changes and watchlist. Testing for speed is tricky and time-consuming. But users will notice! The difference was between half a second and a second, but people did notice it.

Structured Discussions fixes - Program 5, Goal 3

Not discussed in the meeting.

Editor modernisation - Program 3, Goal 3

Not discussed in the meeting.

Contributor tool maintenance - Program 3, Goal 4

Dan: We've been working on converting CX to the visual editor platform. We hope that will help us pay down technical debt and standardize. We will be wortking this coming quarter on making sure all features that work in the existing CX work in the new version, and we will be turning it on as a beta for wikis that request it.
Dan: CX is a widely-used beta feature which helps fill in content gaps. It's currently in beta and will be maintained in beta for the time being.
Dan: Month-on-month statistics. Articles created increased in July and decreased in August—this is normal seasonailty which we saw last year. Deletion rate in Sept spiked because one user on one wiki created a bunch of bad articles; excluding that it was pretty normal. Of users who try the new tools, about half successfully publish a translation. As with all such tools, there's a small number of users who are very prolific and produce a large number of articles.
Toby: I appreciate this project has been a difficult one for Contributions, and I just want to acknowledge the team coming together to have a dialogue and work through these issues.
James: In particular, team members who've been working on this for a long time and new ones have come together, across many different time zones.
Roan: https://www.reddit.com/r/AskReddit/comments/72itfh/how_do_you_contribute_to_society_besides_having_a/dniysct/ is a nice bit of praise about ContentTranslation and VisualEditor from David Richfield (a prolific South African Wikimedian)

Other update slide not discussed in the meeting.

Consistency and Accessibility - Program 3, Goal 5

Volker: Goal is to modernise user interface technologies for our desktop and mobile platforms. Continuation of style guide. We finished initial implementation of design elements.
Volker: Designing and implementing first-class mobile support in widget library OOUI. Implementing based on analytics data. We've been testing based on browser support metrics.
We're supporting other teams, and responding fast to feedback from volunteers and other teams. Example here is of Content Translation to use OOUI.
Volker: Strengthening accessibility and ensuring consistency across areas such as chapters. Example of this is the standardisation across a few different products as a result of this work.
Volker: Performance: consistent interface. OOUI is used in 2017 wikitext editor, Content Translation, edit review. Upstreaming findings from user research into the library without blocking rollout schedules. Highly accessible; removing issues for people using screen readers. Strengthening the brand of the Wikimedia movement using the standardisation. Important from both branding and accessibility standpoints.
James: Thanks Volker, and also all design team and engineers that keep the work going.
Katherine: Saw a thread on Wikipedia Weekly about engaging blind editors. The work you're doing is critical for accessibility. Thank you!
Joe: At Wikimania, in our session, we had a blind user using our review tools. Even a complex system like that is available for people to use! There were a few small issues that he helped us understand, but it's amazing that we've done as well as this.

Parsing improvements - Program 3, Goal 6

Subbu: Worked on two different projects. One is migrating from HTML4-based Tidy, to HTML-5 based Remex HTML. Support editors using linting tool that helps users find errors in wikitext. Work was done in collaboration with Community Liasons and MediaWiki Platform. A bunch of coordination with editors was required, and Erica and Sherry helped a lot with that. Second part is to support multi-script wikis (e.g. Chinese, Serbian), Parsoid did not have support for these wikis. Initial support was added for this. This was required by the Reading Team, e.g. mobile apps do not support Chinese Wikipedia variants with Parsoid (they use old APIs for that), and this helps with that. This quarter we will continue with both these projects; replace Tidy on more wikis, and figure out process to roll out on all other wikis. Language variants, implement the API endpoint that lets the Android app, and anyone else, to use the language variants.
Subbu: Context: in terms of why we're doing this. We have two parsers: PHP which has been around forever, and Parsoid initially developed for VE. Parsoid is used by Android app, Flow, Kiwix, and more. Reading plans to use Parsoid for more of its product. We want to move everything to use Parsoid. Parsoid is based on a slightly different approach to the PHP parser; it looks at it more structured, and exposes that to clients, but it breaks rendering on articles that have broken wikitext. The process: making fixes to Parsoid so it's fully compatible with current PHP parser, and minimising the rendering differences. Language variants, and supporting extensions, and fixing long-tail rendering issues. There are lot of rendering differences between PHP and Parsoid, due to differences between HTML4 and HTML5. By the middle of next year, we expect to have that done. We're creating tools for editors to help them fix broken markup, to identify broken markup and identify how to fix it. There is a fair agreement that Parsoid should replace PHP parser, and there are discussions about the best process to do that.
Subbu: We announced mid-July that we wanted to replace Tidy. Over 300 wikis have no Tidy-specific linter errors, and a bunch have been making fixes using the linter. We've rolled out Remex HTML on a bunch of wikis, e.g. Norwegian. A lot of wikis have been making excellent progress on fixing issues, and we expect they can migrate to Remex in a few weeks. Swedish wiki is using a bot to fix mistakes, and now we want to figure out how to get bigger wikis to make changes to fix it. Language variants: Parsoid supports this, and VE was changed to support it as well. Once the API is complete, Android app can use this too.
Subbu: Snapshot of how the linter looks. High priority issues are ones that block us from replacing Tidy. They don't have to go down to 0, but it must be a small number. Five categories were identified for the blockers. Medium and low priority are not related to our immediate goals, but will come along once we work on our project to streamline wikitext as first class language with fewer errors.
Subbu: We've learnt that wikis have a LOT of markup errors. Literally millions of errors. Without fixing these errors, it's hard to evolve wikitext to support new features. If you provide editors with the right tools, they can and will help. We must go about this incrementally. We've found this process works very well. We've built tools for both editors and us to help.
Toby: Thank you for this work that sometimes goes unnoticed but is critical for a fast experience for our editors, and thanks for the discussions to migrate.
James: Thanks to Maggie and her teams for helping us work on this, and it's great to see how many millions of fixes have been made!
Maggie: They enjoy working with you, and enjoy working to make sure the wikis feel included!
Amir: Thanks to you, your work make Content Translation possible. The linter is really useful for people, and there are a lot of editors that are interested in fixing these things. Sometimes fixing one template fixes thousands of errors.
Subbu: I should also call out Kunal's help with the Linter in our work even after he moved from our team to the new MediaWiki Platform team.

Metrics preview

Neil: There will be a full metrics update presentation in the coming weeks. Here we'll give you a quick preview!
Neil: Monthly active editors, one of our most important movement-wide metrics. Active editors is essentially flat, as they've been for several years, at around 81,000 accounts per month. However, new active editors has been slowly decreasing. It's a volatile metric so seeing trends are hard, but it's gone from around 20,000 3 years ago to 17,000 today. Half of this is due to the introduction of anonymous mobile editing in 2015, which meant that new editors on mobile could edit without registering and therefore without being counted in our editor statistics. The other half is not yet understood..
Neil: investigating that is high on our list of priorities, but we're limited by resource constraints. I'm spending about 80% of my time co-leading New Editor Experiences with Abbey, which is an excellent strategic investment in research but limits our ability to make more tactical investments at the same time. We're working on expanding our resources.
Neil: Existing active editors remains highly stable.
Neil: Also working on a new metric, new editor retention rate. Looking at which accounts have made at least one edit in the 30 days since the day they registered, and what %age of them also edited in the second 30 days. The global number (12-month average) is 5.9%. Doesn't quite match up with active editors numbers because this definition only requires 1 edit, while active editors requires 5. Considering tweaking these parameters, but have to have discussions about that. New metric made possible by Aaron Halfaker and Analytics Engineering.
Neil: Could benefit from standardization across WMF. L&E team is looking at editor retention for programs, but they use differerent parameters (3 months vs 6 for example). Standardizing this could benefit both of us.
Neil: Global new editor retention rate does change but stays between 5% and 7%.
Neil: Broken out by wiki, the rate is much higher on Japanese Wikipedia and Chinese Wikipedia than on others.
Neil: Recurring metrics are in the board update deck and the mw.org Audiences page
Toby: Thanks Neil. This begs the question of "What should the retention number be?" In other industries it's considered to be around 30%. Right now, only 1 in 20 editors that starts editing then edits in their second month. Helping to boost that number is an area of focus for the team.

Audiences Design

Abbey: Human-centred design process that both New Readers and New Editors go through. We do research, we learn about the context and people, do analysis on the data collected to see what it means, see patterns, do synthesis, explore concepts, and then go into implementation, building testing and implementing solutions.
Abbey: This slide describes where New Readers and New Editors are in the process. not the exact same process, it's high-level. New Readers are working on access and affordability, and on awareness. On awareness they're heavily into implementation. For access and offline, Jorge and Anne are working on strategy for that, and moving into phase where concepts are explored.
Abbey: New Editors aims to learn from two medium-sized wikis to try ideas to better recruit and retain new users in those medium-sized wikis. Neil and I started project in February, conducted research in South Korea and Czech Republic in May and June. We built ? personas. Wikipedia's prominence is greatest strength and greatest weakness. New editors challenges are not technical, but conceptual, and struggle to learn about policies and how to shape content in the Wikipedia way. We worked with Legal to make sure report was appropriately anonymised. The report is now published, and the personas. We're assembling a cross-departmental team to work on this, to support collaboration and make sure ideas aren't restricted to one discipline, and create better solutions, e.g. how could there be engineering solutions to support programmatic efforts? First workshop has been held to pull people together, and we've gathered open questions. We're designing the rest of the workshops to decide which of the 11 findings we're going to focus on. We're working with Korean and Czech communities. We have an ambassador in Korea to help us communicate with the communities, and he will work with them to learn what their top three priorities are and why, and it'll help us know what to focus on. We're hiring the Czech ambassador now. We want to take community priorities into account when defining our priorities.
Abbey: These workshops are designed to be fully remote to make sure everyone can participate evenly. We're doing them virtually and synchronously, collaborating across time-zones. We're focussing on Korean and Czech. Last two workshops are designed, then we'll work on concept generation, pull communities in, then do evaluation of those concepts to decide what they should do.
Abbey: Anne, Jorge and I worked on prescriptive and descriptive value webs. These frameworks are to visualise the context, identify entities working in that context, and the value flows between them. Anne and Jorge developed a set of strategies. There's a prescriptive value web to see what needs to change to support that strategic possibility.
Abbey: This image shows what needs to change using the value webs.
Abbey: We realised we needed community ambassadors a little late in the process. If we do this again, we will hire them from the beginning. They will be useful right from the start.
Abbey: Did a legal review for the report. In future we'll have release options for people, so that they can decide how we use their data about their wiki identity. From the beginning, we wanted to release the corpus under open access, but we decided it's better not to do this. The balance between openness and protecting personal identity, so we wanted to be cautious and protect personal identities.
Toby: Thank you Abbey and Neil for leading, working cross-functionally, and taking us above and beyond. I'm excited to see Contributors working on shared problems and solutions.
James: I'm very grateful for taking my comment about having tons of remote staff, and committing to fully remote processes.
Abbey: It's been a big challenge, but it's been worth it. We think other teams can benefit, and we're exploring tools for that.
Toby: A lot of these methodologies assume people are in the same room, and this is really difficult to do remotely. It has been successful, and it will become embedded.
Abbey: Visual aspects are really difficult.
Katherine: I'd love to see you write about that methodology once you've figured out the best practices!
Abbey: Documenting process of contextual queries and learning how to do that in a better way.
Toby: Initially we relied on external consultants, and now we're bring that expertise in-house so we can do ourselves and become more self-reliant.
Heather: Allows it to be more integrated with other people's work.
James: This only works when people have the ability to engage, and anything we can do to open this up to all our colleagues (Foundation, chapters, communities) is great.