Wikimedia monthly activities meetings/Quarterly reviews/Editing, October 2016

Notes from the Quarterly Review meeting with the Wikimedia Foundation's Editing team, 26 October, 09:00–10:00 PDT.

Please keep in mind that these minutes are mostly a rough paraphrase of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Present (in the office or remote): Trevor Parscal (head), James Forrester (product lead), Volker Eckl, Neil Quinn, Santhosh Thottingal, Michelle Paulson, Katherine Maher, Joe Matazzoni, Joel Aufrecht, Wes Moran, Subramanya Sastry, Runa Bhattacharjee, Pau Giner, Amir Aharoni, Ed Sanders, Lindsey Anne Frankenfield, Mark Holmquist, Jaime Villagomez, Sherry Snyder

Mission

James: The Editing Dept. has clear audience in mind: everyone who edits — heavily, attends Wikimania AND those who do not, e.g. only potential editors.

Overall, our work falls under the annual plan Product program 4. 

Goal 1: Maintain & incrementally improve current content creation & curation interfaces

2017 wikitext editor

James: Consistent behavior is the goal: 2017 wikitext editor and VE will have a consistent editing experience.

Goal for the quarter was to bring it out as a beta feature. Held back for bugs; those have now been squashed, user research is being done; should come out as beta feature soon.

Katherine: Can you say more about those rollout plans?

James: Potential to be fairly disruptive, so we're taking it slowly with lots of opportunity to opt-out and steer outcome. The initial Beta Feature will be a wholesale replacement of the wikitext editor, for those people who opt-in.

Katherine: Plan to totally replace the current wikitext editor?

James: Will actually be our seventh editor (plan is to turn four of them off relatively quickly, and eventually have only one). But no plan to turn it off for years, as community members slowly move over. For example, existing gadgets will not work with new editor. 

Katherine: would like a presentation on new wikitext editor before rollout.

James: More advanced editor. Should be a better experience. But will still be disruptive.

Wes: Goal is consistent interface, right? Could you report later on the rate of current switching between editors, so people understand usage patterns?

James: Yes. Note that there's also people "switching" editors between content and e.g. talk pages, which is also important to address.

Goal 2

"Improve editing experience by reducing technical and product debt"

FileAnnotations

James: Structured metadata for Commons is a really big goal. So big there's still heavy discussion. For us, key part of reducing technical and product debt. Currently a Commons gadget, which lacks most of the structured features that could bring significant value. Only available on Commons, only available on desktop. Working to replace the gadget with something that covers existing use cases and also expands.

Rich experience for annotating images with links to other pages, etc. Pretty early in the design phase. Probably not going to end up with a design that looks like this. Reading interface for these annotations will be different. Planning to work on this with the Reading department.

Work for this quarter is to send out for user feedback.

Trevor: importance of this just isn't this feature. We want to start engaging with Wikimedia Deutschland on structured data, Wikidata. But we need to take baby steps to start learning. This project is a good opportunity for this; since the gadget is already liked, this is low risk.

James: Author and maintainer of gadget on Commons is eager for us to replace this system he built. Next steps are to release this in a small way; convert 5 or 6 images and let people decide if they're happy with this. Also, tagging for different kinds of content: video clips, etc.

Language converter in Parsoid

James: Language converter is used for simplified-traditional Chinese, Kurdish (written in three scripts), etc. Until this quarter, Parsoid didn't recognize language converter markup. Now understands it much better. Will provide a better reading experience in these cases. This doesn't get us to the point where you can edit zhwiki with VE. That's upcoming work. Includes a lot of design work; what does parallel text editing look like? No one else does it. Android app uses Parsoid HTML for read mode; with this, should hopefully be able to start doing this with Chinese, etc. as well.

Tidy replacement

James: Tidy cleans up invalid HTML input by our users. Pretty fast, but the only great thing about it. Outdated. But migration is difficult; have millions of pages written assuming that they'll be output using Tidy. Want to do it slowly, give users opportunity to fix pages one by one. ParserMigration extension allows this.

Next step, deploy to production on opt-in basis. We expect vast majority of pages to work identically, but we don't want to spring any surprises on community members. In progress.

Style guide

Volker: Chose to get work from next quarter accomplished this quarter. Very successful cross-team collaboration; involved people from Reading, Discovery, and Editing. Came up with basic fundamentals. Actual documentation still in the works. First part is the new colour palette (screenshot); VE is first consumer. Has also been brought to Portal and MobileFrontend. Better for accessibility. Have involved community early on, with positive feedback. 

James: Accessible not just for people who are colour-blind or have impaired vision, but also for people using phones in bright sunlight. Important to serve all our users.

Wes: Great to see this consistency. 

Translate

James: Last slide on reducing tech and product debt. Translate extension is both Language's most used extension, but also the oldest. Critical to cross-language support. Team felt they were falling behind on supporting it. Decided to focus on their backlog of code review, etc. Now have a really great set of improvements, including better mobile support (some languages spoken by people with largely only mobile phones). 

Runa: One important thing was to have a code review statement of intent. Led mostly by Niklas (creator of this extension). Clear guidelines about what our responsiveness is, etc. for reviewing patches, so the person contributing the patch knows. Will tweak statement of intent based on our experience, may extend to more of our projects.

James: From the graph, you can see a clear improvement in backlog of patches. Not just numbers going down; more things are actually getting fixed.

Goal 3: Invest in new types of content creation & forms of curation & collaboration tools

ReviewStream

Joe: Notably, this is the first project productizing the ORES AI system. Priority is to help turn new users into productive contributors by creating a better edit review process for them. First step is to find good faith new users. Have collaborated with Analytics, Research and learned a lot.

Have two specific tools we want to build next quarter: (1) ReviewStream, which will make it easier for third-party developers to build review tools by providing a stream similar to RCStream but enhanced with review data, and (2) an enhancement to the RecentChanges pages. Should improve edit review for anyone using that page and enable better targeting of their work. Communicating this page's capability has turned out to be difficult; invested time in designing a good UI. Piloting a new filtering interface that's more clear, more scalable for new filters, more powerful for users. We have a strong prototype now that's been through two rounds of user testing.

Joe: The demo video shows how reviewer who's decided to help new users might use the prototype. She opens filter panel; reviews tools; selects filters to find problematic edit and highlight most problematic edits; filters for good faith as well, and highlights newcomers; looks at tooltips to remember what she picked; and looks for where colors mixed (they dots are a key).

Michelle: How do we label good-faith and bad-faith edits?

Joe: We use Aaron Halfaker's machine learning service. The predictions based on sample of edits that were hand-coded by editors. 

James: That coding is an ongoing process (patterns can change). I'm very excited about this project—force multiplier for people already using this page.

Joe: RecentChanges is just a start. I don't anticipate it's the only, or even the best, place for these tools to be. We're piloting a new tool type.

Trevor: Similar to what we're doing in Multimedia. Providing a new feature that builds on an existing, proven feature, but integrates a new technology. Lot of learning coming out of that. 

Katherine: Will you be tracking adoption? Obviously it's a very complex, elegant solution, but I look at this and say "gosh, how would I ever know how to use it?"

Joe: Will probably be an expert tool. Edit review in general is not a beginner activity. We are instrumenting all these tools to see if people use them. This is one of these projects where it's much harder to measure high level effects (e.g. on new user retention) until you have more pieces of the system in place.

James: Also, the prototype in the video starts out with three filters selected by default. These are already default options for RecentChanges, but this better exposes that they are, and that they can be changed. Likely to lead to significantly increased user understanding of this page; for existing users, the main use case is just refreshing the page.

Joe: This is confirmed by user studies of long-time RecentChanges users. Some weren't even aware that filters already existed. Regarding the prototype, people have been saying "wow, this is really aimed at my workflow" and also "wow, this is really fun".

Michelle: Which languages will this first go out to?

Joe: We'd start somewhere among the ~20 wikis where ORES predictions are available (most of the big ones). Not sure which one will be first.

Content Translation (CX)

Santhosh: Want to address missing pieces in CX to prepare to go out of beta. Template translation is one of these. Addressing this in multiple steps. Design for translating template parameters done by Pau. Demo in next slide.

This is first iteration. Next step is adding new translation. Parameters will be machine translated wherever that's available.

James: Language team has also been working on fixing show-stopper bugs (e.g. publication failures). Good work there.

Lisa: How many language is this available in? 

Santhosh: CX is available to all language Wikipedia but machine translation is only available for a much smaller set.

Michelle: Is this coming out of beta now?

Santhosh: Not yet—we're working on it. Other steps beyond template translation required as well.

Santhosh: Graph shows usage of CX is in a steady state now, with about 2000 articles created per month. Campaigns will be coming in a few days; can be used to organise translation campaigns (Wikipedia Asian Month). We expect a jump in use from that.

Metrics

Neil: Non-bot edits. Thing to note is that overall there has been an increase of 15% in the last year. Still strongly suspects that it's related to non-human editing. So don't read too much into this number: we want to replace it as soon as we can.

Mobile editing continues to increase.

IP edits have declined, due to gradually increasing restrictions.

Overall active editors over last five years show a gradual downward trend. I've broken this into sub-groups. 

Subgroup 1 is existing active editors. Core community. Probably doing bulk of the work. Very flat from 2010 to 2013. Drop in 2013. Then rising a little from 2014. What explains drop in 2013? Don't know. Main take-away is stability. 

Subgroup 2 is new active (registered) editors. (First month after registering). Very volatile. Decline 2011 to 2013, then rise to 2015 Then dramatic drop in 2015 -- switch to anon mobile editing. Then this summer fairly dramatic dip which we've partially recovered from. Some seasonal influence. But need to investigate. 

Subgroup 3 is second-month active editors. A new stat. Lower number than first month. Some decay from first month. Concerning that this number is also quite volatile; suggests that declines in new active editors are affecting their conversion into existing active editors (which would not be the case if, for example, the fluctuations were just among the least-interested new active editors).  

Survival rate: second month as a % of first month editors. Around 25%. But if you calculate differently, looking only at people active in second month who were also active in their first month, only about 10%.  Pattern in 2013 onwards of a mid-year dip. 

Katherine: Can you talk about how this bears on your strategy? 

Neil: Core challenge is maintaining and growing existing editors. There are three main avenues for addressing this: 

• First, can improve productivity of those existing editors. Some evidence to suggest this is fruitful line of work. Research shows that amount of productive content added per hour of labor has gone up (although overall productive content added has declined slightly). 
• Second avenue is to increase retention: from one month to next, only half of existing active editors stick around. Could try to increase retention. Hard, because this rate is pretty stable. Also, biggest answer people give when asked why they leave is "lack of time." Users who leave because of harassment might be a productive subgroup to focus on here. 
• Third way to increase: acquire new users. On one hand, this is something we've focused on but not seen strong results. On other hand, it seems like new users is most volatile of three, so perhaps more opportunities for us to influence that. 

Katherine: We're at time for this meeting. But I'd like to hear more in the near future about two questions: of those three avenues, how would you describe your current approach? Also, how do these editor trends look when you break them down into different communities?