Wikimedia monthly activities meetings/Quarterly reviews/Editor engagement experiments/September 2013

The following are notes from the Quarterly Review meeting with the Wikimedia Foundation's Editor engagement experiments (E3) team on September 6, 2013, 1pm–4pm.

Present: Howie Fung, Steven Walling, Erik Möller, Sue Gardner, Dario Taraborelli, Tilman Bayer (taking minutes), Ori Livneh (from 2:30)
Participating remotely: Matthew Flaschen, S Page

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Agenda

Review of Q4 Plans (What we planned on shipping)
What was actually shipped
What we learned about our users

Our 2013-14 Target and Goals
The future of the team (Steven will talk about the mandate of the team, how that's evolved, and what the mandate could/should be going forward.)

Slide deck

Review of Q4 plans

Steven: Welcome
This is our third quarterly review. Not so tightly tied to fiscal year quarters
Changes to the team since last time: Added Dario (shared with Analytics), Pau (shared with Language Engineering), Munaf departed.
last review: had soft launch of account creation, 5th iteration of GettingStarted
had planned:

0-1 editor features: rapid prototyping of GS. Happened less rapidly than planned, due to shift in who was working on it
- Guided Tours outside GS: not done
- use Echo notifications: done

1-5 editor features:

- suggest next steps: not done
- build GS into site nav: not done
- SuggestBot logic re-implementation done, but not evaluating editor's previous edits as planned
data and infrastructure:
- Eventlogging support (Dario): done Wikimetrics. Sue: Wikimetrics has had good response, no? Dario: yes. Sue: E.g. Frank said he is really happy with it.
- (Steven:) reimplemented campaign tracking (i.e. where new users came from)

What we shipped:

GS v5 (100% on enwiki since June)
New account creation and login on all wikis (since May)

Video snippet from user testing

Prototyping:
Used static prototypes and remote usability tests
e.g. to decide about design of progress bar that counts edits 1-5 (shows user testing video)

Data and tools
- Campaign tracking
- CoreEvents (separating things out from Eventlogging. e.g. tracking of preference changes)
- NavigationTiming (clientside performance), lots of MediaWiki Vagrant improvements, supported first A/B test of VisualEditor (this took us a lot of time)

Matt: and the gender signup survey
Problem: WMF doesn't have capacity for concurrent tests by 3 features teams (VE, Echo, E3)

What we learned about our users

Acquisition:

Campaign tracking (where signups come from - e.g. from login page, or mobile upload CTA. Howie ). Erik: do we plan to create a dashboard for that? Steven: kind of risky - anyone can create new campaign strings, e.g. to vandalize the dashboard ;). Howie: about a third of signups now from a campaign [?]
Gender survey - gender gap among signups. Originally the motivation was for VE.

Not quite supporting theory that gender gap is due to many women joining and then being rejected disproportionally
Sue: Do we know of general research on differences in propensity to self-report gender?
(Steven, Sue, Howie, Tilman: discussion about gender self-reporting, survey participation biases, etc.)

Activation:
(Steven)

Simplified landing page to offer only 3 article choices
In usability testing, all clicked help button - but in reality, only 9% opted in to guided tour. (User testers seem to have needed more help, being less familar with site.)
GS users are of high quality, as measured by reverts.
20-30k users/month get notifications per email or web.
Highest rate of VE edits (67%) during July

Retention

power users... (see below)
GS editors a lot like "natural" Wikipedians. Conversion rates to 5+ edits are slightly lower on mobile

Reactivation (get people back on the site):

currently, 19% of users get reactivation notifications, pointing to GS

What slowed us down?

only did one A/B test (1 week long) in the last six months, dramatically less than earlier for this team

Matt: we also had an additional A/B test at the beginning of that period

team capacity:

(Steven:) Ori lot of work on infrastructure/analytical, but not front-end work
Erik: Ori mainly worked on infrastructure, we might make this official. eg. Vagrant
Steven: some of that infrastructure work *was* E3 related
also S overloaded, moving to Flow now
designer transition from Munaf to Pau, but Pau still much involved with Language Engineering team
Howie: so team transitioned from 3 FT developers to effectively 1 (due to S's and Ori's other commitments)
and 1 dedicated designer to part-time designer (also, 8 timezones away)
Steven: also need more data analyst capacity, Dario has a lot of other things on his plate
but now Aaron Halfaker might provide capacity too
Erik: 2 new engineers are going to join E3, already in hiring pipeline
Steven: optimistic, candidate quality for both E2 and E3 seems higher now. These hires will fill the engineering need.
Matt: Agreed in general, will take a couple months to fully onboard people though
Sue: So design remains main concern?
Erik: have mapped this out now. historically, design grew organically, where people asked first. Pau could shift more over once e.g. ULS bugs fixed, e.g 20% lang, 80% E3
Steven: apart from Pau and Brandon, most of our designers kind of float around, and it's not effective and fair use of time
Erik: Jared runs a pretty collaborative process among design team, with several designers providing input on one work area
Sue: sounds like solvable problem - embedding worked with Munaf, ...
Howie: so assuming we will have 3 FT engineers, will that be enough for the team to reach goals?
Steven: yes - there may be some projects which could grow in scope if successful, and then require up to 4 engineers
Erik: Matt, S and Ori all were Wikipedia editors before joining; new hires might not and therefore need more onboarding
Sue: impact of Wikimania on team? a week for travel? a month overall?
Howie: with all preparation for talks etc., a month
Steven: travel kills at least two deployment windows
Matt: I also worked on GSoC mentorship
Erik: which is valuable work, GSoC important for our hiring pipeline
S: CentralAuth was an unplanned distraction, took several weeks
S: I was really productive due to not going to Wikimania ;)
S: A/B test is a lot of effort
Matt: haven't done small iterative tests. E.g. OB6 - made our own dialogue instead of using e.g jQuery.
Steven: yes. some of the upcoming acquisition stuff might actually be smaller
Howie: back to resourcing - hearing that current plan seems reasonable
originally, conceived team as running quick experiments for other teams
but actually, became product team itself
what is the right way to resource a team like this?
Steven: also learned that implement production features is different from "shallow" experiments
Erik: varying existing features in a "shallow" way vs. reimplementing from ground up
might be useful to do more quick UI variation tests for e.g. GS

[15 min break]

(Ori joining)

2013-14 Target and Goals

Steven: our annual plan commitment: 2,400 more active editors/month
our team only one to commit to numbers instead of activities
how to get there:
framework (slide): user lifecycle:
reader / anon editor / registration / .../ very active editor
acquisition / activation / retention & reactivation

(Steven): lot of low-hanging fruit in anon editors acquisition
"you are not logged in .. please log in or create an account" message mid-edit: surprisingly large number come from this CTA; percentages differ by wiki

we know very little about anon editors - even though they contribute e.g. 20% of enwiki edits:
how many are repeat vs new?
reasons for not signing up?
"anonymous" term is misleading them about privacy issues
Sue: could simplify the CTA (e.g. leave out the incentives link)
(Steven): surface registration incentives? (e.g https://commons.wikimedia.org/wiki/File:Edits_anonymous.png )
Howie: also simple workflow issues
e.g. CTA only after anon edits have already been made? Dario: difficult
Steven: e.g. Mobile team piloted idea to show logged-out users the watchlist star
could do this with edit links on semiprotection pages
or the create page link
only example so far: upload file link in sidebar
we are not looking at upload CTAs because mobile, VE, multimedia teams already do that
...
Other small ideas: e.g. use icons for section edit links
Link GT in maintenance templates?
Contextual signup: integrate login or signup in workflow, instead of separate page
current work on activation: ...
who the new users are: 4 different personas
what they want to do: has task/ no task, vs intends to edit/not. GS now covers "has no task & intends to edit" quadrant
semi-protected page notices only resulted in 77 signups in July

Article creators:
landing page improvements ("Wikipedia does not have an article with this exact name. ...")
Draft namespace: community support for software supported tools, e.g. enwiki just had RfCs agreeing that AfC is broken
Sue: discussed sandbox idea long ago, concern was that it adds complexity.
many people who don't edit yet *assume* that there is some kind of apprenticeship etc. first
Steven: creating new page is inherently more difficult, community will always punish bad new article creators
Sue: sandbox with "publish" button?
Erik: AfC process is already using a draft status, frustrating
Dario: Problem: AfD results in complete removal without opportunity for improvement
Steven: enwiki community thinks about this as review process problem
Matt: anyone thought about giving article creation right back to anons, since workflow was established?
Ori: main namespace is already the draft namespace
Steven: whatever we do, need to make sure that the feature is actually attractive/worthwhile for new users

Active editor retention:
1. fix pain points in collaborations
2. make contributing easier and more rewarding

re 1.:

number one pain point is reverts . try to give more information about revert reason, or surface it better
pain point: deletion processes
pain point: bot interactions. false positive rate of bots when reverting / warning new editors? solution: bot audit, regarding impact on new editors. develop standards with bot operators and community

primary metric here would be improved retention of active editors
harder to make smooth process in these areas

Erik: community health dashboard?
Steven: idea: suggest interesting tasks to active editors, show how they are taken up
e.g. enabling "power gnomes" - example from GS toolbar's spelling error fixes task: one user hit "try another article" repeatedly, resulting in 103 edits
GS limitations regarding this:

restricted to only two [?] interfaces
i18n/l10n

...
user-specific page list to MediaWiki core, besides just watchlist, so that we can remember tasks for users
Plans: roll out GS to more languages
deliver tasks across a user's different devices and sessions
more sources for tasks (e.g. Wikidata)
..
make workflow more "addictive" (next steps, rewards)
..
compare http://www.wikihow.com/Special:CommunityDashboard
Ori: RfC https://www.mediawiki.org/wiki/Requests_for_comment/Support_for_user-specific_page_lists_in_core . I think watchlist is already overloaded with use cases.
Erik: separating use cases would need clear definition

Past work on reactivation: ...

Theory of growth:
1. Onboarding (GS & GT) across languages, devices session
2. acquisition campaigns
3. support those who want create articles
4. suggest easy and addictive tasks for active editors (rather than attacking pain points)

backburner:
1. fix reverts re new editors
2. bot audits
3. mobile product experimentation (lots of possible work areas witout interfering with existing work of mobile team)

focus on these three personas (use cases):

"Max": anonymous editors interested in signup
"Adam": made 1 edit, next step
"Sarah": interested in new article creation- 20-30% of all signups, reject 80% of the time. (e.g. fix landing page)

Sue: "power gnome" user story with 103 GS edits is inspiring. GS currently optimized for wikignoming. but article creators important
Matt: those who start with gnoming might graduate to more substantial edits
Steven: coded ~100 GS editors: one group edited various GS articles. second group got very involved in topic of one GS article. third group (maybe reactivated editors): make 5-10 GS edits, then went on to create article. I.e. I don't think we can separate types clearly (gnomes vs. creators)
Erik: are we going to reframe GS, so as not to focus exclusively on newbies?
Steven: cross that bridge when we come to it
hard to press too much into that one GS page
Erik: worth experimenting a lot here
Steven: also interesting to focus on topics, using Special:RandomInCategory. might be hard to effectively match interest area to article, though
Dario: very interested in doing better modeling of user editing patterns to assist task recommendations
Steven: Big view:
Flow and VE improve how editors contribute / collaborate
E3 should develop products to show editors why to join, and what they can do to help

Steven: we propose to change the team's name to Growth. Current EE/E3 distinction vague and confusing to most, and they would become Core Features per consensus with Maryana.
Matt: agrees about confusion caused by E3 name
Sue: however, rest of the organization works (should work) on growth too
Getting a sense that past Analytics difficulties are getting solved across the organization.
Steven: Should talk about blockages in design more in future, including at Tech Days
Steven: need to surface our team's needs to Analytics though. Erik: not yet infrastructure supporting rapid A/B testing though
Dario: but have other existing tools now which would enable some rapid testing for e.g. notifications. low-hanging fruits there
Sue: glad to hear about team's collaboration with Mobile (Maryana)
(Sue leaves)
Steven demos [1] / [2]