Wikimedia monthly activities meetings/Quarterly reviews/Growth/February 2014

The following are notes from the Quarterly Review meeting with the Wikimedia Foundation's Growth team on February 28, 2014, 12:40pm–3pm

Present: Howie Fung, Erik Möller, Sue Gardner, Tilman Bayer (taking minutes), Dario Taraborelli, Steven Walling, Aaron Halfaker

Participating remotely: Matthew Flaschen, Sam Smith, Moiz Syed (listening in)

Please keep in mind that these minutes are mostly a rough transcript of what was said at the meeting, rather than a source of authoritative information. Consider referring to the presentation slides, blog posts, press releases and other official material

Agenda:

Review Annual Plan target and 2013–14 goals
What Growth shipped over the last three months
What we've learned about our users
What's next on the roadmap
Discussion about team's progress and needs

What we shipped since the last review

Steven: Since last review [December 4], some people joined: Moiz (secondary designer, joining Pau), Sam Smith
Last time, talked about: onboarding, A/B tests, plans to roll out outside enwiki, do first tests on anonymous editor acquisition. For article creation, we planned to focus on drafts, ongoing rfc on enwiki

Since then, we shipped drafts on enwiki – without fancy UI, but some particulars like NOINDEXing and allowing anonymous article creation.
In January, Sam joined, we productized GettingStarted and made it to use new API
and (led by Pau) new prototypes, usability tests for drafts enhancement. not yet live on enwiki, but got a lot of community feedback
Sue: Is Pau still on the team? How much?
Steven: yes; still primary designer, but shared with other teams – as usual with UX designers. Moiz is secondary designer. Pau had to spend more time supporting language engineering team in the last month or so
Howie: is that a bottleneck right now?
(Steven:) no, we have Moiz when Pau is not available and design is far enough ahead to not block implementation work usually.

Onboarding (GettingStarted and GuidedTour)

In February, launched GS and GuidedTours on 30 Wikipedias – top 10 and 20 others
edit tags ("via GettingStarted edit suggestions")
Not much outreach work required – affects new editors mostly
Based on 1. internal referer (which page the user was on) 2. category for recommending tasks (e.g. copyediting) – if available on the wiki
from this, three CTA types:

edit the current article (if not protected, or e.g. a help page – then no CTA)
edit the current or a suggested article
edit a suggested article

Dario: emphasis on current article?
Steven: yes, most users prefer that
OB6 test: 60% dismiss vs. 40% choose one suggestion, primary CTA more popular
Erik: when returning after signup stage, can I get these CTAs again?
(Steven:) no, for now it's immediately after the signup flow only

Ongoing tracking:

which users see which CTAs (impressions)
which edits are made via GT (via edit tags – when dismissing CTA and editing same page, no tag)

First two weeks of data:

62,138 users given onboarding CTAs (72% of signups). The rest either had no CTA e.g. no secondary task category available or JavaScript off, or were on a Special page, such as performing search (assume user want to complete that mission instead of seeing CTA.)
~7000 users made 1+ edits via GS
993 reaching 5+

Sue: how does that compare to non-GS users?
Aaron: good question, hard to compare without controlled experiment
Steven: overall activation rate for users shown the single CTA is good but normal (20–25% reach 1+ article edits within 24hrs), for dual CTA it's higher (34%).
Howie: Dario's analysis last year: 17% edit during first week on desktop, 11% on mobile
Steven: not measuring exactly the same thing
Erik: you mentioned # of wikis we're on, but what % of users are we reaching with this deployment to these 30 WPs?

Steven, Dario, Aaron: probably 70–80%
Steven: releasing to the rest will be easier, but won't get enormous ROI
Dario: what about other projects?
Steven: some, like Commons or Wikisource, not appropriate. Others, like Wikidata and Wikivoyage, probably so.
Matthew: might get good ROI finding alternative maintenance categories on the big WPs where we currently don' thave one
Steven: yes, or e.g. Catalan – they didn't have copyedit category when we came, but created one
we know copyediting is simplest for new people, but other option is to enable local admins to create custom task categories (?) or fallback to general cleanup cat – almost all of these projects have one. but that won't be as clear-cut a task for new users, even when e.g. limiting to articles below a certain lentgh
Howie: how high is this on the priority list?
Steven: category stuff is not complicated, but other tasks first – better icons etc.
Erik: should invite community input on categories
Dario: what if category exists, but contains no (unprotected) pages
Steven: we don't serve a suggestion, i.e. do not show the user a button for that in the CTA
Howie: Wouldn't it be nice if we had semantically related categories across projects? ;-)
Matthew: would only need config update if category is renamed
Erik: do you have community liaison available?
Steven: no, will talk about this later in team needs section

Drafts and Article creation data

Learned a lot about article creation ((Research:Wikipedia article creation)
stats.wikimedia.org has good data about overall creation rate, also by bots – but not by other user types (like new / experienced users)

Aaron: who creates articles?

55%–85% experienced (>1month) editors, 55% being the exception (most wikis more like 80%)
5–15% new registered users
10–20% anons (if possible – i.e. outside enwiki)
most new users who create article do so on day 1
3.3% of new users create article on day 1

Steven: then we looked at whose articles survived
Aaron: as newcomers get more experienced, their articles are more likely to be kept (except autocreated users – most of whom probably have experience on other wikis)
anons are doing better than day 1 newbies
two ways to look at anon editing:

even less experiened than newly registered users
also experienced users who prefer not to log in/register (Sue: e.g. who prefer not to have a certain edit/article attached to their contribution history)

Howie: but also, day 1 newbies may have anon editing experience

Steven:

on zhwiki, anon article survival rate particularly high
on plwiki, high survival rate for new users – backlog might be lower there, i.e. more time to treat newbie articles better, perhaps
on jawiki, high survivial rate – notability might be less strict there

Dario: should look at retention rates for jawiki then
Aaron: what impact does workflow have?
tracking page moves was painful, only got it to work on dewiki and enwiki
pages started in user space by new users had higher survival rate
but not for user space creations by experience users on dewiki – unclear why
AfC creations had much higher survival rate – if published, but massive backlog – like 20–25k pages
Steven:

design implications: anon users better than perhaps assumed – how can we serve them?
need to avoid a backlog, which means avoiding pre-publication review. The math just doesn't work, comparing the number of new articles/new authors and potential reviewers.

Erik: how many active AfC reviewers?
Steven: don't know, but regarding participation in community feedback on development work, it's a fairly active WikiProject
Steven: backlog contains lot of pages that would pass as new article

Survival rate per user pre/post AfC

AfC drafts created, 2008–2013

Aaron: shows chart with impact of AfC start (see right)
Erik: might be worth proposing and discussing solution before investing further development work
Steven: learned from mw:Draft namespace/Usability testing (Pau):
1. simple improvements to entry points can make a difference
Matthew: announced these experiments on village pumps
Steven:
2. draft workflow made sense to new users, even without technical understanding of e.g. namespaces
exception: the action of abandoning a draft seems hard to understand (e.g. in notification after 30 days)
3. still hard to teach concepts like refs or notability, or inviting collaboration
Sue: clear that process is public? (thinking of e.g. workflow on Yelp where drafting is private until hitting the publish button)
(Steven:) probably yes
AfC curators want reviewer tools like page curation
but won't be able to implement that right now
probably hard to increase number of reviewers
Erik: want shift from reviewing to collaborating
Steven: Yes.

Steven: we also created new schemas for page creation/move/deletions, should give us much better data down the line (e.g. dashboards instead of bulky SQL queries that take a week...)
Drafts namespace is still tiny e.g. in one week 113 created, 18 published, 5 deleted
Sue: is self-publishing a draft possible?
Steven: yes

Guided Tours reuse

there exist 16 GTs in the wild apart from those created by the team
on 4WPs, wikiHow, Commons, Outreach wiki
Matthew: working to make GT usable with eg. VE, UploadWizard
Steven: i.e. more nonlinear stuff, e.g. waiting for a certain user action before displaying GT

Target for the team, impact on Total Active Editors

target was 2400 additional active editors across wikis
problem: prove "additional"
impact on TAE?
optimistic: 993 in a two-week cohort
conservative: the 2% increase from previous tests
not certain yet about (provably) hitting the target
either extrapolate (guess) from A/B tests or compare with previous status
Aaron: compare change after rollout with change observed in A/B test. not ideal as scientific experiment, but best we can do ethically
Steven:
Product roadmap: acquisition / activation / retention & reactivation
earlier, were thinking a bit bigger about "keep going"
Moiz working on this in current sprints
earlier, sent everyone back to same Special: page
but actual question: where do editors go to find things to do? (don't want to add cruft to UI)
can easily send notification inviting further editing
but more ambitiously, working with Moiz on "home page"
Erik: my opinion: let's use "My contributions". could add sidebar with edit suggestions there. Experienced users might like that, or just turn it off. Not so fond of creating a completely new page that we have to convince users to go to.
Steven: agrees, but wanted to explore the alternatives too
Sue: clicking on one's user name goes to user page – this is kind of weird, because the actual center of activity is more on talk pages. and actually, newbie experience suggests (wrongly) that discussions are more important than article edits
Dario: yes, contribs page is hard to find
important question: how to tailor to users in very different states of experience
subscription model?
subscribe to other channels when progressing in experience
Steven: won't go for complete interest graph etc.
for GuidedTour, need to pay some tech debt and do enhancements
but then focus on anon acquisition in rest of quarter
explore options, narrow down. article creation improvements, redlinks,...?
more complicated features, task creation

10-minute break

Steven: (coming back to future work discussion) many editors who don't get reverted/yelled at still don't come back
onboarding workflow is fine without reactivation features, they are optional
article creation: lot of community demand for work in this area, even though we were clear about managing expectations
Sue: so GettingStarted is more of a "green field" than article creation?
Steven: yes
Sue: regarding encouragement of anons to register, we should be realistic – they include users from all experiences layers, problem users, etc.
Steven: serve both users who purposefully register to do something, and those who just aimlessly create an account
but there's a third category: users who follow campaign invitations, e.g. education program, editathons by chapters
working with Andrew Green from Terry's team on Education Program extension, some volunteer work from Adam White
Sue: possible to use this with email invitations?
Steven: currently solely based on campaign identifier in the URL, using Wikimetrics
related opportunity (with Fundraising team): emails from donors who said they are interested in editing, as side project with design
Erik: education program extension was custom built, has its own cohort analysis, activity fields – to better support them, we're working to make things more generic / reusable, less bespoke. seen smaller projects reusing the EducationProgram
extension for non-education purposes already.
to some extent, this is a distraction for the growth team, but it's necessary to avoid long term technical debt
Steven: yes, might not bring high volume, but higher quality signups

Things brought up in last review

get irons out of the fire (i.e. team focus)
hiring – Sam has brought a lot more velocity.
remote team practices, changed to standups via hangups instead of irc

better track velocity (#of cards in Trello)
Matthew: also, we put everything into Trello (instead of Bugzilla)
Steven: focus on one thing at a time getting better
all remote except me and Moiz. definitely a cost to remoteness, e.g. made a lot of progress on anon acquisition when Pau was here in SF
Additional ideas to consider:

improve agile practices
in some other teams, someone else does Scrum work instead of Product Manager
get community liaison work off of Steven's plate

Sue: might be more of a separate role
Steven: I'm fulfilling 3 roles instead of one

front-end architecure improvements. we fully support the recent architecture considerations in other areas, but this is important too

HTML templating, UI library, respsonsive grid
Erik: yes, but can only do so many things at a time
Steven: see so much work that's implemented separately in each team or not at all, instead of once cross-team – like in a modern website ;)
Matthew: UX team should get some engineers to implement such things consistently
Erik: UX team engineers would rather work on different things, should be located elsewhere
moving forward, will try to get a better picture about opinions in this area
Dario: makes sense to focus on these separate products, to be able to ship. but worry that we limit the available services to where we can make (immediate) impact, instead of building something more universal
Erik: we know it takes a long time to implement and ship a dedicated service in just one particular area (e.g. Parsoid, PDF)
will have Analytics partner with Platform to map this out
Growth team will be customer from the start
just like Wikimetrics had customers from the start
Erik: any other questions?
Sue: wondering about scrummaster work
Steven: scrummaster needs to be either senior person from outside, or we train existing team members
Erik: either hire shared scrummaster, or have Steven in the role
Sue: possibilities to offload CL work?
Steven, Howie: can probably figure something out
Howie: re scrummaster, how does the rest of the team think about the idea?
Matthew: would be valuable, agree with Erik about options
Sue: do engineers not want to be scrummaster? ;)
Steven: it depends
Matthew has spent some time being technical lead, doing it well – without scrummaster title
Erik: basically, people who care about processes, not just code
Dario: question about Analytics support
Steven: worry a bit about anon acq tests
Dario: rest is well supported with Eventlogging