Research:Onboarding new Wikipedians

(Redirected from Research:OB)
This page documents a completed research project.


The purpose of this growth project is to build an onboarding experience for Wikipedians, starting in English, that shows new users how to get started with making productive contributions to the project. See more about what this means in the context of Wikipedia in our design document.

Research questions

edit
  • How many participants in an onboarding experience contribute to the encyclopedia?
  • Which onboarding experience is most effective at helping newly-registered people edit?

Overall metrics

edit

The primary metrics we are interested in are:

  1. Increased the number of newly-registered editors who complete one edit
  2. Increased the number of newly-registered editors who complete five edits

The reason for these metrics is because they are directly relatable to the active editors measurements of Wikipedia overall.

Event logging

edit

We've used the following schemas to log events related to this experiment, in roughly chronological order:

Changelog

edit
See mw:Onboarding new Wikipedians/Recommender system#Changelog
Shortcut Changes Start End Schema rev id Version
OB0 Launch! [screenshot] 2012-12-13 2012-12-20 - -
OB0 Minor design updates [screenshot] 2012-12-20 2013-12-22 - -
OB0 No deploy, but fully randomized the list of tasks, creating much higher refresh rate 2012-12-22 2013-01-10 4876947 -
OB1 Added returnTo funnel logging 2013-01-10 2013-01-25T00:00:00Z 4993508 -
OB2 50/50 split test vs control (no GettingStarted) 2013-01-25T00:00:00Z 2013-01-31T22:45:00Z 4993508 -
OB3 50/50 split test: GettingStarted + GuidedTour vs GettingStarted only 2013-02-08T01:16:56Z 2013-02-14 5219269, 5222838 ob3
OB3 50/50 split test: GettingStarted + GuidedTour vs GettingStarted only (stronger enforcement of control treatment) 2013-02-14 2013-02-21T20:59:39Z 5243394, 5222838 ob3-split
OB3 GuidedTour at 100% for all users 2013-02-21T20:59:39Z 2013-02-22T20:17:16Z 5243394, 5222838 ob3-all
OB3 50/50 split test: GettingStarted + GuidedTour vs GettingStarted only (fixed bug with urlencoded page titles) 2013-02-22T20:17:16Z 2013-02-25T20:56:01Z 5243394, 5222838 ob3-split
OB3 50/50 split test: GettingStarted + GuidedTour vs GettingStarted only (reinstate test after merge disabled it) 2013-02-28 2013-03-08T00:20:00Z 5285779, 5222838 ob3-split-retest-split
OB4 Launch of new landing page 2013-03-08T00:20:00 2013-03-14 5285779 ob3-split-retest-split
OB4 50/50 split test: GettingStarted v.2 vs control (no GettingStarted) 2013-03-14 2013-03-21 5320430 1 (version replaces experimentId)
OB4 GettingStarted v.2 at 100% 2013-03-21 2013-04-18 5320430 2
OB4 GettingStarted v.2 at 100%, version bump to note removal of BLPs 2013-04-18 2013-04-30 5320430 3
OB4.5 GettingStarted v.2 at 100%, version bump to note launch of notifications 2013-04-30T20:00:00Z 2013-05-16 5320430 4
OB5 Split test of GettingStarted with new landing page and new nav bar 2013-05-16 2013-05-20 5467309 1
OB5 Split test of GettingStarted with new landing page and new nav bar, update logging 2013-05-20 2013-05-23T20:31:14Z 5496876 2
OB5 End split test, deliver GettingStarted new landing page, nav bar, and Echo to all users 2013-05-23T20:31:14Z 2013-06-26T09:33:23Z 5496876 3
OB5 Improve appearence of task toolbar on low width screens and print, and log whether the toolbar was shown with the page-* impression. 2013-06-20T20:50:39Z 2013-08-08 5588671 4
OB6 A/B test of the next design iteration of the GettingStarted workflow 2013-09-08 2013-09-15 5658668 1
Rollout Continued rollout of onboarding across languages 2014-02-06 7355552 N/A

Analysis

edit

Primary metrics:

  1. How many users in the funnel complete 1+ NS0 edits in 24 hours
  2. How many users in the funnel complete 5+ NS0 edits in 24 hours
  3. Conversion rates in each funnel (GettingStarted, control, returnTo and other comparable funnels)
    • As a proportion of total impressions
    • As a proportion of the previous step in the funnel
  4. How many edits made through the funnel are reverted?

We also have conducted further qualitative analysis about editing activity by users during this experiment. Individual experiments also may measure the above metrics for subcohorts within each experiment, to compare the effect of different user interface modifications within an experiment.

Overview of results

edit

Results refer to the whole experimental groups from split tests only. All reported effects are significant (p < .05), full reports are available under each test page.

24 hours live account rate

edit
Difference in proportion in 24h live accounts
Test Effect size
OB2: GettingStarted v.1 vs default welcome page
4.3
OB3: GettingStarted v.1 + GuidedTour vs GettingStarted v.1 only
2.2(net effect of guiders)
OB4: GettingStarted v.2 vs default welcome page
3
OB5: GettingStarted v.3 vs default welcome page
4.4


1+ edit in 24 hours

edit
Difference in proportion in 24h 1+ ns0 editors
Test Effect size
OB2: GettingStarted v.1 vs default welcome page
1.9
OB3: GettingStarted v.1 + GuidedTour vs GettingStarted v.1 only
1.1(net effect of guiders)
OB4: GettingStarted v.2 vs default welcome page
1.5
OB5: GettingStarted v.3 vs default welcome page
1.7


User experience and technical documentation

edit

See also

edit