Research talk:VisualEditor

Updates on VE data analysis

Split test: Editing ease

There have been several threads discussing a 43% relative drop in editing activity, based on a 10% effect size between the test and control condition that we initially reported in the draft of the results of the A/B test. It turns out that, as discussed on the test's discussion page, this figure was based on unreliable instrumentation data (we previously removed from the results other parts of the analysis based on the same data source). We reran the analysis using the revision table and the correct effect size between test and control is of the order of 1% (control: 31.3% test: 30.3%), i.e. a 3.2% drop in editing activity for the test condition (VE enabled) when measured relatively to the control condition (VE disabled). The difference is marginally significant, meaning that based on data collected during the test we cannot conclusively answer the question of whether VE is affecting editing ease.

Split test: Attached users

In order to get a more accurate picture of the impact of VE on new contributors we filtered out from the sample "attached users" (i.e. accounts of users previously registered on other wikis who happened to navigate to enwiki during the test) who got randomly assigned to either of the experimental buckets. After filtering out these users, 7,602 users were left in the control condition and 7,572 users in the test condition (see the complete sample). We reran the whole analysis and updated all the figures in the draft of the report and we didn't find any substantial change worth reporting (for example: proportion of reverts by tool before and after filtering attached users).

Real-time edit dashboards

We released a number of dashboards visualizing hourly edit activity for those projects on which VE has been enabled to date: enwiki, dewiki, eswiki, frwiki, hewiki, itwiki, nlwiki, plwiki ruwiki and svwiki. You will find graphs relevant to VE both under the Edits and Reverts tabs. We're adding new dashboards as I write (breaking down attached and new registered users, adding hourly revert rates for wikis other than English etc.), watch this space for more updates.

Instrumentation

The original instrumentation of the edit funnel was designed to test our ability to consistently instrument VE and wikitext and to primarily collect data for QA purposes. It was not meant to produce usable data for measuring completion, bounce, clickth-rough rates or other funnel metrics, but was accidentally included in earlier versions of the analysis, which has now been fixed (no other piece of the analysis of the split test relies on instrumentation data). We're now working on instrumenting the software to get a better sense of completion rates for VE as a function of different browsers, to identify performance issues and to correctly handle edits made on mobile devices, where VE is not enabled.

Anonymous user activity

We released some preliminary descriptive analysis of anonymous activity. We'll be expanding this draft report with new data.

Raw data

We created a dedicated entry in our open data repository to host datasets from the split test. We uploaded the complete sample of users from the split test and we will be releasing other data collected during the rollout of VisualEditor.

All relevant links can be found on the main VisualEditor analytics page, which you can watch for further updates.--DarTar (talk) 21:00, 26 July 2013 (UTC)

Thanks

Thanks for the page and the updates above. The limn dashboards are always extremely hard to find, so an index is very useful. I've linked it from Statistics. --Nemo 14:19, 27 July 2013 (UTC)

Per-wiki latency

You may want to link https://gdash.wikimedia.org/dashboards/ve/ as well. Jdforrester, is there a per-wiki breakdown? I'm interested because of Research:The sudden decline of Italian Wikipedia. --Nemo 10:21, 1 June 2014 (UTC)

@Nemo bis: Interesting issue. No per-wiki breakdown for that dashboard, no; it might be possible to create one. Jdforrester (WMF) (talk) 16:38, 2 June 2014 (UTC)

Ok, then unless someone comes up with alternatives I'll submit a patch to add one for the top10 most edited wikis (cf. analytics list). --Nemo 10:44, 3 June 2014 (UTC)

Links

• Tool for pt.wp: https://tools.wmflabs.org/ptwikis/Editor_Visual --Elitre (WMF) (talk) 11:54, 24 September 2014 (UTC)
• Aaron H.'s research, January 2014: Research:Visual editor usage (January, 2014)
• Amir A.'s proposed research, July 2014 Research:Ideas/How does the availability of VisualEditor affect the rate of edits and editor acquisition in different languages?