Research:WikiGrok

This page documents a completed research project.

About

edit

WikiGrok is an experimental MediaWiki feature designed by the Wikimedia Foundation's Mobile team, allowing readers of a Wikipedia article to extract and fill in structured data from this article. This feature is inspired by Magnus Manske's Wikidata game.[1] (read more). As of May 2015, the project is no longer in active development, however, community and involved developers are currently evaluating the experiment thus far, do define lessons learned and possible next steps.

Research questions

edit

The high-level research questions that we want to answer are the following:

  • What are the aggregate conversion and drop-off rates for each version of the widget?
  • What's the relation between article traffic and conversions?
  • Does the position of the WikiGrok widget affect conversions?
  • What's the distribution of observations per statement obtained with each version of the widget on the Beta site?
  • How much do distinct WikiGrok contributors agree on individual statements?
  • Does the version of the WikiGrok widget affect the quality of contributions?
  • Does logged-in status or edit count affect the quality of contributions?

Tests

edit

The goal of the initial set of experiments is to dry-run the test with a subpopulation of users opted into the Beta mode and validate that the instrumentation and experimental design is sound, as a pre-condition for running the test on a larger population of editors and readers. All times are UTC.

Test name Start End Target Notes
Test 1 2014-11-08 01:00:00 2014-12-02 01:00:00 all users, alpha+beta mobile site, enwiki first dry run, testing the instrumentation
Test 2 2014-12-12 00:41:00 2014-12-18 00:00:00 logged in users only, stable mobile site, enwiki small release on stable on logged in users only, testing patches to instrumentation and measuring conversions
Test 3 2015-01-13 01:00:00 2015-01-? 16% of all users, stable mobile site, enwiki first release on the stable mobile site with a sample of readers

Data

edit

Data for these tests is collected via the following logs:

Schema:MobileWebWikiGrok — logs user interaction with the WikiGrok widget
Schema:MobileWebWikiGrokResponse — stores the response submitted by the user via the WikiGrok widget
Schema:MobileWebWikiGrokError — error log for the WikiGrok widget

Links to qualitative data reports the Design Research team gathered via guerrilla usability testing:

Guerrilla Testing Wikigrok Version A
Guerrilla Testing, November 2014: Wikigrok Version B
Guerrilla Testing, December 2014: Wikigrok Version B v.2
edit

References

edit