FlaggedRevs Report December 2008

Background

One of the fundamental challenges with wiki-based collaboration is the tension between the desires for immediacy and accuracy. The fact that edits are immediately reflected and new information can be incorporated instantly is an incentive for new writers to contribute, and it makes the information more valuable and less likely to be stale at the time at which it is consulted. But especially in a wiki that's open to be edited by anyone, such as Wikipedia, it also makes it more likely that a version that a reader is looking at contains inaccurate information or obvious nonsense.

The wiki immune system relies on radical transparency of all transactions to defend against the insertion of problematic information, and usually responds quickly. This does not, however, address the issue of information already transferred to people consulting the wiki. There are also certain edit situations which are not well-captured by the wiki's traditional mechanisms of examining edits and reverting problematic ones. For example, people patrolling the page for changes might not notice a single problematic change within a set of changes.

The Wikimedia Foundation and the German Wikimedia chapter (Wikimedia Deutschland) have collaborated to develop the FlaggedRevs extension, a powerful module for the MediaWiki software powering all websites operated by the Wikimedia Foundation. Like MediaWiki itself, FlaggedRevs is open source. It is also highly flexible in its configuration. For example, a wiki could implement 10 different stages of review, or just a single one. It could limit reviewing to a small class of users, or it could allow it to almost everyone.

As of June 2008, this module can be enabled when a community of users in a Wikimedia wiki decide (by consensus, by vote, or through some other decision making process) that they would like the feature to be activated. This flexible, bottom-up model is meant to ensure that configurations are chosen which are appropriate to a given community and project. For example, Wikinews (a project to collect the news of the day) obviously may have different needs from Wikisource (a project to collect and document digitized source texts that are often hundreds of years old).

Capabilities of the FlaggedRevs extension

As of December 2008, the FlaggedRevs extension has the following key features:

• A set of rating criteria such as "accuracy" and "completeness" can be defined.
• Within each of these categories, multiple levels of review can be defined (such as "highly accurate", "highly inaccurate").
• Different groups of users can be given permission to assign ratings in these categories to specific versions of articles, images, and other content.
• Review permission can be given automatically to users once they have made a certain number of changes.
• Readers can be shown always the latest version, or always the most recently reviewed version. This can also be configured on a per-page basis.
• Various reports can be generated to find, for example, the oldest pages with out-of-date reviews.
• A more basic "reader feedback" module is available to allow unregistered readers to give instant feedback such as "This article was highly useful to me". Reader feedback can be plotted as graphs

Current deployment

The following Wikimedia projects currently use the FlaggedRevs extension, sorted by the number of pages where at least one version has been reviewed (all numbers December 15, 2008):

1. German Wikipedia: 772,942 articles reviewed (90.77% of all articles)
2. Russian Wikipedia: 124,197 articles reviewed (36.37% of all articles)
3. Polish Wikipedia: 74,060 articles reviewed (13.14% of all articles)
4. German Wiktionary: 54,309 entries reviewed (60.36% of all entries)
5. Hungarian Wikipedia: 9,101 articles reviewed (8.02% of all articles)
6. Ukrainian Wiktionary: 4,356 entries reviewed (57.57% of all entries)
7. Esperanto Wikipedia: 3,889 articles reviewed (3.60% of all articles)
8. Russian Wikisource: 2,564 source text pages reviewed (12.43% of all source text pages)
9. English Wikinews: 2,263 stories reviewed (16.10% of all stories)
10. French Wikinews: 1,782 stories reviewed (36.29% of all stories)
11. English Wikibooks: 534 chapters reviewed (1.32% of all chapters)
12. Russian Wikiquote: 71 quotation pages reviewed (1.55% of all quotation pages)
13. Portuguese Wikinews: 26 stories reviewed (0.68% of all stories)

The German Wikipedia experience

The German Wikipedia implemented a FlaggedRevs configuration in May 2008. In the German configuration, edits by anonymous and new users have to be patrolled by another longer term editor with reviewer rights before becoming visible as the default revision shown to readers. In the long run, proponents of this system also want to implement the use of FlaggedRevs to validate the accuracy of articles beyond basic change patrolling, but a policy for this use of the technology has not yet been agreed upon.

The currently used configuration was later confirmed in a project-wide poll. As of December 2008, more than 90% of all articles in the German Wikipedia have at least a single reviewed version.

Various statistics on the use of the system are available. The most in-depth analysis can be performed in real-time by using a script developed by a volunteer and running on the Wikimedia toolserver (a server sponsored by Wikimedia Germany exactly for such purposes). One of the most important questions for the use of the system has been whether edits can be reviewed within a reasonable timeframe. So far, the system has scaled well to accommodate changes as they are made, and the number of articles with out-of-date reviews has been kept at a reasonable number. As the relevant statistics show, the number has been kept below 10,000 for most of 2008.

This is largely the result of repeated and concerted efforts of the community to mobilize reviewers. Specialized scripts have recently been developed to identify articles with out-of-date reviews in specific subject matter areas, so that users who are experts in those areas can self-select to review changes. Improvements like this, as well as generally increasing acceptance of the system, will likely lead to greater efficiency in patrolling changes over time.

That said, another important number to look at is the number of days that it takes, at maximum, for an edit to be reviewed. This number has climbed as high as three weeks (statistics), and concerted efforts have been made to reduce it.

It is not yet fully understood what the impact of the implementation of FlaggedRevs has been on the number of contributions by new users.