Research:FlaggedRevisions investigation/Results
This work is tracked in Phabricator: task T381044
This study sought to understand
- how the tool is used by patrollers across three modes of configuration as well as
- patrollers' perception of efficiency.
This report details the risks associated with FlaggedRevs’s lack of maintenance by the three core features most useful to patrollers, as well as potential areas where product development may benefit from applying lessons from how FlaggedRevs is currently used.
(1) Page Stabilization: Many editors worry the overall credibility of Wikipedia - specifically their home wiki - would decline without page stabilization.
Recommendations:
- Expand watchlist capabilities to include “types” of articles, e.g. colors, dates, years to speed up fast edit reviews to overlooked article types
- Provide further information and guidance on when and where to set pages in Semi-Protect mode
(2) Unexpiring queue: Relying on RecentChanges filters alone would dramatically shorten the patrolling window.
Recommendations:
- Make it easier to recreate the FlaggedRevs queue in RecentChanges with a premade filter
- Expand watchlist capabilities to include “topics”, allowing known experts or Wikiprojects groups to add the relevant “topics” to their watchlists
(3) Codebase for automation bots: Vandalism would be more difficult to control.
Recommendations:
- Build new automated anti-vandalism tooling that both
- Rejects high probability “bad” edits to decrease the need for human review
- Approves high probability “good” edits to prioritize human review on more ambiguous edits
Overview and background
editWhat is the Flagged Revisions tool (FlaggedRevs)?
editEnabled on 32 languages across 48 Wikimedia projects as of 2024, Flagged Revisions facilitates primarily two workflows: patrolling of individual edits, and hiding new contributions from readers until an experienced editor has reviewed the edit. Highly customisable, the extension enables communities to configure a wide array of user rights, user groups, edit flags, and other settings, to set the behaviour of its edit patrolling and visibility features. In addition, review rates vary widely, spanning less than an hour to ~2896 days.
Brief history of FlaggedRevs
editFlaggedRevs was developed in 2008 as a new ‘milestone’ in how Wikimedia projects could review and moderate incoming edits. Framed as a way to balance the open nature of editing Wikimedia projects with the need to ensure that readers see reliable, vandalism-free content, the extension was made available to any Wikimedia project to enable and configure.
Between 2008 and 2015, Flagged Revisions was deployed to dozens of Wikimedia projects, including 25 Wikipedia languages. In 2014, though, Wikimedia Foundation staff announced that the WMF would no longer be approving deployment of the FlaggedRevs extension to more Wikimedia projects.
As noted in the original release announcement (and still in use today):
| Announcement text | True today? |
|---|---|
| This [Flagged Revisions] has two key advantages compared to the current patrolling model: | |
| "It reduces duplicate effort in basic change patrolling, allowing users to focus on un-reviewed changes and thereby directing their attention more effectively." | ✅ Yes |
| "It ensures higher coverage of changes. In particular, when malicious changes are followed by good faith edits, malicious changes are sometimes overlooked. In the FlaggedRevs model, reviewers can systematically examine every change." | ✅ Yes |
The original release announcement also listed other potential future uses of FlaggedRevs, which you can note below are not currently true today:
| Announcement text | True today? |
|---|---|
| "Use for identification of article versions which meet standards of accuracy and quality as determined by experts. Potentially, FlaggedRevs could interface with external expert communities (such as universities or expert-driven encyclopedia projects like the Encyclopedia of Life) to identify versions of Wikipedia articles which meet scholarly standards of quality." | ❌ No |
| "Use for identification of article versions which meet internally defined standards of quality beyond the simple check for vandalism. The original German Wikipedia proposal for FlaggedRevs includes a more in-depth community quality review stage, which is still being discussed. A simple way to tie into community review mechanisms would be to use FlaggedRevs to “tag” versions which have passed through processes like “Featured Article Candidates”." | ❔Partial |
| "Use to collect basic reader feedback on articles. Asking our readers whether information in Wikipedia articles is useful to them, and whether it meets their quality standards, could be a good way to track reader satisfaction over time. The lead developer of the FlaggedRevs extension, Aaron Schulz, is currently implementing such reader feedback tools.” | ❌ No |
*Emphasis added
How is FlaggedRevs configured across the wikis?
editFlaggedRevs is mostly used in three modes: Overwrite, Protect and Inactive.
- Overwrite Mode: Edits hidden on all pages by default, e.g. German, Polish
- German Wikipedia: Reviewed Versions
- Polish Wikipedia: Marked Version
- Protect Mode: Edits hidden on flagged pages by default, e.g. English
- English Wikipedia: Pending Changes
- Inactive Mode: Edits not hidden by default, e.g. Finnish, Ukrainian
- Ukrainian Wikipedia: Patrolled Version (with “patrolmen” as the user group)
- Finnish Wikipedia: Evaluated pages (with “sifter” as the user group)
An example of different protection levels can be found on English Wikipedia's Protection policy#Protection levels policy page.
Study Overview
editResearch goal
editUnderstand how FlaggedRevisions are used by experienced Wiki editors in patrolling and editing workflows across the three major models of feature usage. This research project fits into the overall goal of providing experienced Wiki editors with the tools they need to patrol (1) quality and (2) vandalism.
There are two problems FlaggedRevs addresses, which need to be interrogated separately.
- User perception of efficiency as explored thought this UXR study
- Actual efficacy in preserving quality control and/or counter-vandalism as explored through behavioral data. Data questions that arose through the analysis of this study have been included throughout as learnings.
Methodology
editWe conducted 6 in-depth Interviews with current FlaggedRevisions editors, in the month of March 2025. Participants in this study:
- Used FlaggedRevisions at least 100 times, between Jan - Feb 2025
- Had one of the following as a primary wiki
- n=2 wikis in Overwrite Mode (German, Polish)
- n=1 wikis in Protect Mode (English)
- n=3 wikis in Inactive Mode (Finnish, Ukranian)
Report Notes
editUser terminology
editAs user permissions and categories vary between wikis, the report aims to use the approximate terminology most representative of the wikis studied. Asterisks (*) are included when more general terms are used. Examples from this report include:
- Newer editors: this group includes logged-out editors, editors with very new accounts (with or without user pages), or those that fall under the threshold for the autoreviewed right (or equivalent)
- New users: as above, but this group also includes readers of Wikipedia who may be less familiar with Wikipedia conventions and policies.
Core functionality key insights and recommendations
editPage stabilization to control public view of newer editors’* edits
editWhy is it useful? For the wikis with the extension, page stabilization provides an added level of control over the content new users can view on all or some articles, depending on the configuration.
What if the feature remains unmaintained? Many editors worry the overall credibility of Wikipedia - specifically their home wiki - would decline without page stabilization.
What alternative is there to this feature, if any? Outside of FlaggedRevs, the closest analogue to page stabilization is to use Protected Pages.
What can we learn from this feature?
- Provide further information and guidance on when and where to set pages in Semi-Protect mode
- Expand watchlist capabilities to include “types” of articles, e.g. colors, dates, years to speed up fast edit reviews, replacing some of the needs for page stabilization.
Unexpiring queue of newer editors’* edits to provide easy, focused access for patrolling quality control and vandalism
editWhy is it useful? As initially intended, the FlaggedRevs queue “[allows] users to focus on un-reviewed changes and thereby [directs] their attention more effectively” (source).
What if the feature remains unmaintained? If FlaggedRevs is unmaintained and editors must rely on RecentChanges filters, it would dramatically shorten the patrolling window.
What alternative is there to this feature, if any? While the FlaggedRevs tool provides an easily accessible queue, there are other (though more manual) ways to configure a very similar list through RecentChanges filters.
What can we learn from this feature?
- Create a ‘FlaggedRevs’-style filter in RecentChanges to make the configuration easier, preferably displaying X months of activity. The timeframe should be determined by behavioral data.
- Expand watchlist capabilities to include “topics”, e.g. academic disciplines (e.g. physics, mathematics) or areas of interest (e.g. video games, manga and anime) would allow known experts or Wikiprojects to add the relevant “topics” to their watchlists.
Codebase for automation bots and tools
editWhy is it useful? As seen in this study, Finnish Wikipedia leverages FlaggedRevs for two automated anti-vandalism bots.
What if the feature remains unmaintained? As these bots are aimed at fast edits, vandalism would be more difficult to control.
What alternative is there to this feature, if any? Without these bots, Finnish Wikipedia editors would need to more closely monitor the Recent Changes queue to manually revert, requiring more “fast” edit reviewers.
What can we learn from this feature? Build new automated anti-vandalism tooling that both
- Rejects high probability “bad” edits to decrease the need for human review
- Approves high probability “good” edits to prioritize human review on more ambiguous edits
Detailed learnings
edit“Fast” and “slow” edits
editThe efficacy of FlaggedRevs relies on each wiki’s editor community to review new “fast” and “slow” edits at the rate they are made
editThe reasons for the longer median review times on Ukrainian and Finnish Wikipedias are not completely known. They could be related to the chosen FlaggedRevs mode, their generally lower editing rates, a lack of unreviewed edit reviewers, or other reasons that are not revealed by this study.
| Wiki | Average review time for logged-out user edits | Median review time for logged-out user edits |
|---|---|---|
| German Wikipedia | 26h 42m | 29m 28s |
| Polish Wikipedia | 21m 6s | 5m 1s |
| English Wikipedia | N/A | N/A |
| Ukrainian Wikipedia | 231d 3h | 14d 15h |
| Finnish Wikipedia | 3d 23h | 3d 3h |
Fast edits: Lower-effort, quick reviews editors can complete without much external verification
- These are often low-character count or the addition/replacement of images or references
- Examples: clear cases of vandalism, or small or incremental edits to a page
Most of the easy cases like you can just pick them from the RecentChanges straight away and be done in 3 seconds with them.
— ukwiki editor
Slow edits: High-effort, labor-intensive edits that require external verification and/or domain expertise
- For edits that require domain expertise, there simply may not be a FlaggedRevs editor that feels knowledgeable enough about the subject to make a decision, leaving the edit unreviewed for long periods of time.
- Examples: edits that add many claims backed by citations, which requires someone to read the citations to see if they verify the claims and meet reliability requirements
I have one on my list which I actually know about but there is a computation to do in order to know whether the change is correct or not and I haven't gotten around to do it. So that's the one on my watch list that is where the flag revision has not been backed because I need more time to do it.
— dewiki editor
- By definition, longer edits can be controversial as the bar to verify is that much higher for the editor
It just takes a lot of time and effort if you want to go through the more difficult stuff especially because some of the bigger changes may be controversial. So even checking the sources and the added text is not necessarily enough to decide if it's good to go to the article or not. So it's labor intensive and sometimes quite complicated to do the bigger reviews.
— ukwiki editor
“Fast” and “slow” patrolling
editFast and slow edit patrolling pose different challenges, and benefit from FlaggedRevs, in different ways.
Fast edits require continuous monitoring by a sufficiently large number of editors relative to the size of the wiki
- Editors are able to review these FlaggedRevs-tagged “fast” edits as they are made through the newest-first RecentChanges queue
Slow edits require domain expertise and/or time-commitment of the editor community
- The FlaggedRevs queue’s unexpiring, oldest-first order provides continued high visibility of “slow” edits that would otherwise not appear on the RecentChanges queue and therefore likely be forgotten
Three core features of FlaggedRevs
editPage stabilization to control public view of newer editors’* edits
editWhy is this feature useful?
editFor the wikis with the extension, page stabilization provides an added level of control over the content new users can view on all or some articles, depending on the configuration. On protected pages, FlaggedRevs provides editors with the opportunity to reject edits for quality assurance and/or vandalism without the content ever being viewable by new users*.
[FlaggedRevs] would, in fact, represent an opening up of Wikipedia rather than a closing down, as many of the affected pages are currently “semi-protected”, meaning that they cannot be edited at all by new and unregistered users due to the perceived risks of malicious edits. Being able to make changes that do not immediately become visible is surely preferable to not being able to make changes at all.
— source
English Wikipedia chose to leverage FlaggedRevs in a Protect Mode configuration, allowing them to closely monitor many basic concept articles (e.g. colors, years, dates) that they believe require higher levels of quality assurances and/or are more susceptible to vandalism. Edits to these articles often go unpatrolled without FlaggedRevs, since they don’t rarely capture the attention of the editor community (e.g. not on many watchlists, RecentChanges moves too quickly for bad edits to be caught). Notably, this is an unintended use of the tool when first released in 2008.
What alternative is there to this feature, if any?
editOutside of FlaggedRevs, the only way to stabilize a page is to use Protected Pages which restricts editing to autoconfirmed users (in Semi-protected mode) and only those in the sysop group (in Fully protected mode). Other levels of protection can be added by changing that wiki’s configuration settings.
As explained in the FlaggedRevs Initial Release Announcement though, the intention of FlaggedRevs’ version of page stabilization was specifically to replace the overly restrictive use of Protection Pages.
What if the feature remains unmaintained?
editMany editors worry the overall credibility of Wikipedia - specifically their home wiki - would decline without page stabilization.
What can we learn from this feature?
edit- Expand watchlist capabilities to include “types” of articles, e.g. colors, dates, years to speed up fast edit reviews, replacing some of the needs for page stabilization.
- Provide further information and guidance on when and where to set pages in Semi-Protect mode
[If FlaggedRevs stopped working] we would want to put certain types of groups of articles under long-term or semi-protection. [The English Wikipedia community] would have an extensive discussion as a community about the pros and cons of doing so.
— enwiki editor
Unexpiring queue of newer editors’* edits to provide easy, focused access for patrolling quality control and vandalism
editWhy is this feature useful?
editAs initially intended, the FlaggedRevs queue “[allows] users to focus on un-reviewed changes and thereby [directs] their attention more effectively” (source). Edits by newer editors* are more likely to be vandalism and/or of lower quality - even if well-intentioned - and therefore require concerted patrolling. In addition, FlaggedRevs provides patrollers with access to oldest-first unreviewed edits regardless of when the edit was made. This configuration provides the increased visibility and additional time required to address “slow” edits.
What alternative is there to this feature, if any?
editWhile the FlaggedRevs tool provides an easily accessible queue, there are other (though more manual) ways to configure a very similar list through RecentChanges filters. The biggest and most impactful difference being that RecentChanges only shows the past 30 days. Outside of the FlaggedRevs and RecentChanges queues, editors are only able to pull > 30 day edits directly from the revisions table which doesn’t currently have a user-friendly frontend. In addition, RecentChanges also only shows edits in chronological order, newest first, while FlaggedRevs’ defaults to chronological order, oldest first.
| ⏰ RecentChanges | 🚩 FlaggedRevs | Queue best for… | |
| “Fast” edits | “Slow” edits | ||
| All activity | Unreviewed edits by newer users | 🚩 | 🚩 |
| 30 day log | All time backlog | 🚩 | 🚩 |
| Live feed | Not a live feed | ⏰ | ⏰ |
| Newest activity on top | Oldest activity on top | ⏰ | 🚩 |
We do have a queue for the PendingChanges as well here, but I would estimate that I go here maybe once a month. Most of my reviewing is done on the RecentChanges… because in the [PendingChanges] queue you first see the big and difficult changes that have been waiting for days. Whereas most of the easy cases like you can just pick them from the RecentChanges straight away and be done in 3 seconds with them.
— ukwiki editor
[I’m looking at] the RecentChanges page, because I'm interested in fighting vandalism, I'm obviously most interested in seeing what's happening now. To stop vandalism right away while it's happening.
— dewiki editor
What if the feature remains unmaintained?
editRelying on RecentChanges filters would dramatically shorten the patrolling window. Even for those Wikis with fast review rates, many slow edits are not addressed in less than 30 days.
What can we learn from this feature?
edit- Make it easier to imitate the FlaggedRevs queue in RecentChanges with a premade filter, preferably displaying X months of activity. The timeframe should be determined by behavioral data.
- Expand watchlist capabilities to include “topics”, e.g. academic disciplines (e.g. physics, mathematics) or areas of interest (e.g. video games, manga and anime) would allow known experts or Wikiprojects groups in that wiki editor community to add the relevant “topics” to their watchlists, similar to much of the quality assurance patrolling which FlaggedRevs currently solves for.
- Currently the closest “mechanism” for this is generally unstructured. Some wikis have an Expert needed template that invites anyone with subject matter expertise to contribute. These experts also need relevant Wikipedia policy expertise though which, definitionally then, is a small group of possible patrollers.
Codebase for automation bots and tools
editWhy is this feature useful?
editAs seen in this study, Finnish Wikipedia leverages the FlaggedRevs code for two automated anti-vandalism bots:
- SeulojaBot (“Sifter Bot”) uses AbuseFilter hits and ORES scores to mark good edits as “reviewed”. This means that SeulojaBot is designed to leave edits that the bot is unsure about, or are most likely to be bad edits, for human review. This is in contrast to anti-vandalism bots on most other Wikipedias, which are designed to revert edits that are identified as “bad” as fast as possible, aiming to leave minimal comments for human review.
- VakauttajaBot (“Stabilizer Bot”) stabilizes pages that are likely to be receiving high rates of vandalism (as determined by AbuseFilter and ORES hits). Five minutes after the first triggering edit the page is locked to the latest stable revision until a human patroller can check its activity.
It is unknown if there are similar bots used on other wikis.
What alternative is there to this functionality, if any?
editIf these bots broke, Finnish Wikipedia editors would need to more closely monitor the RecentChanges queue to manually revert, requiring more “fast” edit reviewers.
AutoModerator could replace some functionality. However, Finnish Wikipedia’s bot tools are made under the assumption that there must always be human review of the edit queue. Their two bots are built around trying to better prioritize what needs human attention, rather than reduce the amount of edits that require human attention as a whole. Therefore, this system works on a different design principle to AutoModerator.
What if the feature remains unmaintained?
editAs these bots are aimed at fast edits, vandalism would be more difficult to control if features these bots rely on go unmaintained.
What can we learn from this feature?
editBuild new automated anti-vandalism tooling that both
- Rejects high probability “bad” edits to decrease the need for human review
- Approves high probability “good” edits to prioritize human review on more ambiguous edits
Mode-specific Learnings
editOverwrite Mode
editOne Overwrite Mode editor expressed the concern that incorporating automated patrolling would decrease motivation for human patrolling.
While this anxiety stems from a slippery slope mentality, it is true that some editors enjoy FlaggedRevs work due to the gamification of reviewing fast edits right as they are made (think wack-a-mole) as well as continuously maintaining/shortening the queue.
I think one argument which is valid is that once you rely on artificial intelligence to do stuff for you it decreases the motivation to do manual patrolling.
— dewiki editor
Potential finding for future developments
editConsider alternative/additional ways of gamifying editing, e.g. create and display a “rate of review” editor-facing metric.
Protect Mode
editProtect Mode provides useful, unintended functionality that expands patrolling coverage to otherwise often unmonitored articles
Editors on English Wikipedia have found that many basic concept articles (e.g. colors, years, dates) that require higher levels of quality assurances and/or are more susceptible to vandalism often go unpatrolled without FlaggedRevs, since they don’t usually capture the attention of the editor community (e.g. not on many watchlists, RecentChanges moves too quickly for bad edits to be caught)
[The tool] is useful at least in some places and it actually has very good rationale for a couple things. The colors and the dates and years - these are articles that are not going to be highly watched, but have additional levels of expectation when information is being added to them that superseded what we might have for [other articles]... they are also articles that tend to have a very low number of watchers. So there is a justification for them.
— enwiki editor
On the flip side, using FlaggedRevs in Protect Mode also creates unnecessary focus on articles that don’t require this extra monitoring - especially since there is currently no process or guidance for adding or removing articles from the list. On English Wikipedia there are currently almost 4,000 articles in FlaggedRevs Protect Mode, many that fall outside this use case as well as many that do but aren’t included.
There are quite a few pages where Pending Changes was applied because it was the new cool tool… [the pages in FlaggedRevs] have never been revisited and [for many pages] there were no problems added since one or two just about the time that they were applied. That tells us that we’re making work for people.
— enwiki editor
If FlaggedRevs goes unmaintained
editSee Page stabilization to control public access to autoconfirmed editors’ edit
Inactive Mode
edit
Many Finnish Wikipedia editors not only use the FlaggedRevs queue, but also have a display (in red) that shows the number of unreviewed edits as well as the age of the oldest.
To address the issue, editors on Finnish Wikipedia have organized concerted one-off pushes to review the oldest edits.
I think sometimes we have had problems with the Pending Changes log getting too long for several months. But then sometimes some group of editors will check the whole log and, yeah, right now we have only 723. So that's quite low… [At one point] I remember seeing 3,000 hours was the oldest edit.
— fiwiki editor
If FlaggedRevs goes unmaintained
editSee Codebase for automation bots and tools
Quality Assurance vs Anti-Vandalism
editEditors report using FlaggedRevs for both quality assurance and anti-vandalism - many edits falling into both categories
- Examples of quality assurance edits: typos, good faith images, strong references
- Examples of vandalism edits: offensive language, pop culture gossip, irrelevant links (spam)
- More ambiguous edits: weak/questionable references, irrelevant/unnecessary text
Notably, many future editors’ first interaction with wikis fall into the “vandalism” bucket as they learn about the project.
My first Wikipedia edit as a teenager was a test edit. I was in school and we had some lessons and I thought okay, what happens if I replace the article text with nonsense…. [People new to Wikipedia] want to confirm the myth that you can actually contribute to Wikipedia because they don't know - they heard it, but cannot really believe it. So then doing a test edit is probably less well intentioned, but also it's not bad intentions.
— dewiki editor
Additional documentation
edit- Original FlaggedRevs release announcement: Quality Assurance in an Open Project
- Extension:FlaggedRevs
- Flagged Revisions - Meta
- metawiki:Talk:Community_Wishlist/Wishes/Regular_support_for_FlaggedRevs