Research:New page reviewer impact analysis
This page is an incomplete draft of a research project.
Information is incomplete and is likely to change substantially before the project starts.
In November 2016 new page patrolling was restricted to editors with a newly created user right called New Page Reviewer. The reason given for this change was that inexperienced patrollers were making incorrect judgements: many new articles that should have been marked for deletion were not being marked; others that needed more scrutiny were being marked as reviewed, and in some cases patrollers were behaving in a hostile manner towards good-faith new article creators.
The volume of incorrect review decisions required other new page patrollers to spend excessive amounts of time re-reviewing previously reviewed articles, rather than assessing the stream of new unreviewed articles, which contributed to a large backlog.
Goals
editIn this study, we will assess the impact of this restriction on who is able to review new articles.
We want to know how this new restriction has affected:
- the proportion of 'incorrect' review decisions by new page patrollers?
- the number of re-reviews that patrollers perform?
- the number of people who perform new page patrol?
- the number of people eligible to perform new page patrol?
- the rate at which new page patrollers can expect to eliminate the current backlog of unreviewed new articles?
Background
editAccording to the New Page Patrol project page, "New Page Review is essentially the first (and only) firewall against totally unwanted content and the place to broadly accept articles that may not be perfect but do not need to be deleted."
Methods
editFindings
edit1. Impact on proportion of incorrect review decisions
edit
Before the new user right was implemented in mid-November 2016, approximately 83% of articles marked for deletion by patrollers were subsequently deleted by administrators. After November 2016, the 'hit rate' for deletion reviews declined by approximately 10%: only 73% of new articles marked for deletion by patrollers with New Page Patroller user right were subsequently deleted.[1]
This suggests that the new restriction on who could review articles may have had the opposite effect than what was intended. Instead of increasing the consistency and correctness of review actions, it actually decreased it.
2. The number of re-reviews that patrollers perform
edit
Some conclusions that could be drawn are:
- The number of re-reviews is highly erratic.
- The number of re-reviews showed a huge spike just after the New Page Review rights implementation, and even after that remained somewhat higher than before.
3. the number of people who perform new page patrol
editThe number of users doing New Page Patrol has been continually decreasing as shown by the plot.
Some useful observations can be made:
- The users performing new page patrol has been constantly decreasing.
- The number of user doing NPP showed a downward spike just after the NPP rights implementation.
- The average reviews per user in each month remained roughly the same before the November 2016, then began to increase a bit. This means that the users having the New Page Patrol right have been doing more work than before.