Research:New page reviewer impact analysis/Incorrect deletion decisions
How has the implementation of the New Page Reviewer right affected the proportion of 'incorrect' review decisions by new page patrollers?
The stated purpose of the implementation of the New Page Reviewer right was to reduce the number of incorrect review actions that resulted in inappropriate articles being marked as reviewed, and appropriate articles being marked for deletion. If the New Page Reviewer right had the desired effect, we should see a decrease in the 'false positive' rate for new articles that are marked for deletion by reviewers: in other words, more of the articles that reviewers mark for deletion should be subsequently deleted by an administrator (other than the reviewer).
Results
edit
Before the new user right was implemented in mid-November 2016, approximately 83% of articles marked for deletion by patrollers were subsequently deleted by administrators. After November 2016, the 'hit rate' for deletion reviews declined by approximately 10%: only 73% of new articles marked for deletion by patrollers with New Page Patroller user right were subsequently deleted.[1]
This suggests that the new restriction on who could review articles may have had the opposite effect than what was intended. Instead of increasing the consistency and correctness of review actions, it actually decreased it.
This finding has implications for quality control and for reviewer workload.
- If more of the deletion requests from reviewers are being rejected, it may be that reviewers are being too stringent in their application of deletion criteria, resulting in the deletion of good faith, promising articles (unless that article is subsequently 'saved' by an admin).
- If more of the deletion requests from reviewers are being rejected, admins and other reviewers may need to re-review more articles, resulting in extra work for them.
Results table
edit| Year | Month | article marked for deletion by patroller was deleted? (True/False) | # of articles marked for deletion | % of articles marked for deletion that were later deleted by an admin |
|---|---|---|---|---|
| 2015 | 12 | FALSE | 433 | |
| 2015 | 12 | TRUE | 1544 | 78.09812848 |
| 2016 | 1 | FALSE | 225 | |
| 2016 | 1 | TRUE | 1196 | 84.16608023 |
| 2016 | 2 | FALSE | 208 | |
| 2016 | 2 | TRUE | 1161 | 84.80642805 |
| 2016 | 3 | FALSE | 251 | |
| 2016 | 3 | TRUE | 1487 | 85.55811277 |
| 2016 | 4 | FALSE | 276 | |
| 2016 | 4 | TRUE | 1548 | 84.86842105 |
| 2016 | 5 | FALSE | 233 | |
| 2016 | 5 | TRUE | 1070 | 82.1181888 |
| 2016 | 6 | FALSE | 163 | |
| 2016 | 6 | TRUE | 783 | 82.76955603 |
| 2016 | 7 | FALSE | 169 | |
| 2016 | 7 | TRUE | 758 | 81.76914779 |
| 2016 | 8 | FALSE | 274 | |
| 2016 | 8 | TRUE | 1259 | 82.12654925 |
| 2016 | 9 | FALSE | 231 | |
| 2016 | 9 | TRUE | 1166 | 83.46456693 |
| 2016 | 10 | FALSE | 228 | |
| 2016 | 10 | TRUE | 1100 | 82.8313253 |
| 2016 | 11 | FALSE | 290 | |
| 2016 | 11 | TRUE | 831 | 74.13024086 |
| 2016 | 12 | FALSE | 442 | |
| 2016 | 12 | TRUE | 924 | 67.64275256 |
| 2017 | 1 | FALSE | 300 | |
| 2017 | 1 | TRUE | 958 | 76.15262321 |
| 2017 | 2 | FALSE | 342 | |
| 2017 | 2 | TRUE | 1071 | 75.79617834 |
| 2017 | 3 | FALSE | 320 | |
| 2017 | 3 | TRUE | 819 | 71.90517998 |
| 2017 | 4 | FALSE | 244 | |
| 2017 | 4 | TRUE | 751 | 75.47738693 |