Research talk:Automated classification of edit quality/Work log/2016-09-26

Monday, September 26, 2016

So, I've been working on making the "prelabeler" work better. I realized that we were marking reverted edits as not needing review if they were performed by a trusted user. Also, both the prelabel and the label_reverted scripts were looking for evidence of reverts. Why not fix the functionality and merge it into a single script?

So here's the old flow of the prelabel script:

1. User in trusted group? --> {"needs_review": false, "reason": "trusted user"}
2. ... or user has more than 1000 edits? --> {"needs_review": false, "reason": "trusted edits"}
3. ... or revision was reverted --> {"needs_review": true, "reason": "reverted edit"}
4. ... or user has been blocked --> {"needs_review": true, "reason": "blocked user"}
5. ... else --> {"needs_review": true, "reason": ?}

So, in this case, reverted edits by trusted users/edit-counts will not need review. I think this is undesirable because we really want some review of reverted edits by "trusted users". Also, it's possible that a user with "trusted edits" was blocked! So, I've changed things around.

Here's the new flow of the autolabel:

1. Revision was reverted --> {"needs_review": true, "reason": "reverted edit"}
2. ... or user has been blocked --> {"needs_review": true, "reason": "blocked user"}
3. ... or user in trusted group? --> {"needs_review": false, "reason": "trusted user"}
4. ... or user has more than 1000 edits? --> {"needs_review": false, "reason": "trusted edits"}

In this order, we pick up all edits that were reverted and all edits by blocked users as "needing_review".

So, I did something else to make it easier to merge the results of autolabel with human-generated labels from Wiki labels. I write a new script called merge_labels that reports inconsistencies between which edits need review and which edits actually have a human label. The output looks something like this:

2016-09-23 17:18:07,059 ERROR:editquality.utilities.merge_labels -- 42799878 has no labels, but was flagged for review
2016-09-23 17:18:07,060 ERROR:editquality.utilities.merge_labels -- 42779921 has no labels, but was flagged for review
2016-09-23 17:18:07,061 WARNING:editquality.utilities.merge_labels -- 44676660 has labels, but was not flagged for review
2016-09-23 17:18:07,062 ERROR:editquality.utilities.merge_labels -- 44360294 has no labels, but was flagged for review
2016-09-23 17:18:07,062 ERROR:editquality.utilities.merge_labels -- 42482136 has no labels, but was flagged for review
2016-09-23 17:18:07,062 WARNING:editquality.utilities.merge_labels -- 44841873 has labels, but was not flagged for review
2016-09-23 17:18:07,065 ERROR:editquality.utilities.merge_labels -- 44621498 has no labels, but was flagged for review
2016-09-23 17:18:07,066 ERROR:editquality.utilities.merge_labels -- 42996223 has no labels, but was flagged for review
2016-09-23 17:18:07,066 WARNING:editquality.utilities.merge_labels -- 44805304 has labels, but was not flagged for review
2016-09-23 17:18:07,068 ERROR:editquality.utilities.merge_labels -- 44914894 has no labels, but was flagged for review
2016-09-23 17:18:07,068 WARNING:editquality.utilities.merge_labels -- 44154518 has labels, but was not flagged for review
2016-09-23 17:18:07,069 ERROR:editquality.utilities.merge_labels -- 42478939 has no labels, but was flagged for review
2016-09-23 17:18:07,069 WARNING:editquality.utilities.merge_labels -- 43934953 has labels, but was not flagged for review
2016-09-23 17:18:07,069 ERROR:editquality.utilities.merge_labels -- 44662371 has no labels, but was flagged for review
2016-09-23 17:18:07,070 ERROR:editquality.utilities.merge_labels -- 43054590 has no labels, but was flagged for review

In this case, I found that the labels for nlwiki had a lot of mismatching. Actually, there were about 860 mismatches! In almost all of these cases, there was a user who has not crossed the "trusted edits" threshold before the Wiki labels campaign, but has now crossed it. In some cases, a user who had crossed the threshold had been blocked. Here's a query that checks on a few of the edits: https://quarry.wmflabs.org/query/12724 --EpochFail (talk) 14:25, 26 September 2016 (UTC)