Objective Revision Evaluation Service/wp10

Article quality in Wikipedia is of critical concern. Many wikis implement a features article process (e.g., en:Wikipedia:Featured article) for identifying high quality content and many WikiProjects use quality assessments to prioritize and direct work. But this assessment work is extremely time intensive and assessments will become out of date. This model makes predictions about article quality to support these processes.

Contexts

English Wikipedia (enwiki)

See en:Wikipedia:Version_1.0_Editorial_Team/Assessment for discussion of the rating scale. See also work by Warncke-Wang et al.^[1]
https://ores.wmflabs.org/v2/scores/enwiki/wp10/?model_info

ScikitLearnClassifier
 - type: RF
 - params: random_state=null, oob_score=false, warm_start=false, center=true, balanced_sample_weight=false, bootstrap=true, max_depth=null, criterion="entropy", balanced_sample=true, scale=true, min_weight_fraction_leaf=0.0, verbose=0, min_samples_split=2, min_samples_leaf=8, max_leaf_nodes=null, n_estimators=320, max_features="log2", class_weight=null, n_jobs=1
 - version: 0.3.1
 - trained: 2016-03-13T14:09:52.020303

Table:
	         ~B    ~C    ~FA    ~GA    ~Start    ~Stub
	-----  ----  ----  -----  -----  --------  -------
	B       378   263     85    164       123       13
	C       178   452     35    126       182        8
	FA       54     7    652    154         1        1
	GA       43    52    279    603         4        0
	Start    57   161      1     14       682      106
	Stub      1     7      0      0       123      873

Accuracy: 0.619
ROC-AUC:
	-------  -----
	'B'      0.829
	'C'      0.844
	'FA'     0.95
	'GA'     0.913
	'Start'  0.908
	'Stub'   0.984
	-------  -----

F1:
	-----  -----
	Start  0.639
	GA     0.591
	B      0.435
	Stub   0.871
	FA     0.679
	C      0.47
	-----  -----

French Wikipedia (frwiki)

See fr:Projet:Évaluation for a discussion the rating scale.
https://ores.wmflabs.org/v2/scores/frwiki/wp10/?model_info

ScikitLearnClassifier
 - type: RF
 - params: warm_start=false, max_depth=null, criterion="gini", oob_score=false, random_state=null, class_weight=null, max_features="auto", center=true, n_estimators=501, verbose=0, min_weight_fraction_leaf=0.0, min_samples_split=2, bootstrap=true, n_jobs=1, balanced_sample=true, scale=true, min_samples_leaf=8, balanced_sample_weight=false, max_leaf_nodes=null
 - version: 0.2.0
 - trained: 2016-03-13T14:37:22.965656

Table:
	       ~a    ~adq    ~b    ~ba    ~bd    ~e
	---  ----  ------  ----  -----  -----  ----
	a      32      84    48     88     19     4
	adq    26     217    11     81      0     0
	b      23      21   144     43     65    11
	ba     30      83    21    171      2     2
	bd      2       1    50      3    161    56
	e       0       0    11      1     53   226

Accuracy: 0.531
ROC-AUC:
	-----  -----
	'a'    0.719
	'adq'  0.891
	'b'    0.838
	'ba'   0.848
	'bd'   0.905
	'e'    0.959
	-----  -----

F1:
	---  -----
	e    0.766
	a    0.165
	b    0.486
	adq  0.586
	ba   0.491
	bd   0.562
	---  -----

Russian Wikipedia (ruwiki)

https://ores.wmflabs.org/v2/scores/ruwiki/wp10/?model_info

ScikitLearnClassifier
 - type: RF
 - params: n_estimators=501, oob_score=false, scale=true, min_samples_split=2, verbose=0, class_weight=null, random_state=null, criterion="gini", max_depth=null, n_jobs=1, min_weight_fraction_leaf=0.0, balanced_sample=true, min_samples_leaf=8, max_features="auto", max_leaf_nodes=null, balanced_sample_weight=false, center=true, warm_start=false, bootstrap=true
 - version: 0.0.1
 - trained: 2016-06-07T17:34:08.180792

Table:
	       ~I    ~II    ~III    ~IV    ~fa    ~ga    ~sa
	---  ----  -----  ------  -----  -----  -----  -----
	I      36     46      24      3     23     47     39
	II     28     75      31      6      8     20     37
	III     7     48     117     36      2      0     22
	IV      1      7      57    157      1      0      5
	fa      6      1       1      1    158     50      4
	ga     19      5       7      3     50    143     16
	sa      6     12       3      0      0     17    207

Accuracy: 0.561
ROC-AUC:
	-----  -----
	'I'    0.73
	'II'   0.782
	'III'  0.868
	'IV'   0.956
	'fa'   0.939
	'ga'   0.888
	'sa'   0.956
	-----  -----

F1:
	---  -----
	IV   0.724
	fa   0.683
	II   0.376
	I    0.224
	ga   0.55
	III  0.496
	sa   0.72
	---  -----

References

↑ Appendix A in Warncke-Wang, M., Ayukaev, V. R., Hecht, B., and Terveen, L. (2015). The Success and Failure of Quality Improvement Projects in Peer Production Communities (PDF). CSCW. , as well as Warncke-Wang, M., Cosley, D., and Riedl, J. (2013). Tell Me More: An Actionable Quality Model for Wikipedia. OpenSym/WikiSym.

[1] Appendix A in Warncke-Wang, M., Ayukaev, V. R., Hecht, B., and Terveen, L. (2015). The Success and Failure of Quality Improvement Projects in Peer Production Communities (PDF). CSCW. , as well as Warncke-Wang, M., Cosley, D., and Riedl, J. (2013). Tell Me More: An Actionable Quality Model for Wikipedia. OpenSym/WikiSym.

[1]