Wiki labels

Documentation FAQ Source code Bug tracker

Wiki labels is a human computation system for use in "Wiki labels" WikiProjects. In order to perform difficult analyses (e.g. measuring the quality of newcomers over time^[1]) and train intelligent wiki-tools like ORES (e.g. for detecting vandalism^[2]^[3]^[4] and assessing the quality of articles^[5]^[6]^[7]), we need labeled data (also known as "hand coding") and lots of it. The Wiki labels human computation system is designed to make collaboratively labeling wiki artifacts quick and easy. The system is implemented as a MediaWiki gadget/server pair in JavaScript/CSS and Python/Flask/Postgres.

Wiki labels gadget. A screenshot of the Wiki labels interface integrated into en:WP:Labels.

Gadget

OAuth authorization. A screenshot of Wiki labels OAuth confirmation is presented.

Editors with the gadget installed will find the wiki labels "home" interface load as a single page app within en:Wikipedia:Labels or their local variant (if their wiki has been configured). This single page app will allow an editor to view active campaigns, request worksets and label revisions and other "tasks".

The system uses mw:Extension:OAuth to integrate with an editor's MediaWiki account. Actions performed in the wiki labels system will be associated with a user's centralauth account.

Installation

No installation necessary. Just point your browser to https://labels.wmflabs.org/ui/ and choose your wiki from the list.

Interface translations

Wiki labels is registered at translatewiki.net. Please see translatewiki:Translating:Revision_scoring to contribute translations.

Server

Behind the gadget, a server manages "campaigns", "tasks", "worksets" and "labels". This server provides a Restful interface for accessing this data. There are 4 basic modules: auth, users, campaigns and forms. The server also provides some user-facing services for designing new forms and access to a stand-alone gadget.

Form builder

labels.wmflabs.org/form_builder

A form builder interface is provided for the development and testing of forms to support new labeling tasks. Forms fields are configured using a YAML format that reflects OOjs UI. Forms can be submitted for review.

Form builder. A screenshot of the form builder.

Stand-alone gadget

labels.wmflabs.org/gadget

In order to ease development and testing, we also provide access to the gadget via a single page app. If for some reason the on-wiki version of the gadget is malfunctioning, users can work from this interface.

Stand-alone gadget. A screenshot of the stand-alone gadget.

Machine-readable paths

/campaigns/ - Lists wikis with available campaigns.

/campaigns/<wiki>/ - Lists out campaigns available for a wiki

Add ?campaign=stats to the query string to include statistics about a campaign in the response.

/campaigns/<wiki>/<campaign_id>/ - Returns data about a campaign.

Add ?tasks to the query string to include the campaign's tasks in the response.
Add ?worksets to the query string to include the worksets for the campaign in the response. Specify ?worksets=stats to include completion statistics with the workset listings.
Add ?assign to the query string to request a new workset assignment for the current user. This will only work if (1) the user is logged in, (2) there're tasks available to be assigned and (3) the user doesn't already have an open workset assignment.

/campaigns/<wiki>/<campaign_id>/<workset_id>/ - Returns data about a workset

Add ?tasks to the query string to list out the tasks for this workset.
Add ?worksets=stats to include completion statistics with the workset listing.
Add ?campaign to the query string to include relevant campaign data and ?campaign=stats to also include campaign statistics.
Add ?abandon to the query string to release the incomplete tasks from this workset. This will only work if (1) the user is logged in and (2) the workset in question is assigned to the user. Only incomplete tasks will be released from the workset.

/campaigns/<wiki>/<campaign_id>/<workset_id>/<task_id>/ - Returns data about a particular task.

Add ?campaign to the query string to include relevant campaign data and ?campaign=stats to also include campaign statistics.
Add ?worksets to the query string to include relevant worksets' data and ?worksets=stats to also include workset statistics.
Add ?label=<encoded json> to the URL. E.g. campaigns/enwiki/1/1/1?label={"damaging":true,"good-faith":false}. This will only work if the user is logged in.

/users/ - Provides access to data from the point of view of a user

/users/<user_id>/ - Returns information about a user's activities

Add ?campaigns to the query string to return all campaigns that the user has been assigned worksets for. Specify ?campaigns=stats to also include campaign statistics.
Add ?worksets to the query string to return all worksets assigned to the user. Specify ?worksets=stats to also include workset statistics.
Add ?tasks to the query string to return all tasks assigned to the user. Note that labels applied by other users will be hidden from this view.

/users/<user_id>/<campaign_id>/ - Returns information about a user's activities within a campaign
Add `?campaigns=stats` to the query string to include campaign statistics. Add `?worksets` to the query string to return all worksets assigned to the user. Specify `?worksets=stats` to also include workset statistics. Add `?tasks` to the query string to return all tasks assigned to the user. Note that labels applied by other users will be hidden from this view.

/auth/ - Provides support for authentication of users.

/auth/initiate/ - Starts an OAuth handshake with MediaWiki.
It is recommended that this path is requested from a pop-up window.

/auth/callback/ - Used by MediaWiki to complete an OAuth handshake.
When a pop-up window is redirected here, it will be automatically closed.

/auth/whoami/ - Prints information about logged-in user
...

/auth/logout/ - Deletes session and forgets about user
...

/forms/ - Provides access to form configuration and internationalization

These forms can be loaded by the labels gadget.

/forms/<form_name> - Returns the form configuration
...

References

↑ Halfaker, A., Geiger, R. S., Morgan, J. T., & Riedl, J. (2012). The rise and decline of an open collaboration system: How Wikipedia’s reaction to popularity is causing its decline. American Behavioral Scientist, 0002764212469365.
↑ West, A. G., Kannan, S., & Lee, I. (2010, April). Detecting Wikipedia vandalism via spatio-temporal analysis of revision metadata?. In Proceedings of the Third European Workshop on System Security (pp. 22-28). ACM.
↑ Smets, K., Goethals, B., & Verdonk, B. (2008, July). Automatic vandalism detection in Wikipedia: Towards a machine learning approach. In AAAI workshop on Wikipedia and artificial intelligence: An Evolving Synergy (pp. 43-48).
↑ Adler, B. T., De Alfaro, L., Mola-Velasco, S. M., Rosso, P., & West, A. G. (2011). Wikipedia vandalism detection: Combining natural language, metadata, and reputation features. In Computational linguistics and intelligent text processing (pp. 277-288). Springer Berlin Heidelberg.
↑ Hu, M., Lim, E. P., Sun, A., Lauw, H. W., & Vuong, B. Q. (2007, November). Measuring article quality in wikipedia: models and evaluation. In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management (pp. 243-252). ACM.
↑ Stvilia, B., Twidale, M. B., Smith, L. C., & Gasser, L. (2008). Information quality work organization in Wikipedia. Journal of the American society for information science and technology, 59(6), 983-1001.
↑ Warncke-Wang, M., Cosley, D., & Riedl, J. (2013, August). Tell me more: An actionable quality model for wikipedia. In Proceedings of the 9th International Symposium on Open Collaboration (p. 8). ACM.

[1] Halfaker, A., Geiger, R. S., Morgan, J. T., & Riedl, J. (2012). The rise and decline of an open collaboration system: How Wikipedia’s reaction to popularity is causing its decline. American Behavioral Scientist, 0002764212469365.

[2] West, A. G., Kannan, S., & Lee, I. (2010, April). Detecting Wikipedia vandalism via spatio-temporal analysis of revision metadata?. In Proceedings of the Third European Workshop on System Security (pp. 22-28). ACM.

[3] Smets, K., Goethals, B., & Verdonk, B. (2008, July). Automatic vandalism detection in Wikipedia: Towards a machine learning approach. In AAAI workshop on Wikipedia and artificial intelligence: An Evolving Synergy (pp. 43-48).

[4] Adler, B. T., De Alfaro, L., Mola-Velasco, S. M., Rosso, P., & West, A. G. (2011). Wikipedia vandalism detection: Combining natural language, metadata, and reputation features. In Computational linguistics and intelligent text processing (pp. 277-288). Springer Berlin Heidelberg.

[5] Hu, M., Lim, E. P., Sun, A., Lauw, H. W., & Vuong, B. Q. (2007, November). Measuring article quality in wikipedia: models and evaluation. In Proceedings of the sixteenth ACM conference on Conference on information and knowledge management (pp. 243-252). ACM.

[6] Stvilia, B., Twidale, M. B., Smith, L. C., & Gasser, L. (2008). Information quality work organization in Wikipedia. Journal of the American society for information science and technology, 59(6), 983-1001.

[7] Warncke-Wang, M., Cosley, D., & Riedl, J. (2013, August). Tell me more: An actionable quality model for wikipedia. In Proceedings of the 9th International Symposium on Open Collaboration (p. 8). ACM.

[1]

[2]

[3]

[4]

[5]

[6]

[7]