Research:Wikipedia Primary School SSAJRP programme/Evaluation

Wikipedia Primary School SSAJRP programme

Timelines & progress

Method

Overview

Project evaluation main goals

Analyzing the state of the art of topics related to Wikipedia Primary School Project
Verifying the project impact in Wikipedia content
Evaluating significant variables of interest for other research projects

Evaluation steps

Current state, Hypothesis, 2014
Intermediate state, Comparison, 2015
Final state, Conclusions, 2016

Hypothesis and questions:

If an article is impossible to find it is useless. How easy is it to find an article?
- Single article
  - Number of templates and categories
  - Number of visits (per categories and portals)
  - Featured in homepage as an article and as a fact
  - Links to the article and interlinks Portals and Wikiprojects Google Rank
- Correlation with other articles
  - Total amount of articles in the same categories

How to “calculate” the quality of an article through quantitative data? Is it possible to define variables in order to extract significant qualitative information?
- Positive variables: Number of references and interlinks, Mentions, Portals and Wikiprojects, ecc.
- Negative variables; Issues Stub, ecc.
- Article: References and external links, Issues Templates, Edits
- Direct evaluations: Features in homepage, Article quality, Monitoring, Mentions
- Indirect evaluations: Links to the article, Number of visits, Page rank
- Assigning points according to direct and indirect evaluations in order to generate a quality rank.

Usually people write Wikipedia articles according to their interests. Can we drive interest?
- Articles size (and variation)
- Editor task force and casual contributors
- Talk page
- New users

Process

Define the list of 100 articles to evaluate
Extract data to visualize
Concept of the data visualization
First draft
Review and second iteration
Final data visualization

Preliminary analysis

Concept

The preliminary analysis aims to give a general view on the current status of the selected articles. The articles under analysis are 171.
The main goals of the intermediate analysis are the following:

Understand the current state of the selected articles
Define and visualize a set of parameters that can lead to the understanding of the ‘quality’ of the selected articles
Explore the relations among articles

Tools

In order to do the visual evaluation, a scraper has been implemented. It’s name is Wikimole and it is available on GitHub for further implementation.
The following is the process Wikimole uses:

Wikimole gets data by using Wikipedia Api, a jQuery page scraper and some other external data sources such as Google Page rank and the Wikipedia article traffic statistics.
After an external section of data filtering and cleaning, Wikimole visualizes data by using Gephi and D3.js.

Intermediate analysis

Concept

The intermediate analysis provides an overview of the state of the selected articles at six months from the previous one.
The articles under analysis are now 176. Among them 36 have been reviewed thanks to the involvement of the Wikipedia Community, 25 articles have been reviewed by expert reviewers.

The main goals of the intermediate analysis are the following:

Understand the impact of Wikipedia Primary School Project in the selected articles
Provide actionable information on how to improve the selected articles
Investigate the user interest in editing Wikipedia articles and how it changes over time

Two new features the data visualizations show:

The benchmarks regarding the previous analysis
The articles reviewed by the Wikipedia Community and by expert reviewers.

Results

Phase 0

Report of the initial status of the selected articles.

Network of incoming links
Network of outgoing links
Incoming and outgoing links
Pageviews
Features of the articles
Edits

Phase 1

Report of the provisional status of the selected articles considering the previous visual analysis. All datasets are open and available online.

Network of incoming links
Network of outgoing links
Incoming and outgoing links
Incoming and outgoing links (link from and to articles excluded)
Features of the articles
Features: comparison before-after of articles improved through the Wikipedia School Project
Features: comparison before-after of articles improved through assessment
Features: comparison before-after of articles improved by Wikipedia community
Features: comparison before-after of articles improved through edit-a-thon
Features: comparison before-after of articles improved by experts
Features: comparison before-after of articles improved through reassessment

Phase 2

Report of the final status of the selected articles considering the previous visual analysis. All datasets and protocols are open and available online.

Articles network in 2015
Articles network in 2017
Network of incoming links
Balance between incoming and outgoing links
Balance between incoming and outgoing links (except links to articles)
Pageviews
Comparison of articles network in August 2015 and August 2017

Findings

The cluster of articles related to South Africa has been enhanced. That is due to the larger number of mutual links between articles belonging to South Africa topic. Some articles that links to the selected articles are the following:

Incoming links has slightly decreased. Outgoing links are increased. The two new articles created under the framework of WIkipedia Primary School project (Kaditshwene and Makhonjwa Mountains) have respectively 13 and 25 incoming links, 46 and 58 outgoing links.
Significant reduction of issues related to articles under examination. Despite reviews, some issue on articles under examination still remain. Among them the following articles have issues:

The number of features of the articles (within WIkipedia Primary School project we consider features the following: references, notes, images, see also) has grown. It emerges the trend to increase the features in the lacking articles and to filter out the relevant features in articles with a rich body. Articles with few features, among those under examination, are:

Between 2014 and 2015 the total amount of editors has slightly decreased (from 845 to 773, -8.5%). The number of edits has slightly increased (from 1463 to 1481, +1,2%). It suggests a collective greater effort in contributing to the develop of Wikipedia articles. Among articles with none or very few edits in 2015 we found: