Research:Standard metrics

This page is about a 2013/14 project for metrics standardization. For overall edit statistics across Wikimedia projects, see Statistics.

Researchers, analysts, and product managers use a wide variety of metrics (from "monthly active editors" to "user's giving proportion in the dictator game"^[1]) track and evaluate phenomena related to the Wikimedia projects. This page collects metrics which are suitable for wide use, which will make it faster to develop new research projects and easier to compare existing ones.

These metrics are mostly quantitive, but qualitive metrics are worth standardizing too. For example, researchers sometimes survey Wikimedia users and contributors about their subjective satisfaction with software. It would be sensible to devise a standard, well-considered way of asking such questions.

Background

Analysis example. An example of sensitivity analysis for the new editor definition: monthly count of newly registered users on the German Wikipedia performing at least one edit in their first day/week in the article namespace or across all namespaces.

Overview

One way to group standard metrics is into 5 categories:

New users: these metrics provide indicators on the acquisition, activation and productivity of users joining Wikipedia or other Wikimedia projects for the first time.
Community: these metrics measure the overall composition, growth and volume of activity of existing communities, including both human and automated activity by bots.
Content: this category of metrics measures the growth and dynamics of content creation, including edits, new articles, uploads.
Curation: these metrics measure the quantity and quality of curation and moderation activities, such as reverts, deletions, blocks.
Traffic: these metrics measure traffic and readership of Wikimedia projects.

Evaluation

Each metric and user class definition comes with supportive analysis whose goal is to understand how sensitive its definition is to specific parameter choices and whether the metric captures the same phenomenon in different projects. We strive to run sensitivity analysis across projects in different languages and of varying levels of maturity, but we welcome feedback to improve these definitions and to identify edge cases, particularly for smaller projects or projects with uncommon policies, where the proposed definition may not accurately capture the quantity it attempts to represent.

We also expect the use of these metrics in the first iterations of the design of Editor Engagement Vital Signs to reveal anomalies and interesting facts that are hard to anticipate until series for each metric are automatically generated for each Wikimedia project.

New users

Newly registered user

A ${\text{newly registered user}}$ is a previously unregistered user creating a username for the first time on a Wikimedia project.

Depends on: none
Used in: New editor

New editor

A ${\text{new editor}}(n,t)$ is a newly registered user completing $n$ edits to pages in any namespace of a Wikimedia project within $t$ days since registration ( $T$ ).

New editor

Standardized definition

$n$ = 1 edit
$t$ = 1 day

Depends on: Newly registered user
Used in: Productive new editor

Productive new editor

A ${\text{productive new editor}}(n,t)$ is a new editor who completes at least $n$ productive edit(s) within $t$ time since registration ( $T$ ).

Productive new editor

Standardized definition

$n$ = 1 productive edit
$t$ = 1 day

Depends on: New editor
Used in: none

Surviving new editor

A ${\text{surviving new editor}}(n,m,t_{1},t_{2},t_{3})$ is a new editor who completes at least $n$ edits within $t_{1}$ time since registration ( $T$ ) and also completes $m$ edits in the survival period $[T+t_{2},T+t_{2}+t_{3}]$ .

Surviving new editor

Standardized definition

$n$ = 1 edit
$m$ = 1 edit
$t_{1}$ = 1 day
$t_{2}$ = 30 days (~ one month)
$t_{3}$ = 30 days (~ one month)

Depends on: New editor
Used in: none

Community

The editor model

The editor model is a suite of metrics which include subclasses of and funnel rates for monthly active editors.

Rolling active editor

A ${\text{rolling active editor}}(T,u,n)$ is a registered user who completed $n$ edits to pages in any namespace of a Wikimedia project between $T-u$ and $T$ .

Active editor (rolling)

Standardized definition

$n$ = 5 edits
$u$ = 30 days

Rolling new active editor

A ${\text{rolling new active editor}}(T,u,n)$ is a newly registered user who both registered and completed $n$ edits to pages in any namespace of a Wikimedia project between $T-u$ and $T$ .

New active editor (rolling)

Standardized definition

$n$ = 5 edits
$u$ = 30 days

Depends on: Newly registered user
See also: Rolling active editor

Rolling surviving new active editor

A ${\text{rolling surviving new active editor}}(T,u,n)$ is a newly registered user who both registered and completed $n$ edits between $T-2u$ and $T-u$ and continued to complete $n$ edits between $T-u$ and $T$ .

Surviving new active editor (rolling)

Standardized definition

$n$ = 5 edits
$u$ = 30 days

Depends on: Newly registered user; Rolling new active editor
See also: Rolling active editor

Rolling recurring old active editor

A ${\text{rolling recurring old active editor}}(T,u,n)$ is a user registered before $T-2u$ , completed $n$ edits between $T-2u$ and $T-u$ and continued to complete $n$ edits between $T-u$ and $T$ .

Recurring old active editor (rolling)

Standardized definition

$n$ = 5 edits
$u$ = 30 days

See also: Rolling active editor

Rolling re-activated editor

A ${\text{rolling re-activated editor}}(T,u,n)$ is a user who completed less than $n$ edits between $T-2u$ and $T-u$ and completed $n$ edits (but was not a R:newly registered user) between $T-u$ and $T$ .

Reactivated editor (rolling)

Standardized definition

$n$ = 5 edits
$u$ = 30 days

Other community metrics

The following metrics do not form part of the Editor Model and are computed daily. These metrics will be delivered in stage 3 (2015-Q1)

Daily unique registered editors

A ${\text{daily unique registered editor}}(D,n)$ is a user who is not a flagged bot and completed at least $n$ edits on date $D$ .

Standardized definition

$n$ = 1 edits

Daily unique anonymous editors

A ${\text{daily unique anonymous editor}}(D,n)$ is an unregistered user who completed at least $n$ edits on date $D$ via the same IP address.

Standardized definition

$n$ = 1 edits

Daily unique bot editors

A ${\text{daily unique bot editor}}(D,n)$ is a user who is a flagged bot and completed at least $n$ edits on date $D$ .

Standardized definition

$n$ = 1 edits

Daily unique page creators

A ${\text{daily unique page creator}}(D,n)$ is a user who completed at least $n$ page creations across all namespaces on date $D$ .

Standardized definition

$n$ = 1 page creation

Daily unique media creators

A ${\text{daily unique media creator}}(D,n)$ is a user who completed at least $n$ media creations on date $D$ .

Standardized definition

$n$ = 1 media creation

Content

these metrics will be delivered in stage 3 (2015-Q1)

Daily edits

${\text{daily edits}}(D)$ is a count of the number of edits saved by any users on date $D$ .

Standardized definition

no parameters

Daily edits by registered users

${\text{daily edits by registered users}}(D)$ is a count of the number of edits saved by non-bot-flagged registered users on date $D$ .

Standardized definition

no parameters

Daily edits by anonymous users

${\text{daily edits by anonymous users}}(D)$ is a count of the number of edits saved by anonymous editors on date $D$ .

Standardized definition

no parameters

Daily edits by bot users

${\text{daily edits by bot users}}(D)$ is a count of the number of edits by flagged bot users on date $D$ .

Standardized definition

no parameters

Daily pages created

${\text{daily pages created}}(D)$ is a count of the number of page creations across all namespaces on date $D$ .

Standardized definition

no parameters

Daily media created

${\text{daily media created}}(D)$ is a count of media creations on date $D$ .

Standardized definition

no parameters

Curation

these metrics will be delivered in stage 4 (2015-Q2)

Traffic

Page views

See Research:Page view.

Unique devices

See Research:Unique devices.

Supplementary resources

Preliminary drafts and background analysis for other metrics can be found in this category
Presentation at October 2014 metrics showcase
Presentation at Wikimania 2014

Notes

↑ Yann Algan et al. (2014), "Cooperation in a peer production economy: experimental evidence from Wikipedia."

[1] Yann Algan et al. (2014), "Cooperation in a peer production economy: experimental evidence from Wikipedia."

[1]