Research:Standard metrics

This page is about a 2013/14 project for metrics standardization. For overall edit statistics across Wikimedia projects, see Statistics.
Metrics standardization, Wikimedia Research & Data Showcase, March 2014

Researchers, analysts, and product managers use a wide variety of metrics (from "monthly active editors" to "user's giving proportion in the dictator game"[1]) track and evaluate phenomena related to the Wikimedia projects. This page collects metrics which are suitable for wide use, which will make it faster to develop new research projects and easier to compare existing ones.

These metrics are mostly quantitive, but qualitive metrics are worth standardizing too. For example, researchers sometimes survey Wikimedia users and contributors about their subjective satisfaction with software. It would be sensible to devise a standard, well-considered way of asking such questions.

A high-level overview of the design of Rolling Monthly Active Editors, June 2014

Background

edit
 
Analysis example. An example of sensitivity analysis for the new editor definition: monthly count of newly registered users on the German Wikipedia performing at least one edit in their first day/week in the article namespace or across all namespaces.

Overview

edit

One way to group standard metrics is into 5 categories:

New users
these metrics provide indicators on the acquisition, activation and productivity of users joining Wikipedia or other Wikimedia projects for the first time.
Community
these metrics measure the overall composition, growth and volume of activity of existing communities, including both human and automated activity by bots.
Content
this category of metrics measures the growth and dynamics of content creation, including edits, new articles, uploads.
Curation
these metrics measure the quantity and quality of curation and moderation activities, such as reverts, deletions, blocks.
Traffic
these metrics measure traffic and readership of Wikimedia projects.

Evaluation

edit

Each metric and user class definition comes with supportive analysis whose goal is to understand how sensitive its definition is to specific parameter choices and whether the metric captures the same phenomenon in different projects. We strive to run sensitivity analysis across projects in different languages and of varying levels of maturity, but we welcome feedback to improve these definitions and to identify edge cases, particularly for smaller projects or projects with uncommon policies, where the proposed definition may not accurately capture the quantity it attempts to represent.

We also expect the use of these metrics in the first iterations of the design of Editor Engagement Vital Signs to reveal anomalies and interesting facts that are hard to anticipate until series for each metric are automatically generated for each Wikimedia project.

New users

edit

A   is a previously unregistered user creating a username for the first time on a Wikimedia project.

Depends on
none
Used in
New editor

A   is a newly registered user completing   edits to pages in any namespace of a Wikimedia project within   days since registration ( ).

 
New editor
Standardized definition
  •   = 1 edit
  •   = 1 day
Depends on
Newly registered user
Used in
Productive new editor

A   is a new editor who completes at least   productive edit(s) within   time since registration ( ).

 
Productive new editor
Standardized definition
Depends on
New editor
Used in
none

A   is a new editor who completes at least   edits within   time since registration ( ) and also completes   edits in the survival period  .

 
Surviving new editor
Standardized definition
  •   = 1 edit
  •   = 1 edit
  •   = 1 day
  •   = 30 days (~ one month)
  •   = 30 days (~ one month)
Depends on
New editor
Used in
none

Community

edit

The editor model

edit

The editor model is a suite of metrics which include subclasses of and funnel rates for monthly active editors.

A   is a registered user who completed   edits to pages in any namespace of a Wikimedia project between   and  .

 
Active editor (rolling)
Standardized definition
  •   = 5 edits
  •   = 30 days

A   is a newly registered user who both registered and completed   edits to pages in any namespace of a Wikimedia project between   and  .

 
New active editor (rolling)
Standardized definition
  •   = 5 edits
  •   = 30 days
Depends on
Newly registered user
See also
Rolling active editor

A   is a newly registered user who both registered and completed   edits between   and   and continued to complete   edits between   and  .

 
Surviving new active editor (rolling)
Standardized definition
  •   = 5 edits
  •   = 30 days
Depends on
Newly registered user
Rolling new active editor
See also
Rolling active editor

A   is a user registered before  , completed   edits between   and   and continued to complete   edits between   and  .

 
Recurring old active editor (rolling)
Standardized definition
  •   = 5 edits
  •   = 30 days
See also
Rolling active editor

A   is a user who completed less than   edits between   and   and completed   edits (but was not a R:newly registered user) between   and  .

 
Reactivated editor (rolling)
Standardized definition
  •   = 5 edits
  •   = 30 days


Other community metrics

edit

The following metrics do not form part of the Editor Model and are computed daily. These metrics will be delivered in stage 3 (2015-Q1)

A   is a user who is not a flagged bot and completed at least   edits on date  .

Standardized definition
  •   = 1 edits

A   is an unregistered user who completed at least   edits on date   via the same IP address.

Standardized definition
  •   = 1 edits

A   is a user who is a flagged bot and completed at least   edits on date  .

Standardized definition
  •   = 1 edits

A   is a user who completed at least   page creations across all namespaces on date  .

Standardized definition
  •   = 1 page creation

A   is a user who completed at least   media creations on date  .

Standardized definition
  •   = 1 media creation

Content

edit

these metrics will be delivered in stage 3 (2015-Q1)

  is a count of the number of edits saved by any users on date  .

Standardized definition

no parameters

  is a count of the number of edits saved by non-bot-flagged registered users on date  .

Standardized definition

no parameters

  is a count of the number of edits saved by anonymous editors on date  .

Standardized definition

no parameters

  is a count of the number of edits by flagged bot users on date  .

Standardized definition

no parameters

  is a count of the number of page creations across all namespaces on date  .

Standardized definition

no parameters

  is a count of media creations on date  .

Standardized definition

no parameters

Curation

edit

these metrics will be delivered in stage 4 (2015-Q2)

Traffic

edit

Page views

edit

See Research:Page view.

Unique devices

edit

See Research:Unique devices.

Supplementary resources

edit

Notes

edit