Grants:IdeaLab/Measure replacement rate among the admins/Notes

An early mockup of a special page for health metrics.

It is possible to graph the metrics, and thus it could be an option to graph it at Wikimedia Statistics, but interpretation of a graph is difficult. It is easier to explain the numbers on a special page, although not much easier. It seems also like the community is more apt to accept pages generated on the wiki rather than pages on some external service.

The mockup should give an idea, but it does not have to look like this, and it does not have to be a special page.

Considerations for where to place the metric

Where to attach the health metric "Admin Replacement Rate" (ARR) in Wikistats 2.0

alignment

• ARR is about users, and so would align with editors
• ARR is about groups, and so would not align with editors

normalization

• Editors has no really clean normalization except against size of internet community within a language
• Health metric for an admin group (migration) has an internal normalization against the admin group and an external normalization against the larger group being managed

split

• Editors can be split in user types, where the admin group can be one of several
• Health metric for an admin group can be split in immigrants and emigrants, and an overall value

It seems like there are more reasons for splitting it out in a separate metric than trying to adapt the editors metric.

How to define the metric

This is the core idea. There should be a separate category "Health" (why?) with a metric "Admin replacement rate".

There should be a normalize pane with entries "admins" and "users", both limited to active admins and active users. Not quite sure if this should be limited to active users, but this is closer to the real group size. It will give higher numbers if there are inactive admins.

If no choice are provided for normalizing factors, then #Minimal replacement migration can be used as an argument for only using the internal change rate as a metric. This gives users leaving and joining the group as ${\textstyle \Delta P_{(t,t+1)}}$ or the change in the total population, and then it is possible to calculate ${\textstyle R_{(t)}^{\prime }}$ or the necessary replacement migration to avoid a decline in administrators.

There should be a split pane entry "Migration", with numbers for immigration, emigration, and migration. The entry "Overall total" uses number for total migration with normalization.

If no choice are provided for normalizing factors, then an additional entry can be provided for "minimal replacement migration".

Implementation

There are several possible ways to implement this, but given the existence of Wikistats 2 it would be better to reuse this. The user part of the existing system is described at Analytics/Systems/Wikistats 2 and the server part of the existing system at Analytics/AQS/Wikistats 2. Those two subsystems will be extended with a small additional user interface for visualizing the collected data, and one or more new endpoints for the server part.

The proposed metrics does not use any private data, and it should not be possible to extract or identify any particular user under normal use. As a border case it is although possible to measure a single admin, if the community has a single admin over a long enough period. This could be solved by dismissing data from periods with fewer than a set limit of admins. Census bureaus typically use a three or five persons as limits. If the limit is reached, then a longer period might have enough data.

User interface

In the user interface there will be an "admins" section similar to "editors", with a "totals" view. (example) This would be close to identical to editors, but the statistics would only be the number of admins. In the "Filter and split" section the previous mentioned statistics will be listed, possible with options to turn on and off additional graphs. Only a single graph for each each of the implemented statistics is part of this proposal.

Metrics to be computed

• #Totals
• #Net migration rate
• #Replacement rate^*
• #Conversion rate^*
• #Minimal replacement migration^*

There will be no additional dashboard as part of this project.

^* this point may be dropped if the project run out of time

Server API

In the server API there will be an endpoint for each of the necessary statistics for the aggregates, that is the health metrics mentioned above.

Statistics to be computed

• ${\textstyle N_{a}}$ the Number of active admins
• ${\textstyle M_{a}}$ the net Migration rate for admins
• ${\textstyle R_{a}}$ the Replacement rate for admins^*
• ${\textstyle C_{a}}$ the Conversion rate into admins^*
• ${\textstyle N_{a|promote}^{\prime }}$ the minimal replacement migration for admins^*
• ${\textstyle N_{u}}$ the Number of active users (or really non-admin users)^*
• ${\textstyle N_{a|promote}}$ the Number of promoted admins (or really users emigrating from non-admins and immigrating to admins)^*
• ${\textstyle N_{a|demote}}$ the Number of demoted admins (or really users emigrating from admins and immigrating to non-admins)^*

^* this point may be dropped if the project run out of time

The statistics provided at the endpoints should align with the metrics visualized in the user interface.

To implement this an endpoint like the following is necessary

get /metrics/admins/aggregate/{project}/{admin-type}/{activity-level}/{granularity}/{start}/{end}

where

• admin-type is one of "total", "active", "emigrating", "immigrating", or "migrating"

or (most likely)

• admin-type is one of "total", "net-migration-rate", "replacement-rate", "conversion-rate", or "minimal-replacement-migration"

Health metrics

These are some of the possible health metrics. I have only listed those that can be solved by slight variations over the same code.

Totals

Standard metric to mimic the other metrics, in particular the "editor" metric. It will only be a single metric, consisting of statistics provided by a single server endpoint.

${\textstyle N_{a}}$ is the Number of active admins

Net migration rate

The first one is the net migration rate into the active admin group. The trend for this number says whether the admin group as such are stable. This metric is the only one where it could be argued that it should be normalized against the total number of admins.

$M_{a,t}={\frac {N_{a|promote,t}-N_{a|demote,t}}{N_{a,t}}}$

where

${\textstyle M_{a}}$ is the net Migration rate for admins

${\textstyle N_{a|immigrate}=N_{a|promote}}$ is the Number of promoted admins (users emigrating from non-admins and immigrating to admins)

${\textstyle N_{a|emigrate}=N_{a|demote}}$ is the Number of demoted admins (users emigrating from admins and immigrating to non-admins)

${\textstyle N_{a}}$ is the Number of active admins

${\textstyle t}$ is an index for the bin at the given time

The number ${\textstyle M_{a}}$ is unitless.

Replacement rate

The second one is the replacement rate into the active admin group. The trend for this number says whether new admins are stable compared to the group of active admins. It can also be described as internal change rate.

$R_{a,t}={\frac {N_{a|promote,t}}{N_{a,t}}}$

where

${\textstyle R_{a}}$ is the Replacement rate for admins

${\textstyle N_{a|immigrate}=N_{a|promote}}$ is the Number of promoted admins (users emigrating from non-admins and immigrating to admins)

${\textstyle N_{a}}$ is the Number of active admins

${\textstyle t}$ is an index for the bin at the given time

The number ${\textstyle R_{a}}$ is unitless.

The metric for active admins is a nearly identical to the present active users, which is already measured, except it has an additional filter to include only the admin group. The metric number of migrating admins is immigrating admins or newly appointed admins, but a separate metric for emigrating admins or newly demoted admins could be interesting too. It is not obvious how the two metrics should be combined, that is the "total" in a split.

Conversion rate

The third is the user conversion rate into the active admin group. It is virtually the same formula as replacement rate, yet the actual numbers will be quite different. The trend for this number says whether new admins are stable compared to the group of active users. It can also be described as external change rate.

$C_{a,t}={\frac {N_{a|promote,t}}{N_{u,t}}}$

where

${\textstyle C_{a}}$ is the Conversion rate for admins

${\textstyle N_{u|emigrate}=N_{a|promote}}$ is the Number of promoted admins (users emigrating from non-admins and immigrating to admins)

${\textstyle N_{u}}$ is the Number of active users (or really non-admin users)

${\textstyle t}$ is an index for the bin at the given time

The number ${\textstyle C_{a}}$ is unitless.

The metric for active users is a nearly identical to the present active users, which is already measured, except it has an additional filtering to exclude only the admin group. The metric conversion rate can also be called replacement rate, turnover rate, or transition rate. A conversion rate is between specific population groups, while the migration rate is into or out of a specific group.

Minimal replacement migration

The fourth is the necessary migration into the admin group, given the number of active admins. minimal replacement migration. The trend for this number says whether the community are able to cope with changes in the admin community.

$N_{a|promote,t}^{\prime }={\frac {-\Delta P_{a,[t-1~t]}}{A_{a,t}}}$

where

${\textstyle N_{a|promote,t}^{\prime }}$ is the Number of users to promote to admins to avoid a decline in the group

${\textstyle A_{a,t}}$ is the retention rate of admin immigrants defined as; ${\textstyle 1-{N_{a|demote,t}}/{N_{a,t}}}$ (1 - instantaneous departure rate)

${\textstyle \Delta P_{a,[t-1~t]}}$ is the difference in the admin population over the time interval; ${\textstyle N_{u,t-1}-N_{u,t}}$

${\textstyle t}$ is an index for the bin at the given time

The number ${\textstyle N_{a|promote,t}^{\prime }}$ has "admins" as units.

The metric for minimal replacement migration use nearly the same statistics as replacement rate, except the number of demotes must also be calculated.

Alternate definitions for admin health metric

It seems like this is several slightly different metrics

Replacement rates

The number of real changes in the group normalized over the total number of members in the group (internal change rate)

This is number the number of new members joining the group plus the number of old members leaving the group normalized over two times the total number of members in the group. Those numbers are straight out of the user rights log given the members of the group. This is perhaps the easiest metric.

Whether previous members should be counted as new members can be debated, but in my opinion they should not. They will not create a real influx of new members, they are only re-appointments of the same members. By counting over a long enough time period re-appointments can easily be canceled out. Still note that the time period must not be longer than the fixed term used at some of the projects.

This metric is probably the one most users would use as a metric for a stale group.

Note that internal change rate is easier to calculate and interpret than replacement rate. The later can only be calculated for incoming new members, as old members leaving the group is nearly impossible to calculate. When has a member truly left the group?

The number of real changes in the group normalized over the total number of active users (external change rate)

This is the number of new members joining the group plus the number of old members leaving the group normalized over two times the total number of active users or the log of the active users.

It is not clear whether the internal or external change rate should be used. The internal one is correct whether the project is oligarchic or not, while the external one should be normalized over the log of the active users if the project is oligarchic. (Ie to be able to predict correct numbers, although if the project is oligarchic it has serious problems.)

Note that this isn't a probability.

Minimal replacement migration

The rate for minimal replacement migration is given by

$R_{(t)}^{\prime }={\frac {-\Delta P_{(t,t+1)}}{A_{(t)}}}$

where

• ${\textstyle R_{(t)}^{\prime }}$ = Replacement Migration avoiding the decline of population in year ${\textstyle (t)}$
• ${\textstyle A_{(t)}}$ = retention rate of immigrants year ${\textstyle (t)}$, defined by (1 - instantaneous departure rate)
• ${\textstyle \Delta P_{(t,t+1)}}$ = change in the total population in the time interval ${\textstyle (t,t+1)}$

This is the minimum to sustain a population over time. The retention rate at time t is compared with a known difference at a future time t+1 and gives the necessary replacement migration to avoid a decline in the population.

It is comparable to the problem given as internal change rate above, but a group does not have changes in the population except for immigration into the group and emigration out of the group. That is immigration and emigration must balance each other over time. It become slightly more complex when modelling retention of admins as users leaving straight out of the group. Eg. a group member "dies on service", but note dies as in leave the project.

Mean time

The mean time the members stay in the group (mean member time)

This the accumulated time for all members in the group, divided by the total number of members in the group, given the current active members. This metric comes from the user rights log.

This is a variant of the change rate as a low number for that will give a long time for mean member time. At some projects this number can approach the lifetime of the project.

Promote

The probability of one new group member in a lower group being promoted to an higher group (promote probability)

For most groups this is simply an extract from the user rights log. Unfortunately it is not possible to do this for the autoconfirmed group, so an alternate method to extract the new group members must be created. Also note that the influx to the higher group must be truly new members, and not just re-appointment of previous members (admins).

Usually this number is very low on Wikipedia, and because it is so low users have stopped expecting to become an admin. This leads to a higher status of being an admin, and people are then less likely to accept being wrong. Ie the social fall has become to large.

The mean time before promotion from one group to another group (mean time before promotion)

This number is simply a variant of promote probability but it is a lot easier to understand.

Because the promote probability is so extremely small the mean time before promotion grows so large it is nearly impossible for an ordinary user to be promoted during his or her lifetime on Wikipedia.

This number can be adjusted for the conditional probability that the mean time is for those that is in fact promoted. The number is still very high.

Activity ratio

The normalized activity level for a group against the total activity in recent changes (activity level)

The activity level of a group can be measured and normalized against the overall activity level in the recent changes feed, thereby giving a measure that say whether there are enough members in the group. That is the activity in the recent changes feed is used as a proxy for the groups overall activity. It does not say anything about the group actually doing their job, but it does say whether the group is large enough compared to some limit.

This number can be calculated from activity in the recent changes feed, with some additional lookup.

Response time

There is a bunch of actions that various groups should take, and some of these has (or should have) a maximum response time. Such actions are quite heterogeneous, but a lot of them are now commonly marked with some kind of Done template. Playing on this it could be possible to detect such markers on a page and calculate response times.

One possibility could be to add a tag on the revision if a special parser variable is incremented, that is a new call is added, and then log the time for the first occurrence of a comment inside a summary and the time for a done-tag on a revision with the same comment. A list of such pairs could be used for creating statistics about the response times on the specific page.

Statistics for a page would include minimum, maximum, mean, and median times. They could be made available as individual values, or as a block of stats. The later could be added automatically if any parser variable is added.

Note that it is not necessary to have a complete picture of response time for all actions to be able to give an estimate for response time, as we can use a proxy into some of the actions.

Alternate definitions for user type health metric

Oligarchiness

A few metrics from the paper of Shaw and Hill (that is the hypothesis’ about oligarchiness reformulated)

The number of new group members (aka administrators) over number of active contributors

Hypothesis 1; "The probability of adding new administrators declines as wikis’ contributor bases grow". The number should probably be normalized over active contributors and not number of registered users.

The number of contributions to project pages by group members (aka admins) over total number of contributions to project pages

Hypothesis 2; "Controlling for the total number of contributions to administrative pages, administrators will contribute more to administrative pages as wikis’ contributor bases grow".

The number of reverts by administrators of experienced contributors over the total number of contributions made by experienced contributors

Hypothesis 3; "Controlling for the total number of contributions made by experienced contributors, the number of reverts by administrators of such contributions will increase as wikis’ contributor bases grow".

These three indicators can be used as trends, comparing over an interval.

Egalitarian contributor

An egalitarian contributor can be modeled as a contributor with nearly linear behavior in H3, while an oligarchian contributor must be modeled as a contributor with non-linear behavior. Any given contributor will be a mixture of the two models, and the optimum mixture to describe the contributor is an optimization problem.

The reasoning goes like "An egalitarian contributor will be equally influenced by any other contributor as he/she is focused on his/her own contributions, thus an increase in other contributions will scale linearly given the users own normalized contributions." Contrary to this "An oligarchian contributor will be increasingly disturbed by other contributors as he/she is focused on his/her own goals, thus an increase in other contributions will scale non-linearly given the users own normalized contributions."

The non-linearness in H3 will probably be a pretty good metric for oligarchiness for a group of users. For a specific user the metric will be quite noisy. More data, that is sampling over longer time, will usually give less noise.

There will be an increase in H2, but I wonder if that is more a result of necessity than an expression of oligarchiness. Individual users will in any case have a wildly different metric for H2. For example will an user active on the deletion pages be flagged as oligarchian according to H2, but i'm not sure this is correct.

Meritocrat–oligarch–egalitarian plane

Over time there should be a part of the community that has clear signs on turning into a w:meritocracy. Those users will show up as ordinary users in group memberships, but they tend to be involved in project discussions too. Meritocrat differ from oligarchs by lesser use of revert tools, they contribute more. Meritocrat get their social capital from contributions, while oligarchs seems to rely more on power-structures.

Simply being an normal user with high activity is not a clear indication of being a meritocrat, the activity must involve project work. There seems to be a kind of plane spanned by meritocrat–oligarch–egalitarian

I'm not sure whether a deletionist can be a meritocrat, but it seems so. Perhaps the inclusionist–deletionist is somewhat orthogonal to the meritocrat–oligarch–egalitarian plane, or slightly oriented along the oligarch–egalitarian axis. A deletionist is more oligarch-like.

If the total group of oligarchs gets to large they will easily overwhelm meritocrats and take control over the wiki.

To much of either meritocracy or oligarchy would probably create a defunc community. A community on the meritocrat-side is probably a lesser evil than an overwhelmingly oligarchic community. An extreme egalitarian community would probably turn into chaos, as it will have little coordination.

Alternate definitions for new editors health metric

Retention of new editors

There is a pretty simple health metric for new editors; relative retention of promising new editors. Assume some new editors arrive at time t₀, and then their activity levels are measured at t₁ and t₂. If the activity level at t₂ is normalized over t₁, then the number should scale pretty well with whatever activity level the wiki has. If it drops to much, the retention gets to low, then something weird is going on and the community is to harsh to newcomers.

A better formula could be

$R_{(t_{0})}={\frac {A_{(t_{1})}-A_{(t_{2})}}{A_{(t_{1})}}}=1-{\frac {A_{(t_{2})}}{A_{(t_{1})}}}$

This relative retention of promising new editors will probably be somewhat dependent on the number of active users, that is normalizing against the number of active users should be enough

$R'_{(t_{0})}={\frac {R_{(t_{0})}}{N_{(t_{0})}}}$

It is likely that there will be some kind of non-linear dependency, so it should probably be modeled as a regression like the oligarchyness.

See also

• mw:Wikistats 2.0 Design Project
  • mw:Wikistats 2.0 Design Project/Vagrant
  • mw:Special:PrefixIndex/Wikistats
• mw:Analytics
  • mw:Analytics/Wikistats
• Wikitech:Analytics
  • Wikitech:Analytics/Systems
    • Wikitech:Analytics/Systems/Reportupdater
    • Wikitech:Analytics/Systems/Dashiki
      • Wikitech:Analytics/Systems/Dashiki/Configuration
    • Wikitech:Analytics/Systems/Wikistats2
      • Wikitech:Analytics/Systems/Wikistats2/Metrics
        • Wikitech:Analytics/Systems/Wikistats2/Metrics/FAQ
  • Wikitech:Analytics/AQS
    • Wikitech:Analytics/AQS/Wikistats 2
      • Wikitech:Analytics/AQS/Wikistats 2/Metrics Definition
• mw:Gerrit
  • mw:Gerrit/Tutorial
• Gerrit: self
• Github: vuejs/eslint-plugin-vue
  • Github: vuejs/eslint-plugin-vue/tree/master/docs/rules
• ESlint
  • ESlint: docs/rules
• Github: wikimedia/analytics-discovery-stats
• Github: wikimedia/analytics-limn-ee-data
• Github: wikimedia/analytics-reportupdater
• Github: wikimedia/analytics-dashiki