Wikimedia Meta-Wiki

Grants:IdeaLab/Measure replacement rate among the admins

< Grants:IdeaLab
Measure replacement rate among the admins
A community has an influx of new editors, and also an influx of admins. The first normalized against the later should be above a given threshold, otherwise something strange is going on.
countryNorway
themeediting statistics
contact email• Email is on my user page.
idea creator• Jeblad
this project needs...
volunteer
join
endorse
created on22:10, 19 July 2018 (UTC)


Note; this project is pretty risky without a proper investigation of how to create the endpoints that provides the data. That is Grants:Project/Experimental endpoints for wikistats.

Project idea

edit

What Wikimedia project(s) and specific areas will you be evaluating?

edit

Is this project measuring a specific space on a project (e.g. deletion discussions), or the project as a whole?

This is a health metrics to measure w:Group dynamics an support w:Group development within and among the admins. Such metrics are important to be able to say something about the staleness of the group as whole. If there are too few new admins the group can become stale. At the same time the group can consist of to many new editors, that too can create problems because the admin groups lacks knowledge. A community can also be unwelcoming towards new editors, and thus will become fragile over time.

There are four related metrics, where only some of them will be implemented. The most important one, and the one that is easiest to interpret, is the #Net migration rate (w:net migration rate). I chose to normalize against active users and active admins as I believe this is correct given the inactive users and admins. Using active users and admins can be viewed as births and deaths in a population.

Because the admin group changes pretty slow it makes sense to smooth the numbers for admin promote and demote over a year. The same could be done for active users and admins to lower the noise.

Describe your idea. How might it be implemented?

edit

Provide details about the method or process of how you will evaluate your community or collect data. Does your idea involve private or personally identifying information? Take a look at the Privacy Policy for Wikimedia’s guidelines in this area.

There are several possible ways to implement this, but given the existence of Wikistats 2 it would be better to reuse this. The user part of the existing system is described at Analytics/Systems/Wikistats 2 and the server part of the existing system at Analytics/AQS/Wikistats 2. Those two subsystems will be extended with a small additional user interface for visualizing the collected data, and one or more new endpoints for the server part.

The proposed metrics does not use any private data, and it should not be possible to extract or identify any particular user under normal use. As a border case it is although possible to measure a single admin, if the community has a single admin over a long enough period. This could be solved by dismissing data from periods with fewer than a set limit of admins. Census bureaus typically use a three or five persons as limits. If the limit is reached, then a longer period might have enough data.

Metrics to be computed

A more in-depth discussion about the metrics are at Grants:IdeaLab/Measure replacement rate among the admins/Notes.

This should be rather straight forward to implement, but as of this writing a large part of the existing base system is implemented on a WMF privat server. That makes it very hard to repurpose the code on existing systems. To implement the described idea a dev system must be set up, and possibly an endpoint defined at Wikitech:Help:Toolforge (previously mw:Tool labs).

What should be a minimal definition of a simple metric (with a somewhat more complex visualization) will be a rather involved investigation of how to create a new endpoint at Toolforge (with Druid and other tools) before any visualization can be made. Fake data could be used, but then the chance is pretty high that the metric does not make sense at all.

(I asked about access at Druid, and got a "No", so I assume this must be reimplemented somehow. It is not completely clear how parts of the system is set up, and whether it can be repurposed.)

Are there experienced Wikimedians who can help implement this project?

edit

If applicable, please list groups or usernames of individuals who you can work with on this project, and what kind of work they will do.

The proposed project would be implemented in cooperation with Trust and Safety for the Community health initiative/Metrics kit.

How will you know if this project is successful? What are some outcomes that you can share after the project is completed?

edit

I wonder if the net result of such statistics would be an increased willingness to get new admins, and that would imply an increasing conversion ratio.

The project will be a partial success if a few of the endpoints are implemented, and one health metric is fully implemented. This should be sufficient to get a feeling whether health metrics for the admin group makes any sense. It will be a full success if all the endpoints are implemented, and at least two of the health metrics is fully implemented.

Functional health metrics would be visible in the Wikistats 2 user interface and be functional. Functional server endpoints will be available as REST endpoints providing JSON structures.

How would your measurement idea help your community make better decisions?

edit

After you are finished measuring or evaluating your Wikimedia project, how do you expect that information to be used to benefit the project?

This measurement gives a quantitative metric about admins, and whether the group is sufficient and long time stable. In particular; if the group is declining, then the community should be warned and find new candidates.

The stats when finalized would be available as part of continued use of the Wikistats2 subsystem, that is the user interface part of the system, and users from Wikimedia communities can inspect them as they need. Users in the communities can visualize how they are trending, and how they compare to other communities. In particular they will get a better understanding of how the admin group evolves.

Do you think you can implement this idea? What support do you need?

edit

Do you need people with specific skills to complete this idea? Are there any financial needs for this project? If you can’t implement this project, can you scale down your project so it is doable?

Probably, but I have not investigated all parts of Wikistats 2 thus I can't be sure about the time estimates. In particular; it is not clear whether all parts related to ingestion is available and can be repurposed. Missing and confusing documentation is probably the biggest risk.

If I or someone else is to invest sufficient time to implement this, then there will be financial needs, but nothing unusual or exceptional.

The simplest way to scale this project is to limit the number of implemented metrics. Most of the core statistics necessary to calculate the health metrics is common for all of them, and should probably be implemented together. Only one of the core statistics is not part of net migration rate which is the one I believe should be implemented first, or at least early. When the user interface for net migration rate is implemented it is possible to stop further development without implementing any of the remaining health metrics.

Project goals

edit

What are your goals for this project? Your goals should describe the top two or three benefits that will come out of your project. These should be benefits to the Wikimedia projects or Wikimedia communities. They should not be benefits to you individually. Remember to review the tutorial for tips on how to answer this question.

The primary goal of this project is to create health metrics that will ease and better the understanding of w:group dynamics and w:Group development in the admin group. It will not end with a complete and exhaustive set of metrics, only some easy to implement metrics, and only metrics that should not be too provocative. There are far better metrics, but they are also a lot more provocative to the community, like measuring individual admins response time to incidents.

The secondary goal is to increase community awareness of the admin group, and the importance that this group is managed well. It is now a tendency to either refuse to accept the existence of a separate admin group in the community, or the other extreme to try to isolate the group from the rest of the community.

A ternary goal is to check out how the community react on such a tool, if this tool is acceptable, and whether there should be any limits on its use. This is not easy to measure, it must be judged by the reception of the tool when it is finished. I doubt this is possible before the tool is available.

Project impact

edit

How will you know if you have met your goals?

edit

For each of your goals, we’d like you to answer the following questions:

  1. During your project, what will you do to achieve this goal? (These are your outputs.)
  2. Once your project is over, how will it continue to positively impact the Wikimedia community or projects? (These are your outcomes.)

For each of your answers, think about how you will capture this information. Will you capture it with a survey? With a story? Will you measure it with a number? Remember, if you plan to measure a number, you will need to set a numeric target in your proposal (e.g. 45 people, 10 articles, 100 scanned documents). Remember to review the tutorial for tips on how to answer this question.

1. I will provide a functional server endpoint for the Number of active admins and the Net migration rate for admins, and visualization in the user interface for the Number of active admins and the Net migration rate for admins.

2. The statistics will be of use to end users, and will support the management of the admin group.

Do you have any goals around participation or content?

edit

Are any of your goals related to increasing participation within the Wikimedia movement, or increasing/improving the content on Wikimedia projects? If so, we ask that you look through these three metrics, and include any that are relevant to your project. Please set a numeric target against the metrics, if applicable. Remember to review the tutorial for tips on how to answer this question.

The project is about health metrics for a part of the community, that is the admin group, and it will not in itself lead to greater participation. It might although lead to a clearer understanding of the group dynamics in the admin group, and what kind of choices impacts the active admins.

Project plan

edit

Activities

edit

Tell us how you'll carry out your project. What will you and other organizers spend your time doing? What will you have done at the end of your project? How will you follow-up with people that are involved with your project?

There will be no public activities for this project until the code is finished.

I'm not sure it is wise to trigger to much discussion about a tool like this before it is available. Such discussions have a tendency to be very speculative and outright bogus. It is better to hold on the discussion until there are some clear examples on how it works.

Budget

edit

How you will use the funds you are requesting? List bullet points for each expense. (You can create a table later if needed.) Don’t forget to include a total amount, and update this amount in the Probox at the top of your page too!

Community engagement

edit

Community input and participation helps make projects successful. How will you let others in your community know about your project? Why are you targeting a specific audience? How will you engage the community you’re aiming to serve during your project?

Finished code will be visible in the user interface for Wikistats 2, together with other health metrics. That is users will find the metrics together with other similar metrics.

Get Involved

edit

About the idea creator

edit

Wikipedian since 2005-ish…

Participants

edit

Endorsements

edit
  • This is definitely an idea that should be implemented. I agree that knowing the replacement/turnover rate would be beneficial. TheSandDoctor (talk) 23:59, 20 July 2018 (UTC)
  • Not only knowing, but also making rules. What of a 1 jear sabbatical leave after 3 years admin, and after that jear submit a new candidature to be voted? --Havang(nl) (talk) 12:15, 28 July 2018 (UTC)

Expand your idea

edit

Would a grant from the Wikimedia Foundation help make your idea happen? You can expand this idea into a grant proposal.

Expand into a Rapid Grant

No funding needed?

edit

Does your idea not require funding, but you're not sure about what to do next? Not sure how to start a proposal on your local project that needs consensus? Contact Chris Schilling on-wiki at I JethroBT (WMF) (talk · contribs) or via e-mail at cschilling wikimedia.org for help!

Retrieved from "https://meta.wikimedia.org/w/index.php?title=Grants:IdeaLab/Measure_replacement_rate_among_the_admins&oldid=18862475"