Community health initiative/Metrics kit/ja

コミュニティの健全性指標キットはコミュニティの健全性イニシアチブと共同で信頼と安全チームによってウィキメディア財団で調査されたプロジェクトです。その究極の目的は、ウィキメディアのプロジェクト単位でコミュニティの相対的な健全性を実証する公開された一連の統計とデータを提供することにあります。このプロジェクトは2018–2019会計年度に調査され、将来さらなるコミュニティ指標の開発をお知らせする予定です。

背景

As a movement, Wikimedians have always measured aspects of their communities. Data points such as editor activity levels, new users, and editor retention have been regularly collected. While these metrics provide some useful indications about the health of a project, they do not give major insights into challenges and specific areas needing improvement. The Community Health Initiative wants to build on the metrics work already done by individual contributors, affiliate groups, academics and researchers, and the Wikimedia Foundation.

This project has two primary goals:

1. to have regularly updated quantitative statistics that provide useful insights into aspects of a community’s health, and
2. to provide better qualitative options for finding insights that can’t effectively be measured through quantitative approaches.

This kit and the data it will contain will be targeted towards two major audiences:

1. The local and global communities, in order to ascertain the relative health of their projects and to more easily identify the early signs of an unhealthy community; and
2. The Wikimedia Foundation, who may make use of the data (now in one centralised place) to better identify trends in community health and direct development efforts towards solving problems.

進捗状況：コミュニティとの協議

プロセス

開催時期	段階
2018年7月–8月	指標の定義、ブレインストーミング、内部で発表
2018年9月	設計の関与、場所の決定、指標についてコミュニティと協議
2018年10月–12月	開発の進展、コミュニティとの協議を継続
2019年1月–3月	技術的な実装、プロトタイプ決定
遅くとも2019年6月末に完了	指標キットを公開

This project is currently at the design and consultation phase. At this point, we have a rough idea of who this project is for (local community members interested in judging the relative health of their communities/projects, and Wikimedia Foundation staff interested in monitoring this health for things like tool or policy development). We also have a rough shortlist of metrics we'd like the kit to include. That is where we need your help.

How to give feedback

While we have set up the framework for this and put down our ideas, this project will suffer without the community's knowledge and expertise. Here, we are asking you to give us feedback on our ideas and to offer your own.

Please use the discussion page of this project for your feedback. To make it easier, we have set up a number of sections there for the aspects of this project about which we are most excited to hear your opinions. One of those sections is titled Other feedback, because we might miss things otherwise. Please feel free to leave your comments and suggestions in one, several or all of the sections! Thank you in advance for your thoughtful insights!

How to engage deeper

In addition to asking for feedback on what we do, we are also looking for volunteers who like to work more closely with us on the project. There are several ways you can engage, if you are interested:

• You can work with us to refine the way we gather the data in one or more fields for all the Wiki communities
• You can help find bugs and mistakes we may make
• You can get our support to understand your own community better through community health metrics
• You can help spread the knowledge about those new metrics throughout the wikiverse

If you are interested in investing time into this project on one or more of those issues on a mid to longterm basis, please sign up under the corresponding section of the discussion page!

How we will use the results

Your feedback at this stage will directly affect our thinking with regards to this project. As one of the major audiences for this work, the community's insight is naturally invaluable. As such, it will all be taken aboard and used to guide the future of this project as we focus on its design and usecases.

Metrics

We're currently looking at metrics that reflect community health. These will be informed by comments and suggestions made during this consultation, as well as by individual user interviews and other design research methods. Right now, we're looking at metrics reflecting things like:

• Active administrator and user statistics;
• Backlog statistics (compared with the number of people working on them);
• Statistics already used to compare Wikimedia projects with each other (such as Wikipedia article depth);
• Rates of vandalism, blocks, and other administrative tasks;
• Other metrics we haven't yet come up with :)

Design

Two screenshots from the prototype as it presently exists.

We've spent a little time thinking about how this kit might ultimately look and be used by members of the community. Our requirements for this project are that it is:

• Findable: It should be easy for community members to find and make use of in their work.
• Readable/Accessible: It should be simple to read and absorb the information it is attempting to impart.
• Up-to-date: It should be updated live (or at least regularly), automatically and with minimal maintenance.
• Translatable: It should be available to users in as many languages as possible (ideally beginning with the UN standard languages), and be available for users to supply translations where none exist.

Right now, we are thinking of hosting this data on Toolforge, as that option would cover most of the above requirements. We are aware that many of the metrics points we'd like this kit to hit are already collected by Wikistats, and we imagine the finished product would likely turn out quite similar to this in look, feel, and target demographic.

発案時の計画

既存の作業から発展させる

コミュニティとの協議の上位案件には、WikiStats ポータルに由来するものを含みます。プラットフォームとして以下を提供します。

• 単一の利用機器数
• もっとも閲覧数が多い記事
• 新規にアカウント登録した利用者
• 新規に記載されたページ
• 編集件数
• 編集者数
• 編集されたページ数
• 差分の合計バイト値
• 差分の絶対バイト値

コミュニティ関与の洞察調査により、コミュニティの健全性の要素が複数、垣間見えたこと、また利用者数ベースで有益な情報を得ることができました。

個々のプロジェクトや投稿者は、さまざまなアプローチで多岐にわたるAPIクエリを用いて特定のワークフローや問題の洞察に取り組みました。一例として英語版ウィキペディアで行った AdminStats ならびに ウィキペディア間の管理者数の分析があります。

これまでの調査プロジェクトの大きな蓄積があり、基本的にコミュニティの健全性に関する話題、あるいは関連数要素や問題を取り扱っています。その一部を表組に抽出しました。もし見落としがありましたら、こちらのサブぺージに補足をお願いします！

Research name	Date begun	Date published	Qual. or Quant.?	Topic	Contact/Author	Relevance
How much do Wikipedians value editing Wikipedia?	2017-12	2018-05	Qualitative	Editor satisfaction	Jana Gallus; Avinash Gannamaneni; Aaron Halfaker	High
Growth and diversity of Technology team audiences	2017-11	2018-05	Qualitative	Technical contributors satisfaction	Jonathan Morgan	High
Holder Administration in anderen Sprachversionen AdminCon 2018	N/A	2018-03	Quantitative	Administrators	User:Holder	High
The Keilana Effect: Visualizing the closing coverage gaps with ORES	N/A	2017-08	Quantitative	Gender gap	Aaron Halfaker; Emily Temple-Woods	High
Wikimedia CH Community Survey 2016	2016-12	2017-02	Qualitative	Community Survey	Wikimedia CH	High
New editor support strategies	2016-05	2016-10	Qualitative	New Editors	Jonathan Morgan; Joe Matazzoni; Pau Giner	High
Becoming Wikidatians: evolution of participation in a collaborative structured knowledge base	2016-05	2016-10	Qualitative	New Editors	Alessandro Piscopo, et al.	High
Impact of The Wikipedia Adventure on new editor retention	2014-11	2016-10	Qualitative	New Editors	Sneha Narayan, et al.	Medium
Online Community Conduct Policies	2016-03	2016-06	Qualitative	Conduct policies in other online communities	Patrick Earley; Karen Brown	Medium
Teahouse long term new editor retention	2015-11	2016-05	Qualitative	Editor satisfaction	Rebecca O’Neill	High
Survey of user in Ireland	2015-09	2016-05	Both	Editor retention	Jonathan Morgan; Aaron Halfaker	High
Wikimedia Nederlands survey (2015)	N/A	2016-04	Qualitative	Community Survey	Wikimedia Nederland	High
Wikimedia CH Community Survey 2015	2016-01	2016-02	Qualitative	Community Survey	Verena Linder	High
Wikimedia Deutschland Editor Survey 2016	2015-06	2016-02	Qualitative	Community Survey	Wikimedia CH	High
Monthly wikimedia editor activity dataset	2015-04	2015-08	Quantitative	Editor activity	Aaron Halfaker	Medium
Active editor spike 2015 (July update)	N/A	2015-07	Qualitative	General - Community Health	Haitham Shamma; María Cruz	High
Community Health learning campaign	2015-07	2015-07	Quantitative	Editor activity	Aaron Halfaker	Medium
VisualEditor's effect on newly registered editors (May 2015 study)	2015-06	2015-05	Quantitative	New editors	Aaron Halfaker	Medium
Active editor spike 2015	2015-03	2015-04	Quantitative	Editor activity	Aaron Halfaker; Dario Taraborelli	Medium
Global South User Survey 2014	2014-09	2015-02	Qualitative	Community Survey	Haitham Shammaa	High
Talk Sentiment Analysis: Editor Retention and Editing Prediction	2014-10	2014-12	Qualitative	Editor retention	Sergio Martinez-Ortuno; Lars Roemheld; Deepak Menghani; Leila Zia	Medium
The Role of Wikipedia Mentorship Programs in the Newcomer Experience	2014-07	2014-12	Qualitative	New Users	Gabriel Mugar with Chris Schilling and Aaron Halfaker	Medium
Wikipedia Gender Inequality Index	2014-06	2014-12	Quantitative	Gender Gap	Piotr Konieczny	Medium
Communicating on Wikipedia while female	2014-11	2014-11	Qualitative	Gender gap	Laura Hale	High
Framing Support for Newcomers	2014-07	2014-10	Qualitative	New Users	Yla Tausczik; Jonathan Morgan	Low
Asking anonymous editors to register	2014-03	2014-10	Qualitative	New Users	Aaron Halfaker	Medium
Non-finite Processes in Human Social Phenomena	2012-10	2014-10	Qualitative	Cooperation	Simon DeDeo	High
Women and Wikipedia	2014-01	2014-09	Qualitative	Gender Gap	Amanda Menking; Jonathan Morgan	High
Women and Wikipedia: Contributions in a Collaborative Online Space	2012-03	2014-05	Quantitative	Gendergap	Melanie Kill	Low
Wikimedia Community Visualization	N/A	2013-12	Quantitative	Editor activity	Haitham Shammaa	High
Newcomer survival models	2013-09	2013-09	Qualitative	Editor retention/New editors	Aaron Halfaker	High
Onboarding new Wikipedians/OB6	2013-09	2013-09	Quantitative	New Editors	Aaron Halfaker	Low
Exploring editing dynamics in different language Wikipedias – Towards a substantive grounded theory	2013-05	2013-08	Qualitative	Social dynamics	Pasko Bilic	High
Getting to know the grassroots	N/A	2013-07	Qualitative	Community Survey	Wikimedia Nederland	High
VisualEditor's effect on newly registered editors/June 2013 study	2013-06	2013-07	Quantitative	New Editors	Aaron Halfaker	Low
Gender micro-survey	2013-06	2013-07	Quantitative	Gender	Howie Fung	Low
WikiProjects as virtual Teams	2013-01	2013-07	Quantitative	Social dynamics	Jonathan Morgan	High
Notifications	2013-05	2013-05	Quantitative	Impact of Technical Feature	Aaron Halfaker	Medium
Anonymity and conformity over the net	2012-01	2013-03	Quantitative	Effects of anonymity	Aaron Halfaker	Medium
Post-registration editor survey	2012-11	2012-12	Quantitative	Editor motivation	Steven Walling	Medium
Wikipedia Editors Survey 2012	2012-06	2012-11	Both	Community Survey	Tilman Bayer	High
Wikipedia Editor Satisfaction Survey	2009-09	2012-10	Both	Editor satisfaction	Denis Barthel; Manuel Merz	High
MoodBar/Email confirmation	2012-07	2012-08	Quantitative	Communication	Dario Taraborelli	Low
Editor Lifecycles	2012-06	2012-07	Quantitative	Editor retention	Shilad Sen	High
Master Thesis Cultural differences in motivations to contribute to Wikipedia	2012-05	2012-07	Quantitative	Editor Motivation	Sjarlot Stal; Nick Geurts	Low
Editor milestones	2012-05	2012-07	Quantitative	New Editors	Shilad Sen; Aaron Halfaker	Low
Shallow Entry: A problem-centric approach to new editor orientation	2012-04	2012-07	Quantitative	Editor Motivation	Maryana Pinchuk	Medium
[1]	2012-04	2012-06	Quantitative	Editor Motivation	Aaron Halfaker	High
Motivations to Contribute to Wikipedia	2012-03	2012-06	Unclear	Editor Motivation	Audrey Abeyta	Low
Necromancy	2012-04	2012-05	Quantitative	Editor Retention	Steven Walling	Medium
Wikipedia Editors Survey 2011 November	2011-04	2011-11	Quantitative	Community Survey	Barry Newstead	High
Women and Wikimedia Survey 2011	2011-05	2011-10	Qualitative	Gender gap	Sarah Stierch	High
Summer of research 2011	N/A	2011-08	Both	mostly New editors, but also Editor retention		Medium
Wikipedia Editors Survey 2011 April	N/A	2011-04	Quantitative	Community Survey	Mani Pande	High
Expert participation survey	2010-12	2011-02	Quantitative	Expert participation	Dario Taraborelli	High
Hebrew Wikipedia satisfaction survey	2007	2007-07	Qualitative	Community Survey	Wikimedia HE	High
100 Questions for Wikipedians	2005	2005-02	Qualitative	Community Survey	Aphaia	Medium
Teahouse group dynamics	2017-05	Ongoing	Quantitative	Impact of harassment	Yiqing Hua; Dario Taraborelli; Leila Zia; Lucas Dixon; Nithum Thain; Cristian Danescu-Niculescu-Mizil	Medium
Detox	2017-01	Ongoing	Quantitative	New Editors	Ellery Wulczyn	High
Study of harassment and its impact	2016-05	Ongoing	Qualitative	Social dynamics	Jonathan Morgan	Medium
Good Faith Newcomer Prediction	2016-05	Ongoing	Qualitative	Social Dynamics	Anna Filippova, et al.	High
Understanding the Dynamics of Hackathons for Science	2016-04	Ongoing	Qualitative	Time commitment and workflow	Kevin Schiroo	Medium
Measuring editor time commitment and workflow	2016-01	Ongoing	Quantitative	Toxic language	Ellery Wulczyn	Medium
Beyond the Gender Gap: Understanding Women's Participation in Wikipedia	2015-10	Ongoing	Qualitative	Gender gap	Danielle McDonald Corple	Low
Online harassment resource guide	2015-07	Ongoing	Qualitative	harassment	J. Nathan Matias, et al.	Medium
Editor Behaviour Analysis & Graphs	2015-06	Ongoing	Quantitative	Editor statistics	Jeph Paul	High
Wikipedia Equality Sweden	2014-10	Ongoing	Quantitative	New Users	Aaron Halfaker	Low
HHVM newcomer engagement experiment	2014-08	Ongoing	Qualitative	Gendergap	Björn Helgeson; Jonathan Morgan	Medium
The sudden decline of Italian Wikipedia	2014-05	Ongoing	Quantitative	Editor Activity Decline	Nemo with Aaron Halfaker	Medium
Dynamics of Online Interactions and Behavior	2012-01	Ongoing	Both	Roles and Functions	Ofer Arazy; Dario Taraborelli	High
Wikipedian and Internet addiction	2010-09	Ongoing	Both	Social dynamics	Dario Taraborelli	Medium
Measuring editor labor hours	2016-04	N/A	Qualitative	Addiction?	OppidumNissenae; Geoide; Alexmar983	Low
Roles and Functions within Online Production Communities	2015-08	N/A	Quantitative	Editor activity	Aaron Halfaker	Low

重点エリア

このプロジェクトの開始時には、それまでに集積したことと未着手だったことを基盤に、このプロジェクトでどのデータポイントと領域を優先課題とするべきか検討しました。それには既存の指標について適用方法の変更 (例：収集済みの情報に対し異なる割合と対照を適用) と併せて、新規の指標を集めることが関わります。指標キットの対象に含めたうち、適法範囲が広い領域は以下のとおりです。

• 関係者の分布
• 管理者の統計
• 利用者の統計
• 新規利用者アカウント
• 活動の頻度
• 破壊行為、付け回し、嫌がらせ行為の定着率
• 不正行為フィルター、破壊行為の通報
• 文化、民族、性別の障壁
• コンテンツの偏向
• 言語の障壁
• ツールを利用できるかどうか (アクセス)
• インターネットへのアクセス
• ライブラリや知的リソースの利用 (アクセス)

今後の計画：質的調査の進展と調査法の改善

コミュニティの健全性イニシアチブにより行われた、英語版ウィキペディアの管理者伝言板への事件報告分析に基づく発展。本プロジェクトは重要な伝言板について利用者の経験を調査しました。調査の支援を改善するとは - コミュニティがそれ自体について調査を行う場合に、より使いやすく適用が簡単なフォームや、回答者を引きつける適切な技術、さらに調査結果の分析を楽にする方策を用意することです。