Grants talk:Project/Eurecat/Community Health Metrics: Understanding Editor Drop-off

Past learning

Maybe I've read too fast, but I'm a bit surprised that this proposal doesn't clarify what exactly it's going to build on, and what previous experiences it has considered and discarded. Statistics links a number of existing tools for visualisation and analysis and the WMF has invested tens or sometimes hundreds of thousands of dollars into other tools for such metrics, including some focused on MediaWiki development.

Is there a separate page where you've listed your complete references and what models you're going to use? I'm shocked for instance that Ortega is not mentioned, although his was the first cross-language analysis we had beyond Zachte's WikiStats, and for many years it was also the only one. I think you cannot start such a project, or meaningfully assess how big it's going to be, before deciding which metrics and methods you're going to use and what tools you're going to need for them. Nemo 12:12, 23 February 2020 (UTC)

Hi Nemo, thank you for your comment. We listed some of the metrics that we plan to compute in the Section "Activities" of the proposal. In particular, we mentioned: edits in different namespaces; reverts, mutual reverts, revert chains; reverts with no personalized edit summary (empty, or auto-generated); having content deleted by another user; comments written, replies received, comments received in personal talk page, mutual replies, reply chains, discussion thread depth. All these metrics have already been defined in the literature (some by us or by Aaron who is an advisor of the project), so we will build on those definitions. Yet, we do not consider this list to be exhaustive and we are open to feedback from the community in terms of which metrics are relevant to understand editor drop-off. That said, the first task of the project is to review the literature and establish a definition of editor drop-off.

About the references, we know Ortega's work and other relevant publications, but we decided to draw a line about the detail of the proposal. --CristianCantoro (talk) 16:03, 24 February 2020 (UTC)

Ok, so it's still not clear to me whether this is more a development project or a research project but I might have misread something. I see 3 months allocated for the "computing" of the metrics and outputs include "toolkit to compute", but that might mean anything from documentation on how to run the wikistats scripts to writing an entirely new software to make those queries on XML or a database (although below in another answer you mention XML). I don't see any mention of reusing existing software.

It would be useful to wikilink the metrics listed, for instance to the various Research:Standard metrics pages or mw:Analytics/Metric definitions, to make sure people are talking about the same things. That should not make the text heavier than you planned to. Nemo 10:41, 8 March 2020 (UTC)

General Observations and Questions

Hello,

First of all I would like to congratulate you for this idea and do endorse it. I have a number of observations/suggestions where I would like to have your clarifications:

1. Gender Gap - You have mentioned explicitly gender-gap as one of the reasons of dropping. While this is of course true, and no doubt about it, I think it is not wise to single out a reason and give it more importance than others, especially that you did not start your research yet, so that you do not fall in the "confirmation bias", where you already put the solution for the problem you did not study yet. In my region, women can for example stop editing after getting married or having children, which is definitely not related to any online harassment for example. Likewise, many people in my region stop editing for economic reasons, as they cannot afford to be volunteers while looking for work. This was not mentioned while it can be as important as the "gender" argument. In my opinion, it is better not to go in details, and not to list any cause (or list all of them) so that you do not already "label" your research, as the grant title is "Understanding Editor Drop-off" and not "Understanding Female Editor Drop-off".
2. Drop-off Reasons - It seems to me that most of your research will be about the "online" reasons on why Editors drop off. While it is true that editing happens online, there were many cases where "offline" problems led to "online drop-off". For instance, I witnessed a number of cases where conflicts in User Groups or gatherings or even personal problems (not at all related to the action of editing) caused people to leave the movement. How do you plan to study this?
3. Type of Drop-off - After Reading your proposal, I have the feeling that it is a "black or white" approach related to drop-off. I do not think that it is true. Many people are multi-lingual and can leave one community but remain with others. I know for instance people who dropped the Arabic community but are very Active in the French speaking. Will the current metrics that you propose consider the people as if the dropped off or not? It does not seem clear to me. Of course, you can have a limitation in that regard, but then please mention it so that we are aware about it.

Otherwise, I believe again that the idea is great and is definitely worth getting started! Best of luck!

Regards -- Anass Sedrati (talk) 18:11, 26 February 2020 (UTC)

Hi Anass Sedrati, thanks for your feedback. Here's our answers about each point:

1. Gender Gap: we want to study gender and other characteristics of users (such as native language and provenance) not as a reason for drop-off, but as a variable. Since we think that in principle there could be different dynamics of drop-off for different user groups - something that would be related to the fact that some group ends up being underrepresented - then we want to study drop-off for that group. In a sense, it is similar to the fact that we study drop-off across language editions. The drop-ff dynamics in the English Wikipedia could be different from the ones in Catalan Wikipedia, maybe because of different community size. So, when we focus on underrepresented groups, we plan to look in-depth at a subset of users to see if for those users the dynamics are different. We do not think a priori that the simple fact that a user belongs to a certain group is a reason for dropping off.
2. Drop-off Reasons: we recognize that there can be a number of off-wiki reasons for leaving the project, including, as you say, interaction with other Wikipedians in real life (user groups, meetups, etc.). We think that the scope of this project is to detect those causes that depend on in-wiki behavior. We think these reason would require further study, using a different methodology.
3. Type of Drop-off: we are aware that a user can edit across multiple wikis, we do plan to take cross-wiki editing into consideration. Also, take into consideration that we plan to have a definition of drop-off as a result of the first activity of the project.

I hope this answers your concerns. --CristianCantoro (talk) 14:09, 27 February 2020 (UTC)

Hello Cristian and thank you for the answer and the clarifications. As a quick follow-up, I understand that gender is only one of the metrics of your study (in fact this was what I mentioned in my first message), therefore it should not be given more than its place (among the many other factors) :)

For the off-wiki and inter-wiki discussion, I understand that the study cannot cover everything (no study does), so I just want you to state clearly this as a limitation when performing your work.

Thank you again for taking time to answer, and a big good luck for your work! -- Regards -- Anass Sedrati (talk) 20:51, 27 February 2020 (UTC)

Classification of edit types

Thank you for planning this interesting project! Although your project focuses on experienced editors, I think we on the Growth team may also learn things that we can apply to newcomers. It doesn't look like you have yet defined exactly what metrics you'll be looking at, but one that I think could be especially valuable is a classification of edit type, and then counting those up for users over time. By classification of edit type, I mean being able to identify whether an edit is a copyedit, minor content addition, major content addition, reversion, content removal, etc. Then we could classify editors based on which kinds of edits they do, and understand their lifecycles in a more nuanced way. Is this the sort of thing that might be part of your project? -- MMiller (WMF) (talk) 22:17, 27 February 2020 (UTC)

Hi MMiller (WMF), thanks for your comment. This fits very well with what we in mind. In the first part of the project we will compute several metrics over time and it would be great for us to be able to discuss with your team to make sure that we are aligned. --CristianCantoro (talk)

That sound good, CristianCantoro. We're happy to talk whenever you're ready. -- MMiller (WMF) (talk) 19:56, 2 March 2020 (UTC)

Endorsement and Suggestions

I have been asked to provide feedback on this project proposal by a member of the project team, online, in a direct contact.

First of all: this is a well formulated project proposal.

The goals are clear and comprehensive. The goals are also not too ambitious (e.g. it is not planned to demonstrate a decline in editor drop-off after some suitably planned intervention based on the results), which I see as positive since we are all aware that many factors whose effects cannot be assessed by this study or from the WMF data sets alone influence editor behavior and one study will not solve the problem.

Next, the authors demonstrate full understanding of the goals of the movement and its strategy, which is exemplified in the way the description of the project is conceptually aligned with the way in which the movement's strategy is expressed. I find this particularly important because it shows that the authors will be capable to manage the research project in a way to deliver the results which will be truly useful both to the Community and to the WMF. Not only results count, but the way they are formulated and communicated too. In a case of a study of complex online editor behavior, where, once again, no available data set will be able to provide for the full understanding of the phenomenon of interest, I find this competence that the authors have demonstrated to be rather important to recognize. Among many ideas on how to approach the problem of editor retention those that plan to communicate what they have learned in a language that the Community and the Foundation really understand should be given some advantage.

That being said, I also find the relevance of this research project to be unquestionable.

What I see as a possible improvement here is a bit more precise description of the health metrics indicators that would be developed. Of course, there would not be any need for a research project if the authors would have precise answers to questions like this all sorted out and ready. However, given that the WMF documents its public data sets in a clear and comprehensible manner on Wikitech - for example, probably the most interesting initial data set in the context of this project: https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Edits/MediaWiki_history - I find that it might be helpful to assess this project better if we could learn a bit about the ideas on how would the authors use the available data sets to develop new indicators. Also, maybe a bit more precise idea on how would the new indicators improve our understanding of editor retention beyond the existing ones. For example, in the Project goals section of the proposal, 2. Increase our understanding of the factors associated with editor drop-off, it stated: "We will collect data about different kinds of interaction over time and study their relationship with editor drop-off", and my question would then be: what kind of interactions would the authors consider? For example, would they consider using available data sets on editor interactions like https://github.com/conversationai/wikidetox/tree/master/wikiconv? Or do they suggest that the production of new data sets on editor interactions is necessary? This is important for the evaluation of this (and any similar) proposal given the high complexity associated with the production of any new editor interaction data sets from Wikipedia and sister projects.

All my comments and opinions expressed here should be seen only as my suggestions to the authors on how to possibly improve this otherwise beautiful submission. I fully endorse this project proposal. GoranSM (talk) 08:10, 29 February 2020 (UTC)

Hi GoranSM, thank you very much for your support and valuable feedback. The core of our contribution with this proposal is not necessarily to propose new interaction metrics (we will mostly rely on state of the art metrics, and extend them only when necessary), but to study these metrics over time to find their relationship with editor lifecycle and drop-off. In the second paragraph of section Activities we have reported a non-exhaustive list of metrics that we will consider, and of course, we welcome suggestions on other metrics that could be included in this stage.

In this description we have focused on the high-level idea of the project, not on the implementation details and datasets that can be (re)used. In this sense, we plan indeed to rely on the Mediawiki History Dumps as a base for mining the edit histories and compute interaction metrics, and on the XML dumps to complement these data when needed. We also plan to take advantage of existing datasets with preprocessed data when available, such as WikiConv and WikiWho for the English Wikipedia. --CristianCantoro (talk) 15:46, 2 March 2020 (UTC)

Eligibility confirmed, Round 1 2020

This Project Grants proposal is under review!

We've confirmed your proposal is eligible for Round 1 2020 review. Please feel free to ask questions and make changes to this proposal as discussions continue during the community comments period, through March 16, 2020.

The Project Grant committee's formal review for Round 1 2020 will occur March 17 - April 8, 2020. We ask that you refrain from making changes to your proposal during the committee review period, so we can be sure that all committee members are seeing the same version of the proposal.

Grantees will be announced Friday, May 15, 2020.

Any changes to the review calendar will be posted on the Round 1 2020 schedule.

Questions? Contact us at projectgrants wikimedia  · org.

I JethroBT (WMF) (talk) 17:48, 2 March 2020 (UTC)

Acknowledgement of Wikimedia's COVID-19 response

We acknowledge the reception of the COVID-19 Notice and the survey from Wikimedia Foundation project officers. Our proposal does include funding for international travel, scheduled to take place in 2021 (Wikimedia Hackathon 2021, TBD spring 2021; Wikimania 2021, Bangkok, Thailand, summer 2021; academic conferences, TBD). Our proposal does not include the organization of off-line events. Notwithstanding the consequences that the COVID-19 pandemic is having on our daily lives where we are based (Barcelona), currently, this proposal is not impacted. --CristianCantoro (talk) 18:50, 21 March 2020 (UTC)

Aggregated feedback from the committee for Community Health Metrics: Understanding Editor Drop-off

Scoring rubric	Score
(A) Impact potential Does it have the potential to increase gender diversity in Wikimedia projects, either in terms of content, contributors, or both? Does it have the potential for online impact? Can it be sustained, scaled, or adapted elsewhere after the grant ends?	7.4
(B) Community engagement Does it have a specific target community and plan to engage it often? Does it have community support?	5.8
(C) Ability to execute Can the scope be accomplished in the proposed timeframe? Is the budget realistic/efficient ? Do the participants have the necessary skills/experience?	7.2
(D) Measures of success Are there both quantitative and qualitative measures of success? Are they realistic? Can they be measured?	6.8
Additional comments from the Committee: The project stresses an important point connected with the lifecycle of editors. Unfortunately the presentation of the project is not helpful to understand it in detail. The project probably fits with Wikimedia's strategic priorities. However I am not sure that its results are going to have any long lasting impact. Most likely the final report will be quickly forgotten. Very good fit with strategic priorities as community health is a part of the new 2030 strategy. The tools this project is developing should be sustainable and scalable. My only concern is regarding the choice of languages, choosing an emerging wiki with "one country = one wiki" like Persian or Polish would give us a better picture. I see the applicable benefits of this investigation. Much can be learned from editor attrition. I like the detail and information provided. I would love to see some preliminary interviews completed to hone this considering the participant sample is rather small. This is the weakest part in my opinion, The project will analyze data and will try to produce data and numbers matching the previous data. Basically it will help to analyze a face of the cube but will not produce a clear answer. These produced data will need an additional analysis to detail the behaviour of these users. Studying reasons why editors stop contributing to Wikipedia is certainly not a new topic of research. So, project is more iterative than innovative. As to risks, they are difficult to estimate as the grantees have not specified the methodology, which they intend to use, in much detail. The only specific measures of success are a toolkit and dash board. Other outcomes and outputs are not specific enough, in my opinion. A rather innovative project based on a lot of small but not systematic past efforts. There is a high potential for impact if the project is adopted by many communities. My concern is that outcomes should be more tangible, e.g. identifying particularly toxic behaviours, problematic users, patterns etc., and/or include remediation strategies. This project’s outcomes will have resonating effects. This study could help prevent editors from leaving and engage with toxic community health issues creating the environment in which people do not feel comfortable participanting. The scope can be accomplished in 12 months. The grantees seem to be competent enough although the roles of team members are not described in much detail. The budget seems to be realistic as well as excessive. To spend 83,400 euro to just learn something that is likely already known is too much, in my opinion. The project can be accomplished within the next 12 months, and participants seem skillful enough. However, budget is a major concern: it is not clear who will do what and how much time it will take. I also think that there should be some better distribution of funding between Eurecat (they are still employed), this grant budget and potentially a better involvement of WMF staff. I have full confidence in this team and their capacity to execute this project. The community is engagement and support are strong enough especially judging by the number of advisors. High community support, with target communities well identified and largely supporting this initiative. The project has a dedicated focus on diversity. The proposal says they will “work with community” but do not clarify how beyond that. I would like to see more details here. (Bi-monthly hangouts, office hours, etc.) In my opinion the costs and the benefits are not proportionated. The proposal should better detail the outputs of the project. What the project will produce exactly? I am not persuaded that spending 83,400 euro on this project is appropriate or worthwhile. The expected results are unlikely, in my opinion, to have any long lasting impact. I would expect more clarity on the budget, roles of participants and distribution of their staff time. I would also expect more clarity on Eurecat's financial participation (if they keep being at least part-time paid by Eurecat costs are becoming quite high). Some more direct participation of WMF staff would be useful as well, as WMF runs similar projects on community health, thus some synergies can be useful. I like the idea and I think it's impactful, the project plan makes sense, but I would expect more detail for a budget of this level mostly dedicated to staff expenses. This is a rather important grant, but it is expensive. This a well-written proposal with clear goals. It also appears to fit neatly with the Wikimedia movement strategy. However, the budget is confusing. It isn't clear from the proposal what 18 personnel would be doing per months. Will all 18 people be engaged at the same time? From the project timeline, the first four months will focus on the activities stated below: 1. Compute editor lifecycle and drop-off statistics 2. Define and compute longitudinal metrics of editor activity and interactions Will all the 18 personnel be involved in these activities at the same time? Who are these 18 people? Are they external experts or members of our community? How many hours of works would each of them commit to the project? I cannot support the payment of remuneration for someone simply because they are on the team. We do need to know what they would be doing and the amount of time they are committing. Can this project be executed with less number of personnel? Maybe 10 personnel for example? In any case, the budget should be broken down (USD/hr) per person.

This proposal has been recommended for due diligence review.

The Project Grants Committee has conducted a preliminary assessment of your proposal and recommended it for due diligence review. This means that a majority of the committee reviewers favorably assessed this proposal and have requested further investigation by Wikimedia Foundation staff.

Next steps:

• Aggregated committee comments from the committee are posted above. Note that these comments may vary, or even contradict each other, since they reflect the conclusions of multiple individual committee members who independently reviewed this proposal.
• If you have had an interview with a Program Officer, you may have orally responded to some of the committee comments already. Your interview comments will be relayed to the committee during the deliberations call.
• You are welcome to respond to aggregated comments here on the talkpage to publicly share any feedback, clarifications or questions you have.
• Following due diligence review, a final funding decision will be announced on May 29, 2020.

If you have any questions, please contact us at projectgrants wikimedia  · org.

--Marti (WMF) (talk) 22:54, 13 May 2020 (UTC)

Response to committee feedback and questions

We have received the following questions from the committee, we share them here - with permission - together with our responses, hoping that this will be useful for the community.

New or unique contribution to editor life cycle/drop-off research

Some committee members pointed out that there is already existing research about editor life cycle and drop-off. They would like you to describe in more detail what this project would add to the existing research. 
Some committee members asked, “Don’t we already know why people stop editing?
 What will this add to that, or how will it somehow help prevent it?
”

While there is abundant literature about what motivates editors to start or continue contributing, much less effort has been devoted to investigating why Wikipedians leave the project. Several studies have focused on new editor retention (Halfaker et al., 2011; Halfaker et al., 2013; Karumur et al., 2016; Karumur et al., 2018; Schneider et al., 2014) and strategies to improve it (Ciampaglia & Taraborelli, 2015; Morgan & Halfaker, 2018), but very few studies investigated reasons for the drop-off of experienced editors. An investigation based on surveys reports interpersonal conflict as one of the two primary reasons for drop-off, together with life changes (Konieczny, 2018), suggesting that in many cases, the history of interactions of an editor may be relevant to explain why they left. We think we still lack a systematic study that may provide substantial evidence and much deeper insights based on editor interaction data. People who have left the project may typically also not participate in surveys. Nevertheless, their record of activity is there, and this offers a unique opportunity to look and try to understand what went wrong if that is the case. In the case of experienced editors, we may typically have a rich history of activity, that allows for an in-depth analysis. If we look at a single user, there can be many confounding factors, such as personal issues, but if we are able to identify common patterns among many users, then this may point us to some aspects of interest, that possibly were not known, or were underestimated by the community.

In the literature we can find many metrics that capture different aspects of editor activity and interaction; however, they have not been studied over time and across languages for this purpose, which we believe is a missed opportunity to better understand the dynamics of editors’ lifecycle and drop-off. One of the most knowledgeable researchers for previous efforts in this area is Aaron Halfaker, who specifically focused, among other things, on reasons behind newcomers’ drop-off (Halfaker et al., 2011; Halfaker et al., 2013; Schneider et al., 2014) and the impact of measures for newcomers retention (Morgan & Halfaker, 2018). Aaron is an advisor of the team and will help us make sure that we build on previous efforts, both in terms of research and of analysis methods and tools.

We have put greater focus on experienced editors because they received less attention by previous research; however, we also plan to include newcomers in our analyses to deepen our knowledge on this fundamental aspect, which is essential for planning new editors’ retention strategies, in line with MMiller’s comment in the proposal’s talk page.

Given the relevance of the gender gap in the Wikimedia community, several studies based on surveys have investigated factors that hinder women’s participation, such as conflict, confidence, and criticism (Bear & Collier, 2016). A few data-driven studies have focused on gender differences, providing insights such as differences in language, emotional expression and leadership style (Iosub et al, 2012). For newcomers, no significant gender difference was found in the probability of an editor to stop editing after being reverted (Lam et al, 2011). No longitudinal study has further investigated editors’ interaction history at a more fine-grained level to identify temporal patterns and provide evidence of factors that may have discouraged women from editing.

Another issue with previous research is that the main focus lies entirely on the English Wikipedia. We believe that this leads to several limitations to the scope and applicability of the results of such studies as they do not account for the diversity of dynamics, community sizes, and cultural backgrounds of other Wikipedias. For this reason, we propose a multilingual approach that considers multiple languages in the analysis phase and develops tools that can be useful to all language communities.

We want this project to become a point of engagement for editors to understand the current state of their community and take preventive action. Examples of such activities could be: starting new forms of mentorship or new collaborations, seeking support from the Wikimedia Foundation or the affiliates, implementing new policies, assessing the health of different spaces over time, etc. Other distinguishing points of this research proposal are that it is aimed at fostering further research and setting community awareness as a priority for continuous improvements. These two goals shape the way research is conducted and communicated. We want the project outcomes to be open and accessible datasets, visual dashboards, and actionable suggestions to improve community health.

References

Bear, J. B., & Collier, B. (2016). Where are the women in Wikipedia? Understanding the different psychological experiences of men and women in Wikipedia. Sex Roles, 74(5-6), 254-265.

Ciampaglia, G. L., & Taraborelli, D. (2015). MoodBar: Increasing new user retention in Wikipedia through lightweight socialization. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 734-742).

Halfaker, A., Kittur, A., & Riedl, J. (2011). Don’t bite the newbies: how reverts affect the quantity and quality of Wikipedia work. In Proceedings of the 7th international symposium on wikis and open collaboration (pp. 163-172).

Halfaker, A., Geiger, R. S., Morgan, J. T., & Riedl, J. (2013). The rise and decline of an open collaboration system: How Wikipedia’s reaction to popularity is causing its decline. American Behavioral Scientist, 57(5), 664-688.

Iosub, D., Laniado, D., Castillo, C., Morell, M. F., & Kaltenbrunner, A. (2014). Emotions under discussion: Gender, status and communication in online collaboration. PloS one, 9(8).

Karumur, R. P., Yu, B., Zhu, H., & Konstan, J. A. (2018). Content is king, leadership lags: Effects of prior experience on newcomer retention and productivity in online production groups. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (pp. 1-13).

Karumur, R. P., Nguyen, T. T., & Konstan, J. A. (2016). Early activity diversity: Assessing newcomer retention from first-session activity. In Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (pp. 595-608).

Konieczny, P. (2018). Volunteer retention, burnout and dropout in online voluntary organizations: stress, conflict and retirement of Wikipedians. Research in Social Movements. Conflicts and Change (Research in Social Movements, Conflicts and Change, Volume 42). Emerald Publishing Limited, 199-219.

Lam, S. T. K., Uduwage, A., Dong, Z., Sen, S., Musicant, D. R., Terveen, L., & Riedl, J. (2011, October). WP: clubhouse? An exploration of Wikipedia's gender imbalance. In Proceedings of the 7th international symposium on Wikis and open collaboration (pp. 1-10).

Morgan, J. T., & Halfaker, A. (2018). Evaluating the impact of the Wikipedia Teahouse on newcomer socialization and retention. In Proceedings of the 14th International Symposium on Open Collaboration (pp. 1-7).

Schneider, J., Gelley, B. S., & Halfaker, A. (2014). Accept, decline, postpone: How newcomer productivity is reduced in English Wikipedia by pre-publication review. In Proceedings of the international symposium on open collaboration (pp. 1-10).

Practical application

For the purpose of this funding program, the committee seeks to prioritize research requests that will have immediate applied value in the Wikimedia community. Some committee members expressed concern that the project does not have sufficient direct application and said they would only want to fund this project if it will lead to tangible outcomes, such as practical applications and remediation strategies
. These committee members were skeptical about the value of data dashboard for its own sake, and were reluctant to make a funding decision based on the possibility of future research based on the dashboard. They noted that they do not believe that communities need a tool in order to understand that they need to be taking actions to support editor retention and preventing editor decline. In light of this feedback, can you describe the immediate applications of the project once it is complete? Especially, what will this project do to help Wikimedia communities achieve better community health? How will it help us achieve the movement strategy?

As immediate outcomes, this project will produce a study of the editor dynamics across several projects that lead to editor drop-off. As a team, we find it essential to initiate remediation strategies to improve community health. We believe that the ultimate aim of our proposal is to support communities deploying strategies to improve safety and inclusion. By helping to understand the current state of community health, we will be supporting the different initiatives that are now working to improve it. When interviewing members of the communities, we will also survey for the community health initiatives supported by their affiliates.

Of course, communities will be in control, and each community will decide whether to implement policy or take specific initiatives based on evidence from the data and the results our study will provide. In this regard, there is precedent for Wikipedia research that did eventually lead to new policies, such as improving semi-automated tools for vandalism detection. It was an outcome of a study, but not a result of the research itself.

To answer both points about “what is the need for a dashboard,” we would like to point out that current metrics tools are mostly focused on new editors — for example, the “Event Metrics” tool measures contributions by users who participated in an event. Still, there are no tools available to the community to understand how effective they are in engaging newcomers and how community dynamics are evolving. Therefore, it is difficult for editors to have a conversation on how to improve retention and general community health without some specific measurements as a common reference.

Of course, every editor has their personal experience of the interactions between community members, including themselves, but our analyses would offer a general, broader point of view that is currently missing. Also, without tools it is not possible to compare different language editions, as editors tend to make the vast majority of their contributions in a single language edition.

We have planned to collaborate closely with different Wikimedia affiliates and user groups to ensure that the dashboards are not only a tool to display and engage with the results of our research project but also a practical resource for those interested in community health assessment. In this sense, we count on advisors whose connections expand to the most active chapters and have been working on initiatives focused on gender gap, harassment, diversity, among others. We believe that once the dashboards are deployed and explained to affiliates and groups of interest, monitoring community health will become a routine, and seeing progress will encourage more and more community health-related initiatives to appear in the community programs.

Finally, we think that this project is aligned with the Wikimedia Strategy 2030 recommendations and goals. The strategy document places overcoming gender gap as an essential goal, and recommendations like “Provide for Safety and Inclusion” present several actions to work on improving community health. Among these actions, one of the most important ones is the Universal Code of Conduct (UCoC), which has been recently put forward by the Wikimedia Foundation Board of Trustees. Our research project may be used to measure the impact of the UCoC when deployed. The UCoC will be crucial to community health as much as notability is in terms of content; however, its impact may not be as easy to measure. In this sense, this project will offer a way to measure the effect of community health initiatives, offering new possibilities because what is measured can be improved.

Community Health

Some committee members aren’t convinced that this project makes a significant contribution to our understanding of community health. They say that pointing to editor drop-off does not go far enough to give useful insight into community health, which is a much more complex concept than just presence or absence of editors. They would like to understand better the ways in which you perceive this project as advancing our understanding of community health as a whole.

We wholeheartedly agree that community health is much more than the presence or absence of editors, and of course, there are many ways in which one can work to improve it. The main idea behind this proposal is not only that we would like to retain editors, but also that looking at what makes people leave the project may help to understand what affects editors and help improve community health in general. For example, we may be able to detect that some interaction dynamics - e. g. edit wars with specific common characteristics, or receiving responses with a particular tone in talk pages - lead some editors to leave the project. However, the same interaction may negatively impact many other editors who don’t leave, but are still affected, feel frustrated, and less motivated after the same interaction dynamics. In other words, drop-off is not only an issue we want to address, but also a lens for diagnosis of broader issues that may be affecting community health.

We don’t have the ambition to cover all aspects that make a community healthy; instead, we aim to get as much knowledge as possible about the circumstances associated with editors leaving the project so that this information can be used to improve community health. Of course, we believe the metrics, findings, and knowledge obtained can be useful beyond editor retention.

In the dashboard, we aim to include indicators and metrics that are currently missing, which can help the communities to be more aware of their status: how things are going in different spaces, of possible issues they may be overlooking, etc. These indicators and metrics will be chosen based on findings from the data analysis, as well as on input and feedback from the communities.

The dashboards will focus on aggregated statistics about the editor lifecycle, and on spaces (pages, groups of pages, projects, etc.) that may be undergoing detrimental dynamics: we do not want our system to detect and point to individual users who might be in risk of dropping off, because this could be potentially harmful and invasive of user privacy.

In this sense, these dashboards will not be directly focused on editor drop-off, but more broadly on community health. We believe the combination of findings from the data analysis and input from the communities will allow us to design and implement dashboards that can help prevent editor drop-off and assess the impact of targeted campaigns and initiatives, as well as to improve community health at different levels.

Clarifying the role of Eurecat

Some committee members wanted more information about Eurecat and its role in this project.

Eurecat (official name: Eurecat - Centre Tecnològic de Catalunya) is a recognized non-profit foundation in Spain. All members of the team (Cristian Consonni, David Laniado, and Pablo Aragon) are employed by Eurecat.

In this capacity, Eurecat will offer administrative support and manage the funds of the project. The funds will be administered by Eurecat as for any other project for which we receive funding as for example European projects or funding from the national or local governments. We are members of the Big Data & Data Science unit at Eurecat, which leads the Big Data Center of Excellence of Barcelona, that will provide a large scale high computing infrastructure for this project. Furthermore, Eurecat has a research communication office to contribute to the dissemination tasks of ongoing projects.

From some comments that we have seen in the aggregated feedback to the proposal, we have realized that some committee members may have been misled by how we used the term “person-month” in the project budget. Person-months are a way to measure commitment and effort in projects. They are very commonly used in project proposals at the European level and for this reason, we are very used to measuring our time in that way, which we also used in this proposal. A person-month is the time equivalent to one person working full-time for a month.

Our project proposal includes 18 person-months worth of work, over a period of 12 (actual) months. In practical terms, this means that 3 people - the 3 members of the team - will work for 50% of their time over the course of 12 months.

Budget concerns

Some committee members are open to funding the proposal but do not feel comfortable with the cost of the request. There was discussion about asking that the project budget be cut. To be clear, the committee has not made a funding decision about this proposal, and it’s possible they could request greater or lesser budget cuts, or they could decide against funding the proposal. We are not asking you to cut your budget at this point, but could you speak at a high level about what it would mean for the project if you were offered a reduced budget. For example, if the committee were to offer to fund 50% of your request, would this be useful or would it render the project not feasible?

Reducing the budget would negatively impact the project overall. As of now, the project combines research on the causes of drop-off with the development of easy-to-understand metrics and dashboards. Sacrificing any part of the proposal would be problematic on different levels: on one hand research findings and conclusions are essential to design remediation strategies or validate the current ones coming out of Strategy 2030; while the dashboards are the outcome we envision to be more useful for communities to monitor and improve community health. The bulk of the data analysis and modeling work will be the basis for both outcomes, and reducing it would weaken the project’s impact.

We believe we have developed a thorough project that includes different activities to address editor drop-off and community health:

• Activity 1 consists of computing lifecycle and drop-off statistics for a set of 30 language editions. This requires coming to a shared operational definition of drop-off, on which the whole project will rely, and will result in data that is currently missing to understand in detail population dynamics and evolution in each wiki. This information will be updated periodically and will be part of what is shown in the dashboards. We believe this task is fundamental for the rest of the project, beyond producing output that is useful by itself.
• Activity 2 consists in computing metrics of activity and interaction over time for selected language editions. This is also a fundamental pillar of the project, on which the following activities are built. We do not aim to focus on developing novel complex metrics, but rather to build on previous literature to implement established metrics that will be useful for this aim. The important aspect here is collecting the output of a comprehensive set of metrics over time, so that they can be used in the model in activity 3. Here we could slightly reduce the scope (and budget) of the project by reducing the number of language editions analyzed (we have foreseen 4), although we believe having a minimum set of diverse communities would help to achieve more robust and reliable results. We could also cut the language-dependent metrics, that require more effort to be applied to multiple language editions; however, they carry relevant information in this context and discarding them might significantly reduce our ability to identify factors associated with drop-off.
• Activity 3 contains the core of the data model, that will allow us to identify factors associated with editor drop-off, and we see it hard to reduce it. We expect to get from this activity the main findings, that will result in recommendations for initiative and policies for improving community health, and in metrics and indicators for the development of the dashboards.
• Activity 4 is focused on deepening the analyses for specific collectives of editors, and in particular to address the gender gap. This activity could be discarded, although we believe that if we remove the focus on minority groups, the project would lose an important pillar, and reduce its potential to foster inclusion and diversity in the editor community, which is an important strategic goal for the movement.
• Activity 5 consists of the development of dashboards to make data and results accessible to the communities. If we discard this, the project will still have valuable outputs but will miss its most tangible outcome, that will help communities in their daily life in several ways, such as evaluating editor lifecycle and retention, assessing the impact of campaigns and initiatives, monitoring different aspects of community health across a wiki.
• Activity 6 is essential for receiving feedback from the communities that will drive the development of the project, and to disseminate the results.

In conclusion, we see the project as a combination of activities that we consider all necessary to fully reach the expected outcomes and impact. We have indicated a few places where we see it would be possible to reduce the effort (and budget) required, with the associated loss in terms of project outcomes, which in general we believe would reduce the impact of the project to help communities increase diversity and monitor and improve community health.

---

On behalf of the team, (CristianCantoro, sdivad, and Elaragon). --CristianCantoro (talk) 22:33, 27 May 2020 (UTC)

Round 1 2020 decision

Congratulations! Your proposal has been selected for a Project Grant.

The committee has recommended this proposal and WMF has approved funding for the full amount of your request, 83,400 €

Comments regarding this decision:
The Project Grants committee is pleased to support your research into editor drop-off. The committee would like you to emphasize creating practical, applied value for Wikimedia communities, such as deliverables that might guide follow-up action by Wikimedia communities.

We would also like to schedule a consultation to consider how to best manage risks of the dashboard, to be arranged with your Program Officer. This may include some oversight from staff from the Wikimedia Foundaiton’s Research Team, depending on their availability..

Next steps:

1. You will be contacted to sign a grant agreement and setup a monthly check-in schedule.
2. Review the information for grantees.
3. Use the new buttons on your original proposal to create your project pages.
4. Start work on your project!

Upcoming changes to Wikimedia Foundation Grants

Over the last year, the Wikimedia Foundation has been undergoing a community consultation process to launch a new grants strategy. Our proposed programs are posted on Meta here: Grants Strategy Relaunch 2020-2021. If you have suggestions about how we can improve our programs in the future, you can find information about how to give feedback here: Get involved. We are also currently seeking candidates to serve on regional grants committees and we'd appreciate it if you could help us spread the word to strong candidates--you can find out more here. We will launch our new programs in July 2021. If you are interested in submitting future proposals for funding, stay tuned to learn more about our future programs.