Grants talk:Project/I School Challenge for Countering Online Harassment

Change status to 'proposed' to submit for 8/2/16 Project Grants deadline

Dear Npdoty,

Please note that if you intend to submit this proposal for the August 2 deadline of the current round of Project Grants, you must change the status from draft to proposed. If you have any questions, our final proposal clinic is from 1600-1700 UTC on August 2.

Cheers,

--Marti (WMF) (talk) 05:13, 2 August 2016 (UTC)

Thanks! That was one of my open questions. (Tried to join the office hours via IRC, but may have missed you.) Npdoty (talk) 20:09, 2 August 2016 (UTC)

Comments / Questsions from I JethroBT (WMF)

Hi Npdoty, I'm Jethro, a community organizer. Thanks for preparing your proposal to conduct comprehensive, research-based approach to tackling harassment on Wikimedia projects. I have a couple of suggestions about places to obtain some feedback and recruit volunteers from Wikimedia community who might be interested in assisting with this initiative. If the target project you are seeking to improve is the English Wikipedia, a few good places to make a general announce to get feedback include:

• The wiki-research-l mailing list is all about scientific research regarding project content and their communities, and would be a great place to announce the project, get feedback and look for potential collaborators,
• The village pump is a general space for alerting the community about community-focused proposals. It's common for feedback to be provided locally there-- feel free to link to the discussion if that is the case.
• Try dropping a suggestion about the grant proposal to The Signpost, a semi-weekly newsletter sent to thousands of English Wikipedia contributors.

I've already dropped a line about the proposal in the #wikimedia-research^connect IRC channel dedicated to research discussions and topics as well.

I had a few questions as well:

• The proposal outlines plans to evaluate multiple platforms, including Wikipedia. What other platforms will be evaluated? In total, do you have an estimate of the number of platforms this research is scoped for?
• One component of the project is to develop a portal for research and tools. The Research namespace here on Meta currently contains an space for folks to submit and document research projects and the tools they develop along with them, and can be categorized under a project. Can you describe what sorts of benefits a new portal could have in this project?

Thanks, I JethroBT (WMF) (talk) 01:02, 6 August 2016 (UTC)

Much thanks, I JethroBT (WMF), for both the suggestions and the questions. I'm reaching out with announcements to those lists/fora now. I'll follow up more here later on your two questions; briefly, we have a few platforms in mind to use as background research and comparison, but I'm hopeful that a research conference would expand that list considerably. Npdoty (talk) 23:21, 17 August 2016 (UTC)

Regarding these two questions, I JethroBT (WMF):

1. what other platforms?

In order to get useful comparison, it would help to provide some background research on, say, 3-6 platforms prior to a conference. In addition to Wikipedia, we have considered the following:

• Twitter, a large, proprietary, public messaging system where harassment has been cited as a particular concern (at our event in April, we both heard interest from Twitter folks and saw experience with self-help tools directed particularly at Twitter)
• Nextdoor, a geographically-focused, invite-only community in US cities, which has had substantial media coverage regarding issues of race and moderation (CTSP fellows began research on this community over the past year)
• other suggestions have included: Reddit (prominently covered in press), Discourse (general software tool, targeted at community), Facebook (different set of tools, including ones for encouraging resolution of disputes), Metafilter

While we would be looking at a handful of platforms as points of comparison prior to the research conference, the focus of the project is not on conducting all the research directly with grant funds, but in providing some case studies useful to researchers (in academia and in industry), and then providing incentives for development of tools and mechanisms for collaboration in order to encourage sharing of research. To that extent, the work should cover a much wider range of platforms, and also to more basic questions, in areas of social psychology or computer science, for example.

2. how would a Research Commons differ from existing Wikimedia resources regarding collecting research?

Feedback from this Talk page and other venues has pointed me to at least the following set of existing or in-progress pages for collecting research or other resources regarding online harassment:

• Research:Online harassment resource guide, annotated bibliography/literature review
• Research:Detox, a research project for ML-driven detection of toxic comments on Wikipedia
• Research:Modeling Talk Page Abuse/Resources, links to discussions, data sources and Wikipedia-related programming tools

What we have proposed for a "Research Commons" would, I think, tend to be broader, in terms of collecting data, policies, tools, related research and discussion for platforms including but not limited to Wikipedia or other Wikimedia-related ones. That being said, I think a good first phase for this part of the project would be gathering both existing resources and a sense of the needs of researchers -- if it would make more sense for researchers, for example, to add catalogs of policies and data and Talk pages for discussion to the existing Online harassment resource guide bibliography, then we would benefit from pooling effort. If instead it would be easier to use a discussion forum like Discourse for more open-ended discussion of open research areas, then a separate portal might better engage academic and industry researchers.

Also an open question, I believe, is whether such a Commons could also be used as a repository not just for researchers, but for users. We have heard interest in more user-friendly resources for those encountering online harassment and for law enforcement in different jurisdictions often not well-educated about the issue. (Those are my personal thoughts, but I'm open to alternative ideas and additions to either of those lists above.)

-- Npdoty (talk) 04:35, 21 August 2016 (UTC)

comments from Thepwnco

@Npdoty: Hello and thanks for this proposal - there are a lot of good details in the project plan and I agree that this is an important and necessary area of research to support if we are to develop robust solutions that address online harassment.

I have a few comments and questions for you as outlined below. Thanks for your time and consideration.

• Are there researchers already working in this space - and specifically with Wikimedia projects - that you have already identified as potential speakers and presenters at the conference?
• You mention the importance of bringing together the findings of researchers from around the world and who are working with various platforms in order to provide a global forum for knowledge exchange as well as account for cultural differences. Given the number of languages and communities across Wikimedia projects, I also see this is as a necessary approach. Can you provide more information on how you will ensure that diverse experiences and perspectives are reflected in conference participation and the literature review/preliminary research gap analysis?
• Is there room in your proposal to tailor some of the analysis and findings to Wikimedia projects? For example, perhaps the most relevant tools, policies, and practices for Wikimedia projects could be highlighted in the conference report and/or made the focus of the iSchool challenge?
• I echo comments made by User:I JethroBT (WMF) - it's not entirely clear to me what the impacts would be of creating a dedicated portal for this area of research, particularly given the other deliverables of the project (literature reviews and a conference report summarizing key findings). Could you further outline your ideas around this? As a sidenote, it might be helpful to think in terms of measures of success (beyond just the launch of the portal), for example, number of researchers subscribed to the portal, number of new discussions, number of new submissions made by researchers, etc.

Cheers, -Thepwnco (talk) 20:25, 14 August 2016 (UTC)

Thepwnco, thanks for these comments and questions. (Some brief replies here now, and we'll try to make further edits, either directly on the proposal page or as added material here soon.)

Regarding current Wikimedia-related researchers who might be a fit for the conference, we have reached out to a couple directly (especially those with a direct relation to UC Berkeley), but I think more importantly we've been able to learn more about existing collaborations through mailing lists and other forms of feedback we've received.

Re: different platforms and cultural differences; I think this is likely to be one of the more challenging parts of any research-driven project in this area. We want to be especially open about asking researchers and users of Wikimedia for help in finding the best ways to reach a diverse set of user contexts and perspectives. I expect part of that to be asking individuals we know have worked with different geographic diversity in Wikipedia (e.g. OII's recent project), part would be using mailing lists and surveys for outreach and part would be finding existing organizations/user groups around, for example, Wikipedia in different countries. We have suggested having student researchers here at Berkeley conduct interviews; those may also be limited by language barriers.

I think it's a great idea to tailor some analysis and results to, for example, Wikipedia. While we think part of the value of a conference of this type would be getting experiences and research with different platforms and different contexts to try to share insights, that doesn't mean that it wouldn't be valuable to customize some of the outputs to a specific platform. Perhaps we could provide a customized executive summary of the conference report / background research findings for Wikipedia in particular.

We should add more detail here on the functionality of a portal/commons and on how we would measure its success. As I mentioned in responding to I JethroBT (WMF) above, I think additional features could be breadth of collection (data, code, policies as well as research), variety of platforms, and different audiences (not just researchers, but also users and platform developers, say). But I think we should get direct feedback from researchers (and from those other audiences) about what would be useful to drive that decision. I'm hoping to talk with some of those existing researchers shortly, so I can provide updates here.

Thanks again for these questions and comments; most helpful. -- Npdoty (talk) 08:11, 30 August 2016 (UTC)

Feedback

I'm a antopodean librarian with a CompSci PhD and a decade editing en.wiki, which undoubtedly colour my perceptions on this, but here's a couple of points:

• I think idea of a physical conference is great, but effectively limits to attendees, the more online the event, the more global the participation can be expected to be.
• If you want to motivate academics and grad students to do stuff (i.e. research harassment) you need to find incentives. Getting papers published, highlighted and/or cited is a great way to achieve that. I think you could focus more on this, whether it's by finding and summarising existing harassment research elsewhere; funding review articles of what the state of harassment research is in different academic fields; finding a forum to publish the proceedings of your conference; find a peer-reviewed forum for a massively-co-authored conference position paper, etc.
• 'Winter' is not a global descriptor, in particular the global south has winter at a very different time to the USA. Find a time period that doesn't alienate half your target audience.
• Consider a deal with a data repository for researchers to publish their data for free if published three months prior to the conference. This incentives researchers to make public their existing data well in advance so everyone can poke it prior to the conference.
• I think clarification of the concept of 'tools' could be elaborated, since it covers everything from ethical guidelines to iron maidens.

Just my 2c. Stuartyeates (talk) 04:51, 18 August 2016 (UTC)

Thanks Stuartyeates for these comments. I believe there's an interesting balance here in providing incentives for academics (faculty, graduate students) to participate and the in providing incentives for developers and companies that run online platforms or build software (typically a different type of incentives). Being at a University, we feel more comfortable with ways to include students; getting people in industry involved with running platforms to share their data, code, or experiences more generally may be a more important challenge. I like the idea of using the possibility of the open data repository (which I believe Berkeley can provide, or could be part of the "research commons") as one incentive for participation by academics, and maybe that would apply to researchers within industry as well. We have had some success in using events as one way to help interested people form groups (see the proposed Day of Code) focused on a particular problem, and hope the Challenge will be a financial and reputational incentive.

I've fixed the usage of seasons (winter, spring) to describe dates. Thanks for catching that!

-- Npdoty (talk) 17:40, 4 September 2016 (UTC)

Eligibility confirmed, round 1 2016

This Project Grants proposal is under review!

We've confirmed your proposal is eligible for round 1 2016 review. Please feel free to ask questions and make changes to this proposal as discussions continue during this community comments period.

The committee's formal review for round 1 2016 begins on 24 August 2016, and grants will be announced in October. See the schedule for more details.

Questions? Contact us.

--Marti (WMF) (talk) 18:04, 23 August 2016 (UTC)

Comments of Ruslik0

I have some comments/questions:

• Can you clarify the title of your project? ("I School Challenge for Countering Online Harassment"), in particular, what does "I" mean?
• You write: "There is little opportunity for researchers from different parts of the world to exchange research findings or to collaborate on new research on this topic." I would not be so categorical.
• You write: "Collaborative work with academics may lead to an occasional publication" but in the next section I see: "There is a growing body of academic research focused on harassment in the online space". There is a tension between these two statements.
• The measures of success are very vague and non-specific.
• As understand the project aims primarily at research of on-line harassment outside the WMF projects and there is no or little of no connections with the Wikimedia movement in general and its goals. Is this statement true?

Ruslik (talk) 11:48, 31 August 2016 (UTC)

Ruslik0, thanks for the comments. Others may be able to help with resolving the comments you've pointed out about language being unclear. However, a quick response to your questions:

• "I School Challenge" is referring to the institutional home (the UC Berkeley, School of Information, known as the "I School") that we would be encouraging to make Countering Online Harassment a year-long theme of work. ("I School" is explained in the text, but I see that it's not near the title; our mistake.)
• We hope that it's not at all the case that the goals of this project have no connection to the Wikimedia movement and its goals! I apologize that that apparently didn't come across clearly enough in the Proposal text. Our argument is that the problem of harassment in online communities inhibits participation, often continues historical discrimination and chills contributions to sharing of knowledge online. Those have been particular concerns with Wikipedia and contributions to the online encyclopedia, but they are also common to many online platforms. To that end, we want to combine and connect research on related fields (sociology, psychology, computer science, etc.) and experience with harassment on different platforms in order to build a research-driven approach to the problem.

There were suggestions in the Discussion above regarding specific, Wikimedia-project-related deliverables. (Apologies if this should be moved over into the Grants page directly.) In particular: providing background research/interviews on Wikipedia users as input to the conference; developing a customized summary of findings from the conference applicable to Wikimedia-related projects.

-- Npdoty (talk) 18:26, 4 September 2016 (UTC)

Reservations about funding academic conferences

We recognize the significant value of the partnership between WMF and the I School, and we are interested in supporting further collaboration. However, because of the limited funds we have to award through this program and because of the substantial attention directed to Wikipedia by researchers, I have reservations about allocating funds for conferences in support of academic efforts. If we set a programmatic precedent of that nature, we open ourselves up to being flooded with requests that we don't have the resources to support, either through this program or the dedicated Conference program. Consequently, I'm writing to ask how dependent the conference is on the funds requested. If you are not awarded funding through WMF, will the conference proceed in some form? If so, it would be more feasible for us to award supplemental funding for a day of code or hackathon around the conference. Generally, I'm wanting feedback about wiggle room in this proposal's focus that might allow us to zero on funding aspects that are the closest fit for our program.

I also want to advise you (independent of whether this proposal is recommended for funding) of the opportunity for scholarship funding for key researchers to attend Wikimania in Montreal next year and/or the Wikimedia Hackathon. I'd be glad to follow up separately about this opportunity at mjohnson wikimedia  · org.

Cheers,

--Marti (WMF) (talk) 22:54, 31 August 2016 (UTC)

Mjohnson (WMF) -- this is useful feedback, and I can understand your concern. First, I hope that we can distinguish this proposal from a traditional academic conference; this is intended to be gathering research/researchers and assessing needs about a specific problem encountered on several platforms (including Wikipedia); rather than formal presentations of basic research, I'm thinking of this as a working event. That's the reasoning behind providing both literature review and user and developer interviews in advance of the event -- to more narrowly define the problems faced and help researchers see where contributions can effectively address ongoing problems.

In any case, I do believe we would be open to funding of some pieces of the proposal that would be more directly in line with the focus of the grant program. We would plan to seek funding from other sources around the idea of a meeting for a research-based approach, and hopefully to build on or learn from other recent discussions among academics and industry. The Challenge Prize or the Day of Code and Research Portal pieces could be more in line with what you're anticipating. (We have also heard the suggestion that a "Day of Code" should really be a weekend in order to give enough time to build actual prototypes.) But we believe that such work will be more successful if it's guided by research (gathered across disparate fields and sectors) and a concrete assessment of needs; that is, solutions will be more effective when informed.

I'll follow up via email regarding the events you mention. It could be that encouraging people working in the area of online harassment to attend Wikimedia events would be another useful direction.

Thanks -- Npdoty (talk) 07:19, 7 September 2016 (UTC)

Feedback from Jonathan Morgan

@Npdoty: I think that the goals of this proposal are laudable, but I have some reservations. Note that although I am a WMF staff member, my opinions are my own. I do not work with the Grants team and I have no say in funding policy generally or in which projects get funded.

• Heads up that the terms of the Wikimedia Foundation Open Access Policy may require that any research materials generated by this project be published free/libre under an open license. Different academic conferences and journals have different policies around copyright and OA licensing, and many require you to pay an extra fee if you want your manuscript to be published open access. UC Berkeley may have policies around how content hosted on their servers is licensed. If your proposal is accepted by the Grants committee, I suggest you set up some time to speak with Mjohnson (WMF) about how to plan and/or budget for this.
• If I'm reading this right, only UC Berkeley students and faculty are eligible for the The I School Challenge program. Which means that WMF is essentially providing $10,000 of unconstrained grant funding to the University of California, which the university will disburse to its own students and faculty. I don't know if our financial policies specifically preclude this, but it seems strange to me that WMF would act as a pass through grantor on an online harassment project, given that we run our own Anti-Harassment Challenge program—one in which anyone can receive funding, regardless of their affiliation.
• How will this project benefit the Wikimedia Movement?

I think this is a worthy project. I'm just not sure WMF is the grantor to fund it. Cheers, Jmorgan (WMF) (talk) 20:28, 3 September 2016 (UTC)

Jmorgan (WMF) -- thanks for sharing your support regarding the goals and for being open about your reservations. Some replies below, but I would also be happy to talk more offline if that would be helpful or if I'm not understanding you.

• Yes, the University of California also has an Open Access Policy, which is implemented at UC Berkeley. In terms of literature review and needs assessment, conference report, development of the Research Commons -- we would host that content (and any code or data) and provide free, open access. (It's great to have such an OA policy when funding research; I applaud having such a requirement.)
• Apologies that we may have been unclear on this point, perhaps because there is a tension on trying to make the Challenge a spur to a significant local community to participate and in trying to engage a larger audience. I would encourage open eligibility for such a challenge prize, as the reputational incentive of participating in such a wide competition is one of its advantages. As I understand it, challenge prizes are specifically not unconstrained grant funding or pass-through arrangements, but a specific use of funds to encourage development towards a particular goal, with a juried process for determining a winner. (As opposed to a sub-grant process, where funds are delegated out to pay for other projects.) We're certainly aware of the IdeaLab/Inspire Campaign; some of the discussions during that month were what got us thinking about this proposal! As I understand it, the Inspire Campaign specifically wasn't a challenge or a prize for either completed work or the best project idea, but an opportunity to try to brainstorm ideas that might become funding proposals. (I'm new to this particular grant process, so apologies if I misunderstood any pieces along the way.)
• We tried to describe the benefits to Wikimedia projects in the proposal Project idea, but in brief, we hope that: gathering research and needs assessments (including specifically from Wikipedia) and encouraging researchers in different fields to look directly at the problem of harassment in different online platforms; motivating the development of projects by collaborative teams; and providing a portal for collecting research, data and discussion for researchers and users; will together contribute to more effective, research-driven solutions to problems of online harassment which have inhibited editing of Wikipedia and contribution to Wikimedia's collaborative projects.

I hope this responds to your comments and question. Cheers, -- Npdoty (talk) 08:28, 7 September 2016 (UTC)

Aggregated feedback from the committee for I School Challenge for Countering Online Harassment

Scoring rubric	Score
(A) Impact potential Does it have the potential to increase gender diversity in Wikimedia projects, either in terms of content, contributors, or both? Does it have the potential for online impact? Can it be sustained, scaled, or adapted elsewhere after the grant ends?	5.7
(B) Community engagement Does it have a specific target community and plan to engage it often? Does it have community support?	5.7
(C) Ability to execute Can the scope be accomplished in the proposed timeframe? Is the budget realistic/efficient ? Do the participants have the necessary skills/experience?	5.1
(D) Measures of success Are there both quantitative and qualitative measures of success? Are they realistic? Can they be measured?	5.3
Additional comments from the Committee: The proposal is set as a sequence of projects to provide a deep understanding of online harassment and develop counter-harassment practices. From this perspective, this proposal fits Wikimedia's strategic priority of creating sustainable communities that interact with civility. Harassment and counter-harassment are topics of utmost relevance within Wikimedia projects, and I support the idea of a thorough research project to tackle harassment and counter-harassment on web-2.0 projects, specifically Wikimedia projects. I recommend that the proposal at this point focus solely on the research proposal (not the prize, that could become the target of a different application). This would definitely make this project less ambitious, but also less risky. The research could then be placed on Commons and be engaged with the community. In the context of the research, I think it would be important to focus on Wikimedia projects, not online projects in general. In order to have a sense of what the research team is able to produce, it would be interesting to have a detail of the methodology that is expected to be used in the lit review. The proposal seems to be aimed at online harassment in general and is not directly related to Wikmedia Projects and the Wikimedia strategic priorities. A concern about the proposal is that its scope of interest might be too large. Wikimedia projects have distinct features and dynamics in comparison to other web-2.0 projects, and an approach that would target specifically Wikimedia projects appears to be more appropriate. The section on Previous Outreach lists documents that have at the end little to do with Wikimedia projects, and it is not clear how this research proposal would directly tackle concerns from our community and not online harassment at large. Moreover, the proposal author appears not to take into consideration how diversity might affect the outcome of interest --diverse cultural backgrounds could affect harassment, so that what is seen on Wikipedia in English might not at all be informative for what happens in other communities. This is close to WIkimedia strategic priorities (healthier communities) but too much content will be of little relevance to Wikimedia. There is no evidence that this will result in any working (not theoretical) solution for Wikimedia projects. The measures of success are abstract and less quantifiable. However, I understand that it is quite hard to come up with metrics for measuring success in a conference of this kind. Apart from the named outcomes, I also expect the grantee to write blog posts regarding their activities and also give significant press coverage. The proposal refers to the need of "extensive literature reviews" on harassment and, I imagine, counter-harassment. In order to have a better assessment of what is expected, I would appreciate more detail on methods and outcomes. Huge innovative potential, but this is more of a general than Wikimedia-related event. It is hard to distinguish potential impact on Wikimedia projects from the total impact, given that we do not know how many participants will work on harassment on Wikimedia projects. I think it can be executed and is the budget is realistic though duration of the project is unclear. The grantees probably have skills to execute it. The proposal is very ambitious, and I am concerned that the risk involved might be significant. If the pre-conference does not succeed in accomplishing what it expects, or that the conference does not provide good support, then the sequence of activities might be jeopardized. It is very hard to say if this will result in any material impact on Wikimedia projects within the next 12 months. This is a rather standard budget for a scientific conference, but it is not clear if any participants are already working on harassment on Wikimedia projects: if they are completely new to it they are unlikely to provide working solutions. The proposal does not specify a target community (I am assuming the focus is English Wikimedia). The proposal has had little endorsements, which might signal it is at the end generally disconnected from the Wikimedia community. And of course community is key to the success of this activity, since one is not expecting to bring "exogenous" solutions to community dynamics, but to consider inputs from the living community that support the projects.. I support this project, but think it would be a good idea to explore if other organizations are ready to jointly fund.

This project has not been selected for a Project Grant at this time.

We love that you took the chance to creatively improve the Wikimedia movement. The committee has reviewed this proposal and not recommended it for funding. This was a very competitive round with many good ideas, not all of which could be funded in spite of many merits. We appreciate your participation, and we hope you'll continue to stay engaged in the Wikimedia context.

Next steps: Applicants whose proposals are declined are welcome to consider resubmitting your application again in the future. You are welcome to request a consultation with staff to review any concerns with your proposal that contributed to a decline decision, and help you determine whether resubmission makes sense for your proposal.

Over the last year, the Wikimedia Foundation has been undergoing a community consultation process to launch a new grants strategy. Our proposed programs are posted on Meta here: Grants Strategy Relaunch 2020-2021. If you have suggestions about how we can improve our programs in the future, you can find information about how to give feedback here: Get involved. We are also currently seeking candidates to serve on regional grants committees and we'd appreciate it if you could help us spread the word to strong candidates--you can find out more here. We will launch our new programs in July 2021. If you are interested in submitting future proposals for funding, stay tuned to learn more about our future programs.