Grants:Wikimania scholars/Reviewer's guide

This guide was created in 2015 by members of the Wikimania Scholarship Committee and has been updated to be applicable to the 2019 Scholarship program as well.

Process & Instructions

Evaluation of scholarship applications will be a three phase process:

Phase 1: Eligibility filtering
- Process:
  - WMF staff will complete an eligibility assessment of all submitted applications, with each application given a pass/fail score based on the eligibility assessment criteria and associated scoring guidelines
  - The WMF will then notify all applicants by e-mail on whether their application has passed or failed once this phase of review is complete
- Instructions to reviewers:
  - In the Scholarship review system, under the "Review" tab select the "Phase 1" report to see all applicants eligible for review; each reviewer will be assigned a set of applications for review. For each application, the reviewer will give a pass/fail score using the drop down menu.
    - The score can be changed by any reviewer at any time, with a new score overriding a previous score.
  - In the editable notes section, each reviewer should leave clear comments as to why an applicant was passed or failed (e.g. noting which criteria an applicant failed to meet as a reason for failing Phase 1 assessment)
    - All comments should be signed with initials or a nickname.
- From previous years, the estimated time per application should be between 1-5 minutes.
Phase 2: Evaluation
- Process:
  - In accordance with the new diversity System created in 2015, Phase 2 applicants will be grouped by their "home country" and "primary language community on wiki" into one of eight "Language categories".
  - The Committee Chair will determine the set of applications each Committee member will review. Each applicant should be reviewed by at least three committee members to determine their ultimate Phase 2 (average) score.
  - After all Phase 2 scoring is complete, the list of applicants will go to WMF for final approval.
- Instructions to reviewers:
  - In the Scholarship review system, under the "Report" tab select the "By language community" report to see the list of Phase 1 "passed" applicants for Phase 2 review by the above categories.
  - In accordance with the Phase 2 criteria, each reviewer will give a score of 0-10 against two main criteria: "Relevant experience" and "Experience sharing". While a few weak and strong examples have been outlined in the Phase 2 scoring guidelines, ultimately it will be the responsibility of each reviewer to synthesize multiple inputs from across the application to give a final score.
    - Previous scholarships recipients will have answered an additional question on the impact of attending Wikimania previously and their goals for attending again. While this is not given a separate score, evaluation of the applicant's answer will be included into the "Experience sharing" score.
  - In the editable notes section, each reviewer should leave clear comments on the application, to guide/highlight/notify other reviewers.
    - All comments should be signed with initials or a nickname
Phase 3: Final approval
- Based on estimated travel costs for each individual (based on home airport), top scoring applicants from each group will be offered a full or partial scholarship, with allocations and adjustments made in accordance with the selection process.

Phase 1 scoring guidelines

Applications will fail Phase 1 if any of the following failing criteria apply. For applications written in languages other than English, translations should be attempted, with the understanding that machine translations may not give the most accurate results.

The applicant is a 2014, 2015, 2016, or 2017 scholarship recipient that did not complete their post-conference report.
- This will be completed by WMF staff before Phase 1 begins.
The application entirely or primarily consists of content which is off-topic or abusive.
- This criterion covers spam, joke applications, and obvious abuse. If an application contains off topic or otherwise inappropriate material, but this forms a minor part of the overall application, then the applicant should be given the benefit of the doubt.
The applicant has failed to make a reasonable effort to answer the questions on the application form.
- This criterion is applicable to any application in which little effort has been made in answering the questions. This criterion is mostly aimed at applicants who give blank, irrelevant, or poor (one short sentence or less) answers to any of the long answer questions.
The applicant has failed to provide evidence of English language abilities that are at a level which would enable them to participate in Wikimania, a conference which is primarily conducted in English. Sufficient English abilities could be demonstrated in the application itself or elsewhere.
- In most cases the application itself should provide sufficient evidence for an applicant to pass this criterion. In the event of doubt (e.g. poor answers or suspected use of machine translators), the reviewer may do any of the following to verify English capabilities:
  - Review the applicant's Wikimedia contributions by using the link (called "cross-wiki contribs") provided in the application, next to the "Username" field. The type of edits (e.g. article writing vs. image insertion) should be taken into consideration if possible.
  - Review any external links provided in the application, e.g. blogs, new articles written by the applicant.
  - Use personal knowledge of the applicant's English capabilities, based on previous interactions in English (e.g. conversations held at a previous event)
- Note that the necessary English language abilities are roughly equivalent to en-2 or better on the babel scale; native or near-native proficiency is not necessary. Individuals often use babel userboxes to note their language abilities, although these should be looked at sceptically as they can be wildly inaccurate.
- In the case that an application is not submitted in English, the above strategies do not work, and the applicant would otherwise pass Phase 1, the applicant's ID number should be flagged to the review organizers so that additional measures may be taken to verify the applicant's English capability.
The applicant has failed to demonstrate significant Wikimedia contributions or activities which may merit the awarding of a scholarship.
- Applicants were asked to indicate if any of following described their engagement in the movement and may be be sufficient to understand if the applicant's engagement with the movement. However, if an applicant selected a few or none of the below options, the rest of their application (particularly the answers to the long-form questions) should be read to decide whether the applicant has demonstrated "significant Wikimedia contributions/activities". Additionally, as with the English criteria, you may review the applicant's Wikimedia contributions by using the link (called "cross-wiki contribs") provided in the application, next to the "Username" field.
  - Active contributor to a Wikimedia project (e.g. Wikipedia, Commons or Wikisource), with at least 50 contributions (edits)
  - Mediawiki code contributor, gadget or other tool-builder for Wikimedia projects
  - Involvement in some form of Wikimedia organization (chapter, thematic organization or user group)
  - Wikimedia CheckUser, Admin, Bureaucrat, Steward or OTRS volunteer (current or former)
  - Wikimedia Foundation Grantee
  - Wikimedia Researcher
  - Participant in a Wikimedia program (e.g. GLAM partnership or education program)
  - Participant in Wikimedia organized events (e.g. photographer contributing to Wiki Loves Monuments (WLM), workshop attendee)
  - Organizer of Wikimedia events (e.g. WLM, edit-a-thons)
The applicant is a paid employee of the Wikimedia Foundation or an affiliate, excluding part-time employees, interns and working students.

Phase 2 scoring guidelines

Relevant experience

There are three main dimensions to consider when evaluating an applicant’s activities and experiences within the movement: Collaboration, Impact, and Community leadership. While only one score will ultimately be given against this criteria as a whole, applicants answer separate questions along each of these dimensions. As such, this guide provides a few weak and strong examples for each dimension, to help with evaluation.

The scoring rubric has remained a 0-10 scale, with the following descriptions provided for the even numbered scores. As in previous years, odd numbered scores may be awarded to those applicants who fall in-between.

0 = Not a participant, with no involvement in any Wikimedia project, organisations or initiative
2 = Very low level participant, with very occasional involvement
- Provided weak examples of collaboration on/off-wiki
- No notable impact on any Wikimedia project, organization or initiative
- No leadership demonstrated in community activities
- No outreach activity indicated.
4 = Low level participant, with occasional involvement
- Provided standard examples of collaboration on/off-wiki
- Small/narrow impact overall on relevant Wikimedia project, organization or initiative
- Minimal leadership demonstrated in community activities
- Minimal outreach activity or no explanation for lack of outreach activity.
6 = Mid level participant, with continued involvement
- Provided a mixture of standard and strong examples of collaboration on/off wiki
- Moderate overall impact on Wikimedia projects, organisations and initiatives
- Provided at least one example of leadership roles played in community activities, though minimal description was included
- Mediocre explanation of experiences with outreach activities, mostly just participant or supporting Wikimedian.
8 = High level participant, with dedication
- Provided strong examples of collaboration on/off wiki
- High and diverse overall impact on Wikimedia projects, organisations and initiatives
- Provided at least one strong, well-described example of leadership roles played in community activities
- Provided at least one example for being an organizer of outreach activities.
10 = Very high level participant, with exceptional dedication
- Provided very strong examples of regular and recurring collaboration on/off-wiki
- Very high and very diverse overall impact on Wikimedia projects, organisations and initiatives
- Provided more than one strong, well-described example of leadership roles played in community activities
- Provided more than one example for organizing outreach activities.

To the best extent possible, additional consideration should given for an individual's environmental context, i.e. environmental factors influencing their ability and interest in contributing to the wiki projects (inferring from an applicant's country, gender, and occupation). A fews examples are:

Governmental / political climate, including favorable government policies, perceived credibility of NGOs
Literacy rates overall
Availability and affordability of internet access
Cultural norms & inculcated behaviors, including:
- “Leisure time” and % time spent online
- Precedence of “participatory” / volunteer initiatives and ability to contribute
- Language status among native speakers (diglossia)

Community leadership

Key question: Has the applicant demonstrated leadership in any of their Wikimedia activities?

Notes on evaluation: While there are many roles across the movement that demonstrate some level of leadership, the amount of effort involved in executing a particular role may vary by context (e.g. many factors impact the ease or difficulty in organizing an off-wiki events). To the best extent possible, these external factors should be considered when assessing this dimension.

**Examples of Community leadership**
Minimal leadership demonstrated	Strong leadership demonstrated
Attendee of community meet-ups Participant in projects, events or programs (e.g. individual contributor, student in the Education program)	Organizer of significant projects or programs (e.g. Wikiprojects, WLM) Holds on/off-wiki governance role (e.g. admin, steward, board member) Organizer for a Wikimedia organization (User Group, Thematic Organization, Chapter) or other informal Wikimedia group Organizer of significant community conferences or outreach events

Collaboration

Key Question: How has the applicant collaborated with other individuals, organizations within the movement?

**Examples of Collaboration**
Weak collaboration	Standard collaboration	Strong collaboration
Applicants who may... Make solo edits or other contributions, but rarely engage in discussions Have activities that involve minimal or limited collaboration with Wikimedians	Applicants who may... Contribute to collaborative endeavors on-wiki (e.g. WikiProjects, Village Pump discussions, Tool Labs, etc), though mostly contributes individually Contribute individually, but actively gives and solicits feedback on new content or creations Regularly engage in talk page or community discussions	Applicants who may... Regularly partner with other individuals on wiki to create new tools, curate or improve on-wiki content, or improve policies or governance of a wiki community or User Group Regularly partner with other individuals or organizations off wiki to launch outreach & events, partner with non-Wikimedian organizations for content curation (e.g. GLAM), or coordinate local or global groups

Impact

Key Question: What was the qualitative or quantitative impact of the applicant’s Wikimedia activities?

"Quantitative" refers to results that could be measured (e.g. # participants, # articles improved, # editors that used a new feature, # of new bots)
There are two dimensions to consider when evaluating impact:

Level of impact: How much did the applicant's activities impact their project, event, or community?
- Remember that contribution to small projects or small communities can have a very high level of impact, even when the amount of work / contribution may seem low overall.
- While citing a large number of edits is demonstrative of a high level of participation, the number alone should not be considered a sufficient indicator of "high impact"; quality, subject matter, and project are just a few pieces of important information to understand the impact of those contributions. The applicant should still describe why and how those contributions have affected Wikimedia projects or initiatives.
Diversity of impact: How many different ways has the applicants activities impacted their project, event, or community?
- One strong indicator of "diversity" would be an applicant who both enumerates and describes their impact/results across the multiple categories we provided as examples in the 2019 programme criteria (reproduced below).

	Online Impact	Offline Impact
Qualitative	Increased awareness on the importance of reliable sources Increased/Improved skills of on-wiki contributors (e.g organized editing workshops) Made it easier for readers and new/experienced editors to engage (e.g. created or participated in on-wiki mentoring spaces) Improved the ability for editors to be more productive on-wiki (e.g. improved or created new MediaWiki features)	Increased awareness about the Wikimedia projects through off-wiki channels (e.g. posted articles in blogs or newspapers, or gave a presentation at non-Wikimedia conferences) Improved the public perception of Wikimedia as a source of reliable information (e.g. gave a talk about Wikipedia processes and policies that ensure reliability) Improved gender, language, or geographic diversity off-wiki (e.g. organized an event targeted at raising awareness for under-represented groups or languages) Increased/Improved skills of off-wiki volunteers (e.g. organized an event where volunteers gained knowledge on policy advocacy or event organization)
Quantitative	Identified/Addressed content or category gaps (e.g. number of new/improved articles in underdeveloped or missing categories) Made reliable sources available to editors (e.g. gained and shared access to previously locked sources) Increased access to Wikimedia by creating/improving a product that addresses access (e.g. Improved QR codes or Kiwix to support offline Wikipedia) New editors (e.g. # new editors due to organizing an editing workshop)	For events organized, number of participants that attended a Wikimedia event you organized (e.g. for organizers of photo contests, the number of contest participants) For Wikimedia programs you are involved in, the number of participants or volunteers supported (e.g. for Wikipedia Education Program campus ambassadors, the number of students supported in a semester)

Experience sharing

The key question to evaluate applicant responses is: Has the applicant demonstrated that they can sharing experiences/learning back with their community / with broader audiences?

The scoring rubric has remained a 0-10 scale, with the following descriptions and illustrative examples provided for the even numbered scores. However, as with the "Relevant experience" criteria, odd numbered scores may be awarded to those applicants who fall in-between.

0 = Applicant provided no answer, vague answers or off-topic answers
2 = Applicant provided very general comments on how to share experiences/knowledge
- Example: I think the best way to share is through face-to-face conversations.
4 = Applicant provided general lists, without any commentary on how they have been / are currently used
- Example: Wiki newsletters, on-wiki reports, personal blog, Facebook/social media
6 = Provided at least one example of how the applicant has shared experiences/knowledge, but only a minimal or general description was provided
- Example: I typically share through my community newsletter, to spread the word about new project or events. This has been pretty successful.
8 = Provided at least one strong example of how the applicant has shared experiences/knowledge
- Example: I have used my community newsletter and on-wiki reports to share about new projects, tools, or events that other communities have used to raise awareness about Wikipedia, including the way our community might be able to adapt the strategies for ourselves.
10 = Provided 3+ examples of channels through which the applicant has shared experiences/knowledge, and how they have used those channels
- Example: I have used my community newsletter and on-wiki reports to share about new projects, tools, or events that other communities have used to raise awareness about Wikipedia, including the way our community might be able to adapt the strategies for ourselves. I've also used my Chapter's general meeting and general meet-ups to speak about and discuss the potential of these ideas. More broadly, I've written about the things that I learn on my personal blog. Here's a link.

Score adjustment for previous scholars

While previous scholars were asked to answer an additional question about their past Wikimania attendance, no separate score will be given. Instead, the following augmentation will be applied to an applicant's Experience sharing score, when applicable.

No change in score: Applicant clearly articulated the results of their previous Wikimania attendance, as well as clear goals for attending again.
-2 point: Applicant included clear results or goals but not both.
-4 points: Applicant included neither clear results nor goals.