Grants:APG/Proposals/2019-2020 round 1/Wiki Education Foundation/Impact report form

Purpose of the report

This form is for organizations receiving Annual Plan Grants to report on their results to date. For progress reports, the time period for this report will the first 6 months of each grant (e.g. 1 January - 30 June of the current year). For impact reports, the time period for this report will be the full 12 months of this grant, including the period already reported on in the progress report (e.g. 1 January - 31 December of the current year). This form includes four sections, addressing global metrics, program stories, financial information, and compliance. Please contact APG/FDC staff if you have questions about this form, or concerns submitting it by the deadline. After submitting the form, organizations will also meet with APG staff to discuss their progress.

Global metrics overview - all programs

We are trying to understand the overall outcomes of the work being funded across our grantees' programs. Please use the table below to let us know how your programs contributed to the Global Metrics. We understand not all Global Metrics will be relevant for all programs, so feel free to put "0" where necessary. For each program include the following table and

Next to each required metric, list the outcome achieved for all of your programs included in your proposal.
Where necessary, explain the context behind your outcome.
In addition to the Global Metrics as measures of success for your programs, there is another table format in which you may report on any OTHER relevant measures of your programs success

For more information and a sample, see Global Metrics.

Metrics from our proposal

Metric	Achieved outcome	Explanation
1. # of total participants	15,476	We nearly made our goal of 16,325, an impressive statistic given the pandemic.
2. # of newly registered editors	13,218	A larger percentage of our participants were repeat participants in our programs than expected.
3. # of content pages created or improved, across all Wikimedia projects	26,690	We exceeded our goal, mostly thanks to some active Wikidata contributors.
4. quantity: words added to the article namespace	11,582,343	We nearly made our goal of 12.8 million, an impressive statistic given the pandemic.
5. quality: number of articles increasing by at least 10 points on the ORES scale	3,681	We nearly met our goal of 4,620, an impressive statistic given the pandemic.

Global metrics

Metric	Achieved outcome	Explanation
1. # of active editors involved	5	These are Wiki Education's staff members.
2. # of new editors	13,218	This is the same number as our "newly registered users" metric.
3. # of individuals involved	16,057	This is the same number as our "total participants" plus the instructors who taught with Wikipedia.
4. # of new images/media added to Wikimedia articles/pages	2,433	We had a total of 7,946 uploaded, but many are not yet used in articles.
5. # of articles added or improved on Wikimedia projects	26,690	This is the same number as our "number of content pages created or improved, across all Wikimedia projects" metric.
6. Absolute value of bytes added to or deleted from Wikimedia projects	76,836,067	We don't measure deletions, so this is only additions.

Telling your program stories - all programs

Wiki Education All Staff meeting via Zoom, July 2020

When the COVID-19 pandemic struck the world early in 2020, it created havoc across every organization's plans. Wiki Education was no exception; the instability in the weeks and months following the initial declaration of a pandemic challenged our organization as a whole. The generous grant from the Wikimedia Foundation in 2020 represented 16% of our planned revenue; however, several of our major funders pulled expected renewals based on the economic instability following the pandemic, causing our annual revenue to be cut in half. To respond to this situation, we made the difficult decision to say goodbye to half of our staff and cut all non-essential spending. These painful cuts enabled us to survive 2020, and we are deeply grateful to the Wikimedia Foundation, the Stanton Foundation, and the Hewlett Foundation, whose renewals of our grants in early 2021 have set us up for success in the next year too.

In the first half of the year, we had made good progress toward our programmatic goals despite the challenges of the pandemic. Wiki Education was more fortunate than others in the Wikimedia movement because the majority of our programs were already based online. The support we provide to college and university instructors in our Wikipedia Student Program was virtual already, and the Wikipedia and Wikidata courses we taught in our Scholars & Scientists Program were already conducted via Zoom. In the second quarter of the year, immediately following the start of the pandemic, engagement in our programs decreased slightly as some participants needed to reduce commitments to adjust to new situations in their lives caused by changes stemming from the pandemic. By the second half of the year, people had adjusted and if they enrolled in our programs they stuck around at similar levels to pre-pandemic; however, our reduced staff capacity meant we couldn't support as many program participants as planned. Nevertheless, given all these challenges, we came very close to meeting our overall goals.

We couldn't be prouder of how our staff responded to the challenges of 2020, nearly meeting all of our impact goals despite the trials of a global pandemic.

Wikipedia Student Program

Quantitative targets

Measure of success	Goal (2020)	Total	% completed	Notes
Total Participants	16,000	15,063	94%	We came remarkably close to meeting our goals, given the challenges of the pandemic.
Newly Registered	15,500	13,039	84%	We came remarkably close to meeting our goals, given the challenges of the pandemic.
Content Pages Improved	15,500	13,910	90%	We came remarkably close to meeting our goals, given the challenges of the pandemic.
Quantity	12,500,000	11,104,310	89%	We came remarkably close to meeting our goals, given the challenges of the pandemic.
Quality Articles	4,500	3,537	79%	We came remarkably close to meeting our goals, given the challenges of the pandemic.

What we did

The Progress Tracker helps guide students through the article expansion process.

The new grading view highlights where each student editor is in their assignment.

The spring 2020 academic term, which roughly correlates to the first half of the year, was supposed to be our largest ever: We'd successfully onboarded 409 courses, our largest cohort to date. To prepare for this large influx of courses, we debuted two new technical features, the Progress Tracker and grading view for instructors. To refresh from our midterm report:

The Progress Tracker scaffolds the article building work students do. When a student chooses an article, the Progress Tracker will appear for that article with a number of useful links. Students can then quickly access a sandbox link for them to craft a bibliography, another sandbox link for them to draft their article, and training modules relevant to each stage of the process. It also links to the sandbox pages where classmates will add their peer reviews.
The new grading view for instructors includes links to the key sandbox pages where students complete the different phases of their assignment — preparing their bibliography, drafting their articles, reviewing the drafts of their peers, and editing live Wikipedia articles. The goal of this new view is to make assessing student work easier for faculty.

While the end result of the term missed our expectations due to the pandemic, we still had meaningful impact on Wikipedia content, and we found these new technical features helpful in supporting students and instructors involved in our program.

This trend continued in fall 2020, where we still supported large numbers of students but with reduced staff capacity. We debuted a new application process, and asked faculty who were interested in teaching with Wikipedia to submit their course page by a certain deadline. As part of the application process, we asked if instructors were okay having students stay in sandboxes until we could evaluate their work, which was likely to happen after the official end of the term. We accepted about a third of the courses under the condition that students would stay in sandboxes. This enabled us to — with significantly fewer staff — still support 322 courses in fall 2020.

The courses produced some excellent work, e.g.:

A chemistry student at the University of Michigan created a new article on RNA theraputics, an important article that's received thousands of page views as people seek vaccine information.
Rice University students collaborated to significantly expand the article on education inequalities in the United States, adding information about the history, racial and other factors, and potential solutions.
A psychology student at the University of California Los Angeles created a new article on social projection, which has received tens of thousands of page views since it was created.
A medical student at Boston University significantly expanded the article on Methylprednisolone, used to treat COVID-19.
A medical student at Vanderbilt University significantly expanded the article on respiratory syncytial virus, or RSV.
Other Vanderbilt University students created the article on Indigenous storytelling in North America.
A group of ecology students from Eastern Washington University collaborated to completely rewrite a high-trafficked but not high quality article, energy flow, adding images from Wikimedia Commons.

Not only did our Wikipedia Student Program produce excellent content for Wikipedia, it also provided a lifeline for faculty facing Zoom teaching, something that's revealed in our end-of-term instructor survey. In the open text field for the survey, faculty shared their experiences:

"Wikipedia worked extremely well as a tool in the online classroom. It offered an asynchronous learning element to my course which was otherwise largely synchronous. Using breakout rooms on Zoom, it gave students an opportunity for collaborative work, which they told me they appreciated enormously as a way to connect with fellow students in a relaxed classroom setting."
"I actually chose to begin this assignment this year because of the pandemic, I was hoping to give the students an opportunity to collaborate and to do some virtual group work — hopefully reduced the feelings of isolation!"
"Because we wrote as a team, the Wikipedia assignment ended up helping us to build an esprit de corps which was so welcome in these COVID-online times."
"In a semester where students were anxious, overwhelmed, and depressed, it was more important than ever to have assignments that students found meaningful, and I believe that this fell into that category."
"This assignment brought energy to a semester that was weighed down by all of the stress of the pandemic."

We are grateful that our program was able to offer engagement for instructors, many of whom were facing the challenges of virtual teaching and learning for the first time.

Celebrating 10 years

We would be remiss in noting that 2020 also marked the 10-year anniversary of the start of the program. To honor the occasion, we published a series of blog posts from staff reflecting on the achievements.

Executive Director Frank Schulenburg took a look back at the program, where it's come from, and where it is now.
Chief Programs Officer LiAnna Davis discussed impacts to Wikipedia's new active editor numbers from the program.
Senior Wikipedia Expert Ian Ramjohn reflected on changes we've made to Wikipedia's content.
Senior Program Manager, Wikipedia Student Program Helaine Blumenthal reviewed student learning outcomes.
Director of Partnerships Jami Mathewson examined our impact to Wikipedia's gender gap.
Wikidata Program Manager Will Kent showcased our expert engagement learnings.
Chief Technology Officer Sage Ross narrated our program's history with technology.
Helaine also reflected on how we've tackled equity.

We didn't just want to reflect on our perspectives of the last 10 years, however. We also wanted to showcase those who helped make it happen:

Max Klein and PJ Tabit joined Wikipedia to support student editors; both are still active today.
Jonathan Obar started teaching with Wikipedia in the pilot program; he's still teaching in our program today!
Amanda Levendowski was a student in the pilot program; now, she's a proud Wikipedian and a law professor who assigns her own students to edit Wikipedia.
Tighe Flanagan took three classes in the pilot program and then worked for the Wikimedia Foundation helping to expand the program globally.
Kasey Baker learned to edit as a student in the pilot program and has gone on to bring hundreds of others to Wikipedia.

What worked well

Help requests from students were less frequent in 2020 than in 2019, and were also more evenly spread throughout the Fall term.

Our strategy of continually identifying the most common points of confusion and adapting the Dashboard to fix them has continued to pay off this year. Our main technical focus in late 2019 and early 2020 was a set of features for providing step-by-step guidance for the key stages of Wikipedia writing assignments, with related updates to standardize the locations for draft and peer review work and improve the interface for instructors to see what students did in each stage of the Wikipedia project.

We are also quite proud of our help structure for students as well. In the 10 years we've been running this program, we've learned a lot about how to socialize new contributors to Wikipedia. As we hoped, the combination of interface improvements and refinements to our help materials is reducing the amount of time staff spend responding to common problems. We found a modest decrease in the rate of help requests throughout 2020, compared with previous years. The requests were also spread more evenly throughout the term, with a lower peak in the busiest month of November — which is especially beneficial for us, as peak periods are the main constraint on how many courses we can support each semester. We were able to succeed in the second half of 2020 in part because of the dedication of our staff, but also because of the groundwork we've been laying for a decade, in terms of quality onboarding materials, clear training modules, and a robust technical platform. With only two full time staff members dedicated to the project in fall 2020, we were able to successfully support more than 6,800 new contributors to Wikipedia.

Our successes in this area are summed up in part of one student's reflection essay:

As a newcomer to this successful online community, I'll be honest, I was really overwhelmed. Writing a Wikipedia article seemed like a daunting task that I wasn't up for. But eventually, I did, and my article is now a hundred times more developed than it was before. The trick? WikiEducation. I don't think I could have done this without it.
WikiEducation is a step-by-step learning resource that teaches students and teachers how to contribute to Wikipedia. It's helpful for Wikipedia users, because it gives insight into what the websites norms and policies are. As a newcomer, this leaves less room for doubt about whether or not "I'm doing it right." I could imagine that feeling overwhelmed and unqualified to write articles for Wikipedia are oftentimes a deterrent for newcomers.
Not only is it enormously helpful for users, but Wiki Education is also helpful for the site of Wikipedia itself, because it does a great job of socializing users. As we learned in class, one of the greatest dangers to an online community is newcomers that break norms. Newcomers that do not have a sense of their role in the community perform poorly, don't commit, annoy experienced users, and turnover quickly. However, it is unlikely that a newcomer that has completed the Wiki Education training modules would break norms, because the spinoff site does a fantastic job of laying the ground rules for the website.
A third reason why I believe Wiki Education is a great tool is because it adds more credibility to the website. If all users were given the opportunity to complete the training module, it might lead to a better reputation of the website. Knowing that the site was only contributed to by trained users might change the minds of those teachers I had in high school, who believed Wikipedia was merely a space for non-academics to mess around and edit articles.

We look forward to continuing to provide support for students like this, so they can have quality educational experiences while improving Wikipedia, for another decade.

What didn't work well

As mentioned earlier, in order to support more courses with our reduced capacity in the fall 2020 term, we attempted something new: encouraging a third of courses to leave work in sandboxes. While this helped us avoid a major challenge on wiki, we found it did not significantly reduce the staff time necessary to support these courses as anticipated, and results from our instructor survey showed students were less motivated by the assignment. We spent significant staff time in the final weeks of December combing through the work left in sandboxes, and reached a few conclusions:

Moving out of sandboxes earlier in the term helps head off obvious problems that doom projects from the start (poor sourcing, improper tone, etc.). If these are caught and re-directed early in the term, students can still salvage the project and create something Wikipedia-ready. The sandbox only process missed that step, leading to large chunks of unusable content left in sandboxes.
For good content, many articles had intermediate edits made between when the students did the work in sandboxes and when we finally had time to review it, making history merges impossible from our end, and students after the end of the term failed to follow-up on moving the work themselves.
Many students ignored our requests to stay in sandboxes and moved content live anyway; challenges emerged, however, because their course pages didn't have necessary instructions about how to do this, leading to some improperly moved content.
We still spent significant staff time answering questions for these courses during the term, and the amount of time to review each article in sandboxes after the end of the term was not worth it for the minimal high quality content we were able to actually move live.
Instructors who had previously had typical assignments noted decreased engagement from students this term, when they couldn't move work live.

For all these reasons, we've significantly scaled back on our sandbox-only courses for the spring 2021 term. A few instructors actually liked it, so we'll continue letting those participate in the program that way, but we no longer believe it will be worth it from a staff time savings and quality perspective, as we'd hoped.

Connections to strategy

The work our Wikipedia Student Program does aligns well with Recommendation 8, Identify Topics for Impact. The sheer scale at which our program operates means it has a large influence on content on English Wikipedia.

In 2020, 44% of all pageviews of Wikimedia projects went to the English Wikipedia. In other words, one Wikimedia project has nearly the same number of page views as all the other projects combined. This outsized impact of English Wikipedia is demonstrated in other ways too: As one of the global languages, emerging communities often use translated articles from English Wikipedia to build coverage on their language Wikipedias.

But English Wikipedia still has significant knowledge equity gaps. And so programs like Wiki Education's Wikipedia Student Program are critical to the Wikimedia strategy of knowledge equity. Addressing content equity gaps on our most read project, a project used as the basis for translation by emerging communities, is critical if we want to achieve knowledge equity.

In 2020, Wiki Education supported more than 60 courses on topic areas related to race, gender, and sexuality content gaps. Some examples:

Delia Steverson's University of Florida students improved articles related to African American literature.
Two Canadian courses improved articles related to Indigenous Canadians.
A University of California Berkeley course improved topics related to African politics.
Heather Sharkey at the University of Pennsylvania led students to improve articles related to Middle Eastern food history.
Military history isn't just about the details of battles; it's also about the victims of nuclear weapons, as one class demonstrated.

Through this program's work, we're able to continually improve English Wikipedia's coverage of topics related to knowledge equity.

As we mark 10 years of the program, it's also a moment to acknowledge this program is succeeding today in part because of following Recommendation 10, Evaluate, Iterate, and Adapt. We've been continually monitoring the program, evaluating it, iterating based on what we've learned, adapting to external factors, and scaling as we go. We believe our program is an existing success story for this model, and we're excited to see it be adopted more widely in the Wikimedia movement.