Grants:APG/Proposals/2017-2018 round 1/Wiki Education Foundation/Proposal form

Proposal by Wiki Education Foundation to support its annual plan with $750,000 USD.

In 2018, the Wiki Education Foundation seeks to improve the quality and quantity of information available on Wikimedia projects by bridging the gap between academia and the Wikimedia projects. We will do this through three main programs: (1) Our Classroom Program, in which we will support more than 16,000 university students in the United States and Canada as they edit the English Wikipedia as a class assignment; (2) Our Visiting Scholars Program, in which we facilitate providing university logins for existing Wikipedia editors, giving them access to sources that would otherwise be behind paywalls; and (3) Our Wikipedia Fellows pilot program, in which we train members of academic associations to edit Wikipedia articles in their areas of subject matter expertise.

Comment on Wiki Education Foundation's proposal here

Return to the community review page.

We ask that you please don't make any changes directly to this page after the proposal submission date, but comment on the discussion page.

If you need to review the edit instructions you will find them in the edit intro.

Use this form if you are eligible to submit a proposal in the current round, to request funding in the current round.
This form must be published on Meta by the proposal submission date for each round to be considered, and no changes may be made after the submission date without staff approval.
This form complements your organization's annual plan, detailed budget, and strategic plan. It won't be considered complete without a detailed budget and strategic plan.
Organizations may apply for a funding period that does not exactly match their fiscal years. Follow the instructions carefully to understand what time period each question refers to.
Refer to the framework, guidance from the Board, and the FDC's recommendations before using this form, so you have an understanding of the Annual Plan Grants process.
Please Email FDCSupport@Wikimedia.org with questions about using the form.

A few terms used in the form:

FDC proposal form terms	Wikimedia terms	Learning & Evaluation terms
Revenue Spending or Expenses Currency requested Staff Long-term contractors Rate of change Annual plan Funding period Fiscal year	2016 Draft WMF Strategy Wikimedia project	Grant metrics Program Objective Goal Target Needs assessment Logic model Impact

Overview

1. In order to support community review, please provide a brief description of your organization's work in the upcoming funding period.

In 2018, the Wiki Education Foundation seeks to improve the quality and quantity of information available on Wikimedia projects by bridging the gap between academia and the Wikimedia projects. We will do this through three main programs: (1) Our Classroom Program, in which we will support more than 16,000 university students in the United States and Canada as they edit the English Wikipedia as a class assignment; (2) Our Visiting Scholars Program, in which we facilitate providing university logins for existing Wikipedia editors, giving them access to sources that would otherwise be behind paywalls; and (3) Our Wikipedia Fellows pilot program, in which we train members of academic associations to edit Wikipedia articles in their areas of subject matter expertise.

2. Name, fiscal year, and funding period.

Legal name of organization: Wiki Education Foundation
Organization's fiscal year: 07/01/2017–06/30/2018
12-month funding period requested: 01/01/2018–12/31/2018
Currency requested: US Dollars
Name of primary contact: LiAnna Davis, Deputy Director

3. Amount requested.

Table 1

	Currency requested	US$
Total expenses for the upcoming year	$2,823,819	$2,823,819
APG funding requested for the upcoming year	$750,000 Classroom Program: $400,000 (40% of total program cost) Visiting Scholars Program: $200,000 (74% of total program cost) Wikipedia Fellows Pilot: $150,000 (56% of total program cost)	$750,000 Classroom Program: $400,000 (40% of total program cost) Visiting Scholars Program: $200,000 (74% of total program cost) Wikipedia Fellows Pilot: $150,000 (56% of total program cost)
Amount of funding received from WMF for the current year	$99,996^[1]	$99,996^[1]

Table 1 notes:
^[1] We received a $99,996 Simple APG that covered the period 1 May 2017 to 30 September 2017.

4. How does your organization know what community members and contributors to online projects need or want? Does your organization conduct needs assessments or consult the contributors and volunteers most involved with its work?

More than half of our staff are experienced volunteer contributors to the Wikimedia projects, especially Wikipedia and Wikimedia Commons, and all programs staff interact regularly on the English Wikipedia. Our ongoing engagement with Wikipedia in both our work and volunteer capacities gives us a solid footing for all organization discussions regarding the Wikimedia community. Nine of our staff have graduate degrees, most have experience teaching college courses, and three have Ph.D.s, meaning we are also extremely well versed in the academic community we serve.

For our Classroom Program: At the end of each term, we conduct an instructor survey, which we send to the 300+ people who teach with Wikipedia each term through our Classroom Program. In the survey, we ask them questions about the support they received from us, the support their students received from us, what additional support would be helpful, and how our Dashboard software worked for them. We have done surveys in the past with student editors as well, but they require us going through an Institutional Review Board, which adds a lot of complications. Instead, we have incorporated lightweight requests for feedback at the end of student training modules, and adapt our support based on their responses. When we develop new materials (such as our discipline-specific handouts), we consult experienced editors in those disciplines through WikiProjects and individual outreach. When an existing editor raises concerns about a student's editing on the Education Noticeboard or through another venue, we track this as an "incident", work to resolve it, and then address whether we could have prevented it through developing better materials. We have an ongoing relationship with the English Wikipedia community through which they know they can provide feedback, make requests, and raise issues knowing that we will work to understand and respond. We monitor students' interactions with the community, which gives us an idea of which individual issues need to be addressed even if editors don't report them, and allows us to see patterns that inform how we develop our support infrastructure. Finally, we track the questions we get from student editors via ask.wikiedu.org and email, and take common questions into consideration as we plan out what additional resources we need to modify or develop.

For Visiting Scholars: The foundation of the Visiting Scholars program is a direct response the Wikipedia community's frequently expressed need for access to high-quality research resources. We've solicited feedback from the community and program participants both on-wiki and off, including needs-finding conference sessions at WikiConference North America and Wikimania.

For Wikipedia Fellows: We will engage in regular dialogue to see how the pilot is working for participants, and to help resolve any issues that come up.

5. Please provide a link to your organization's strategic plan, and a link to your separate annual plans for the current and upcoming funding periods if you have them.

Our Annual Plans for the three years of our existence are all available on our website. Our last strategic direction, developed in collaboration by our board and senior leadership during a 6-month strategic planning exercise in the first half of 2015 (see our Annual Plan for 2015–16), expired in 2017. We had initially planned to launch the next round of strategic planning in 2016 but decided to postpone the process to 2017, which allows us to sync our thinking with the new Wikimedia movement strategy. Board and staff started working on our new strategic plan in July 2017. At the end of October 2017, we'll have an in-person meeting of our board and senior staff in order to set the course for the next three years. We expect our board to confirm our new strategy during its in-person meeting in January 2018.

Financials: current year

The purpose of this section is to give the FDC an idea of how your organization is receiving funds and spending funds toward your current plan. Your current funding period is the funding period now in progress (e.g. 1 January 2016 to 31 December 2016 for most organizations).

Revenues or expenses	Planned (budgeted), total	Planned (budgeted), YTD	Actual, YTD ^[2]	Projected, total
Financials for the current funding period ^[1]
Revenues	$2,738,000 ^[3]	$350,000	$403,702	$2,738,000
Expenses	$2,427,800	$535,757	$412,531	$2,427,800

Table 2 notes:
^[1] The numbers in Table 2 reflect revenues and expenses of our current fiscal year 2017–18.
^[2] As per September 28, 2017.
^[3] The difference between revenues and expenses will go toward next fiscal year 2018–19.

Programs: upcoming year's annual plan

This section is about your organization's programs. A program is a defined set of activities that share the same objectives and a similar theory of change. Please share the general goal of each program, and then list the specific objectives that the program will meet. Please do not include information about your organization's operating activities in this section. You may provide information about activities like administration, staff and board training, fundraising, governance, and internal IT in another section or in a supplementary document, but please do not include these activities here as programs.

1. For each program, and overall

Include targets for each of the three shared measures for each program, and overall. If one or more of these required metrics are not relevant to any of your programs, please consult your program officer.
Also choose at least two grantee-defined metrics to highlight in this section, and include targets for each of these grantee-defined metrics for each program, and overall. (Other program-specific metrics may be included in your program objectives, in the detailed program sections below.)

Table 3

Shared metrics

Participants: The number of people who attend your events, programs or activities, either in person or virtually. This definition does not include people organizing activities, social media followers, donors, or others not participating directly.
Newly registered: The number of participants that create new accounts on a Wikimedia project. These include users who register up to two weeks before the start of the event.
Content pages: A content page is an article on Wikipedia, an item on Wikidata, a content page on Wikisource, an entry on Wiktionary, and a media file on Commons, etc. This metric captures the total number of content pages created or improved across all Wikimedia projects.

Grantee-defined metrics

Quantity: Number of words added to the article namespace.
Quality articles: Number of articles that have at least a 10-point improvement in an ORES-based quality prediction score, which indicates significant improvement of the "structural completeness" of an article. See the "Defining quality content" sections in each program section for a description of how this applies to that program.

Program	Participants	Newly registered	Content pages	Quantity	Quality articles
Classroom Program	16,000	15,500	19,200	11.2 million	5,000
Visiting Scholars Program	15	0	680	810,000	435
Wikipedia Fellows Pilot	20	20	80	24,000	40
TOTAL FOR ALL PROGRAMS	16,035	15,520	19,960	12 million	5,475

Table 3 notes:

See additional targets for each program as connected to the logic model at the end of each program's section.

2. Please list your goals and objectives for each program. Please be sure your objectives meet all three criteria for each program

The objectives listed are each SMART: specific, measurable, attainable and relevant, and include time-bound targets.
Include both qualitative targets and quantitative targets, and remember to highlight your baseline metrics.
Provide any additional information that is important to our understanding of this program. For example, you may include needs assessments, logic models, timelines, tables, or charts. Share how this program will contribute more broadly to movement learning, or explain how your program aligns with important Wikimedia priorities such as increasing participation and improving content on the Wikimedia projects.

Classroom Program

Connection to the draft Wikimedia Movement Strategic Direction

The Classroom Program fits into both the knowledge as service and knowledge equity directions in the current draft. Creating the infrastructure to bring external partners into our movement has been a key role the Wiki Education Foundation has been playing for several years; our Dashboard software is a key example of this. It provides the structure to enable the management and tracking of programs to bring newcomers to Wikipedia.

Those newcomers represent the knowledge equity element. Our Classroom Program brings in a diverse group of students from community colleges to Ivy League schools. Our work to target content improvement on topics related to gender, race, and sexuality has meant that more people globally have access to high quality free knowledge on these topics. In addition to providing all of our students with the resources they need to successfully contribute to Wikipedia, regardless of their backgrounds and academic experience, we provide additional support through our discipline-specific handouts. These handouts ensure that students editing in certain fields, such as gender studies, can contribute more equitably to Wikipedia.

In our Classroom Program, student editors at universities in the United States and Canada edit Wikipedia articles as a class assignment. Our theory of change is that by providing support for instructors and student editors, we will improve the quality and quantity of Wikipedia content in underdeveloped subject areas.

Background of program

When we started the pilot of the program (the Public Policy Initiative) in 2010, we had seen university instructors around the world assign their students to edit Wikipedia as a class assignment — with good results when the instructor was a Wikipedian themselves, and with rather mixed results when the instructor didn’t know much about Wikipedia. We wanted to create a sustainable, volunteer-driven program in which university instructors who didn’t know much about Wikipedia could get the support from our program to successfully teach with Wikipedia.

We started the program at the Wikimedia Foundation; a few years in, we knew what we had was a good program but it wasn’t the focus of the Wikimedia Foundation, and it needed to have more attention to realize its potential. A group of volunteer Wikipedians and professors proposed a new organization, the Wiki Education Foundation, and in late 2013, the program officially moved from WMF to the Wiki Education Foundation.

Eight years of growth in number of classes participating in the Wiki Education Foundation's Classroom Program, 2010–17.

Nearly eight years in, we’ve learned a lot about this program in the United States and Canada. One of our big learnings is that there is a lot of enthusiasm for teaching with Wikipedia among university faculty here. Higher education trends point toward providing more project-based, active, or service learning projects; Wikipedia assignments meet all these criteria. Universities are desperately trying to teach students crucial information literacy skills. As a 2016 Stanford University study measuring information literacy skills of middle school, high school, and college students noted, “Overall, young people’s ability to reason about the information on the Internet can be summed up in one word: bleak.” Again, Wikipedia assignments build these skills. And finally, the academy has embraced Wikipedia as a starting place for research, and begun to recognize the importance of improving the information resource the world uses. As our slogan says, “Don’t cite it. Write it!” This enthusiasm for our program has enabled a sharp growth trend from 14 classes participating in our program in Fall 2010 to 358 classes participating in Spring 2017.

We’ve also learned that with the right support, Wikipedia class assignments can be successful even if the instructor has never edited an article. Assignment design is key; so are training modules, brochures, a searchable question-and-answer platform, and experienced Wikipedians on standby to provide a human backup when something goes wrong. The wide range of support offered by the Wiki Education Foundation has enabled more than 1,000 individual faculty members to incorporate Wikipedia into their classes since 2010; fewer than 5% of them had ever edited Wikipedia before our project.

Another learning is that the volunteer-driven element of our program simply didn’t scale effectively. We had initially created “Wikipedia Ambassadors” who would have a 1:1 relationship with students to provide support, with a “Regional Ambassador” volunteer who would coordinate the work in a particular region. This model worked well when we were supporting 20–30 classes, but started breaking down shortly after we started scaling. We weren’t able to find enough volunteers to meet the demand from instructors. There was a wide quality and quantity gap in the support individual volunteers provided to courses. Many well-meaning volunteer editors were also students themselves; they were eager to help during the beginning of the term when their own course work was lighter, but when students wanted help often naturally coincided with the biggest crunch time of their own academic schedule, when they had less time to give. And not everybody wanted to do all of the tasks for a class, making it extremely difficult to pair the right volunteer with the right class. We realized we were spending dramatically more staff time trying to coordinate volunteer Ambassadors, and with inconsistent support to classes, than we would be if we simply brought that role into our organization as staff; we did this in 2014. While we believe the volunteer model works well for small-scale education programs, we find it to be a better use of resources to have a staff-driven program at our scale.

A scalable education program

While the amount of staff time devoted to the Classroom Program in FTE equivalent terms has stayed relatively flat over the last four years (+/- 1.5 FTEs), the number of students has dramatically increased, showing that we are able to increase the impact of our program without adding staff (our biggest expenditure for the program).

The Wiki Education Foundation’s organizational goal continues to be building a scalable model: Rather than increasing our budget as we increase the number of students we add to the program, we want to instead keep our spending on the program stable while we increase our impact. To do so, this required a large investment (in our 2015–16 fiscal year) to build out our Dashboard course management software tool. We’ve tripled the number of classes we supported over a three year period (from 117 in spring 2015 to 358 in spring 2017) without increasing the staff dedicated to the Classroom Program, clearly demonstrating the success of this approach.

Process map describing the Wiki Education Foundation's Classroom Program as of 2014. (Click through for a larger version.)

In 2014, we approached how to scale the program by creating a process map of our program. In the process map, we walked through each stage of the Classroom Program experience, identified where there were current bottlenecks to our scaling, and then identified what potential technical solutions would be to each of those, when there was a necessity for human intervention, and identifying any dependencies that were our assumptions. Creating this process map was a key step for our organization, because it created a roadmap for what we needed to create in terms of our technology and our communications materials to support a growing number of students. Today, our Dashboard course management platform has resolved most of the bottlenecks identified in our original process map.

A note about the Program & Events Dashboard

Our Dashboard software is released under an open license, and in 2015, we installed a version of it on WMF Labs called the Program & Events Dashboard, or P&E Dashboard. Thanks to some work from WMF staff, some collaborations at past Wikimania Hackathons, some Google Summer of Code and Outreachy interns, and some development work from the Wiki Education Foundation, the P&E Dashboard has a pared-down feature suite, but works across all languages and Wikimedia projects. Program leaders can use it to see what participants in programs including education, edit-a-thons, writing contests, and photo contests have added to Wikimedia projects. Many of the program management features on our Dashboard (the Assignment Design Wizard, week-by-week timeline functionality with temporally assigned training modules, the alerts system, the survey system) are not available on the P&E Dashboard, and program leaders who’ve used the P&E Dashboard have requested a number of other features that are useful for programs like edit-a-thons that the Wiki Education Foundation doesn’t do. That means there is a separate technical roadmap for the P&E Dashboard.

We are not including any money for development for features on the P&E Dashboard's technical roadmap in our FDC proposal. Instead, we are only seeking funding for technical projects that will help the Wiki Education Foundation's version of the Dashboard. If we develop a feature for our Dashboard that is also relevant for the P&E Dashboard, we will deploy it to the P&E Dashboard. But specific technical features for the P&E Dashboard that aren't relevant to the Wiki Education Foundation's programs will be funded separately.

Here’s how our Dashboard software works:

A new instructor creates a Wikipedia account, and is guided through an online orientation that introduces them to key Wikipedia policies and what teaching with Wikipedia looks like.
After successfully completing the orientation, the instructor is offered a series of assignment types that they can create. Most instructors choose the “Expand or create a Wikipedia article” assignment, although some do other options like copyediting, adding images, or translation.
The Dashboard’s Assignment Design Wizard walks the instructor through a series of questions about their course and how they would like the assignment to work (e.g., does the instructor want to create a list of articles for students to choose from, or will the students need to identify their own articles for improvement? Are students working individually or in groups? Are students possibly working on medical topics?).
Based on what the instructor has inputted in the Assignment Design Wizard, the Dashboard creates a suggested timeline for the Wikipedia assignment. Relevant training modules are automatically added (so, for example, students who might be working on medical content will get a special module on MEDRS). The instructor can edit this timeline if they choose. Once they are happy with their assignment, they’ll click the submit button, which causes the course page to get copied to the English Wikipedia for transparency’s sake, and notifies our staff that a course page is available for review.
Our Classroom Program Manager reviews the course page to ensure any edits the instructor made meet our best practices (so, for example, if the instructor has added that they will grade only on what sticks on Wikipedia, not on what students did, we won’t approve the course page until that language is removed). This is also the point where we will discourage classes that are not appropriate for full-length writing a Wikipedia article assignments (e.g., those with more than 100 students), pushing them to consider an alternative assignment.
We assign each course to one of our two Wikipedia Content Experts, who are longtime Wikipedia editors responsible for overseeing student work and helping resolve problems.
Once the course is approved, the instructor receives a link to distribute to their students. Students click on this link, create Wikipedia accounts, and enroll in the Dashboard. We automatically put links on their user page to their course page, and we create their sandbox, replacing the typical sandbox template with a template that directs them to our staff for questions.
On a week by week basis, the Dashboard will assign brochures to read, training modules to complete, or other references for students as needed. They learn new things about Wikipedia the week they need to know them, so for example, students learn about notability and sourcing early in the term, but they’re not served the training module about how to move out of their sandboxes until several weeks in, when they’re supposed to move their article.
The Wikipedia Content Expert assigned to the class sees which classes are active each week, giving them an opportunity to proactively look into what students are doing. There are also a series of automated alerts that the Dashboard detects and notifies the Wikipedia Content Expert about, including: if a student edits an article with a discretionary sanction tag, if an article a student edited is nominated for deletion, and if an article a student edited is nominated for the Did You Know section on Wikipedia's main page. These alerts enable us to head off any potential problems without burdening valuable volunteer time.
When students have questions, they’re prompted to search our help forum (ask.wikiedu.org), which is a Q&A format that offers answers to the most frequently asked questions. While anyone can log into ask and answer questions with their Wikimedia accounts, the majority of program participants simply use the “Get Help” feature in our Dashboard to reach out to the Wikipedia Content Expert assigned to their course. At the end of each term, we add additional questions and answers to ask.wikiedu.org that we received throughout the term, so the resource continues to grow.
At the end of the term, instructors use the Dashboard’s metrics, authorship highlighting (which shows which students added what to each article), diff viewer, and other tools to easily visualize and grade students’ work.
When the course is finished, the Dashboard emails the instructor a link to our end-of-term survey, where we ask what went well, what didn’t, and what we could do better next term.
At the beginning of the next term, the instructor can either go through the Assignment Design Wizard again for their next course, or clone their previous course page and simply adjust the dates, and the process starts again.

Through this process, many instructors and students go through our program without ever interacting with one of our staff. Those who do interact with us mostly do in their first term teaching; once they have a good feel for how the program works, our tools enable them to run successful Wikipedia assignments without taking our staff time. The Dashboard has enabled us to scale such that we are able to serve them the appropriate training modules and brochures so they can be self-sufficient, but with enough monitoring and alerts that we are able to jump in to help when students run into trouble.

Which classes we support — and which ones we don't

We've learned seminar-style classes where students have a close connection to the instructor work best in the program; our average class size is 21 students.

We've learned in doing this program for many years that some courses are a great fit for full Wikipedia writing assignments, and some — no matter how much the instructor or we might want it to work — just aren't. The Wiki Education Foundation has worked with faculty at more than 500 universities in the United States and Canada, supporting classes at institutions ranging from community colleges to small liberal arts colleges to Ivy League schools to large public research universities.

Our experience has taught us that the most important factor in whether a course adds good content to Wikipedia is how engaged the instructor is in providing a meaningful learning experience for their students, and exceptional teachers can be found across the academic spectrum, from adjuncts to tenured professors. We avoid classes where that engagement is difficult: the class has more than 100 students or the class is primarily taught by teaching assistants (TAs). We also avoid survey courses for non-majors, where students likely haven't learned enough depth about a topic to make a good contribution to Wikipedia. We also screen carefully to ensure instructors aren't expecting students to do things like original research, requiring that students submit to the Good Article or Did You Know process even if the article doesn't qualify, or grading on what "sticks" on Wikipedia. This course evaluation process honed over many years enables us to identify the courses with proper assignment design that are most likely to have a positive contribution to Wikipedia, and head off those that are likely to end poorly.

A powerful tool to target content gaps

More than 60% of the Wiki Education Foundation's students in our Classroom Program identify as women.

The beauty of our education program isn’t just that we are improving the quality and quantity of information available on Wikipedia: It’s that we can target the subject of that information to match the English Wikipedia’s existing content gaps. One of the most well-publicized gaps is the gender gap; not only does this manifest itself in the contributor base on Wikipedia, but it also manifests itself within the content: topics related to race, gender, and sexuality have significant gaps on the English Wikipedia. The Wiki Education Foundation’s Classroom Program has made a major impact in the gender gap on Wikipedia over the last several years. Around 60% of our student editors are women. And thanks to our partnership with the National Women’s Studies Association (NWSA), we’ve improved more than 3,500 articles related to race, gender, and sexuality on Wikipedia in the last two years.

These partnerships are a key strategy for several of the Wiki Education Foundation’s programs. For the Classroom Program, our partnerships with academic associations prove to be fertile recruiting ground for instructors. For example, with our NWSA partnership, they provide us a workshop slot and a free exhibit hall booth at their annual conference. Our staff travels to wherever the conference is being held, leads the workshop, and collects names and email addresses of instructors who are interested in teaching with Wikipedia. We follow up with these instructors after the conference to encourage them to start teaching with Wikipedia by logging into the Dashboard software as noted above. NWSA also sends out email blasts to their members, features us in their communications, and invites us to speak at department chair sessions. The endorsement of the association gives another reason for instructors to participate. The result of this partnership for the Classroom Program is that 159 courses in women’s studies have taught with Wikipedia since the start of our partnership.

Partnerships take time to flourish, but the success of the NWSA partnership demonstrates the power of professional associations in helping us close content gaps on Wikipedia.

Demonstrated impact to student learning

Having data on student learning outcomes is key to bringing more faculty into the program.

While most of our instructors are motivated to teach with Wikipedia because they want to see the encyclopedia’s coverage of their field of study get better, they also need to be able to justify the decision to deans, tenure committees, and other university administrators who are more concerned with student learning than with improving Wikipedia. That’s why it’s so important to have peer-reviewed literature that measures the student learning outcomes from Wikipedia assignments. The Wiki Education Foundation is uniquely positioned to lead this research for the Wikipedia Education Program globally simply because we have a large enough sample size of students to get meaningful results.

Student Learning Outcomes using Wikipedia-based Assignments Fall 2016 Research Report

In fall 2016, we conducted our first student learning outcomes research project, with the results published in a report on Commons (and submitted to a peer-reviewed journal). The study results indicated:

96% of instructors thought the Wikipedia assignment was more or much more valuable for teaching students digital literacy than traditional assignments
85% of instructors thought the Wikipedia assignment was more or much more valuable for teaching students the reliability of online sources
79% of instructors thought the Wikipedia assignment was more or much more valuable for teaching students to write clearly for the general public
Wikipedia assignments shift students’ perceptions of Wikipedia’s reliability to show more trust in Wikipedia
Students are more motivated to complete Wikipedia assignments, particularly because they perceived work to be useful beyond the classroom
Students’ skill development from Wikipedia assignments maps well to the Association of College and Research Libraries’ Information Literacy Framework, particularly:
- Authority is constructed and contextual
- Information creation as a process
- Information has value
- Scholarship as conversation

The results of this study have given instructors the data they need to prove the value of teaching with Wikipedia from a student learning outcomes perspective to recalcitrant university administrators. We’ve heard from several Wikimedia program leaders that the research results are valuable for them to convince decision-makers in their countries of the educational value of teaching with Wikipedia.

Defining quality content

The three core FDC metrics all measure outputs: participants, new participants, and how many articles they edit. Our organization is focused on impact as a measure of success, which is why we chose quality and quantity as our two grantee-defined metrics. Quantity is quite easy to measure for the Classroom Program: Our Dashboard software automatically calculates the number of words students add to the article namespace. Quality is a lot harder to measure.

In 2010–11, the pilot year of our program, we conducted an extensive research study. We brought in subject matter experts and Wikipedians, and asked them to assess student articles on a 26-point scale, based on the English Wikipedia's 1.0 Assessment scale. The results showed that students make substantive improvements to articles through their work. We repeated this study again in spring 2012; again, the results showed significant quality improvement. But both of these studies took an immense amount of volunteer time to assess articles. It was not scalable to keep asking Wikipedians to assess before and after articles each term.

More recently, we've started using ORES as a way to measure "structural completeness" of articles. ORES generates a "wp10" score for each version of an article, based on how well an article matches the typical structural features of a mature Wikipedia article. The machine learning model calculates scores based on the amount of prose, the number of wikilinks to other articles, the numbers of references, images, headers and templates, and a few other basic features, on a scale from 1 to 100. While ORES does not take the quality of sources and writing into account, it does give the best machine estimate of the quality level of an article currently available in the Wikimedia movement to our knowledge. Getting the first 20 or so points in an article is pretty easy: add a couple section headers, put a few sentences and references, add an infobox. And then the real work begins. At the Wiki Education Foundation, we use the ORES scores as an imperfect metric for the quality of student work: How much did existing articles improve on the ORES scale through student work? We purposefully exclude new articles from this count because getting an article to stub or start class automatically gives it so many points that it distorts our overall numbers, and we want the most accurate reflection of the impact students have.

We define the baseline for student work as articles where student editors in the Classroom Program added at least 1,000 bytes. That's almost 200 words — enough to exclude edits where students are doing copy edits, adding wikilinks, or other minor edits. In spring 2017, our students added at least 1,000 bytes to 3,253 articles (they also added at least 1,000 bytes to 806 new articles, but we have excluded that from the chart since there's no "before"). The chart below shows the ORES score of articles before (teal) and after (lavender) the students worked on them:

As you can see from the chart, students take a lot of stub and start class articles, and make major enough improvements to move these articles into mid-quality range. On average, our students improved these articles by 13 points on the ORES scale.

But the impact our program has is even more dramatic when you focus solely on the articles where students added at least 6,000 bytes of content. That's about 1,150 words — a significant amount of content added to an article by a student or group of students working together. In spring 2017, our student editors added at least 6,000 bytes to 1,156 articles. The chart below again shows the ORES score of articles before (teal) and after (lavender) the students worked on them:

The impact of our program is even more stark here: Articles went up an average of 23 points on the ORES scale. Student editors in the Wiki Education Foundation's Classroom Program are making a major difference in the quality of content on Wikipedia, as measured by ORES.

Another indicator of quality is how many student edits were reverted or deleted; the English Wikipedia community is generally pretty good at catching and reverting or deleting poor quality content. In spring 2017, only 2.6% of our students' mainspace edits were reverted and only 1.9% were deleted. This percentage is quite small, especially considering the high quantity of content our student editors add compared to other first-time editors, and is another indication that the content our student editors add to Wikipedia is of high quality.

But what about retention?

Unlike other education programs, our focus is not on retaining student editors. Our system teaches them what they need to know about Wikipedia to contribute successfully to Wikipedia as a class assignment. On student surveys, we’ve asked if students plan to keep editing; the majority say they do, but very few actually stick around. Some student editors find a new home in the Wikipedia community, and they stick around, but they are the exception rather than the rule. Nevertheless, a recent study conducted by researchers at Carnegie Mellon University and the University of Pittsburgh has proven that students who joined Wikipedia through the Wiki Education Foundation’s programs edit more, improve the articles more, and stayed longer than did newcomers who joined Wikipedia individually.^[1] While we are encouraged by this study’s results, we believe our time is more valuably spent retaining instructors.

When we retain an instructor in our program, they will continue to teach with Wikipedia year after year, bringing a new crop of students to improve content on Wikipedia. As an example, Timothy Heningsen, an English professor at the College of DuPage, taught with Wikipedia for the first time in spring 2015. He had such a great experience that he’s integrated it into most of his classes since; he’s taught 17 courses with us, brought 366 students to Wikipedia who have added 279,000 words to more than 400 articles. The impact of retaining instructors amplifies over time, so we focus on ensuring we are retaining a large percentage of instructors whose students added good content to Wikipedia.

Year over year retention of instructors in the Wiki Education Foundation's Classroom Program.

Measuring quality of a course is something we’ve focused a lot on recently. We have a "still-being-perfected" formula that calculates a course’s ranking at the end of the term, taking into account the amount of content added, the number of students, the staff time we spent on the course, and our Wikipedia Content Expert’s ranking of the quality of content students added to Wikipedia. Then we focus our retention efforts on the top classes; for low-ranked courses, we try to determine if there’s something we can fix to make the course run more smoothly in a future term, and if not, we discourage that instructor from teaching with Wikipedia again. Retention is further complicated by academia’s scheduling, where professors might not teach a Wikipedia-appropriate course every term, go on sabbatical, have a term off from teaching while they serve on committees, or leave academia. That means we will never achieve 100% retention, but our focus on retaining our best courses’ instructors helps us ensure our programs are adding quality content to Wikipedia year over year.

Last year, we dug into the numbers, looking at the individual 177 instructors (not courses) who taught with us during the 2014–15 school year. A total of 103 of those individual instructors taught with us at some point during the 2015–16 academic year, representing a raw 59% retention rate of instructors. For the other 74 instructors, we looked into email exchanges we had with them and dug into online course catalogs on university websites to determine if they were teaching the course they'd taught with us again. From that, we determined 18 (or about 10%) were either on sabbatical or had left academia, meaning there was no way we could have retained them. Another 23 (or 13%) did teach that academic year, but did not teach the same course from when they participated in our program; the course(s) they did teach may or may not have been appropriate for a Wikipedia assignment (large survey classes led by TAs, for example, are not appropriate for Wikipedia assignments). We couldn't determine the status of 25 (or 14%) of the instructors; they may have left academia, but we weren't able to find out, and they did not respond to our emails. Finally, we found that six (or 4%) of the instructors had taught the same course again, but this time without a Wikipedia assignment. That means we failed to retain only a small percentage that we know of. Overall, we believe the numbers indicate that we are doing well in our efforts to retain instructors in our program.

References

↑ Robert Kraut et al., Production or commitment? Dilemmas in bringing newcomers on-board (currently under review)

Supporting the global education community

The Wiki Education Foundation's Director of Programs LiAnna Davis presents to heads of representatives from the Research, Information, and Dissemination Centers that are part of the São Paulo Research Foundation (FAPESP), in an effort to encourage more science classes to join the Grupo de Usuários Wikimedia no Brasil's education program.

The Wiki Education Foundation takes its role as the largest education program in the Wikimedia movement seriously. Education is a big part of a lot of Wikimedia affiliates' activities, but since it is the entire focus of our organization, we've been able to scale immensely. We've learned a lot along the way, and we believe it is important to share our learnings regularly with other Wikimedia movement entities. We publish regular blog posts, extensive monthly reports documenting our activities and learnings, and full evaluation reports of our pilot programs. These evaluation reports are particularly useful for global community groups who are looking to repeat our work; for example, the Grupo de Usuários Wikimedia no Brasil is interested in doing a Year of Science initiative in Brazil, modeled after our initiative last year. We periodically contribute to the Education newsletter, share our learnings to other program leaders in the Education Collaborative (which started transitioning to a user group), and have facilitated workshops at the last two Learning Days during Wikimania 2017 in Montréal. We stay in contact with other education program leaders globally, offering mentorship when we can, and learning from what they do.

Another key area where we provide leadership to the global community is in development of resources. We've ported the content of our online training slides to the Program & Events Dashboard, and with support from some community members, are working to make them translatable. We've provided our source files for our printed brochures to groups looking to translate and localize our materials for their own educational contexts.

We strongly believe in documentation and not re-creating the wheel, and we spend a fair amount of staff time and energy ensuring we are transparently sharing our successes, failures, and learnings with others in the movement.

Logic model

Activities in 2018

A student uses the Wiki Education's "Editing Wikipedia" brochure as she completes her course assignment.

In 2018, we will continue to scale the Classroom Program, growing the number of classes participating in the program and thus increasing the quality and quantity of content added to the English Wikipedia in academic subject areas.

This year, we are kicking off an initiative called Future of Facts, in which we endeavor to help close content gaps on politically relevant topic areas. In the age of “fake news”, purposeful misinformation, and lack of information literacy skills, Wikipedia stands as a beacon of fact-based truth. We want to ensure that when citizens turn to Wikipedia to inform themselves about politically relevant topic areas, they will find neutral, well-cited facts. As part of the Future of Facts initiative, we will:

Create partnerships with academic associations in topic areas like public policy, political science, law, history, and environmental sciences
Recruit new instructors to our Classroom Program through existing partnerships with the Midwest Political Science Association, American Sociological Association, and National Women’s Studies Association
Staff booths at exhibit halls for academic conferences for politically relevant subject areas to recruit new faculty
Target outreach activities at historically black colleges and universities (HBCUs). By bringing students explicitly studying topics related to African-American culture and history, we can improve Wikipedia’s coverage of these topics. This ties in to our efforts to have citizens who understand American history, culture, and topics often overlooked in primary education
Support instructors and students who want to assign their students to edit Wikipedia in these subject areas
Create new discipline-specific handouts on how to edit Wikipedia in politically relevant topic areas

The Wiki Education Foundation's Jami Mathewson talks to a potential instructor a the 2017 American Sociological Association conference.

We will also continue to support classes in other subject areas; our growth rate for new classes continues to increase each term due to the visibility we’ve achieved, and we anticipate the program will continue to expand. We will:

Support faculty and student editors who are participating in the program, intervening if there are any problems
Develop additional software features for our Dashboard course management software that enables us to support more students with the same number of staff
Create new training modules and ask.wikiedu.org content to address common student problems and questions
Continue to measure the impact of our classes, and focus on how to retain the best instructors
Publish regular blog posts highlighting the impact of our work

The "Guided Editing" project

One of the biggest pain points about our student editors for existing Wikipedia editors is that sometimes they struggle to get the tone right for an encyclopedia article. Issues with plagiarism, too few citations, and failure to meet the manual of style on Wikipedia frustrate existing editors. We want to create a Guided Editing experience that uses machine learning to review students’ edits as they make them, making suggestions to avoid plagiarism, citation issues, tone problems, and manual of style errors, before the students’ edits are made on the live article namespace on Wikipedia. Creating this Guided Editing system will enable us to address many of the most common challenges student editors face.

In 2018, we will solicit external funders to seek a restricted grant to develop the Guided Editing software. If money comes through for this in 2018, we will begin the early stages of what we expect to be a major technical project that will span more than one year. But because the timing of this is dependent on when this grant funding comes through, we are not including it in our FDC application.

Finally, we started investigating student learning outcomes with Wikipedia-based assignments, as described above. This research acts as an example, model, and framework for future research, but it is far from complete and requires more time and effort to continue. In 2018, we will expand the study. This study will benefit researchers and students by narrowing the focus of previous research and increasing the amount of data on this underrepresented subject. More data on this subject will allow a better understanding of the students' contexts that affect attitudes, productivity, digital literacy, collaboration, and skills transfer. This information is beneficial to Wikipedia and Wikipedia researchers because it helps triangulate attitudes and contexts for understanding how students engage Wikipedia, and it helps create knowledge about increasing student learning while collaborating on and creating Wikipedia articles. We believe this research is critical for our program as well as for programs globally who need this data to convince ministers of education, deans, and other decision-makers of the validity of Wikipedia assignments producing student learning outcomes; endorsements from global program leaders on an earlier unsuccessful project grant speaks to this as well. In 2018, we will

Design survey for both repeated and term-specific research questions
Continue data collection and analysis
Publish data in open formats
Present data and findings at national and international conferences
Publish analysis and presentations in open venues

Quantitative targets, 2018

Measure of success	Type	Baseline (2016–17)	Goal (2018)
New discipline-specific handouts developed	Input	4	4
New academic association partnership agreements signed	Input	1	2
Academic association conferences attended to recruit instructors	Input	20	15^[1]
Number of students	Output (participants)	14,136	16,000
Percentage of students who identify as women	Output (participants)	60%	60%
Number of articles edited	Output (direct product)	15,156	19,200
Amount of content added	Impact (short term)	10.6 million words	11.2 million words
Number of articles edited with at least 10 ORES point improvement	Impact (short term)	4,240	5,000
Instructor satisfaction rate (percentage who indicate interest in teaching with Wikipedia again)	Impact (short term)	98%	98%
New student learning outcome study completed	Impact (long term)	1	1

^[1] We have a better idea now of what conferences are most effective for us, so we believe we can achieve an increase in impact while attending fewer conferences.

Risks

As our program has grown, our individual connection to each faculty member participating in the program has decreased. The less personal connection faculty have to our staff, the higher the risk that they will not return to the program. We actively track instructor retention and spend time engaging on how to retain more faculty, so we do not believe this will be a problem, but we acknowledge it is a risk.
As we move toward higher quality contributions from our Classroom Program student editors, we may see a decrease in the overall number of students in the program. By targeting smaller classes that produce better content, we believe will have a deeper impact on Wikipedia, but this impact may come from fewer student editors. We believe the amount of content added and the quality improvement is more important than the number of students as a metric of our success.

Visiting Scholars Program

Connection to the draft Wikimedia Movement Strategic Direction

The Visiting Scholars program exemplifies the knowledge equity desire to break down barriers preventing people from accessing free knowledge. To build Wikipedia, editors need access to high-quality sources. There's more free content available than ever before, but a great deal of what editors need is locked behind paywalls, in expensive books, or only accessible through a particular institution. Through the Visiting Scholars program, we use our networks and connections to put these resources into the hands of people who cannot otherwise access them. Everything our Visiting Scholars add to Wikipedia is content that would not have been there without access to sources behind paywalls, which provide a literal barrier to free knowledge. By adding the content to Wikipedia, Visiting Scholars promote knowledge equity.

In our Visiting Scholars program, the Wiki Education Foundation facilitates relationships between existing Wikipedia editors and educational institutions to provide access to sources for the existing Wikipedian. Our theory of change is that by providing access to sources otherwise behind paywalls, existing editors will be able to edit more and with higher quality contributions because we provide them with the reliable sources they need.