Learning and Evaluation/Case studies/WWHM/el
Part of the Wikimedia Resource Center
Learning and Evaluation
WikiWomen's History Month Edit-a-thons
This evaluation looks at two events, both titled "WikiWomen's Edit-a-thon." The two events took place in March 2012 and June 2012, in San Francisco, California. Both events were held at the Wikimedia Foundation offices. The idea to host these two events came out of the desire to get more women to edit Wikipedia. A gender gap exists in Wikipedia - more men edit than women, which causes a systemic bias in the content. The goal of both events was to engage more women to edit Wikipedia and to improve content on English Wikipedia about women's history and related subjects.
Τι & Γιατί
This section gives a brief summary about each event, including outputs.
First event briefing
The first event, held in March 2012, came out of the idea of WikiWomen's History Month (WWHM), which was developed by Wikimedia Foundation Community Fellow Sarah Stierch and Wikimedia New York City President Pharos. WWHM takes place every year during Women's History Month in the United States, Australia, and the United Kingdom. This international event calls upon volunteers throughout the Wikimedia community to host edit-a-thons focused around writing and improving articles related to women's history on any language Wikipedia. Stierch led the planning for the Wikimedia Foundation (WMF) event remotely, as she working out of Washington, D.C. at the time.
The team consisted of six volunteers total, who spent approximately 51 hours of volunteer time planning and executing the event. The team was able to secure $250 in support from WMF for food and drinks. Volunteers developed to-do lists for participants of articles needing creation and improvement and also gathered the appropriate reliable secondary sources needed to contribute to Wikipedia. They worked with the WMF Information Technology (IT) department, who donated their time to making sure internet access, power strips and laptops, participants who needed them could borrow, were available. In total, 36 participants were recorded as attending. The event consisted of 33.3% new editors. Participants added over 30,000 characters to the English Wikipedia article space, totaling 247 edits. Those contributions total 20 standard sheets of paper. A total of 11 new articles were written and 22 articles were improved. This first event inspired the planning of a second event.
Second event briefing
The first event inspired the volunteers to plan a second event for June. This time, Sarah Stierch was working in San Francisco, and able to play the lead role at planning the event. The second event, which was primarily planned by Sarah, with additional support from three other volunteers, took less time to plan due to experience with the first event. Volunteers totaled about 31 hours, with Sarah putting approximately 20 hours into planning.
This event replicated many aspects of the first event. WMF donated $250 for food and drinks and WMF IT provided laptops for those who needed them, extra power strips and stable internet access for the event. Volunteers made a to-do list and gathered resources to improve subjects about women's history on English Wikipedia. 31 participants attended the edit-a-thon. This event had a considerably higher percentage of experienced attendees, with an overwhelming 96.7% of the participants having edited Wikipedia before the event. Only one new editor was noted as attending, and they made only one character contribution and have not edited since the event. Despite a high previous editor participation rate, 35.4% of the participants edited. Those editors contributed over 23,000 characters, totaling 250 edits. Those character contributions are the equivalent of 15 copy paper pages. A total of five new articles were written and 24 were improved related to women's history and other subjects.
Logic model for WikiWomen Edit-a-thons
Διαδικασία
This section provides the step by step process for executing the program.
First event
Pre-event
- Volunteers (6) expressed interest in doing an edit-a-thon after calls to action for WikiWomen's History Month participation was posted to the Wikimedia-SF mailing list.
- Volunteers moved off the mailing list to create their own private email thread, which served as the primary planning space for the event.
- Volunteers started by choosing a date and making sure that the Wikimedia Foundation office space would be available for the requested date. They confirmed with the human resources department and it was placed on the event calendar for WMF so no one else booked the space.
- Volunteers created an event page on English Wikipedia. The page was then linked from the main WikiWomen's History Month calendar.
- One Volunteer sent invitations via Wikipedia talk pages to Northern California Wikipedians, inviting them to the event.
- Two volunteers created an EventBrite allowing interested participants who don't have Wikipedia accounts to sign up to attend.
- Volunteers promoted the event through mailing lists (Bay Area technology lists, women/feminist mailing lists, Wikimedia lists) and through social media like Twitter and Facebook.
- Volunteers in San Francisco secured financial support ($250) from WMF to fund food and non-alcoholic beverages for the event.
- Volunteers worked with WMF IT to make sure that loaner laptops would be made available for those who needed them. They let IT know when the event was taking place and that they also needed extra power strips and wifi information.
- A group of volunteers created a to-do list for the event, with a list of women's history related subjects needing articles created or improved. This list was the start of the ongoing list maintained by international volunteers involved in WWHM.
- A few days before the event, volunteers let WMF human resources (who donated the $250) know the head count for attendees so that human resources could order catering. They ordered a bit of extra food over the headcount provided just in case extra people signed up after the catering order was made.
- Volunteers created signs giving directions to where the event was for placement in hallways and the elevator to guide attendees.
- Promotional merchandise (swag) was acquired to giveaway.
Day of event
- Volunteers arrived one hour before the start time to prepare the space. (19:00)
- To-do lists were printed out for participant use, with specific tasks being needed (i.e. "these articles need to copyediting")
- Guidance signs were printed out and posted for attendees.
- Caterers arrived and catering was set up.
- Tables were set up with specific tasked themes, so experienced editors could help new editors - i.e. "how to make an account," and "how to copyedit". Signs were placed on tables.
- Swag was placed at the sign in table and distributed on tables throughout the event area.
- Event starts! (20:00-00:00)
During event
- Experienced editors helped new editors at specific tables guiding them in various tutorials.
- Volunteers maintained that the space was clean and tidy throughout the event.
- Volunteers and participants made sure that the event page was updated with who attended and what articles were worked on.
- Volunteers took photographs of the event to document it.
- Volunteers rotated at the door downstairs to let attendees in. WMF is located on the 6th floor of an office building, and the main door is locked on the weekends for security.
Post event
- Photographs of the event were uploaded to Commons.
- Outcomes section on the event page was updated.
- Lead volunteers shared feedback about the event. This was developed into a lessons learned and successes page.
- Volunteers followed up with participants with welcomes and reminders of where they can find help on Wikipedia if needed. (Example)
- Volunteers decided to do a second event.
Second event
A few changes were made to the process for the second event. Here we will list the changes. Sarah had relocated to San Francisco and was able to take on the majority of planning herself. Also, the Wikimedia Foundation remained as the sponsor and location of the event.
The only changes that took place from the original planning process were:
Pre-event
- Decision to do a second event and the date selection took place on a private email thread, not a public mailing list.
- Three volunteers created a to-do list, since this event wasn't associated with WikiWomen's History Month.
- The same signs used for the first event were used again, after being saved to a shared drive for future use.
- View the Eventbrite invitation here and the on wiki event page here.
Day of event
No changes.
During event
- No photographs were taken to document the event.
Post event
- Outcomes section was updated.
- A third event was not immediately planned after the second event was completed.
- Sarah added templates to the pages created and improved at the event, reminding experienced editors that a new editor might have created it, and to be friendly and helpful if making changes or suggestions. Example here.
Results
This section features results from the event - information about the participants and outputs.
Συμμετοχή
This table includes data about the types of participants who attended both events.
- Experienced editors, in this evaluation, is defined as an editor having more 100+ edits before the event.
- Participants with accounts before the event are editors who had Wikipedia accounts, regardless of it they have edited in the article space or not.
- New editors are defined as those who did not have accounts until the day of the event.
| Δεδομένα | Edit-a-thon 1 | % of total | Edit-a-thon 2 | % of total | Combined total |
|---|---|---|---|---|---|
| Total participants | 36 | — | 31 | — | 67 |
| Experienced editors | 13 | 65,0 % | 20 | 64,5 % | 33 |
| Participants with accounts before event | 25 | 69,4 % | 30 | 96,7 % | 55 |
| New editors | 12 | 33,3 % | 1 | 3,2 % | 13 |
| Participants who edited at event | 26 | 72,2 % | 11 | 35,4 % | 37 |
- Key results
Event #2 had an overwhelming majority of participants who had edited Wikipedia before. In fact, 42% of the participants who attended event #2 had attended event #1. Event #1 was more successful in attracting new editors, who totaled 33.3% of the participants. Event #1 also had a higher participant editing rate, meaning more people actively edited during the event, despite a 13.8% difference in total attendance between the events.
- Questions
- What about event #2 attracted about the same amount of attendees, but less new editors?
- Event #2 was successful at retaining attendees from the previous event. What motivated them to be repeat attendees?
- Why did such a less number of people edit at event #2? Do editors who have edited before edit less at events than new editors?
Direct products
Characters and edit count
This table includes outputs that came from the event.
| Data | Edit-a-thon 1 | % of total | Edit-a-thon 2 | % of total | Combined total |
|---|---|---|---|---|---|
| Total characters added by all participants | 30.594 | — | 23.251 | — | 53.845 |
| Total characters added by new editors | 5.977 | 19,5 % | 1 | 0,0 % | 5.978 |
| Total characters added by participants with accounts before event | 24.740 | 80,8 % | 23.250 | 99,9 % | 47.991 |
| Average characters added by all participants | 849 | — | 750 | — | 1.599 |
| Total edits made by all participants | 247 | — | 250 | — | 505 |
- Key results
At both events, participants with accounts before the event edited significantly more, which is understandable due to the skill set of editing that they bring to the event. With less editors editing during the second event, they still contributed a total of 15 pages of paper to the Wikipedia article space, just 5 pieces of paper short of event #1. The one new editor who attended the second event was significantly unproductive compared to others at the event. They only contributed one edit, totaling one character.
In total, the combined events totaled 40 pages of standard copy paper.
- Questions
- Until we know what the average edit-a-thon of this size (about 32 participants) produces, how do we know if this is below, above or average in the scope of edit-a-thons?
- Why did the new editor at event #2 add so few characters and make so few edits?
Subject improvement
| Data | Edit-a-thon 1 | % of total | Edit-a-thon 2 | % of total | Combined total |
|---|---|---|---|---|---|
| Total articles created | 11 | — | 5 | — | 16 |
| Total new women related articles | 9 | 81,8% | 3 | 60 % | 12 |
| Total articles improved | 22 | — | 24 | — | 46 |
| Total women articles improved | 15 | 68,1 % | 9 | 37,5 % | 24 |
- Key results
In total, both events created 16 new articles and improved a total of 46 existing articles. The intended outcome of both events was to not only retain new editors, but, to also improve content related to women's history and related subject areas. Both events succeeded at the latter, despite organizers not having a specific number goal of articles written and improved. The majority of articles created at both events were about women subjects. The second event had two more articles improved, totaling 24, compared to event #1 which had 22 articles improved. Despite an increase of production at event #2, which had 11 participants (out of 31) who edited at the event, versus event #1 where 26 out of 36 edited during the event, less articles were created, and less articles specifically about women subjects were improved.
- Quality of new articles
| Event #1 | Event #2 | Total | |
|---|---|---|---|
| Good article | 0 | 1 | 1 |
| Did You Know | 1 | 0 | 1 |
- Questions
- Why did event #1 have more articles created with a stronger emphasis on women article outputs than event #2?
- Did event #2 have more articles improved because it had a higher experienced editor participation count?
- Do events with specific themes and to-do lists make a bigger impact than events without?
- What is the average creation and improvement rate for edit-a-thons on English Wikipedia?
Retention
This evaluation used Wiki Metrics to see if participants at both events edited one month and six months after each event, with the intention of understanding the retention of editors.
One month
- A retained editor is considered an editor that makes 5+ edits a month.
| Data | Edit-a-thon 1 | % of total | Edit-a-thon 2 | % of total | Combined total |
|---|---|---|---|---|---|
| Total attendees | 36 | — | 31 | — | — |
| Total retained editors | 17 | 47,2 % | 17 | 54,9 % | 34 |
| Total editors that were retained who were new at event | 3 | 17,6 % | 0 | 0,0 % | 3 |
| Total characters added by all participants | 277.020 | 184,7 sheets of paper | 190.431 | 127 sheets of paper | 467.451 |
| Average characters added by all participants | 7.695 | — | 6.143 | — | — |
| Total characters added by participants that were new at event | 1.371 | 0,5 % | 333 | — | 1.704 |
| Average characters added by editors that were new at event | 114 | — | 333 | — | — |
| Total characters added by participants who had accounts before event | 275.649 | 99,5 % | 190.098 | 99,8 % | — |
| Average characters added by participants who had accounts before event | 11.025 | — | 6.336 | — | — |
| Total edits made by all participants | 1.732 | — | 1.338 | — | 3.070 |
| Total edits made by participants that were new at event | 62 | 3,6 % | 1 | 0,07 % | 63 |
| Total edits made by participants who had accounts before event | 1.670 | 96,4 % | 1.337 | 99,9 % | 3.007 |
- Key results
Both events retained the same number of users, with possible overlap due to 42% of the attendees from event #1 attending event #2. However, event #1 had three new editors successfully retained after one month, when the one new editor who attended event #2 made 1 edit, totaling 333 characters, but did not make further edits to be considered retained. For both events, editors who had accounts before the events contributed over 99% of the characters added after one month. They did contribute more characters and make more edits after the first event.
In total, combined character additions from both events equals 314.7 pages of standard copy paper. That's enough pages to fill one copy of the first Harry Potter book, Harry Potter and the Philosopher's Stone, US edition!
- Questions
- Why were those three editors from event #1 retained?
- Why did new editors from both events not edit, or not edit more to be considered retained, after one month?
- What motivated that one new editor from event #2 to make one 333 character edit, and not edit anymore?
- Why did participants who edited one month later from event #1 contribute more bytes and edits than the 17 retained editors from event #2?
Six months
| Data | Edit-a-thon 1 | % of total | Edit-a-thon 2 | % of total | Combined total |
|---|---|---|---|---|---|
| Total attendees | 36 | — | 31 | — | — |
| Total retained editors | 16 | 44,4 % | 18 | 58,1 % | 34 |
| Total editors that were retained who were new at event | 1 | 2,7 % | 0 | 0,0 % | 1 |
| Total characters added by all participants | 1.594.906 | 1.063,3 sheets of paper | 2.214.464 | 1.476,3 sheets of paper | 3.809.370 |
| Average characters added by all participants | 44.302 | — | 71.434 | — | — |
| Total characters added by participants that were new at event | 1.505 | 0,9 % | 0 | 0,0 % | 1.505 |
| Average characters added by editors that were new at event | 125 | — | 0 | — | 125 |
| Total characters added by participants who had accounts before event | 1.593.401 | 99,9 % | 2.214.464 | 100,0 % | 3.807.865 |
| Average characters added by participants who had accounts before event | 63.736 | — | 73.815 | — | — |
| Total edits made by all participants | 11.152 | — | 13.170 | — | 24.322 |
| Total edits made by participants that were new at event | 67 | 0,6 % | 0 | 0,0 % | 67 |
| Total edits made by participants who had accounts before event | 11.085 | 99,3 % | 13.170 | 100 % | 24.255 |
- Key results
Little changed outside of the one month retention rate of 17. The events retained 16 and 18 editors, respectively. The count of new editors retained declined, from three to one from event #1, and stayed at zero from event #2. However, that one retained editor made more edits than the three previously retained editors did in one month. Event #1 participants averaged about the same contribution wise, over six months, than they did one month later. Editors from event #2 were more productive after six months time. They added 38.8% more characters to the English Wikipedia article space, and their monthly average was 11,905, higher than the one month later average of 6,336. They made up for their lower productivity seen one month after.
In total, combined character additions from both events, six months later, equals 2.539,6 pages of standard copy paper. These contributions could fit into two copies of War and Peace with a little text left over. That is an average of about 423 sheets of paper (bigger than that Harry Potter book!) written a month.
- Questions
- Why did that one editor retain after six months?
- Why did those two editors fail to be retained?
- Why did the event #2 retained editors "make up" for their lower contribution numbers that appeared one month later.
Limitations
- Lack of data on the average of characters added and edits made at edit-a-thons around the world, of this average size (30-35 participants). This would allow us to learn more about the impact of individual events and gain a broader sense of if these events meet their goals (i.e. retaining new editors, exciting experienced editors, people have fun, content gets improved)
- On occasion, editors create new accounts (perhaps due to losing password information, socks, (legal and not legal), bots, etc.) and we're unable to track those changes. So, we are unaware if perhaps a new editor who edited at the event, made a new account after the event and was retained.
- It was very hard to do an evaluation almost a year after the event took place. It's best to do it in the "present" of the event time.
Next steps and reflection
What worked well
- Having experienced editors on hand to help train and work with new editors as needed.
- Reaching out to non-Wikimedia channels, such as universities and organizations, can help bring new and curious participants.
- Pre-preparing materials such as print outs and templates (even the event page template) is helpful, which made the second event take less time to prepare.
- Having refreshments and giveaways always makes an event more enjoyable!
- Taking photographs at the event makes for helpful images to promote the event in the future.
- Having a to-do list prepared is helps participants make easy decisions on what to edit.
- Allowing people to bring children, especially at women-centric events where it might be hard for someone to find child care.
- Having regular events - we held the events with a few months of each other around the same time of the month which makes consistency in the event and can attract same attendees.
What didn't work well
- Properly track who does attend the event. Both events failed to track all attendees. Which can lead to data being incorrect.
- Parents brought children, but, no child care was provided and children eventually got bored.
- Not evaluating throughout the entire process. Thus an evaluation over a year later.
- We only took photographs at one event. Always take photos.
What data we would seek next time
- Complete listing of who attended and made accounts.
- Survey participants before (perhaps brief survey upon arrival) and a post-event survey to learn about:
- General attitude and feeling about event
- Learn about participants skill, experience and knowledge with Wikipedia
- Learn about what participants want to experience at the event
- Learn if participants believe they will edit after the event
- Gender/background information
- Reach out personally to participants, especially new editors, not just on talk pages (email, phone) to see why or why not they have edited.
- Return on investment.
Conclusion
With intended goals to improve women related articles and to teach and retain new women editors, these two events succeeded easily at one, and struggled at the other. In total, both events created 12 new articles and improved a total of 24 existing articles. Article subjects that were created include American journalist and activist Eva Valesh and feminism in Russia, the latter which became a good article. Improved articles include the core article women's history and the biography of Russian writer Irina Ratushinskaya. Additional articles were also written and improved. Total article creations and improvements for both events were 16 and 24, respectively.
The intended outcome of retaining new editors who attended the events was a clear struggle. Event #1 had a total of 36 attendees with 33.3% being new editors, event #2 struggled to attract new editors, bringing only one who made an account at the event. The event #2 failed to retain that editor. However, event #1, after one month, succeeded at retaining three out of their 12 new editor participants. Averaging 114 characters added each, those three declined to one retained editor, after six months time. However, that one editor averaged 125 character contributions each month, that is a higher average than those three editors had after one month!
There are many questions to still be answered, including what do we celebrate? Do we celebrate the retention of that one new editor? Why did they stay, and what can we learn from their retention. Is it a certain type of person who edits Wikipedia - whatever that is - and they fit the mold? Or did they feel inspired and get more involved in the community after the event? Did informing new editors about the Teahouse and other similar help systems help to retain that one user? All we can do is ask. And of course, let us celebrate the contributions made by editors of women's subjects who attended the events, and celebrate the experienced and new editors (even if it is only one!) who continue to build and improve the world's largest free online encyclopedia.
Edit-a-thons often serve a few specific purposes such as allowing people interested in editing Wikipedia to socialize in a relaxed setting, educating potential new editors, encouraging editors to contribute content in a short period of time about, generally, specific subject matters. All good faith goals. A lot of work still needs to be done in order to see if edit-a-thons, as a whole, are meeting those goals and making an impact. And perhaps, we will learn if these goals aren't being met, but other successes are - participants have fun and make friends, curious potential new editors get to see Wikipedians in action and learn more about Wikipedia, or organizers feel good and special afterwards. All of these successes, are no less important than the goal of new editors and improved content. It just means we might have to shift our ideas about and missions around investing in, planning, and executing edit-a-thons.