Wikimedia Meta-Wiki

User:MCruz (WMF)/Sandbox/Program Reports/Executive Summary

< User:MCruz (WMF) | Sandbox | Program Reports
Wikimedia Programs Evaluation 2015


Wikimedia Program Reports is a collective effort of program leaders all over the world.
Wikimedia Evaluation initiative started two years ago, with the aim to bring forward successful programs, understand what works best for our shared goals, and how to best build capacity in the movement towards a culture of accountability and data-driven decisions. Learn more about Evaluation... the Wikimedia way!



Data and analysis

edit
We had many questions to ask program leaders!
We have a lot of questions, and the data reported has helped us answer
  • What do these programs costs in terms of dollars and hours invested and what other costs may be hidden in donated resources used?
  • What is the reach of these programs in terms of accessing new and existing editors/contributors?
  • How much content do programs produce in terms of bytes pages or photos/media added?
  • What are the costs in terms of dollars and hours input Tper unit of content (text pages or photos/media added) or per participant/recruit (for workshops which produce no content)?
  • To what extent do program outputs increase the quality of Wikimedia projects?
  • To what extent does program participation produces new active editors/contributors, or retain active editors, at 3- and 6- months retention points?
  • To what extent does the program have examples for easy sharing and replication?

Priority goal setting

edit

We worked with the community to discover the most commonly seen goals for programs. The June pilot workshop in Budapest served as a way for the our team to identify 18 commonly seen outcomes across programs, which were discovered through conversations, break out sessions, and logic modeling. Participants in the Data Collection Survey were asked to select outcomes and targets the programs they reported on had. These 18 priority goals are:

  • Building and engaging community
  • Increasing accuracy and/or quality of contributions (i.e. clean high resolution photographs which are placed in the proper articles)
  • Increasing peoples awareness of Wikimedia projects
  • Increasing peoples buy-in for the free knowledge/open knowledge/culture movements
  • Increasing contributions to the projects
  • Increasing diversity of contributions and content
  • Increasing diversity of contributors
  • Increasing positive perceptions about Wikimedia projects
  • Increasing reader satisfaction
  • Increasing the usefulness, usability, and use of contributions
  • Increasing the use and access to projects
  • Increasing peoples editing/contributing skills
  • Increasing volunteer motivation and commitment
  • Increasing respect for the projects (i.e. higher education acceptance)
  • Making contributing fun
  • Making contributing easier
  • Recruiting new editors/contributors
  • Retaining existing editors/contributor

In the survey, program leaders could also write in other goals in a section titled "other". This set of 18, with the "other" option, were presented for each program that program leaders reported on. They were asked to select priority goals for their reported programs. The number of goals program leaders reported ranged from five to 11. The overall mean for any given program was nine selected priority goals. In general, program leaders demonstrated difficulty in "prioritizing": Out of all reports, only 12.5% selected five or fewer priority goals.

Inputs and participation

edit

Inputs

edit
The majority of program leaders reported some type of budget, but the majority didn't report data about hours it took to implement their programs.

Regarding inputs, program leaders were asked to report:

  • Budget – how much it cost them to produce their program in US dollars
  • Staff and volunteer hours – How many actual or estimated hours staff and volunteers put into their program from beginning to end
  • Donated resources – Including equipment, prizes, give-aways, meeting space, and other similar things donated by organizations or individuals to support the program

Most program leaders reported budget data, while a larger number did not provide data about hours. Across the 119 report responses:

  • 55% included budget data (22% of budgets reported were zero dollars)
  • 34% included staff hours (51% of staff hours reported were zero dollars)
  • 44% included volunteer hours (2% of volunteer hours reported were zero dollars)


Participation

edit
The majority of program leaders were able to report on how many and what kinds of participants were involved in their programs. It takes all kinds!
The majority of program leaders could report how many people participated in their program, when little over half were able to tell us how many new editors made accounts for their programs.

Regarding participation, program leaders were asked to report:

  • Total number of program participants
  • Number of participants that created new user accounts during the program

The majority of participants reported the total number of participants (98%), when little over half (57%) reported number of new user accounts created during their program.

GLAM content donations had a slightly different reporting request about participation:

  • Total number of GLAM volunteers involved in the program (78% reported)
  • Total number of GLAM staff involved in the program (89% reported)

Program leaders were also asked to provide the dates, and times, if applicable, for their program.


Content production and quality improvement

edit

Content production

edit
Most program leaders were able to tell us how much media was added during their program, but a minority were able to report on how many characters (bytes) were added during their events, let alone how many editors actually editing during their programs.
Read more

Regarding content production, program leaders were asked to provide various types of data, about what happened during their program, depending on the level of data they were able to record and track. These data types were:

  • Total number of characters added (33% reported)
  • Average number of characters added (33% reported)
  • Number of participants that added characters (8% reported)
  • Number of photos/media added (80% reported)
  • Number of Wikimedia project pages created or improved (50% reported)

Content production metrics were not requested of those who reported about editing workshops, since content production is not the main goal of that type of program.

11% of program leaders were able to report on how many Featured Images their program produced.

Quality improvement

edit
Most program leaders were able to report how many and much of their images, uploaded during their program, were used in the projects after the program ended. However, most were unable to report about the quality of articles and images, and most that did report on it stated that no featured, good, or valued articles or images came out of their event.
Read more

The survey also asked that program leaders report on the quality of the content that was produced during the program. They could report:

  • Total number of good articles (38% Reported, 51% of reported no good articles)
  • Total number featured articles (34% Reported, 77% of reported no featured articles)
  • Use count of photos added that were being used in Wikimedia project pages (63% reported)
  • Number of unique images used on Wikimedia project pages (63% reported, 9% of reported that none were being used)
  • Number of Quality Images (27% reported)
  • Number of Valued Images (29% reported)
  • Number of Featured Pictures (28% reported)

Those who reported about edit-a-thons and workshops were not asked to report about image use and quality.

Recruitment and retention

edit
Just over half of respondents were able to tell us how many of their participants were active 3 months following their program and less than half were able to do so 6 months after.
Read more

Tools like Wikimetrics can make this possible, which means tracking usernames is important to learning about retention. For edit-a-thons and workshops, the majority of those reported on did not retain new editors six months after the event ended. A retained "active" editor was one who had averaged five or more edits a month.[1]

Regarding the recruitment and retention of active editors, program leaders were asked to report two areas of data. An "active editor" is defined as making 5+ edits a month.[2]

Read more
  • Total number of contributors still active 3 months after the event (55% reported, 15% reported zero retained)
  • Total number of contributors still active 6 months after the event (45% reported, 19% reported zero retained)

If the program reported on was an edit-a-thon or workshop, program participants may have been split into two groups: new editors or existing editors, in order to learn the retention details about each cohort. This is important, since both edit-a-thons and workshops often attract new and experienced editors, unlike on-wiki writing contests that generally target existing contributors, and the Wikipedia Education Program that generally targets new editors.

In terms of recruitment and 6-month retention of new editors (those who made accounts at or for the event):

  • Edit-a-thons (44% reported, 85% of those reported zero retained)
  • Editing workshop programs (56% reported, 78% of those reported zero retained)
  • Wiki Loves Monuments recruitment and retention data was mined using the entire set of uploader usernames for the 2012 events.
We asked different questions about recruitment and retention for Wikipedia Education Program and GLAM content donations.

We asked about the retention of partnerships between educational institutions or cultural organizations, instead of editor retention. Wikipedia Education Program respondents were asked to identify how many instructors were participating in the program, and GLAM content donation respondents were asked if the GLAM they worked with would continue their partnership with Wikimedia and if the content donation would lead to other GLAM partnerships.

We wanted to know if program leaders had or produced materials to hand out, and had the ability to support other program leaders to do the same.

Replication and shared learning

edit

We wanted to learn if program leaders believed their program(s) could be recreated (or replicated) by others. We also wanted to know if program leaders had developed resources such as booklets, handouts, blogs, press coverage, guides, or how-to's regarding their program. We asked if the program:

  • had been run by an experienced program leader who could help others do;
  • had brochures and printed materials developed to tell others about it;
  • had blogs or other online information written to tell others about it (published by yourself or others);
  • had a guide or instructions for how to implement a similar project.

Reporting format and goals

edit

References
  1. ↑ This was measured for each three-month window for the first quarter year and second quarter year following the program to allow for minor month-to-month fluctuations as it was often observed that a user name would surpass the threshold in two of the three months and just miss it on one of the particular months, but not necessarily in a chronological order. For, example, an editor making 8 edits in Month 1, 10 edits in Month 2, 4 edits in Month 3, 8 edits in Month 4, 4 edits in Month 5, and 6 edits in Month 6 would be counted as a retained "active" editor at both 3- and 6-months follow-up, rather than inactive at 3-months and active at 6-months if the measure was based on a strict single month comparison. It will be important to further examine this approach and design to simultaneously report those that may start off highly active and drop off more rapidly than this quarterly perspective.
  2. ↑ mw:Special:MyLanguage/Analytics/Metric definitions#Active editor

Retrieved from "https://meta.wikimedia.org/w/index.php?title=User:MCruz_(WMF)/Sandbox/Program_Reports/Executive_Summary&oldid=11630296"