Learning and Evaluation/Evaluation reports/2015/Writing Contests/Outputs

Peer Review in Process now through July 3
The Learning & Evaluation from the WMF will be actively reading and responding to comments and questions until July 3.
Please submit comments, questions, or suggestions on the talk page!

Writing Contests

Summary

Limitations

Inputs

Outputs

Outcomes

Key findings

Appendix

On this page, you will find answers to:
How many participants are involved in contests?
How much content is typically produced?
What are some relationships between budget and participants or hours and content?

Participation	Event Length The event length of contests varied, with most contests lasting one month or less. Still, 46% of contests lasted more than one month and 21% lasted more than 3 months. Start and end dates were reported for 32 of the 39 implementations (82%). A broad range of contest lengths is included in these data: the shortest contest lasted one week and the longest lasted 43 weeks. As illustrated in the graph below, most events (78%) lasted less than 12 weeks. All contest lengths were rounded to the nearest whole week. Event length. Most contests lasted between one and four weeks. All contest lengths were rounded to the nearest whole week. Participation The number of participants ranged from 1 to 146 participants, with the average number of participants being 18 participants. In total, 1,027 participants were reported across 39 contests. Participant counts were obtained for 39 (100%) of the events. The number of participants in each contest or event ranged from 1 to 146, with an average of 18 participants Of the total participant base (1,027 users), 114 (11%) were newly registered. Participants counts in this figure are not unique; if a participant attended more than one contest, they are counted twice. “Some judges are people who used to participate in the contest...people change, they graduate from college and do other things. Maybe they don’t have enough time to edit substantially enough for six months to be competitive, but they have time to judge for a couple of weeks.” User:Shipmaster Producer Prize, Arabic Wikipedia Cost per Participant From the 10 programs reporting, the cost per participant was wide ranging with an average of $37 USD per participant. Measuring cost-per is a way to help inform setting reasonable goals and targets and to compare performance. It is especially useful if one has no experience with running a writing contest and you can see what others have done. It is important to use extreme caution in applying across international contexts. In the figure below, you will find a box plot that represents the distribution of cost per participant. The cost per participant ranged from $0.66 USD to $140.74 USD, with an average of $36.82 USD. "[C]ommunity on IRC suggested that they have a contest to correct typos to improve the quality and credibility of Wikipedia. During the competition, people were encouraging each other through micro-contests on IRC, for example, seeing who could complete the most edits in 10 minutes. In the end, a lot of people wanted to donate their prizes, which were small anyway, because they don't like the idea of paid editing." User:Newyorkadam Tyop Contest, English Wikipedia Cost per participant. The average cost per participant was $37 USD. Only 10 budgets are represented in this graph. Use caution when applying these values to different contexts.
Content Production	Articles Created and Improved A total of 15,156 articles were created or improved as part of the contests, and 9,404 (62.8%) of these were articles created. The average contest resulted in 134 articles created or improved. Contest participants create and improve many articles. From the 35 contests reporting this metric, the range was 12 to 3,741 articles created or improved, where the average contest produced 134 new articles. "The community on Hebrew WP doesn't have a lot of physicists, so they can’t contribute a lot to the content. They try to help with the styling of the pages and things like that, but it is not as involved as the regular competition. If you are someone who likes to write about physics, the competition is like a celebration, a festival, because all of the sudden there are a lot of new pages about the topic." Physiwiki Contest Hebrew Wikipedia Pages of Text From the 30 out of 39 contests reporting (80%), 13,330 pages of text were added and 1,814 pages were removed across more than 15,000 articles created or improved as part of the contests. The average contest resulted in 238 pages of new text. Text pages are a way to frame how much content is added to articles. Bytes added to or removed from an article are converted into language characters and then into text pages. Generally, 1 or 2 bytes equals 1 character depending on the language, and one text page is defined as 1,500 characters. From the 30 contests reporting bytes added, the average contest resulted in about 238 total text pages. By counting pages of text added and removed in the contests, we see that a total of 13,300 pages of text were added (88% of pages), while 1,814 pages of text were removed (12% of pages). The average number of pages removed was 21. These numbers are specific to articles submitted for the contest only, and do not include other articles created or improved for the project by the participants included in the cohort during the contest time period. Text pages added or removed. Text pages are one way to understand how much content is added or removed as a result of writing contests, with one text page defined as 1,500 characters. Writing contests result in a large amount of information to articles: from the 30 contests reporting this data, 13,330 pages worth of text were added and 1,813 pages removed in order to improve articles. Budget & Content Created Cost per article created or improved The cost per article created or improved was less than $8.30 for six (75%) of the eight contests reporting data, while the average cost per article created or improved was $0.59 USD. With access to so few budget, it is very important to take the context of the into consideration when interpreting the numbers and should not be generalized. Eight contests had data on budget and articles created or improved. The cost per article created or improved has a wide range $0.04 USD to $85.11, but a narrow distribution, where 75% of the reported cost per article created or improved is below $8.30 USD. For the average contest, the cost per article created or improved is $0.59 USD. Cost per page of text The cost per text page is less than $6.75 for all eight implementations where the average is $1.14 USD per text page. There were eight contests with both budget and text pages. The cost per text page has a small range and narrow distribution, where the cost per text page for any program is less than $6.80 USD and the cost per text page for 75% of the contests is less than $2.05 USD. For the average writing contest, the cost per text page is $1.14 USD. Cost per content. This graph shows the cost per content created in terms of both articles created/improved and pages of text. While this data can be helpful in planning a context, use caution when applying to other contexts since there are only eight contests represented. Weeks, Participation, Text pages The length of a contest was positively correlated with both participation and amount of content affected. The longer the contest, the higher the participation count and amount of text added. The bubble graph below illustrates the length of the event in association with participation and absolute text pages added or removed. The correlation analysis shows a significant positive relationship for 30 contests. As the length of writing contests increases, contests tend to include more participants. Likewise, as the length of contests increases, the numbers of absolute text pages also tend to increase. In examining the correlation coefficient, the length of each contest in number of weeks had a positive correlation with total participants. and text pages. Weeks, participants and text pages. This graph compares the length of contests with numbers of participants and absolute text pages. This graph shows that as the length of contests in numbers of weeks increases, the numbers of participants and the numbers of text pages also increase. Budget, participation, and text pages (per week) A positive relationship was seen between budget and pages of text produced, but not budget and number of participants. Only 10 contests reported both budget and participation and 8 reported both budget and text pages. We conducted correlation analysis with extra caution since having few data points makes it difficult to draw conclusions. We divided each measure (budget, participation, text pages) by the length of each contest in number of weeks in order to have a standard weekly unit of measurement for comparison. The graph below compares budget, number of participants, and absolute text pages, dividing each by the length of the contest in number of weeks. Budget per week does not show a significant relationship with number of participants per week. However, budget per week tends to increase as absolute text pages created or improved per week increases. Hours, participation, and content No relationships were found among hours, participation and content. This likely due to only five contests reporting data about these measures. In terms of relationships between hours invested and numbers of participants or amount of content, there were only five contests for which both hours and numbers of participants were reported, and five for which both hours and amounts of content were reported. The correlation analysis demonstrated no relationship between hours and absolute text pages with this very small set of comparison points. or hours and participation. This means that there may or may not be a relationship and we need more and better data to find out. Participation, articles created or improved and text pages (per week) From the 30 contests reporting, we found a positive relationship between numbers of participants per week and articles created or improved per week, as well as a positive relationship between numbers of participants per week and amounts of text pages per week. Knowing that there is a positive correlation between the length of the event in weeks with numbers of participants and with the amounts of text added or removed, we look here at how participation is related to how much content was added or removed while considering the lengths of the contests. The numbers of participants per week tend to increase when the numbers of articles created or improved per week increase. Similarly, we find that numbers of participants per week increase as text pages per week increase. Participation, articles created or improved, and text pages (per week). This graph compares the 30 programs that report numbers of participants, pages created or improved and text pages, per week. Even when taking weeks into account, we find that as participation increases, more content is produced.