Research:Wikimedia Summer of Research 2011/Newbie teaching strategy trends
There is a steep learning curve to contributing productively to Wikipedia. New editors, even those who edit in good faith, often make mistakes when they start out. We were interested in finding out how more established community members responded to these initial edits by new editors:
- do they take the time to tell them personally what they had done wrong, and why?
- Is the criticism they offer specific enough to be constructive, instructive and actionable?
- Do they give them any kudos for the things they'd done right?
- And, most importantly, how have these teaching practices changed over time?
We were interested in various approaches to teaching new editors how to contribute most productively to Wikipedia. We coded messages on new editors’ talk pages for moments of teaching/instruction, praise/thanks, criticism, and warning. We noted when these were templated or personalized interactions, and when editors referred to specific edits, types of edits, or editing Wikipedia generally.
By attending to positive and negative, impersonal and personal, and specific and general dimensions of these talk page messages, we expect that:
- there will be a general rise in the overall use of templates over personalized messages between 2004 and 2011
- there will be an increase in the percent of messages that contain criticisms and warnings
- there will be a decrease in personalized messages that attempt to ‘teach’ new members rules, norms, or effective editing strategies
Question
editHow have community strategies for teaching new editors to be effective contributors shifted since 2004?
Methods
editWe coded a sub-set of the data used in a previous research sprint that analysed the rise of warning templates on new users' talk pages. We adapted the coding scheme used in the previous sprint to focus specifically on identifying messages related to teaching.
Coding categories
edit1. Message Purpose
- Teaching: Explanation of how or why to do something: “Please include this stub template when creating a new stub article, it helps to draw other editors attention to the article and encourages them to contribute.”
- Criticism: “You did something wrong!”
- Praise/Thanks: “I really liked your last edit!”
- Warning: Something bad will happen: you will be sanctioned/banned, your article/image will be deleted
2. Message Specificity
- Specific edits: directly or indirectly refers to individual actions; such as “don’t keep adding that citation”
- Edits to pages: “stop adding unsourced images; thanks for good spellchecking; you’ve been editing on Canada articles, so join WikiProject Canada”
- Edits to Wikipedia: “Thanks for your contribution to Strawberries, here are 50 rules; your edits to England are unconstructive and have been reverted”
3. Message type
- Templated: Message is template, with little or no personalization.
- Personalized: Message is either a) completely freeform or b) a template that has had some personal message included (besides just a username or article title)
Coder Agreement
edit| code | % agreement between coders |
|---|---|
| teaching | 68% |
| criticism | 83% |
| praise | 81% |
| warning | 94% |
| template message | 78% |
| personal message | 84% |
Agreement notes:
- each coder coded the first 20 talk page messages for 2005, 2006, 2007, 2008, 2010 and 2011, resulting in a total of 60 multiply-coded posts. We assessed our agreement using ReCal a free online tool for computing inter-coder reliability.
- in the interest of time, we only computed inter-coder agreement on some of our codes
- agreement was lower on some categories than others because it took a while for us to come to a common understanding of the definitions of these codes. We are currently working on documenting more precise, detailed definitions of our codes, and expect that agreement will be higher in subsequent coding projects.
Results and discussion
editTotal messages coded:
Message features
 |
 |
Personalized versus templated messages
Personal versus templated teaching messages
Teaching messages that contain criticism versus praise
Message specificity
Summary
editWe found Wikipedian teaching strategies shifting in two significant ways:
- a significant drop in messages including praise and thanks corresponded with an increase in the overlap of teaching with criticism
- a decline in personalized teaching corresponded with an increase in templated instruction.
Future work
editFor future work, we would like to chart the subsequent editing behaviors of new editors who received more- vs. less-personalized teaching, as well as users who received multiple templated warnings (for example, deletion notification for articles they wrote or media they uploaded) before they received a single note written by a human and addressed specifically to them.
We are also interested in a couple of trends that we noted anecdotally, but failed to capture in our coding scheme due to time constraints:
- the possible decline of posts that contained task suggestions for the new users.
- the possible rise of templates composed almost entirely of generic link-lists (to policy pages, new user guides, and open tasks), rather than 'teaching templates' that offer richer and more detailed explanations of what and what not to, what Wikipedia is all about, and where to go for help.