Wikimedia Meta-Wiki

Wikimedia Fellowships/Project Ideas/Badges for Wikipedia Community Engagement

< Wikimedia Fellowships | Project Ideas

Updated Proposal

edit

Overview

edit

Badges could provide community members of all stages, novice to expert, with goals to work toward and feedback about whether their goals were attained. The main goal of the Wikipedia Badges Fellowship is to see if there is a Wikipedia appropriate badge system design that can help acknowledge the work of editors, be verifiable and help support overall editor retention.

I propose to use WP:Teahouse to serve as a pilot project for a more targeted badge system design. Creating badges on a pilot project will create an opportunity to explore key questions surrounding badges without affecting the rest of the Wikimedia community. These questions include:

  • How to create a badge system that isn’t game-able through linking of evidence.
  • How to create a badge system that maintains recipient anonymity.
  • How badges can interact with existing barnstars.
  • How we can encourage members to use badges as a way to reward and acknowledge good work.
  • Whether badges really help to increase participation and engagement in a project.

The pilot design will include

  1. A curated set of barnstars that are appropriate for the projects
  2. An augmented collection of additional badges (maybe we call these mini-barnstars? or maybe these will be just regular barnstars) that specifically acknowledge the work done through these projects
  3. Mechanisms to make the barnstars/badges more accessible to project participants.

Additionally, I would like to explore ways by which badges can be made more meaningful encouraging the linking of evidence, working toward making badges that can be pushed to an Mozilla OBI Backpack, and surfacing information for when badges are the most effective as encouragement tools.

The deliverable of this project would be a document of best practices and community responses as to how and when badges are most effective as tools for encouraging acknowledgement and editor retention.

WP:Teahouse Badges Project

edit

The WP:Teahouse project, active since February 2012, was created to help new editors find a supportive editing community. The project created a welcoming environment where experienced editors, themselves supported with a community and a set of best practices, could connect with new editors and guide them through editing. Findings showed that creating a more welcoming environment did help support editor retention, particularly among women, as “28% of Teahouse participants are women, up from 9% of editors on Wikipedia in general and 2 weeks after participating 33% of Teahouse guests are still active on Wikipedia, as opposed to 11% from a similar control group.” However, female participants also reported “feeing discouraged that their contributions were not acknowledge by barnstars or in other ways.”

The aim of this pilot is to build a collection of existing barnstars and additional new badges to serve as symbols of acknowledgement on the WP:Teahouse project and to monitor their impact on the WP:Teahouse community. Ideally, badges would provide a mechanism for participants to reward each other’s work, decrease the feeling of discouragement, and help support the Teahouse’s success in increasing editor retention. A successful Teahouse badge pilot would help to establish the process of identifying an appropriate barnstar/badge set and a more straightforward process for rewarding and acknowledging project work.

Creating the Badge Collection

edit

I propose that the badge collection for the Teahouse project consist of two parts that directly reflect the work done through the Teahouse:

  1. Barnstars that are appropriate for the Teahouse such as the “Random Acts of Kindness Barnstar” or the “Teamwork Barnstar.”
  2. Additional badges that acknowledge work done on the Teahouse specifically such as “exceptional work on the QandA forum,” or that acknowledge work done on a small scale such as “first successful article.”
Why badges and barnstars? (What's the difference between the two?)
Barnstars and badges are similar. Badges tend to be understood as symbols of accomplishments, identity, and feedback. Barnstars are acknowledgements and “thank you’s” of exceptional work. They are like badges because they recognize accomplishment and can be used to identify contributors that are particularly active on their project. From my perspective, barnstars are actually a type of badge, just named and used in a Wikimedia context.
Yet, barnstars don’t accomplish all of the things that could potentially be accomplished with a robust badge system. For example, barnstars don’t tend to acknowledge work that is much less than exceptional. Barnstars don’t acknowledge someone’s first edit or first full article (though these can be great accomplishments for new editors). As pinnacles of achievement barnstars don’t show a path to becoming a better contributor and poorly recognize progress. And they are not always consistent. Barnstars may not always be given out regularly to all contributors doing equally exceptional levels of work.
Barnstars could be revamped and redesigned to be better as badges – barnstars could be created for small scale achievements, they could be organized into nested pathways, and they could be more consistently given out. But it also makes sense to preserve barnstars as reward and acknowledgement of exceptional work and to not cheapen or dilute them. To show new editors pathways to success, a separate class of badges could be created to recognize work done on a smaller scale. These badges could be awarded automatically once someone reached a certain level or awarded by people for doing work of a certain level of quality. Additionally, they could be the necessary prerequisites to earning a barnstar. The idea would be to create a badge system that works together with existing barnstars in a way that acknowledges achievement at all levels of engagement.

The badges should directly reflect the work done throughout the Teahouse and align with participants’ motivations for doing the work. To ensure a good tie to appropriate accomplishments at the Teahouse, I would like to conduct interviews of project organizers and/or project participants, review past qualitative data collected by the Teahouse project, and ask community members to submit relevant badge ideas. A good badge collection would represent acknowledgements for a diverse amount of tasks, reward accomplishments of varying difficulty, and be perceived as meaningful.

The presentation of badges would involve establishing a good badge aesthetic that incorporates the aesthetic of the Teahouse and the chosen barnstars. Badges should also be closely tied with their criteria, titles and descriptions, and should be presented in ways that allow for easy discovery of these meta-data elements.

Distributing and Accessing Badges

edit

Experienced contributors, or Teahouse hosts, should be encouraged to give badges to each other and to other Teahouse guests where appropriate. To make sure people know about badges and to encourage their use, badges should be explicitly made accessible on the Teahouse project. To increase accessibility, I propose that available badges be presented at the Teahouse hosts’ lounge and a tally for who-got-what badge be presented to all Teahouse guests at a public location.

To maintain verifiability, badge awards should specifically link badges with concrete evidence of the work that the recipient did to qualify for the award. For example, a “great QandA answer” badge should link to that answer.

Potentially, a script or a set of scripts could be developed to make it easier for hosts to give out badges, particularly badges that can be pushed to the Open Badge Backpack. Integrating badges with the Backpack would allow badge recipients to collect, group, and share the badges and barnstars they’ve received together with other badges received at other locations. However, anonymity is an important consideration for using the Backpack service as it requires that badges be associated with a recipient’s email.

Research and Evaluation

edit

Because of the many different dynamics that could contribute to successful editor retention at the Teahouse over time, such as the ongoing success and popularity of the project itself, evaluating whether badges have a positive effect will require a combined qualitative and quantitative approach. I propose that participants be surveyed before the introduction of badges to gain a baseline measurement for their own perceived levels of engagement and interest and then surveyed again after badges have been allowed to run over a period of time to see if perceived levels of engagement and interest have changed.

Additionally metrics should be collected for:

  • Host activity before badge pilot vs. host activity after badge pilot
  • Guest activity before badge pilot vs. guest activity after badge pilot
  • Badge award activity
  • Badge sharing activity/How many badges end up in the Open Badge Backpack

These metrics will serve as indicators of health of the system and early signals of any problems with the design or its unpopularity. Community response such as badge idea suggestions and general feedback will also serve to indicate system health. We will want to see that badges are being used and used appropriately (ie without too much gaming). Things that could go wrong to watch out for:

  • Poorly administered badges. Ie. People not receiving badges they deserve due to technical or social errors
  • Badges that motivate people to do something that is harmful for the project or take away focus from constructive editing and collaboration
  • Badges that are ignored or completely not used
  • Badge recipients trying to collect badges without doing the necessary work
  • Badge recipients receiving badges for work not done or achievements not achieved.
  • Increase in the gender editing gap (potentially due to increased competition or misplaced acknowledgement)
  • Decrease in editing or overall participation

Project Plan

edit

Phase 1:

  • Look through past metrics, interviews, and other research to better understand participant motivations and goals on the project
  • Engage community and project leaders in brainstorming potential badges for a badge collection.
  • Survey participants to gain a baseline measurement for perceived levels of engagement and interest.
  • Brainstorm designs for how badges should be presented to participants

Phase 2:

  • Build badges and any associated pages
  • Create any related scripts to enable badge awarding
  • Work on integrating with Open Badge Backpack

Phase 3:

  • Recruit select participants to start awarding badges on a consistent basis
  • Monitor badge awarding, adjust where necessary

Phase 4:

  • Interview and survey participants
  • Synthesize and report findings

Background Research

edit

The following is a list of some of the information available on badges and how they relate to reputation systems, games, and motivation. Not an exhaustive list by any means and more research will be added as it becomes known:

Mozilla Open Badges Project
Games and Gamification
Badges in Reputation Systems, Community, and Identity
Badges are one of several ways for people to display their reputation on or offline (others include leaderboards, ratings or special names). If they are properly awarded for mastering or achieving something they can signal to others information like skill-level, dedication, age (newbie or expert) or even friendliness of a user. When thought of in the context of reputation systems, the value of badges to users lies in their ability to solve the problem of information asymmetry.
Badges in Learning and Badges as Certification
Examples of 'Failed Badge and Extrinsic Award Systems
Motivation on Wikimedia Projects

Original Proposal

edit

Overview

edit

This project proposal is to build badge systems to help encourage participation on Wikimedia projects. Although a robust system of barnstars and personal rewards exists to reward “above and beyond” contributions to Wikipedia and its related projects, there is currently no similar way to reward participation on a less-than exceptional level. Badges could be used as a reward for contributions that may not necessarily be extraordinarily significant from a Wikimedia project’s point of view but may still be meaningful to an individual.

Research has shown that set goals and specific asks can help encourage participation. Rewarding community members for smaller contributions could lead to significant aggregate contribution to a project. Badges could provide community members of all stages, novice to expert, with goals to work toward and feedback about whether their goals were attained. For example, badges could be given out for best new article in a given category, or badges could be given out for X number of tickets cleared on an OTRS queue. Additionally, badges could help signal to new potential participants appropriate points of entry for a project and outline the pathways they could take to grow to more advanced participants. What is a badge? What is a badge system?

A badge can be understood as a graphic packaged together with an explanation of how it is earned. It can also be combined with information on who earned the badge and how many badges were awarded. Although each badge represents a discrete accomplishment, badges can be hierarchical with some badges being earned only after certain pre-requisite badges have been earned. Depending on the context, badges can be thought of as simple rewards, as feedback of accomplishment, or as accreditation of skills earned.

A badge system is how badges are displayed to participants and how participants interact with the badges. Designing a badge system means considering the total number of badges to be created, types of badges in the system, types of goals set and accomplishments rewarded, and roles of participants within the system. Badge systems can allow participants to share their badges widely on social media sites or to display locally within the system. Badge systems can also vary based on who does the badge evaluation and assignment -- the wider community, an authority, or an automated system.

Goals

edit

Because of the wide range of possibilities for designing badges and badge systems, the goal of this project is to provide a means for understanding badges, badge system how-to’s, and best practices within the Wikimedia context. The intent is to empower meaningful growth and implementation of badges on Wikimedia projects by exploring the contexts in which badges are appropriate for the Wikimedia community. Different community spaces and projects should only have badge systems that are appropriate for the scope of their projects and meaningful to their participants. Badge systems can be created and evaluated on some projects of varying scale to create a proof of concept that can be applied throughout the Wikimedia ecosystem. If proven successful, creating and documenting a badge framework will help the badge initiative continue and grow.

Some questions that this project aims to explore:

  • Can badges be used to motivate users to participate more on Wikimedia projects?
  • Are badges appropriate for all Wikimedia projects or only on some types of projects? What are the different types of Wikimedia projects?
  • What is the best way to display badges and participation goals to users?
  • Does integrating with the Mozilla Open Badges Initiative make sense for Wikimedia?

Rationale

edit
Targeted. Badges can be used to increase overall participation and the quality of participation.
Actionable. As deliverables, this project should have a trial badge system(s) with badges, a documented set of best practices and how-to’s, and any other material and artifacts necessary to make badge systems replicable on different Wikimedia projects.
Impactful. The intent is to allow badge systems to be versatile and applicable to different projects throughout the Wikimedia ecosystem. The proposal is to only affect some projects while the badge system designs are being evaluated.
Sustainable. Part of designing a successful badge system would be to make it possible for community members to create their own badges, evaluate achievement and assign badges to recipients.
Scalable. A documented set of best practices will help badges be easier to implement in different contexts.
Measurable. A successful badge system would be used by the intended participants and would increase contribution. It would also be straightforward to implement by a project community.

Idea submitted by

edit

Anya (talk) 14:54, 28 March 2012 (UTC)[reply]

Prospective fellows

edit

Would you like to be considered for a fellowship to work on this project?

If you'd like to take an active role in this project, either alone or in a team of fellows with complementary skills, please add your name below. Note that in order to be considered for a fellowship, you must also submit an application to the program.

  1. Anya (talk) 14:54, 28 March 2012 (UTC)[reply]
  2. Ocaasi (talk) 21:19, 11 September 2012 (UTC)[reply]

Endorsements

edit

This section is for endorsements by Wikimedia community volunteers. Please note that this is not a debate, vote, or poll, but is rather a space for volunteers to describe in detail why they think a project idea is of value. If you have concerns or questions rather than an endorsement to make, please use the idea Talk page. Endorsements by volunteers willing to work in collaboration with a fellowship recipient on a project are highly encouraged.

  • I think there is useful space for building from barnstars into something that is a bit more formal and encourages best practices. Important that this not become a calcified structure, and be designed from the outset to be flexible, but the advantages to having a system of recognizing contributors is worthwhile, I think. -- Halavais (talk) 17:44, 2 April 2012 (UTC)[reply]
  • Mostly just echoing some of the above comments. This proposal feels to me like a method to formalize barnstars and other awards that already exist. Although I do worry that some users may feel as though a badge system is overly rigid where current awards systems are extremely flexible. However, for some badges I think there could be great value, and assuming the system was built on something like Mozilla's OBI, one huge benefit would be the ability to display the badges elsewhere. MyNameWasTaken (talk) 21:36, 9 April 2012 (UTC)[reply]

Hi Anya! I love this idea, and think it shows great promise. My big question is... where's the background research? Gamification and reputation systems are hot-hot HCI research topics these days. There's lots of good work out there on what works and what doesn't, from both academic and in industry research institutions. See, for example, this presentation on some of the pitfalls of gamification, which also includes a few useful tips of how it can be done right. I don't share Sven's concern about 'hat collectors' harming the project, btw. There are plenty of intrinsic motivations for contributing to Wikipedia, and I doubt that people who enjoy editing for its own sake will get so wrapped up in collecting badges that they lose interest in editing. If it's badges you want, there are always going to be plenty of other sites that will give you more immediate badge gratification than Wikipedia. If it becomes apparent that massive gaming is taking place, the badges themselves may lose credibility but I don't think that will sully Wikipedia's good name or anything. Mostly, I've just seen a lot of badge systems ignored or abandoned, and I wouldn't want that to happen on Teahouse! Jtmorgan (talk) 20:58, 28 June 2012 (UTC)[reply]

Great point! I've added a section on the background research above. You're right, there's a ton of research on gamification and reputation systems out there (trying to keep up with it all is fun but can be quite a task!). There are also quite a few thoughts and opinion pieces on the concept of badges themselves, though not very much published academic research (yet). I will keep adding more resources as I find them. Please feel free to suggest/add more. That presentation was great!
To speak to your point about badges and somewhat to Sven's concerns. There are indeed badge systems that have failed. A documented example of a case where badges failed is at the Huffington Post. And there is also a write-up of a failed game/point-system at Consumating. On the other hand, there are several cases where badges seem to be working out alright, for example StackOverflow and the GirlScouts. The difference seems to be whether badges end up providing value to the community or are a misplaced attempt by the community administrators to get people to do something they wouldn't normally want to do through a system of extrinsic rewards. From my reading on reputation systems and games I have come to think of badges as being valuable when they help solve the problem of information asymmetry and when they provide feedback of skill mastery. The idea is that as someone learns something or is on a path to master a skill, for example how to edit, they need feedback of how well they are doing to keep motivated (I believe a good resource into motivation and mastery to be this book. Badges can provide feedback of progress and a way to acknowledge good work. They can also help others identify how well someone is doing on their way to mastering something or how active they are in the community, thus creating a kind of reputation system that can be useful both internally within the community or externally in the outside world. Yes, it's possible to get hat collectors or to make badges reward behavior that isn't necessarily preferred or to create badges that are completely ignored. I'd like to minimize that by creating badges that are likely to be useful in terms of feedback, reputation and acknowledgement. And if that isn't possible and badges end up not working out then it will make sense to stop the pilot and write up the findings of what happened. Anya (talk) 22:21, 13 July 2012 (UTC)[reply]
Sweet! Now that's a nice lookin' bibliography! :) Thanks for clearing things up. I'd love to chat more about your design & implementation ideas. Jtmorgan (talk) 15:23, 14 July 2012 (UTC)[reply]
Yes definitely! I am currently in brainstorming mode and I will keep posting updates here as I have them. Your suggestions and ideas on that front would be welcome. Anya (talk) 17:49, 17 July 2012 (UTC)[reply]
  • I like this idea but Teahouse may not be its best use. It seem most suited to: 1) Recognizing working through a new editor tutorial. adoption or mentorship program. Many en:WP editors who participate in Adopt-a-User already reward their adoptees with custom barnstars as they accomplish goals. See the programs by en:WP:User:Worm_that_turned and en:WP:User:Pluma. 2) Participation in various activities of WikiProjects. Some already have awards. The en:WP:Guild_of_copy_editors awards barnstars for active participation in backlog elimination drives, special awards for the winner and second place in a Copy Edit of the Month contest, and so on. en:Wikiproject:BIRDS has no awards, not even a project barnstar. It might benefit from awards to encourage participation in various activities. Doctree (talk) 17:07, 5 August 2012 (UTC)[reply]
There are definitely other projects that might benefit from awards, most of which are unknown to me so I appreciate these kinds of suggestions. I may look into a project like Wikiproject:BIRDS as things move more forward. For now, part of the reason why I chose to do the Teahouse was because I already know that it has a mechanism for gathering data on what works to engage editors and what doesn't. The key to this pilot is to try and gather data to support the claim that awards are helping with engaging participants and to think through how to make those awards portable to the outside world. Also, Teahouse female hosts noted that there wasn't as much support and acknowledgement on the project as they would have liked. To me that says that despite the mentorship program inherent within the Teahouse there is still potentially a need for some kind of award/acknowledgment mechanism. I'd be interested in learning more about how mentors give out custom barnstars, thanks for pointing that out to me. If you have any other ideas/suggestions/thoughts in that area, please post. Anya (talk) 13:11, 23 August 2012 (UTC)[reply]
  • I strongly support this idea. Indeed in March 2012 I submitted, together with other researchers and experts, a proposal to the University of Granada (Spain) to gamify a wiki project called Descuadrando, the open encyclopedia on Business. This project is based on MediaWiki and follows the Wikipedia philosophy. The project was accepted in September. We want to develop a proposal to gamify wikis in order to improve engagement of users and editors. The prototype we design is intended to be transferable to other MediaWiki and finally to Wikipedia, as a proposal. That's why to make people think about the problem we have set the project challenge as "how to Gamify Wikipedia". The advantage of experimenting in a small project is that we can implement our ideas in the full encyclopedia. We'd like to be in touch with the responsible of this proposal and try to get your advice and expertise. Participation on the project and outcomes are open. Here you can see the first documents: Synthesis of the project How to gamify Wikipedia; Resources on gamification. As long as I discover you project proposal some days ago I took some of the resources you include here in our wiki (including the reference to this page). eromerof (talk) 23:22, 9 January 2013 (UTC)[reply]
Retrieved from "https://meta.wikimedia.org/w/index.php?title=Wikimedia_Fellowships/Project_Ideas/Badges_for_Wikipedia_Community_Engagement&oldid=24927241"