Grants talk:Project/Racso/Knowledge Defenders

An initial comment about the project

Fixing vandalism [...] is one of the easiest things you could give a newbie to do [...] And it's fun!

— Raph Koster. Wikipedia is a Game, 21:09. I agree with him; however, some care should be taken so the newbie doesn't do more harm than good when trying to fix vandalism.

If you want to have new content for newbies, you should encourage vandalism.

— Raph Koster. Wikipedia is a Game, 28:25. While it's a terrible idea if taken as-is, vandalism can be simulated in sandboxed environments to allow people to "play" with it without harming Wikipedia

Wikipedia is a Game, presentation by Raph Koster at Wikimania 2014.

When I began working with Wikipedia in 2007, I almost immediately focused a great part of my volunteering time in dealing with vandalism. I began patrolling Recent Changes in order to identify bad faith edits as soon as they were made, and soon after that I discovered some tools that helped doing that work. It was conforting to know that I was somehow helping to protect the encyclopedia from harm. Also, I was having a lot of fun.

In Wikimania 2014 in London, Raph Koster, author of A Theory of Fun, gave a nice speech about engaging people to Wikipedia an the Wikimedia projects. He talked about the various stages of fun that can be achieved when working with the projects, and warned about the risk of having all the funniest activities being made only by old users. At some point, he mentioned that dealing with vandalism was a fun activity, and asked why aren't newbies introduced to it instead of always being introduced to editing articles.

I totally agree with him about this. Getting engaged with the Wikimedia projects requires, a lot of times, fun. At least in the beginning, it's not only about free knowledge, but also about feeling that the activity being done can be considered a nice hobby.

Wikipedia vandalism is not a game. It harms the projects, and may affect a lot of innocent people who daily use it to look-up for information. Vandalism shall be treated with full responsibility, by using all the means available to combat it.

This doesn't mean that the activity itself cannot be approached with a playful attitude. Sometimes, dealing with vandalism seems like a game, especially when tere are several people doing it at the same time. You're taking meaningful decisions. You're getting immediate feedback of your actions. You're trying to beat your fellow Wikimedians in the amount of caught vandals. For a moment, you become a member of "Team Good", dealing with the bad guys.

I believe that combating vandalism can be, truly, a nice (albeit uncommon) way to gen people involved with Wikipedia's dynamics. Therefore, and in order to enhance the experience and provide new tools for dealing with this problem, I propose the development of this project: a mobile and desktop game that focuses around the mechanics of combating vandalism in Wikipedia. --Racso ^¿¿¿??? 18:12, 14 March 2017 (UTC)

[F]AQ

In this section, I'll try to provide more detail to [frequently] questions asked in this talk. Of course, everything written here is open to critique and comments, but I would appreciate if questions or comments are written in other sections in order to keep this [F]AQ as readable as possible. --Racso ^¿¿¿??? 17:57, 26 March 2017 (UTC)

How would the training work

This is a rough and simplified explanation on how would the game train users to deal with vandalism. Specifics such as terminology or game elements are just temporary placeholders, as the game is still in an ideation stage and the first stage of the development process would be aimed are defining those specifics. They're here just for enabling the explanation of the idea.

The game would rely on a set of previously cataloged diffs. As a simplified example for this explanation, consider that the diffs may have two labels: "must be reverted" (e.g. vandalism or gf damaging edits) or "should not be reverted" (any other kind of edit). This is similar to the sets used to train machine learning bots, but would probably be smaller as humans learn patterns more effectively than bots.

In the first level, players would only see edits from the training set. For each one, they would need to make a choice: to "revert" the edit or to ignore it. The game would be able to provide feedback on the choice (as it knows the "right answer") and either reward or punish the player for it. Of course, no real action is applied on-wiki. False positives ("reverting" a good edit) would be punished way more heavily than false negatives (not "reverting" an actual vandalism) so the player is forced to be cautious and "revert" only when sure.

Only after the player has shown proficiency in the first level (i.e. a low rate of mistakes, specially false positives), he's able to access the second level. In this level, the user is also shown real recent on-wiki edits. However, his choices (reverting or not) are not applied directly on-wiki, and he isn't given feedback immediately. Instead, they are checked by another human first, and only then are given feedback. This "checking by another human" may take many forms. One idea is to have a place where this kind of choices are posted, so experienced users can check them and act accordingly (e.g. revert if necessary or somehow tell the system that the edit shouldn't be reverted). That page would act as a "possible vandalism alerts" page, which would be useful for advanced users to quickly find possible vandalisms (e.g. if an edit is tagged as "revert" by several players, it's probably a vandalism").

Level changes are seamless, and happen progresively without the user noticing. For example, when the user beats the threshold for getting into the second level, he begins to progressively receive more and more real edits, but he doesn't know in advance which ones are real and which ones are from the training set (well, he doesn't even know the training set exists).

Only after the player has shown proficiency in the second level, he's able to access the third level. Again, the change is seamless, progressive and without notice. In this level, the user begins to work directly on-wiki, with realtime edits and choices (reverts) that are really executed.

Comments of Glrx

I would decline this proposal.

Yes, vandalism is a problem on WP, and it would be good to have more vandal fighters. It would also be good to train vandal fighters.

I've seen several good edits improperly reverted as vandalism, and those reverts can discourage new users. I'm not convinced that improper reverts are a serious problem. I am thinking of one prodigious vandal fighter who has been told many times to slow down because he's executing false positives and false negatives. I don't think anything will fix his skill level.

Fundamentally, I'm don't believe a game is the appropriate venue for training vandal fighters. Vandal fighting should be deliberate and serious. Training involves spotting problems and learning tricks. An essay (rather than a game) can teach a lot of that material, but even so, there's no education like seeing vandal tag teams pull a fast one. There's also an open problem with robot edits masking vandalism.

In addition, the proposal is vague. It says it will make vandal fighting fun, but how will it do that?

Vandal fighters can be quite productive: thousands of edits per month. The goals seem off.

Recruiting non-Wikipedians as vandal fighters does not seem wise. It takes some skill as an editor to start dealing with vandals. A lot of vandalism looks like a legitimate edit. The trivial vandalism is caught by Cluebot. There are many subtleties. I'm not reassured by the "I am not a Wikipedian" stance.

Glrx (talk) 23:23, 20 March 2017 (UTC)

Hi. Quick initial question: what do you mean by 'the "I am not a Wikipedian" stance'? Thanks. --Racso ^¿¿¿??? 18:59, 25 March 2017 (UTC)

Thanks for checking the proposal, Glrx, and for your comment. I'd like to address some of your objections about the proposal. I'll enumerate my comments in case you'd like to further discuss them separately.

1. First of all: vandal fighting is already fun. Yes, it should be deliberate and serious, but doing anything on Wikipedia should be deliberate and serious. This doesn't mean, however, that you can't have fun while doing it. In fact, a lot of people are probably here (in the WM projects) because they have fun (in their own ways) while contributing. Fighting vandalism is simply another kind of activity you can perform; an important but often-overlooked-by-newbies one. The game would simply enhance that experience and make it easier for new people to get started (among other things I mentioned in the proposal, such as serving as a mobile tool for dealing with vandalism by advanced users).
2. The goals aren't too ambitious because 1) newbie vandal fighters may not be able to revert thousands of monthly edits, 2) those newbie vandal fighters may get involved with other tasks in Wikipedia after getting started; I would only measure reverted edits as the indicator for the objective, but we may get users active in other kind of tasks as a side-effect, and 3) advanced vandal fighters may prefer to use other tools for the job after all.
3. Of course it takes skill to deal with vandalism. That's why the game would remove a lot of the technical hazzle to start reverting edits, and would take measures to 1) show those newbies how to actually revert vandalism rather than legitimate edits, and 2) avoid those newbies to harm the project while learning. I wasn't more explicit on details about the game because the format for requesting the grant asks people to be succint and I couldn't enter in too much detail, but you can check a bit more of detail in the Phabricator task I linked in the "Community notification" section.
4. About the "I am not a Wikipedian" stance thing: I'm not sure if you understood that I wasn't a Wikipedian, or if I somehow said that in the proposal, but that's wrong. I've been a Wikipedian since 2007, and I've reverted vandalism almost since I started. I've been a lot less active lately because I've switched my Wikimedia-related work to the Colombian user group, but I don't consider at all that I stopped being a Wikipedian.
5. Finally: almost anything can be explained through essays and text. That's how we've done it during all this time. However, I personally believe that if we can enhance the ways we bring people closer to the projects and/or the overall experience of advanced users, we should give it a try.

Regards. --Racso ^¿¿¿??? 19:28, 25 March 2017 (UTC)

First line of the linked Raph Koster video is "I am not a Wikipedian".

I need to see details of what you would do. Phab:T155074 says essentially nothing more than it's an idea. Linking to an entire conference is a bad idea. I need more meat than just turning it into a training game.

I need to be convinced about applicability. Does it appropriately address a real problem. (Your statement "first of all, vandal fighting is already fun" undermines your position.) How would it compare to other possibilities such as recruiting drives, meetups, or mentorship.

I also need to be convinced about effectiveness. Saying "game" and "fun" does not persuade me. How is a game better than other possible training methods? Can a game backfire: if reverting "vandalism" is too much fun, then good edits might be reverted just to experience the additional "fun".

As I said, I'm not happy about the measurement metric.

I also come with a bias. I don't want WP to be recruiting newbies as vandal policemen. I take the typical career arc of a Wikipedian as someone who contributes a lot of content and then drifts into reverting vandalism and doing new page patrols. I'm leery of editors who want to jump into anti-vandalism because they don't have much experience or perspective about what editing means. It's one thing to think about vandalism in the abstract, but it's quite another to experience a hour's worth of earnest work being reverted in an instant or a new article start being tagged for speedy deletion.

Glrx (talk) 21:49, 25 March 2017 (UTC)

1. I forgot to address your comment about ClueBot catching trivial vandalism. That's true, of course, but only for those wikis in which the bot operates. For example, ClueBot doesn't operate in the Spanish Wikipedia, and we didn't have any anti-vandalism bot running for a while (a whole year) because the previously existing bot was broken after some MediaWiki updates and its creator wasn't active anymore. Getting any machine learning-based bot to work in a specific language requires a good and big enough training set, which isn't always available, and even when the bot is ready to work, there's still a big amount of vandalism that won't be detected by it. Don't get me wrong: I think that ClueBot and ORES are simply fantastic, but they can't do everything.
2. Please find more information on how would the training process work here. You'll note that the proposed dynamic would reduce the "backfiring" risk. Hopefully, it would also reduce your bias. In my opinion, your bias is valid, but I think it exists because of bad previous experiences with newbie with loose triggers. I think it wouldn't be the case of this game.
3. Sadly, I can't prove that the game is the best method for training people on this. I simply don't have data for that as there are no previous experiments in this sense. I could link you to some papers on serious games, but I don't think that would be meaningful for this specific project. I'm talking based on my personal experience on motivation, collaboration in the Wikimedia projects, vandalism fighting, games and gamification. I believe this would be a great alternative, that I can make a good job, and that it should work. However, there's no "secret formula" for making games fun or successful. I only know that the result must be excellent, as good tools enhance work but bad tools make it even more difficult.

--Racso ^¿¿¿??? 19:42, 26 March 2017 (UTC)

Comments of Antur

I consider the idea interesting, but at the same time it generates me some doubts:

1. I think that pure and hard vandalism is not complicated at Wikipedia to fight with: it is massive, but most can be reversed (and indeed it is, including spanish Wikipedia) by a bot, which also warns when three or more vandal interventions are made to request a blockade. The bot is so fast that a human patroller can´t deal with. Then: Do we really need this kind of training?, or -in other words- Will the game engine have enough intelligence to teach how to handle with self-promotion articles, conflicts of interest, particular purpose accounts, lack of neutrality, which are the really difficult maintenance problems?
2. Many years ago in Spanish Wikipedia it was usual to talk about patrol cars, patrols, wikiproyects, contests, prizes, all identified with a police cap or a gun. There were several problems generated by this role-play game, a "Far West Saga" within Wikipedia, so the third question is: Are we really talking about a game, or will it be more a simulator, an automated form of training with an exciting Title, but soon to be replaced by something more playable, such as Grand Theft Auto. ;)?

These doubts may be generated by the novelty of the proposal; I do not deny that there is a way to give a clever answer to all of them, but a deeper opinion about the proposal would require a more complete development of the idea. Best regards --Antur (talk) 04:31, 4 April 2017 (UTC)

The "role-play" issue has been raised up during the developer Summit, at least by me. French Wikipedia, where I'm active as a volunteer, has had the same issues concerning the military wording. My suggestion was to focus the training on human respect, and reward good behavior. Trizek (WMF) (talk) 13:24, 4 April 2017 (UTC)

Thanks for taking the time for writing, Antur. Here are my answers to your questions:

1. Yes, the idea would be to teach people how to deal with several kind of vandalisms. Of course, the idea is that players would develop their skills as they play the game, so in the end they would be able to identify cases that weren't specifically covered by the training. The hard part would be how to give players enough confidence to try to deal with "obscure" vandalism while at the same time keeping them from being careless or risky when reverting.
2. The idea would be to actually have a playable, fun and competitive game. It would be similar to what you call "Far West Saga" in terms of somehow enhancing the experience with a playful experience, but in a controlled environment instead of on-wiki. I believe that previous experiences of people "playing" with vandalism (such as the one that Trizek mentioned during the Dev Summit) weren't positive because users 1) could risk being moderately careless without facing strong consequences, and 2) were acting directly on-wiki since the beginning, so their actions affected real users immediately. The game would address both issues (I tried to explain how here). In general, I don't think the problem is seeing vandalism-control as a game, but is seeing it as a game without rules. When you play any game, you have rules and boundaries that you agree to respect and comply with; if you don't you simply lose. The proposed game would have those rules and enforce them, so users won't simply "revert everything they see to win points", as they will simply not have an in-game motivation to do so.

I hope that answered your questions. --Racso ^¿¿¿??? 03:42, 7 April 2017 (UTC)

Eligibility confirmed, round 1 2017

This Project Grants proposal is under review!

We've confirmed your proposal is eligible for round 1 2017 review. Please feel free to ask questions and make changes to this proposal as discussions continue during the community comments period, through the end of 4 April 2017.

The committee's formal review for round 1 2017 begins on 5 April 2017, and grants will be announced 19 May. See the schedule for more details.

Questions? Contact us.

--Marti (WMF) (talk) 19:53, 27 March 2017 (UTC)

Dear Racso,

Thank you for submitting this proposal. I have one quick piece of feedback. Though the idea of gamifying antivandalism is very interesting, $24,000 is a significant ask for a proof of concept. Consequently, I think it will be very important for you to get as much feedback as possible here on your talkpage from editors in Wikimedia community who are focused on fighting vandalism, especially in whichever projects you are likely to work on first. We will want to hear their opinions about this project and whether they see this as an effective strategy for addressing the most pressing problems related to vandalism. If you like, you can reach out to I_JethroBT_(WMF), our community organizer, for ideas about how to do this.

Kind regards,

--Marti (WMF) (talk) 01:42, 3 April 2017 (UTC)

Licence of the game

Hi Rasco and thanks for this interesting proposition. You talk about purchasing audio files ; does that mean that you do not intend to publish the game under a free licence, such as MIT, BSD or GPL ? Léna (talk) 08:13, 16 April 2017 (UTC)

Hi! I do intend to publish the game under a free license. Buying assets would be my last resource after trying to make them myself, getting free options and paying someone to make them. If I end buying something, I would check the licensing to see what can I do in terms of open-sourcing the game (for example, I may release the game source code without the assets, but distribute the binaries with them).

I hope that answers your question. Regards! --Racso ^¿¿¿??? 16:06, 25 April 2017 (UTC)

Comments from Ruslik0

Thank you for and interesting proposal but I have some comments:

1. I am not sure that "Vandalism is probably Wikipedia's biggest problem". It is a problem but one of the many. Moreover currently there exists plenty of anti-vandalism tools and processes and it is not clear how the proposed game will fit in the broader picture.
2. The fun that some people have when fighting is to significant extent related to the fact that they fight real vandals and make real difference. So, working in a sandboxed environment is unlikely to be attractive - the anti-vandalism activity itself is quite boring.
3. From the proposal it is not clear how the proposed application will look like. Can you point to other similar games?
4. The budget is not very detailed. You should probably clearly separate what should be paid to you and what to other people or to purchase the multimedia content. Though I am not sure that relying on unfree content is a good idea.
5. How the application will be licensed? Will its code be open?
6. As you should know that all automatic or semi-automatic editing tools are tightly regulated on many projects. Why do think that any project will allow you to test your game on it? Your community outreach is quite limited.
7. Another related question. As I understand at the third level your tool will allow users to make high speed reverts of the real edits. Will not this create additional avenues for vandalism?
8. "One idea is to have a place where this kind of choices are posted, so experienced users can check them and act accordingly" And it is likely that such a place will become hopelessly backlogged quickly.

Ruslik (talk) 20:45, 16 April 2017 (UTC)

Thanks a lot for your thoughtful questions and comments!

1. I should have begun that sentence with "In my opinion". I do think that vandalism is our biggest problem. Of course, we have several problems that we must constantly tackle, but I think vandalism can make the biggest [negative] impact if not dealt with adequately. What would happen if Wikipedia stopped working for one week? Probably not so much. What if we stopped dealing with vandalism for the same time?
2. This is an interesting observation, and you may be right. This would be a quite interesting conversation. For now, what I can say about this is that the sandboxing of the game would be quite invisible, so the player would transition from the sandboxed environment to the real one without knowing. In practical terms, the player wouldn't know at all that there was a sandbox in the first place.
3. Not really. The exact details don't exist yet; the initial stage of the project would be focused on giving the game its definitive shape. I have some preliminary ideas, though. As an example, I created a quick demo to illustrate my point during the Wikimedia Developer Summit (screenshots in the right). I'd like to stress that this is just a preliminary, quick, crude, basic sample I created as a proof of concept, and everything (from the general aesthetics to the core mechanics) can (some things will) change during the first stage of the project. This sample (which behaves more like a gamified vandalism tool than like a game) aims to provide a 3D zen-like experience, in which Wikipedia is illustrated as an ever-changing blob of knowledge, and edits are constantly appearing and trying to merge with it. By clicking in an edit, the player is taken to it to check its content, and he can act on it accordingly. Edits can be attacked, left alone or marked as positive.
4. Again, there are details that I don't have yet. For example, right now I don't even know what assets will I need. Anyway, when in need for an asset, I would try to 1) make it myself, 2) get a free option, 3) pay someone to create the asset, 4) buy assets.
5. Please see my answer to Léna's question above.
6. I have experience in testing automated tools in the Spanish Wikipedia. I have created some already. I think I'll be able to get the community's approval for testing the game on it. As the wiki would only be modified when the player is in an advanced stage in the game, the "invasive" testing would only be made in the end, and with already experienced players.
7. Yes. However, a vandal who [correctly] plays the game until it reaches a high enough level to be able to vandalize the wiki from within the game wouldn't be a very efficient vandal, as he would be quickly blocked in the wiki. Of course, the game wouldn't be able to bypass blocks, and the game doesn't make a player invisible to the rest of the community; there would be no difference between this hypotetical player and a standard good user who suddenly begins to vandalize. It would be way more efficient to simply vandalize directly on-wiki.
8. I don't think so. Keep in mind that the choices reflect possible vandalism that hasn't been reverted yet, so the list would be a "list of probably vandalic edits that haven't been reverted yet". For a sysop, that kind of list is heaven. Also, keep in mind that the list would be dynamic and mantained by the game itself, so no extra action would be required to remove processed items from the list or to keep it up-to-date.

I hope that addresses some of your questions. I'll be more than happy to keep discussing about the project, so if you have anything else or need clarification, please shoot it :) --Racso ^¿¿¿??? 23:32, 24 April 2017 (UTC)

Round 1 2017 decision

This project has not been selected for a Project Grant at this time.

We love that you took the chance to creatively improve the Wikimedia movement. The committee has reviewed this proposal and not recommended it for funding. This was a very competitive round with many good ideas, not all of which could be funded in spite of many merits. We appreciate your participation, and we hope you'll continue to stay engaged in the Wikimedia context.

Next steps: Applicants whose proposals are declined are welcome to consider resubmitting your application again in the future. You are welcome to request a consultation with staff to review any concerns with your proposal that contributed to a decline decision, and help you determine whether resubmission makes sense for your proposal.

Over the last year, the Wikimedia Foundation has been undergoing a community consultation process to launch a new grants strategy. Our proposed programs are posted on Meta here: Grants Strategy Relaunch 2020-2021. If you have suggestions about how we can improve our programs in the future, you can find information about how to give feedback here: Get involved. We are also currently seeking candidates to serve on regional grants committees and we'd appreciate it if you could help us spread the word to strong candidates--you can find out more here. We will launch our new programs in July 2021. If you are interested in submitting future proposals for funding, stay tuned to learn more about our future programs.

Aggregated feedback from the committee for Knowledge Defenders

Scoring rubric	Score
(A) Impact potential Does it have the potential to increase gender diversity in Wikimedia projects, either in terms of content, contributors, or both? Does it have the potential for online impact? Can it be sustained, scaled, or adapted elsewhere after the grant ends?	5.3
(B) Community engagement Does it have a specific target community and plan to engage it often? Does it have community support?	5.8
(C) Ability to execute Can the scope be accomplished in the proposed timeframe? Is the budget realistic/efficient ? Do the participants have the necessary skills/experience?	5.5
(D) Measures of success Are there both quantitative and qualitative measures of success? Are they realistic? Can they be measured?	3.8
Additional comments from the Committee: The project doesn't fit with Wikimedia priorities but tries to develop a new way to increase the quality of articles. It could have an online potential if the idea and app were well developed. Interesting idea, but the patrolling is not a game. I am not sure that creation of such games is within Wikimedia's strategic priorities. The potential online impact can even be negative. It *might* be a good fit with strategic priorities (participation) and have a high online impact (more users fighting vandalism), but there is no guarantee it will have exactly this result and not the opposite one. The project isn't innovative, but tries to adapt the idea about Wikidata game (gamification) to collaborate with Wikipedia. The proposal is innovative in a sense. However the risks are high, in my opinion, and they are not in code development itself but rather the basic assumptions underlying the project may be incorrect. An exciting idea! Measures of success are reasonably good as well. Not very useful, however, as ClueBot and similar solutions seem to be more efficient at fighting vandalism. The participant probably has necessary skills/experience though the project is rather complex but the budget is not clear enough. The applicant himself is not sure he will be able to accomplish the project on time, unfortunately. He has good experience, however. The community engagement for a project, which success depends on how the community accepts the new game, is very low. Community engagement seems really low on this project, which can seriously damage the potential impact. What is considered a proper way to deal with vandalism is hugely community dependent. Community is only notified of the submitted grant. Missing public discussion. Support is small. Major concern here: the target community (Spanish Wikipedia) is not sure this will be beneficial to them, they tend to think that, on the opposite, this may support vandalism or even not be of any use, as fighting real vandalism is way more interesting. Designing a game to get people involved in handling vandalism seems like it wouldn't transfer since the game concept isn't fully explained. While they evidently attended the developer summit and got good feedback there, this isn't reflected in their grant application. I do not think that some basic assumptions underlying the project are correct. For instance, I am not sure that vandalism is currently "Wikipedia's biggest problem". So, I can not support this project. The result is that the online impact can be even negative. Ready to fund only if significant community support from several communities. Interesting but the request lacks clarity and informational detail. Although the idea in itself is lovely, unfortunately I do not see a significant interest from the local community. On the other hand, I see some criticism that this will not help the real vandal fighters (as fighting real vandalism is more important). If this programme will be able to get and detect vandalism edits automatically, I hope there will be a better application of this data.

Thanks a lot to everybody who participated in this process. To me, it was also clear that the project lacked from community support. I hope I'll be able to contribute with other, more appealing ideas in the future. Regards! --Racso ^¿¿¿??? 16:02, 12 July 2017 (UTC)

Undue focus on vandalism

The main problem with this proposal is that it stems from the assumption that vandalism and vandalism fighting are the biggest thing, while it's just the loudest. Raph Koster also states that it's the main game-like infrastructure, with rankings and a growth path, but in reality most of the work is invisible and thankless and you can only deal with vandalism after gaining considerable experience.

I'd argue that the biggest game available out there is small fixes. For instance, language fixes: most articles have tons of poor language, from typos which are easy to fix to badly constructed sentences and paragraphs which need some writing skills. And then there are many small factual errors or out ofdate information, which can be a source of additional participation as shown by Benjamin Mako Hill in research on the effects of importing a lot of TIGER mistakes into OpenStreetMap. --Nemo 06:56, 16 August 2018 (UTC)