Research:Deletion notice effectiveness

Contact

This page documents a completed research project.

This short research project is a qualitative look at editor experience with the deletion processes on English Wikipedia, and specifically whether user talk page notifications adequately explain the process.

Project Summary

Along with looking at the quantitative data from our Template A/B testing experiment with deletion notices sent via Twinkle and SDPatrolBot, we have gathered some qualitative feedback about deletion notifications (CSD, AfD, and PROD) from users who received them over the course of the test.

Contacts

Methods

We sent talk page messages to all registered editors who received a deletion notification as part of a randomized A/B test of deletion notifications, and who made at least one edit after receiving the template. In the talk page message, we asked them some brief questions about Articles for deletion (Afd) and Proposed deletion (PROD) notices and their experience with the relevant process. They were free to answer these questions onwiki or via email (email answers will only be used anonymously and/or in aggregate, unless responders feel comfortable having them published onwiki).

Notifications tested

Proposed deletion

Default

Proposed deletion of Article

The article Article has been proposed for deletion because of the following concern:

concern

While all contributions to Wikipedia are appreciated, content or articles may be deleted for any of several reasons.

You may prevent the proposed deletion by removing the {{proposed deletion/dated}} notice, but please explain why in your edit summary or on the article's talk page.

Please consider improving the article to address the issues raised. Removing {{proposed deletion/dated}} will stop the proposed deletion process, but other deletion processes exist. In particular, the speedy deletion process can result in deletion without discussion, and articles for deletion allows discussion to reach consensus for deletion.

New

Proposed deletion of Article

Hello, $USERNAME, and thanks for your contributions to Wikipedia!

I wanted to let you know that I’m proposing an article that you worked on, Article, for deletion because I don't think it meets our criteria for inclusion. If you don't want the article deleted:

edit the page
remove the text that looks like this: {{proposed deletion/dated...}}
save the page

It helps to explain why in your edit summary or on the article's talk page. If you have any questions, feel free to ask on the Help Desk. Thanks again for contributing!

AfD

Default

Nomination of Article for deletion

A discussion is taking place as to whether the article Article is suitable for inclusion in Wikipedia according to Wikipedia's policies and guidelines or whether it should be deleted.

The article will be discussed at Wikipedia:Articles for deletion/Article until a consensus is reached, and anyone is welcome to contribute to the discussion. The nomination will explain the policies and guidelines which are of concern. The discussion focuses on good quality evidence, and our policies and guidelines.

Users may edit the article during the discussion, including to improve the article to address concerns raised in the discussion. However, do not remove the article-for-deletion template from the top of the article.

New

Deletion discussion about Article

Hello, $USERNAME, and thanks for contributing to Wikipedia!

I wanted to let you know that some editors are discussing at Wikipedia:Articles for deletion/Article whether the article Article should be in Wikipedia. I encourage you to comment there if you think the article should be kept in the encyclopedia.

The deletion discussion doesn't mean you did something wrong. In fact, other editors may have useful suggestions on how you can continue editing and improving Article, which I encourage you to do. If you have any questions, feel free to ask at the Help Desk.

Thanks again for your contributions!

Findings

Of the 372 editors we talk page messaged, a little over 40 responded, with roughly equal proportions of new and very experienced Wikipedians. The feedback from these two groups was very different, suggesting that there is a disconnect between how established Wikipedians view deletion notices and how these notices are perceived by new and less active users.

New and less experienced Wikipedians

Overall, the responses we got suggest that this group is very confused and frustrated by the deletion processes; however, there is some encouraging evidence that the new Proposed deletion notice we tested was received very positively by these users.

New and less active Wikipedians were most positive about our new Proposed deletion template: every user in this group who received our new template said that the message was clear and helpful:

It was good and the message helped me fix the article to avoid getting it deleted
Instructions were clear and easy to follow.
It was easy.

This group of editors was most negative about the default AfD notice, with most users who received the notice speaking negatively about the experience:

I have give up the article content after that notice.
There are far to many internal links to other article about how to go about the process of contesting a deletion of an article. There was far to much information. I had skim through the material and try to make sense of it all.
in my opinion was really unfair. It dissapointed me much... My article survived, but it rather discourage me from writing articles. (sic)

The rest of the specific template feedback tended to vary. This is probably to some extent due to the fact that many of these users were so frustrated or confused by the deletion process that they had not processed the information on the template notification and couldn’t give feedback on it.

Adding to the confusion is the fact that many of the people who received our test notices also received other deletion notices, including speedy deletions of files, lists, and articles. The process for contesting deletions is different in all three cases (AfD, PROD, CSD) and notability/referencing/licensing requirements differ for articles, lists, and files. To these users, however, all these different processes appear to be interchangeable. As a precursor to this survey, we asked people who had been warned for removing speedy deletion tags from articles they were working on to comment on the templates they received. Their feedback also suggested that they had trouble differentiating between the different kinds of deletion on Wikipedia, especially given that the method for contesting one kind of deletion (PROD) becomes a blockable offense in other cases (CSD, AfD):

I don't think the messages were clear. They indicated that I could remove the tags and add justification. And then I received a threatening accusation that I would be unsubscribed/banned for having removed the tags.

It is evident from the feedback we received that the deletion processes on English Wikipedia are incredibly confusing, frustrating, and/or demotivating to new and less active editors:

I strongly felt that there was some merit for the page to be up there (for example because several other similar pages are active), but I found it quite hard to understand how to do it correctly.
I dont remember what the instructions specifically said, but I do think they need to be a little more detailed or have some sort of example on how to approach this. If someone doesn't know how to respond, their page may be deleted without just cause.
Perhaps there should be a ‘wikipedia for idiots’ type of help (the current help pages, albeit useful, are really complicated) with only the main functions.
the deletion experience made me feel like an unwanted outsider, not a welcome potential resource.
Wikipedia now feels to me like every article should be subject to speedy deletion until a lot of time and discussion is made about the validity of the article can be verified through some sort of a judicial process. This is very good for those who have a lot of time to dedicated to the continued reliability of wikipedia. I on the other hand cannot always cater to this intricate process.

Moreover, it is important to note that some of these editors not only do not understand how to contest deletions, but don’t understand that deletion can be contested, or that the people responsible for nominating their article for deletion are not a paid review staff, but are simply other volunteer editors.

I don't quite understand the second part of the third question ("Do you understand how to contest a deletion?"). Can you give me more information about that?
I understand that the deletion proses is designed to limit Wikipedia to articles about things that are popular in the US.

Highly active Wikipedians

Unlike new and less active Wikipedians, most active Wikipedians don’t seem to pay very close attention to AfD/PROD notifications. When discussing the effect of these templates on themselves and other users, they are divided between a) feeling very strongly that these messages need to look urgent and sound serious in order to be effective; and b) considering simpler, friendlier messages to be better for new and established editors alike.

Most of the highly active Wikipedians we surveyed admitted to not paying much attention to deletion notices:

As far as the wording goes, I probably didn't read it, I just see the template, and generally wonder why people don't leave a proper message about the page.
I frankly didn't notice any difference as I cared most about the issue, not the notification's text.
I see them so rarely on my own talk page that I didn't pay the form of it much attention, just went to the article to go fix it.
the message I got happened to be for an article I created a long time ago on a non-notable topic, so I didn't exactly read the message.

However, there are some highly active editors who are not familiar with these deletion processes:

Ah, so I assume prod means 'proposed deletion'. But I still don't know what the difference is between afd and prod.

Many felt that deletion notices need to stress the gravity of the situation and look urgent, rather than being friendly:

'The new one Template:AfD-notice-rand/new is much too harmless. It sounds like someone is giving you a friendly notice, saying you did nothing wrong (soothing you into feeling safe again...) and Thanks for the contrubutions (sic)... I think a newbie would be entirely surprised to see that after that his or her article were deleted! So, rather a bit more dramatic rather than all-too-friendly.
The proposed versions are, in my opinion, not alarming enough. No matter what, something ugly is happening with your article. "Victims" should be properly warned about this.
In my personal opinion, template messages like this should be somewhat officious and impersonal; we are trying to build an encyclopedia, not a Facebook group.

However, there were notable exceptions to that sentiment, as well:

In the PROD one, I like the clear bullets. The user will be freaking out, and will want clear step-by-step instructions on what to do... only 2 blue links is better than the current, and overwhelming, 7.
I found my (new) PROD message to be friendly and not intimidating at all. It was calm, rational, and was easy to understand. As a person who likes Wikipedia, I hope that more PROD messages are like this in the future so that new editors can easily transition into becoming frequent editors of Wikipedia.

While several editors had specific feedback on the deletion process, most did not mention any negative or demotivational consequences to having articles deleted, in sharp contrast to the new/less active editors. As mentioned above, at least one active editor did not understand the difference between AfD and PROD; furthermore, interpretation of the current PROD policy seemed to vary.

Summary

Though our new Proposed deletion template improved the experience somewhat for new and less active users, deletion appears to be an overall negative experience for this set of editors. Those who understand that deletion can be contested, and even those who know where to go to find the appropriate policy steps (which is a very tiny subset of this population) still have a hard time contesting deletion and are very stressed out by the process.

While we may have made some improvement to PROD by simply making the template clearer, AfD appears to be a more difficult and painful process, because it involves a greater investment of time and energy debating the merit of their article. Users who have never engaged in a deletion discussion feel disadvantaged and tend to give up, and unfortunately it seems the original attempt to be much more encouraging was not effective.

In contrast, experienced editors don’t pay much attention to deletion notifications and don’t appear to be as negatively affected by having their articles AfDed or proposed for deletion. In addition, language that seems friendly and helpful to new editors may end up seeming condescending or simply unnecessary to very experienced authors; one solution here is to encourage no templating of regulars, or to develop a special "short and to the point" template for very experienced Wikipedians. There is some disagreement about how deletion notices should look: some feel they need to look more urgent and serious, and some feel they need to look simpler and friendlier.