Research:Necromancy

Contact

Growth team

---

Research:Projects

This page documents a completed research project.

This is a project of the Editor Engagement Experiments team at the Wikimedia Foundation.

Background and rationale

"Necromancy (/ˈnɛkrɵˌmænsi/) is a claimed form of magic involving communication with the deceased – either by summoning their spirit as an apparition or raising them bodily..."

When an active contributor leaves a Wikimedia project, their talk page typically attracts comments imploring them to return. Particularly beloved editors can get dozens of messages, and a few can get messages months and even years in the future (See example).

It's also not uncommon for contributors who disappear to receive emails about their absence, based on the entirely logical assumptions that...

• they're not checking their talk page
• they may not wish to discuss their reasons for leaving in public

The E3 team believes that such personal appeals (pun very much intended) to lapsed editors are likely the most effective way to motivate people to return to the wikis. Frustrated editors who declared they were done with Wikipedia for good often return after a relatively short wikibreak. Sometimes it also seems as if people leave just to see if anyone notices.

There are, however, big downsides to relying solely on other editors to make these personal appeals to return:

1. this almost always excludes any editor too new to know many people
2. in practice, this leaves out any editor who is not particularly well-known, especially editors who work hard in less crowded areas of the wikis

Our hypothesis

While surveys of former contributors have been conducted several times, we know of no direct effort at testing "please return to editing" emails to lapsed editors in a systematic way. Therefore, we intend to conduct a very simple experiment to test whether A) people even read such emails and B) whether such are useful for understanding what ways we might encourage people to come back.

Methodology

The experiment in a nutshell:

• we randomly sample from two groups of lapsed English Wikipedia editors (more below about who they are); The lists are randomized by the number of seconds modulo 2 in the timestamp of the user's last edit
• we email them asking them to return to Wikipedia, and measure the open rate, clickthrough rate, and the activity on-wiki for each of these two groups after sending an appeal email requesting that they return to editing Wikipedia
• The email was sent to both groups at Thu Apr 12 21:07:00 UTC 2012

Sample groups

Who exactly qualifies as a lapsed or gone editor is entirely ambiguous to us at this early stage of experimentation. We are hoping to understand more about the lapsed-time and edit-counts thresholds -- in search of the ranges in which lapsed users are most likely to be convinced to return to editing. Therefore, we chose to begin with two separate groups:

The first group are editors who...

• Have not edited at all since January 9th, 2012 (hence "3 month group")
• Made at least 100 edits to articles in the 90 days prior to that date (ignoring edits to other namespaces)

There are 380 total editors who meet the above criteria (190 in the control group, 190 in the group emailed).

The second group are editors who...

• Have not edited at all since April 9th, 2011 (hence "1 year group")
• Made at least 100 edits to articles in the 90 days prior to that date (ignoring edits to other namespaces)

There are 189 total editors who meet the criteria (94 in control group, 94 in the group emailed).

We chose these two groups through trial and error. Our original metric was 300 article edits in the 90 days prior to leaving, but this produced an extremely small sample size. Further experimentation with group inclusion metrics to may be fruitful, depending on whether results from the first test are completely dismal or not. However, emails to editors with lower activity levels would almost certainly need to be delivered sooner than months or years after they lapse, as their commitment to the project is likely to drop off much faster and selling a return to editor would be much more difficult as time passes.

Email text

Key elements of both emails:

• Short and to the point: the email is less than 100 words
• Signed by people, as opposed to MediaWiki or a generic address like info@wikimedia.org
• Subject line is a strong request
• Emails to 3m lapsed editors were sent from/reply to Steven Walling
• Emails to 1 year lapsed editors were sent from/reply to Maryana Pinchuk

3 month lapse group

Template:Letterhead Subject:

Wikipedia still needs your help!

Body:

Hi $username,

We noticed you haven't edited Wikipedia for a while. You've made some fantastic contributions, but the encyclopedia still needs your help.

Millions of people depend on Wikipedia, but only a handful ever edit. It's people like you who have built the site, and there’s great things still do be done: new articles to start and existing ones to improve.

Please dive back in and see what you’ve missed! If there’s anything we can do to help you get started again, let us know.

Thanks,

Maryana and Steven, at the Wikimedia Foundation

Key elements:

• A thank you for past contributions
• A reminder that being a Wikipedian is something special
• A statement about how without editors to sustain the project, Wikipedia's readers will be underserved
• An appeal to people's sense of ownership of Wikipedia, as an author/editor

1 year lapse group

Template:Letterhead Subject:

Wikipedia still needs your help!

Body:

Hi $username,

Hey, we just saw all the edits you made in 2011. You did fantastic work! Where did you go?

Please dive back in again and contribute to Wikipedia! There’s still great things to be done: new articles to start and existing ones to improve.

If there’s anything we can do to help you get started again, let us know.

Thanks!

Maryana and Steven, at the Wikimedia Foundation

Key elements:

• A reminder about past contributions to Wikipedia, and accolades for it
• A much more conversational tone, e.g. "Where did you go?"

3 month lapse group (round 2)

Template:Letterhead Subject:

Wikipedia needs you to be WP:BOLD again

Body:

Hi $username,

We noticed you haven't edited Wikipedia for a while. You've made some fantastic contributions, but the encyclopedia still needs your help.

Millions of people depend on Wikipedia, but only a handful ever edit. It's people like you who have built the site, and there’s great things still do be done: new articles to start and existing ones to improve.

Please dive back in and see what you’ve missed! If there’s anything we can do to help you get started again, let us know.

Thanks,

Maryana and Steven, at the Wikimedia Foundation

Key elements:

• Subject line changed, in order test the importance of it in conversions. The new version we tested is still a strong call to action, but is much more insider-centric
• Link changed to Special:Watchlist instead of Special:UserLogin. We wanted to change the &returnto= function to point to the watchlist, but in reality the link allowing people to return there is not very prominent.

1 month lapse group (round 3)

Template:Letterhead Subject:

Wikipedia needs you to be WP:BOLD again

Body:

Hi $username,

Thank you for all the great work you've done to improve Wikipedia. There were more than 3.5 million changes to the encyclopedia last month. That's why editors like us are the engine that keeps the project going.

Log in and see what’s new on your watchlist! We need your help to make the free encyclopedia even better.

See you on the wiki,

Steven Walling

Key elements:

• Link is to login, with a "return to" link to the watchlist
• Starts with a thank you
• Reminds editors of the stats on how many edits are made each month to English

1 month lapse group (round 4)

Using the same email subject line and body text as the previous round, we emailed our control group at a different time in order to measure the importance of time of day an email is sent. The available literature suggests that emailing people within your timezone in the morning is preferable to the afternoon times we usually sent the emails (from 2-4pm PDT).

Technical requirements

• Wikimedia's installation of contact CiviCRM is used to send all emails and facilitates tracking of open rates. (The emails will be sent using individual email addresses and replies will go to a human being.)
• Emails were batch imported into CiviCRM via .csv file
• UDP filters are used to see how many people log in after clicking the decorated link in the email Update: since filters were throttled to tracking 1 in 10 clickthroughs and the sample size is very small, we decided to not use these.
• Redundancy with CiviCRM click-through tracking allows us to verify results

Results

Round 1: 3-month and 1-year

Emails were sent on April 12th, 2012 at 2:06 PM PDT. Results as of April 19, 2012 at 3:25 PM PDT were...

• 3 month email group
  • 186 emails sent
  • 47 opened the email (25%)
  • 14 click throughs to the login page (29% of opens)
  • 5 users edited (2.7%)
  • 23 total edits

• 3 month control group
  • 2 of 194 users edited (1%)
  • 646 total edits

• 1 year test group
  • 95 emails sent
  • 23 opens (24%)
  • 10 click throughs (43% of opens)
  • 0 editors
  • 0 edits

• 1 year control group
  • 2 of 94 editors (2%)
  • 3 edits

Though low, these results compare quite favorably compared to industry standards and some fundraiser emails from Wikimedia.

Round 2: 3-month only

Emails were sent April 23, 2012 at 4:04 PDT. As the 1 year lapse group's rate of return (as measured by edits) was pretty dismal, we opted not to include them in a second round. Our main hypothesis when sending a second round to the controls in the previous test was the effect of a change in subject line.

Results as of one week later (April 30, 2012):

• Total recipients: 191
• Open rate: 51 (27%)
• Clickthrough: 16 (31% of opens)
• Editors: 9 (5% of total)
• Number of edits after: 372

Note: 4 additional editors from this group began editing again in the time between our generating the sample and sending the email (they are not included in the "editors" group above).

Round 3: 1-month only

Emails were sent the afternoon (PDT) of April 26th.

• Total recipients: 251
• Open rate: 74
• Clickthrough: 21
• Editors: 28 (11% of total sample)
• Number of edits: 637

A separate control group (with a sample size of 287) that was not emailed had a return rate of:

• Editors: 26 (9% of total sample)
• Number of edits: 194

Note: the increased edit count in the group that was emailed was mostly due to two editors, who each contributed 175 and 222 edits.

Round 4: 1-month only

Emails were sent 9am (PDT) on May 8.

• Total recipients: 216
• Open rate: 51 (24%)
• Clickthrough: 20
• Editors: 2
• Number of edits: 11

55 editors from this group returned and made edits in May on their own before we sent out the email. We see a few factors that are likely contributing to the lack of success in this round:

1. Existing literature recommends emailing people in the morning in their timezone. Since we do not know the timezones of the people in our sample, we may have been emailing them at any time.
2. Unlike traditional non-profit campaigns or Wikimedia Foundation email to donors, editors are participating in a leisure time hobby for the most part. Even for people in the same time zone as us, the morning of a working day is likely an inappropriate time to contact them about Wikipedia editing.
3. We do not know what the timezone distribution of this group compared to the others is.

Research questions and answers:

Will lapsed editors open an email from Wikipedia/Wikimedia staff?

About 25% will open an email from WMF staff. There was no discernable difference between staffer names or gender.

Will lapsed editors log in again if we ask them to?

7% of 3 month lapsed and 11% of 1 year lapse clicked through to the login page.

Will lapsed editors edit again if we ask them to?

1 out of 281 editors edited again. Generally, no.

What is the change in post-email edit rate for control vs. test groups?

too few of both the control and test group editors returned and made contributions to be able to declare any kind of statistical significance.

Is there a measurable difference between editors who recently left and who left a year or more ago?

No edits came from editors who'd lapsed a year ago. We presume that they will never come back. A single edit came from the 3 month group. Not a statistically significant difference.

How many email addresses from lapsed editors are junk?

Unknown. Tracking tools broke down and were unable to track bounces. Civi also did not properly track unsubscribes; we do not know if anyone tried to unsubscribe

Will anyone reply to our emails?

Yes, we received emails with specific questions and concerns about Wikipedia editing (not about our emails themselves) from every test group.

When do people mostly take action on email we send them?

The bulk of email opens and click-through occurred in the first 12 hours after the email was sent; almost no traffic was seen after 3 days

Comparison of tests

Editors we tried to resurrect

Editor Group	1 year Lapse	3 month lapse	1 month lapse
Editors emailed (test 1)	95	186	251
Editors emailed (test 2)	--	191	216
Control group	94	194	287
Total editors lapsed	189	380	538

Edit/Editor Counts and Percentages

Time of Lapse	1 year	3 months	1 month
# of editors	189	380	538
Return to edit rate: control vs. test	2% vs. 0%	1% vs. 5%	9% vs. 11%
Returning editors: control vs. test	2 vs. 0	2 vs. 9	26 vs. 28

Other ideas

• Changing how we send: if form letter style emails are not effective, more personalized approaches may work, such as...
  • Recreating the missing Wikipedians page as a dynamic list
  • Generating a social graph of lapsed editors (via talk page edits or article collaboration) and asking editors who interacted with them to author emails
  • Asking OTRS volunteers or other active editors to sign up to email a small, curated list of lapsed editors
  • Providing personalized metrics about the activity to articles a Wikipedian worked on substantially after they have left (e.g. "There have been 500 edits recently to articles you wrote. Would you like log in and review them?") [requires productization]
  • Leveraging the upcoming notification system to automate these types of emails for editors that are opted-in (or not opted out)

Conclusions

• Editors who have lapsed for one month are not worth emailing; the test group and the control group didn't show significantly different behavior. They may only be on Wikibreaks.
• Editors who have lapsed for one year are unlikely to be enticed by an email to return to editing. They may be good candidates for another experiment or approach.
• Editors who have lapsed for three months seem 5-6% likelier to return to editing than our control group. Should we productize, we believe that 3 month lapsed editors are a good target group. For example, if we can entice 4% of the nearly 400 editors who lapse in a three month period to return to ENWP, over one year that could mean:
  • We retain approximately 64 active editors
  • If these people returned to their previous edit rate of at least 100 edits over 3 months, we could add around 25,600 contributions to articles!

(This is very basic math that just multiplies what we saw in three months over a 1 year period. Doesn't account for seasonal fluctuations or other edit count influencers.)

Do we productize?

Maybe.

• In the short term, we can continue to pull numbers for the 3 month group every 3 months and manually send email inviting editors to return.
• In the long term, we should ensure that the overhaul of the notification system can accommodate sending and opt in/out of lapse notification emails.