Research:Donor engagement/Thank You campaign

Overview

edit

The campaign ran 3 different pairs of banners tested concurrently. During each test, visitors randomly received one of the two banners up to a maximum of 5 impressions, after which the banner was automatically disabled. Visitors also had the ability to manually disable banners via the close button. Each banner included a direct link to the account registration landing page as well as thumbnails pointing to a series of landing pages with editor stories, which in turn pointed to the account creation page (indirect link). Referrals from the direct link where logged with a fr12b-{banner_id} campaign id, while referrals from the indirect links embedded in editor stories where logged with a fr12c-{page} campaign. The campaign was launched at 2012-12-27T21:15:00Z and ended approximately at 2013-01-01T00:10:00Z.

Banners

edit
2012-12-27
B12_1227_ThankYou_5pillars
B12_1227_ThankYou_Short
2012-12-28
B12_1228_ThankYou_Nonprofit
B12_1228_ThankYou_Thrilled
2012-12-31
B12_1231_ThankYou_Createacct
[1]

Results

edit

How many new users registered as a result of this campaign?

edit

The campaign resulted in a total of 1,107 successful account registrations referred from the banners. The vast majority of these registrations (N=1,057, 95.4%) were referred via the direct link, while only a tiny fraction originated from the landing pages (N=50, 4.5%) . During the period of the campaign (as defined by the time of the first and the last account registration referred by the banners), a total of 20,958 users registered an account via the regular Create Account link, in line with the expected signup rate for this period of the year.

How productive are these users compared to organic registrations?

edit
 
Proportion of users with 1+ ns0 edits within 24 hours of registration in the two groups, plotted with 95% confidence intervals

To compare the productivity of the two groups of referred and organic registration we looked at the proportion of users in each group who completed at least 1 edit in the main article namespace within 24 hours of registrations. A total of 183 users (17.3%) from the referred group completed 1+ main namespace edits in 24 hours compared to 4,633 users (22.1%) from the control group of organic registrations (+4.8%). A chi-square test rejects the null hypothesis that the true proportion of editing accounts is the same in both groups (X² = 13.24, N = 22,015, p < .001). Referred users were significantly less likely to complete an edit than the control.

Which banner had the highest conversion rate?

edit

Banners had fairly low conversion rates (account creations by raw account impressions), with mean conversions ranging from 0.00037% to 0.00056%. We compared the raw clickthrough rate, completion and conversion rate for each pair of banners that ran concurrently (we didn't include the last pair of banners (B12_1231_*) as they ran for a substantially shorter amount of time compared to the previous ones). The plots below compare the campaigns against these three metrics.

Conversion rates for each banner campaign (with binomial proportion CI)

      Estimate        Lower        Upper  Campaign
1 0.0005489544 0.0004825809 0.0006244566 B12_1227_ThankYou_5pillars
2 0.0005655095 0.0004927165 0.0006490568    B12_1227_ThankYou_Short

      Estimate        Lower        Upper  Campaign
1 0.0004575854 0.0004052429 0.0005166885 B12_1228_ThankYou_Nonprofit
2 0.0003737361 0.0003332572 0.0004191317  B12_1228_ThankYou_Thrilled

Conclusions

edit

Funnel conversion data

edit

The following data was collected at the end of the test (events logs imported as of 2013-01-02T23:30:00Z, raw event counts, non-deduplicated).

edit

(1/100 sampled impressions)

420800	B12_1227_ThankYou_5pillars
357200	B12_1227_ThankYou_Short
568200	B12_1228_ThankYou_Nonprofit
781300	B12_1228_ThankYou_Thrilled
76300	B12_1231_ThankYou_Createacct
23200	B12_1231_ThankYou_Getstrtd

Account creation: impressions

edit

(excluding error impressions, i.e. form impressions logged when an error is returned after submitting the form)

edit
2298 fr12b-B12_1228_ThankYou_Thrilled
2145 fr12b-B12_1228_ThankYou_Nonprofit
1517 fr12b-B12_1227_ThankYou_Short
1332 fr12b-B12_1227_ThankYou_5pillars
 623 fr12b-B12_1231_ThankYou_Getstrtd
 288 fr12b-B12_1231_ThankYou_Createacct

Landing page

edit
 110 fr12c-Poongothai-Balasubramanian
  92 fr12c-Thank-You
  84 fr12c-Ken-Thomas
  73 fr12c-Mei-Jiun
  57 fr12c-Jimmy-Wales
  55 fr12c-Ruben-Hilare-Quispe
  53 fr12c-Oarabile-Mudongo
  15 fr12c-Abhishek-Hingnikar
  12 fr12c-Melisa-Parisi
  12 fr12c-Benjamin-Chen
   8 fr12c-Peter-Gallert
   7 fr12c-Ravan-Jaafar
   6 fr12c-Zinaida-Tebaykina
   6 fr12c-Peter-Coti
   6 fr12c-Noopur-Raval
   6 fr12c-Dumisani-Ndubane
   6 fr12c-Aliona-Bogdanova
   5 fr12c-Q-Miceli
   5 fr12c-Netha-Hussain
   4 fr12c-Viswa-Prabha
   4 fr12c-Elena-Strelnikova
   4 fr12c-Anastasia-Lvova
   4 fr12c-Amina-Duale
   3 fr12c-Susanna-Mkrtchyan
   3 fr12c-Ganesh-Paudel
   3 fr12c-David-Shankbone
   3 fr12c-Andrea-Zanni
   2 fr12c-Tony-Santiago
   2 fr12c-Judy-Tuan
   2 fr12c-Juan-Ignacio
   2 fr12c-Ivan-Meetin
   2 fr12c-Howard-Morland
   2 fr12c-Chanitra-Bishop
   2 fr12c-Benoit-Rochon
   2 fr12c-Amit-Kapoor
   2 fr12c-Alejandro-Linares
   1 fr12c-Srikeit-Tadepalli
   1 fr12c-Mina-Nagy
   1 fr12c-Leslie-Carr
   1 fr12c-Gideon-Digby
   1 fr12c-Gereon-Kalkuhl
   1 fr12c-Emmanuel-Engelhart
   1 fr12c-Deryck-Chan
   1 fr12c-David-Thomsen
   1 fr12c-Brandon-Harris
   1 fr12c-Adrianne-Wadewitz

Account creation: error impressions

edit

(form impressions logged when an error is returned after submitting the form)

edit
1735 fr12b-B12_1228_ThankYou_Thrilled
1314 fr12b-B12_1228_ThankYou_Nonprofit
 981 fr12b-B12_1227_ThankYou_5pillars
 900 fr12b-B12_1227_ThankYou_Short
 239 fr12b-B12_1231_ThankYou_Getstrtd
 164 fr12b-B12_1231_ThankYou_Createacct

Landing page

edit
 52 fr12c-Poongothai-Balasubramanian
  40 fr12c-Oarabile-Mudongo
  31 fr12c-Thank-You
  31 fr12c-Amina-Duale
  25 fr12c-Ken-Thomas
  19 fr12c-Ruben-Hilare-Quispe
  17 fr12c-Jimmy-Wales
  15 fr12c-Abhishek-Hingnikar
  11 fr12c-Mei-Jiun
   4 fr12c-Ganesh-Paudel
   4 fr12c-Deryck-Chan
   3 fr12c-Viswa-Prabha
   3 fr12c-Peter-Gallert
   3 fr12c-Dumisani-Ndubane
   3 fr12c-Amit-Kapoor
   2 fr12c-Zinaida-Tebaykina
   2 fr12c-Ravan-Jaafar
   2 fr12c-Noopur-Raval
   2 fr12c-Benjamin-Chen
   2 fr12c-Adrianne-Wadewitz
   1 fr12c-Elena-Strelnikova
   1 fr12c-Andrea-Zanni

Account creation: submits

edit

(excluding multiple submit attempts after the form returns an error)

edit
 814 fr12b-B12_1228_ThankYou_Thrilled
 765 fr12b-B12_1228_ThankYou_Nonprofit
 590 fr12b-B12_1227_ThankYou_Short
 557 fr12b-B12_1227_ThankYou_5pillars
 117 fr12b-B12_1231_ThankYou_Getstrtd
  91 fr12b-B12_1231_ThankYou_Createacct

Landing page

edit
  35 fr12c-Poongothai-Balasubramanian
  19 fr12c-Thank-You
  17 fr12c-Ken-Thomas
  12 fr12c-Jimmy-Wales
  10 fr12c-Ruben-Hilare-Quispe
   9 fr12c-Oarabile-Mudongo
   7 fr12c-Mei-Jiun-Kwek
   5 fr12c-Abhishek-Hingnikar
   3 fr12c-Zinaida-Tebaykina
   3 fr12c-Amina-Duale
   2 fr12c-Viswa-Prabha
   2 fr12c-Peter-Gallert
   2 fr12c-Ganesh-Paudel
   2 fr12c-Dumisani-Ndubane
   2 fr12c-Benjamin-Chen
   1 fr12c-Tony-Santiago
   1 fr12c-Ravan-Jaafar-Altaie
   1 fr12c-Peter-Coti
   1 fr12c-Noopur-Raval
   1 fr12c-Juan-Ignacio-Iglesias
   1 fr12c-Howard-Morland
   1 fr12c-Elena-Strelnikova
   1 fr12c-Deryck-Chan
   1 fr12c-Andrea-Zanni
   1 fr12c-Anastasia-Lvova
   1 fr12c-Amit-Kapoor
   1 fr12c-Aliona-Bogdanova
   1 fr12c-Adrianne-Wadewitz

Account creation: error submits

edit

(submit attempts after the form returned at least one error)

edit
1035 fr12b-B12_1228_ThankYou_Thrilled
 875 fr12b-B12_1228_ThankYou_Nonprofit
 725 fr12b-B12_1227_ThankYou_5pillars
 595 fr12b-B12_1227_ThankYou_Short
 165 fr12b-B12_1231_ThankYou_Getstrtd
 119 fr12b-B12_1231_ThankYou_Createacct

Landing page

edit
  31 fr12c-Poongothai-Balasubramanian
  31 fr12c-Amina-Duale
  29 fr12c-Oarabile-Mudongo
  18 fr12c-Thank-You
  15 fr12c-Jimmy-Wales
  11 fr12c-Ken-Thomas
  11 fr12c-Abhishek-Hingnikar
   9 fr12c-Ruben-Hilare-Quispe
   5 fr12c-Mei-Jiun-Kwek
   3 fr12c-Viswa-Prabha
   3 fr12c-Thank-You
   3 fr12c-Ganesh-Paudel
   3 fr12c-Deryck-Chan
   3 fr12c-Amit-Kapoor
   2 fr12c-Poongothai
   2 fr12c-Peter-Gallert
   2 fr12c-Noopur-Raval
   2 fr12c-Benjamin-Chen
   2 fr12c-Adrianne-Wadewitz
   1 fr12c-Zinaida-Tebaykina
   1 fr12c-Ravan-Jaafar-Altaie
   1 fr12c-Dumisani-Ndubane

Account creation: success

edit
edit
 292 fr12b-B12_1228_ThankYou_Thrilled
 260 fr12b-B12_1228_ThankYou_Nonprofit
 231 fr12b-B12_1227_ThankYou_5pillars
 202 fr12b-B12_1227_ThankYou_Short
  38 fr12b-B12_1231_ThankYou_Getstrtd
  33 fr12b-B12_1231_ThankYou_Createacct

Landing page

edit
   9 fr12c-Poongothai-Balasubramanian
   7 fr12c-Jimmy-Wales
   6 fr12c-Thank-You
   4 fr12c-Oarabile-Mudongo
   4 fr12c-Ken-Thomas
   3 fr12c-Amina-Duale
   2 fr12c-Viswa-Prabha
   2 fr12c-Ruben-Hilare-Quispe
   2 fr12c-Mei-Jiun-Kwek
   2 fr12c-Abhishek-Hingnikar
   1 fr12c-Zinaida-Tebaykina
   1 fr12c-Ravan-Jaafar-Altaie
   1 fr12c-Peter-Gallert
   1 fr12c-Noopur-Raval
   1 fr12c-Ganesh-Paudel
   1 fr12c-Deryck-Chan
   1 fr12c-Benjamin-Chen
   1 fr12c-Amit-Kapoor
   1 fr12c-Adrianne-Wadewitz

Raw signup completion rates

edit
Referral Raw account creation impressions Account creations Raw completion rate
fr12b-B12_1227_ThankYou_Short 1517 202 13.3%
fr12b-B12_1227_ThankYou_5pillars 1332 231 17.3%
fr12b-B12_1228_ThankYou_Nonprofit 2145 260 12.1%
fr12b-B12_1228_ThankYou_Thrilled 2298 292 12.7%
fr12b-B12_1231_ThankYou_Getstrtd 623 38 6.1%
fr12b-B12_1231_ThankYou_Createacct 288 33 11.4%
fr12c (landing pages) 674 50 7.4%

Notes

edit

This data is interesting to compare to the results of our previous account creation experiments, where conversion rates were between 28-32% for both the control and test versions of a new account creation interface. It should not, however, be necessarily used to decide which of the two banners was more successful. To do so, the conversion rate would be account creations over total banner and landing page impressions.