Fundraising/Chapters/WMDE 2013

This page is for discussion and informational purposes about fundraising tests from Wikimedia Germany. Discussion should occur on the discussion page. If you want to leave a note in German, the German portal site might be a better place.

Tests on de.wikipedia.org

Test 1: facts banner: appeal vs. no appeal

Tuesday, 5.8.2013, 10:00 UTC - Friday, 9.8. 2013, 9:00 UTC (only 1% of page views)

Last year we saw that the success of the facts banner is not much affected by changes of the content inside the drop down. Because of that finding, WMF has reduced additional content inside the drop down facts banner. So to start this years campaign prepaperation we tested two versions of the facts banner: one with the well performing appeal of Dr. Cueppers inside the drop down against one with solely the donation form inside. The results suggest that the text of the facts banner (before clicking) is the key element in convincing readers to donate: Both banner versions perform almost on the same level.

Banner	Landing-Page	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
with_appeal	5f	1082000	18500	1113	828	14190.77	17.14	0.017098	0.001029	0.06016	0.0007652	0.04476	0.74394	0.0131153	19.34
form_only	5f	1055500	18485	1184	865	14185.39	16.40	0.017513	0.001122	0.06405	0.0008195	0.04679	0.73057	0.0134395	18.31

• donation/click: +4,55% (p-value = 0.3704)

--Tobias Schumann (WMDE) (talk) 10:42, 21 August 2013 (UTC)

Test 2: facts banner vs. 5€ banner

Tuesday, 27.8.2013, 14:15 UTC - Thursday, 29.8.2013, 17:00 UTC

Sometimes less is more. Last year's best performing banner only featured one sentence: the 5€ slogan. Yet, it was not clear to us if the reason for the success was related to the end time of the fundraising campaign when readers assumingly already had recognized the facts banners with it's important information. So this year we wanted to check the performance of the 5€ banner right at the beginning of our campaign testing. The results show us that the success of the banner is not due to the point in time of the campaign. A 45% increase in donations/impressions clearly shows the power to convince readers to support Wikipedia. Still there is a difference to the tests in 2012. While this year the success of the 5€ banner lies in a profound increase of click rate in last year's campaign this banner type received far less clicks than the facts banner but a much higher percentage of productive clicks. Maybe this difference is related to point in time of the campaign.

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
facts banner	379600	9725	493	418	7668.52	18.35	0.025619	0.001299	0.05069	0.0011012	0.04298	0.84787	0.0202016	15.06
5€ banner	378300	15576	698	604	9250.32	15.32	0.041174	0.001845	0.04481	0.0015966	0.03878	0.86533	0.0244523	20.67

• click rate: p-value < 2.2e-16
• donations/imp: p-value = 4.382e-09
• donations/click: p-value = 0.1127

--Tobias Schumann (WMDE) (talk) 12:53, 3 September 2013 (UTC)

Test 3: dear readers

Wednesday, 4.9., 10:00 UTC - Saturday, 7.9., 12:00 UTC

Last year's test results revealed the power of directly adressing Wikipedia readers. In the current test we wanted to see if we could use this effect to rise our constantly low female donor rate (roughly about 20 % of all donors) and by that the donation rate in general. So we modified the form of adress to "Liebe Leserinnen und Leser" (impossible to translate that directly into English due to missing grammtical gender in nouns - it could be translated as: "Dear female and male readers" though). The results show us that this modification has no impact on donation rate nor female quota.

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	1448000	19912	963	852	16496.29	19.36	0.013751	0.000665	0.04836	0.0005884	0.04279	0.88474	0.0113925	20.24
var	1452000	19784	1097	886	18534.21	20.92	0.013625	0.000756	0.05545	0.0006102	0.04478	0.80766	0.0127646	21.76

• donations/imp: p-value = 0.4486

--Tobias Schumann (WMDE) (talk) 12:18, 12 September 2013 (UTC)

Test 4: 5€ banner: appeal vs. no appeal

Wednesday, 11.9., 9:45 - Monday, 16.9., 9:30 UTC

As in this year's first test here we looked at the effect of showing a personal appeal inside the drop - this time in combination with the 5€ banner. And again there was no real difference in donation rate with and without the personal appeal.

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
appeal	1435400	12427	770	632	9813.79	15.53	0.008658	0.000536	0.06196	0.0004403	0.05086	0.82078	0.0068370	18.31
no_appeal	1460500	12299	776	636	9345.00	14.69	0.008421	0.000531	0.06309	0.0004355	0.05171	0.81959	0.0063985	16.73

• donation/click: +1,7% (p-value = 0.7607)

--Tobias Schumann (WMDE) (talk) 09:49, 23 September 2013 (UTC)

Test 5: size of donation button

Thursday, 19.9., 12:00 am - Thursday 24.9., 9:00 am UTC

In the coming weeks we will focus more on concept and layout of the donation process. As a first step this test evaluated four different sizes of the donation button. According to the results it looks like changing the size doesn't affect the conversion rate.

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	1178200	14922	789	667	12969.00	19.44	0.012665	0.000670	0.05287	0.0005661	0.04470	0.84537	0.0110075	22.12
big	1183200	15215	811	675	13449.48	19.93	0.012859	0.000685	0.05330	0.0005705	0.04436	0.83231	0.0113670	22.73
bigger	1188200	14854	803	675	12896.59	19.11	0.012501	0.000676	0.05406	0.0005681	0.04544	0.84060	0.0108539	22.39
biggest	1175600	15031	806	674	12664.31	18.79	0.012786	0.000686	0.05362	0.0005733	0.04484	0.83623	0.0107726	25.90

Test 6: rollin' rollin' rollin'

Thursday, 26.9., 10:00 am - Friday, 27.9., 10:00 pm UTC

By evaluating last year's test results we have learned that one of the key elements for success is to win attention by readers. Many augmentations of the donation rate in 2012 were generated by modifications of banner layout focussing on this aspect. In this test we hoped to benefit from these findings trying a new layout where the banner is not simply displayed but rolls down kind of like a jalousie. The test results confirm our presumptions: the new layout increased the click rate profoundly and by that also the donation rate is raised by 22%.

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	848500	31211	1011	841	16243.62	19.31	0.036784	0.001192	0.03239	0.0009912	0.02695	0.83185	0.0191439	23.85
roll down	817500	35920	1175	988	19383.55	19.62	0.043939	0.001437	0.03271	0.0012086	0.02751	0.84085	0.0237108	21.71

• click rate: +15,1% (p-Wert < 2.2e-16)
• donation/imp: +21,9% (p-Wert = 2.275e-05)

Test 7, 8 and 9: small contribution, 1€, 5€ or 10€?

Throughout October we ran several tests focussing on the amount mentioned in our most succesful call to action "If everybody donated x €, ...". We wanted to know which amount works better: 1€, 5€, 10€ or the more open wording "small contribution". The concept behind this: with a smaller amount, we assume to lower the threshold for the support of Wikipedia so that more people decide to donate. On the other hand there exists the risk of reducing the average amount too much. Indeed, the test results shows that the average amount correlates with the mentioned amount in the banner text and the highest average results from the abstract term "small contribution". Also, the donation rate (numbers/impression) changes clearly depending on the banner text. With the "1€"-variation so many readers are being convinced to donate that the smallest average amount is far compensated.

test 7, Tuesday, 1.10., 2:30 pm - Monday, 7.10., 4:00 pm UTC

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
5€	2953100	20813	1413	1202	21055.43	17.52	0.007048	0.000478	0.06789	0.0004070	0.05775	0.85067	0.0071299	18.78
small contribution	2932200	22060	1052	832	16914.83	20.33	0.007523	0.000359	0.04769	0.0002837	0.03772	0.79087	0.0057686	21.38

test 8, Friday, 18.10., 10:15 am - Tuesday, 22.10., 2:00 pm UTC

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
5€	2182600	15738	1030	823	12660.66	15.38	0.007211	0.000472	0.06545	0.0003771	0.05229	0.79903	0.0058007	20.18
1€	2210700	18132	1438	1168	15061.13	12.89	0.008202	0.000650	0.07931	0.0005283	0.06442	0.81224	0.0068128	21.44

test 9, Wednesday, 23.10., 10:00 am . Tuesday, 29.10., 10:00 am UTC

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
1€	3203800	23320	1801	1403	18152.33	12.94	0.007279	0.000562	0.07723	0.0004379	0.06016	0.77901	0.0056659	20.81
10€	3262600	21032	1135	880	16387.12	18.62	0.006446	0.000348	0.05397	0.0002697	0.04184	0.77533	0.0050227	21.55

--Tobias Schumann (WMDE) (talk) 10:23, 4 November 2013 (UTC)

Test 10 und 11: form banner

In this two tests we tried the new WMF banner concept of integrating the donation form into the banner. At first we wanted to see the effect of the form as isolated as possible. So we modified a facts banner (drop) to adapt it graphically to the form banner. As expacted the click rate decreased strongly. On the other hand the donation rate increased dramatically by 98%. In a following test we ran the form banner against our current best performing 1€-banner. Still the increase of donation rate added up to 79%.

test 10

Thursday 30.10., 13:00 - Friday 01.11., 13:30 Uhr

Banner	Landing-Page	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
no_form	5F	1034000	10862	663	513	9733.50	18.97	0.010505	0.000641	0.06104	0.0004961	0.04723	0.77376	0.0094134	23.04
form	5F	1023700	0	1904	1006	18746.33	18.63	0.000000	0.001860	0.00000	0.0009827	0.00000	0.52836	0.0183123	14.56

test 11

Friday 01.11. 1:30 pm - Thursday 05.11. 3:00 pm

Banner	Landing-Page	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
1€_no_form	5F	2276500	17836	1466	1131	14962.16	13.23	0.007835	0.000644	0.08219	0.0004968	0.06341	0.77149	0.0065724	19.41
form	5F	2276400	0	4515	2027	38018.39	18.76	0.000000	0.001983	0.00000	0.0008904	0.00000	0.44895	0.0167011	20.66

• p-value donation/imp (chi-squared test) = 2,80648183729183E-57

Test 12: rolling banner II

Friday 08.11., 10:30 am - 5:20 pm

This time we simply retested banner display with the new roll-effect of test 6. The roll-effect seems to increase attenion on the banners by readers. To be sure about the results we tried it on the form-banner resluting in similiar results. The click rate is increased by 17% (test 6: +19%) and the donation rate by 12% (test 6: 22%).

Banner	Landing- Page	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
no_roll	5F	2396200	0	6640	3163	57011.12	18.02	0.000000	0.002771	0.00000	0.0013200	0.00000	0.47636	0.0237923	23.18
roll	5F	2368200	0	7697	3485	65047.40	18.66	0.000000	0.003250	0.00000	0.0014716	0.00000	0.45277	0.0274670	24.92

• statistical significance don/imp: chi-squared-test p=0,00000936

Test 13: blue link

To highlight the important "If everybody..." we retested a successful design of 2012: formatting the sentence as a blue link. In test 39 we saw an increased donation/click while the click rate didn't change. This lead us to the assumption that the whole banner text gets more convincing by highlighting this so well-proven sentence.

In our current retest we saw totally different results: an increased click-rate (+49%) but a decrease of donation/lp visit on a similiar scale (-33%) so that the overall donation rate didn't change. It looks like the formatting as a link has a totally different impact in the context of the new banner concept (donation form, no drop with personal appeal). Obviously the landing page doesn't match the expactions of people clicking on the link and perhaps looking for more information. So here we see big potential in adjusting the landing page.

Monday 11.11., 11:45 am - 7:00 pm

Banner	Landing- Page	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
no_link	5F	1043100	0	3189	1326	22973.51	17.33	0.003057	0.0012712	0.00000	0.41580	0.0220243	23.43
link	5F	1052400	0	4788	1332	24933.00	18.72	0.004550	0.0012657	0.00000	0.27820	0.0236916	24.69

--Tobias Schumann (WMDE) (talk) 14:12, 15 November 2013 (UTC)

Test 14: banner text form banner: facts vs. 1€

Tuesday 12.11., 4:45 pm - Wednesday 13.11., 09:45 am

From our second test from this year we know that reducing the banner text to the "If everybody..." sentence can be very powerful. So we needed do try this too on our current form banner. However in the banner design of test 2 the facts text was shown inside the drop and still before filling out the donation form. From our current test we know now that this is crucial. Reducing the banner text of the form banner to the "If everybody..." sentence strongly reduced it's succes: the donation rate decreased by ~30%. Combined with a smaller donation average the success of the banner fell even more.

Banner	Landing- Page	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
facts	5F	1195100	0	3835	1680	30328.84	18.05	0.000000	0.003209	0.00000	0.0014057	0.00000	0.43807	0.0253777	23.11
1€	5F	1202200	0	3094	1190	14973.45	12.58	0.000000	0.002574	0.00000	0.0009899	0.00000	0.38462	0.0124550	19.26

• significance donation/imp: p = 1,26892988247306E-20 (chi-squared-test)

--Tobias Schumann (WMDE) (talk) 14:24, 18 November 2013 (UTC)

Test 15, 16, 28, 30, 53: modifications facts text

Throughout the campaign we tested several changes to the banner text. At first we tried a new version of the facts text inspired by the current WMF text (test 15). This new text version performed significantly better (donation rate e.g. donation/imp: +20%).

In test 16 we again examined the difference between "1€" and "small contribution". As always "1€" results in an increase of donations but a decrease in average amount. Unlike previous tests of the "If everybody"-sentence it looks like within the facts the "small contribution" text works better than "1€".

To get a shorter banner text in test 28 we cut out the text part about the temple of mind. Our assumption was that a shorter text would have a higher chance to be read by Wikipedia users. In contrast the shorter version performed slightly worse in the overall donation rate (-6,7%).

In test 30 we added some aspects we expected by experience to be convincing. In fact the text changes improved the donation rate, but on such a small scale (+2,7%) that the results were not statistically significant.

An appeal gets more compelling when it is concrete and refers to a clear occasion or cause. Thats why on christmas we tried a banner text which asks the readers to give something back "on this christmas day" (test 53). That small modification improved the donation rate by around 13% - a very instructive success which gives a clear hint to communicate more focussed on occasions or causes in the next fundraiser.

test 15, Friday 15.11., 9:00 am - 1:00 pm

Banner	Landing Page	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	5F	1273900	1452	701	12162.50	17.35	0.001140	0.0005503	0.48278	0.0095475	24.50
text_new	5F	1282800	1667	866	18364.49	21.21	0.001300	0.0006751	0.51950	0.0143159	25.40

chi-squared-test donation/imp: p = 0,00005538

test 16, Friday 15.11., 4:00 pm - Sonntag 17.11., 5:30 pm

Banner	Landing Page	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
small_contrib	5F	8055800	19299	10180	207503.16	20.38	0.002396	0.0012637	0.52749	0.0257582	24.50
1€	5F	8065800	21927	11807	194619.86	16.48	0.002719	0.0014638	0.53847	0.0241290	23.96

chi-squared-test donation/imp: p = 0,00000000

test 28, Monday 02.12., 00:15 pm - Tuesday 03.12., 00:15 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	4182300	5391	2696	49653.99	18.42	0.001289	0.0006446	0.50009	0.0118724	29.08
no_park	4228700	5481	2652	49195.98	18.55	0.001296	0.0006271	0.48385	0.0116338	30.48

chi-squared-test donation/imp: p = 0,314689

test 30, Tuesday 05.12., 11 am - Friday 06.12., 5:30 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
text_ctrl	5221100	6015	2876	54755.30	19.04	0.001152	0.0005508	0.47814	0.0104873	28.41
text_var	5169400	6160	2925	57276.78	19.58	0.001192	0.0005658	0.47484	0.0110800	29.26

chi-squared-test donation/imp: p = 0,3065

test 53, Monday 23.12., 11 pm - Wednesday 25.12., 9 am

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	2042900	4126	2156	45632.07	21.17	0.002020	0.0010554	0.52254	0.0223369	19.59
xmas_textvar	2041600	4395	2439	51387.08	21.07	0.002153	0.0011947	0.55495	0.0251700	19.58

chi-squared-test donation/imp: p = 0,00003123

Test 17, 19, 20, 23, 41, 44, 55, 56: time delay

During the Fundraiser we put some effort to understand the effect of a time delay of the banner display. Why did we do that? With our banner communication we interupt the reader in his search for information. Wouldn't it be much better if the banner is displayed after the reader found what he was looking for? He would probably be less distracted by the banner and much more thankful so that his disposedness to donate would grow. As plausible this sounds as difficult this is to implement. Different information needs go along with different durations of reading. And the average retention period on Wikipedia probably differs with the time of the day and the weekdays. In the end we had no other choice than to test different time delay periods by trial and error. After the total of eight tests we now can say that a time delay between 7,5 and 15 seconds improves the donation rate significantly.

We tested these time intervals: 60, 30, 15 and 7,5 seconds. The results suggest that the click rate drops with growing time delay. On the other hand, the conversion rate on the landing page improves with time delay, but seems to stagnate at 15 seconds. This goes along with our assumptions about the impact of the time at which the banner appears. And it is reflected on the impact on the overall donation rate (donation/imp): While a longer time delay is not successful a delay between 7,5 and 15 seconds improves the donation rate by up to 14%. Because of the expected variation of the average retention period on the different weekdays we retested some time intervals plus ran intervals against each other. In the end a time delay of 7,5 seconds performs best and was implemented most time during the campaign.

Test 17, Monday 18.11., 9:00 pm - Tuesday 19.11., 8:00 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
no_delay	3926400	6226	2873	58464.06	20.35	0.001586	0.0007317	0.46145	0.0148900	28.76
delay_30s	4006000	6018	3090	61968.11	20.05	0.001502	0.0007713	0.51346	0.0154688	29.14

• chi-squared-test visitor/imp: p = 0,002766
• chi-squared-test donation/visitor: p = 0,000000
• chi-squared-test donation/imp: p = 0,041739

test 19 Friday 22.11., 11:45 am - 5:30 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
no_delay	1322800	1687	748	15063.53	20.14	0.001275	0.0005655	0.44339	0.0113876	25.33
delay_60	1399000	1308	643	13453.00	20.92	0.000935	0.0004596	0.49159	0.0096162	28.30

• chi-squared-test visitor/imp: p = 0,000000
• chi-squared-test donation/visitor: p = 0,008710
• chi-squared-test donation/imp: p = 0,000112

test 20 Friday 22.11., 5:30 pm - Sunday 24.11., 1:00 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
no_delay	5090000	8329	4088	83313.43	20.38	0.001636	0.0008031	0.49082	0.0163681	25.05
delay_15	4990400	8549	4570	90724.19	19.85	0.001713	0.0009158	0.53457	0.0181797	24.90

• chi-squared-test visitor/imp: p = 0,002885
• chi-squared-test donation/visitor: p = 0,000000
• chi-squared-test donation/imp: p = 0,000000

test 23 Thursday 28.11., 7:00 pm - Friday 29.11., 10:30 am

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
no_delay	1593500	1982	841	17522.50	20.84	0.001244	0.0005278	0.42432	0.0109962	27.12
delay_15	1602300	1961	964	20222.34	20.98	0.001224	0.0006016	0.49159	0.0126208	27.50

• chi-squared-test visitor/imp: p = 0,611696
• chi-squared-test donation/visitor: p = 0,000022
• chi-squared-test donation/imp: p = 0,005454

test 41 , Sunday 15.12., 1:45 pm - Monday 16.12., 1:15 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
delay_15	3564200	4361	2062	43676.14	21.18	0.001224	0.0005785	0.47283	0.0122541	25.12
delay_7,5	3529100	4809	2233	45783.13	20.50	0.001363	0.0006327	0.46434	0.0129730	25.69

• chi-squared-test visitor/imp: p = 0,000000
• chi-squared-test donation/visitor: p = 0,415859
• chi-squared-test donation/imp: p = 0,003341

test 44 , Monday 16.12., 10 pm - Tuesday 17.12., 4:30 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
no_delay	2474800	2840	1157	24763.58	21.40	0.001148	0.0004675	0.40739	0.0100063	27.41
delay_7,5	2385900	2513	1106	22826.46	20.64	0.001053	0.0004636	0.44011	0.0095672	25.34

• chi-squared-test visitor/imp: p = 0,001727
• chi-squared-test donation/visitor: p = 0,015594
• chi-squared-test donation/imp: p = 0,839830

test 55, Saturday 28.12., 2:45 pm - Sunday 29.12., 6 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
no_delay	3356500	6884	3873	82483.04	21.30	0.002051	0.0011539	0.56261	0.0245741	23.45
delay_7,5	3318700	7114	4260	88047.13	20.67	0.002144	0.0012836	0.59882	0.0265306	23.98

• chi-squared-test visitor/imp: p = 0,008877
• chi-squared-test donation/visitor: p = 0,000014
• chi-squared-test donation/imp: p = 0,000002

test 56, Sunday 29.12., 6 pm - Monday 30.12., 11 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
delay_15	3771900	6877	4290	90130.42	21.01	0.001823	0.0011374	0.62382	0.0238952	22.15
delay_7,5	3761100	7304	4344	91444.13	21.05	0.001942	0.0011550	0.59474	0.0243131	21.55

• chi-squared-test visitor/imp: p = 0,000170
• chi-squared-test donation/visitor: p = 0,000391
• chi-squared-test donation/imp: p = 0,474747

Test 21, 31, 52: new landing page layout

During each fundraiser one of our main concerns is the drop out rate on the landing page. Even now with the donation form in the banner, still about 50% of the people filling out the form break up the process on the landing page and evenutally don't donate. That's kind of strange. After all, filling out the form on the banner indicates a clear intention to donate.

In this light the approach for the 2013 fundraiser included a completely new design and relaunch of the landing page featuring well-known factors supporting online payment processes. For instance the layout should be clean and clear structured, the payment process should feature positive feedback (e.g. green check marks after filling out a form element) and informations should be provided right where they are needed. Being curious about of the outcome, we started to test the new landing page layout. A first test with a slightly negative result was not trustworthy because afterwards it appeared that a technical error occured on the new landing page (test 21). A second test afterwards featured a new designed donation form in the banner to provide more continuity in design to the landing page (test 31). This produced somewhat strange results: a 15% decrease in click rate and a 13% increase in landing page conversion with a the final effect that both test groups performed more or less equal in donation rate. We don't have an explanation for the decrease in click rate plus this test didn't bring more clarity about the impact of the new landing page layout. Finally, a third and fourth test on wikipedia.org and wikipedia.de over long testing periods brought certainty: the new landing page layout indeed decreases the drop out rate. An increase of landing page conversion of 6% and 5% is not so much as expected but still is a continous improvement.

test 21 , Wednesday 27.11., 1:00 pm - Thursday 28.11., 12:00 am

Banner	Landing-Page	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	layout_old	3513000	4036	1897	37924.00	19.99	0.001149	0.0005400	0.47002	0.0107953	29.05
ctrl	layout_new	3484600	4040	1851	37210.64	20.10	0.001159	0.0005312	0.45817	0.0106786	26.81

chi-squared-test donation/visitor: p = 0,285606

test 31 , Friday 06.12., 5:30 pm - Monday 09.12., 11:30 am

Banner	Landing-Page	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
form_old	lp_old	8482300	11442	5747	112457.79	19.57	0.001349	0.0006775	0.50227	0.0132579	26.39
form_new	lp_new	8314500	9501	5392	108190.45	20.06	0.001143	0.0006485	0.56752	0.0130123	25.41

chi-squared-test donation/visitor: p = 0,000000

Test 52, Monday 23.12., 9 am - Friday 27.12., 00:15 pm

Banner	Landing-Page	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	layout_old	6713400	13451	7437	157678.19	21.20	0.002004	0.0011078	0.55290	0.0234871	24.25
ctrl	layout_new	6773400	13080	7662	160878.99	21.00	0.001931	0.0011312	0.58578	0.0237516	18.56

chi-squared-test donation/visitor: p = 0,00000006

Test 22, 34, 35, 40, 42: form banner <--> facts_drop/Jimmy_drop/5€_drop

The integration of the donation form into the fundraising banners brought a huge boost in the number of donations. But we couldn't be sure that this banner concept, which gives relatively few informations, would work best over the whole course of the annual campaign. So throughout the fundraiser, we repeatedly tested the form banner against drop down banners which provide more information and a donation appeal. We used different drop down banners which featured the facts text, the "if everybodey would donate 5€,..." and the Jimmy appeal. But at no time the drop down banners outperformed the facts banner. Only the Jimmy appeal came close while the facts banner still had an increase in donation rate by +2%.

test 22 , Thursday 28.11., 12:00 am - 7:00 pm

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
facts_drop	1702100	16669	849	695	14382.10	20.69	0.009793	0.000499	0.05093	0.0004083	0.04169	0.81861	0.0084496	30.77
facts_form	1678600	0	1794	821	16807.50	20.47	0.000000	0.001069	0.00000	0.0004891	0.00000	0.45764	0.0100128	27.31

chi-squared-test donation/impression: p = 0,000452

test 34 , Tuesday 10.12., 4:00 pm - Wednesday 11.12., 9:00 am

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
facts_drop	2200900	20954	1302	1039	20584.55	19.81	0.009521	0.000592	0.06214	0.0004721	0.04958	0.79800	0.0093528	31.63
facts_form	2132200	0	2432	1140	23216.45	20.37	0.000000	0.001141	0.00000	0.0005347	0.00000	0.46875	0.0108885	31.32

chi-squared-test donation/impression: p = 0,003674

test 35 , Wednesday 11.12., 11:30 am - Wednesday 11.12., 5:00 pm

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
pa_Jimmy_drop	1452200	35205	751	569	11637.00	20.45	0.024243	0.000517	0.02133	0.0003918	0.01616	0.75766	0.0080134	35.42
facts_form	1420300	0	1349	567	11454.00	20.20	0.000000	0.000950	0.00000	0.0003992	0.00000	0.42031	0.0080645	30.40

chi-squared-test donation/impression: p = 0,752729

test 40 , Friday 13.12., 5:45 pm - Sunday 15.12., 00:00 am

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
5€_drop	4026900	70876	3517	2878	49334.77	17.14	0.017601	0.000873	0.04962	0.0007147	0.04061	0.81831	0.0122513	26.42
facts_form	3263200	0	5113	2581	49811.50	19.30	0.000000	0.001567	0.00000	0.0007909	0.00000	0.50479	0.0152646	24.87

chi-squared-test donation/imp: p = 0,000182

test 42 , Monday 16.12., 1:15 pm - 10:00 pm

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
5€_drop	2257200	42680	1444	1128	20286.29	17.98	0.018908	0.000640	0.03383	0.0004997	0.02643	0.78116	0.0089874	24.95
facts_form	1926400	0	2379	1053	22332.01	21.21	0.000000	0.001235	0.00000	0.0005466	0.00000	0.44262	0.0115926	26.10

chi-squared-test donation/imp: p = 0,036274

Test 24 and 38: banner layout: text highlighting

Very often we could see in our tests how compelling the "If everybody" sentence is. For higher visibility of the sentence we tested the yellow text highlighting from the WMF banner layout. This was really successful: The donation rate improved by 46%. Thats why we wanted to use this effect also to highlight our bank account in the banner. We know from surveys that there are plenty of Wikipedia readers which want to donate but don't trust online payment methods in generell. During every fundraiser we receive tons of emails where readers ask for our bank account number to donate by bank transfer. That's why we had put these informations on our banners (see test 29), but even after that people still asked in emails for them. So we decided to highlight them. In test 38 we only wanted to see if this layout modification has an effect on the donation rate, but it hasn't much.

test 24, Friday 29.11., 12:00 am - 6:45 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
no_highlight	1320200	1461	733	15002.00	20.47	0.001107	0.0005552	0.50171	0.0113634	25.98
highlight	1288100	1894	1047	21336.95	20.38	0.001470	0.0008128	0.55280	0.0165647	24.62

chi-squared-test donation/imp: p = 0,00000000

test 38, Tuesday 12.12., 7:00 pm - Friday 13.12., 1:30 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
no_highlight	2105300	3183	1510	29505.31	19.54	0.001512	0.0007172	0.47440	0.0140148	27.17
highlight	2084700	2956	1403	26351.69	18.78	0.001418	0.0006730	0.47463	0.0126405	27.07

chi-squared-test donation/imp: p = 0,08583915

Test 25: salutation in capital letters

Friday 29.11., 6:45 pm - Saturday 30.11., 5:00 pm

For the purpose of getting more attention on the important salutation we tried a version in capital letters following the layout of WMF. Unfortunately, it did not effect the conversion rates positively.

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	2250500	3848	2130	39670.52	18.62	0.001710	0.0009465	0.55353	0.0176274	23.60
caps	2254300	3873	2137	40256.00	18.84	0.001718	0.0009480	0.55177	0.0178574	24.13

chi-suared-test donation/imp: p=0,95846222

Test 26, 27, 29: hover element

With the payment process starting directly in the banner, the form banner provides relatively few information before the decision to donate. We expected at least some people would like to have more information before donating. This is why we thought about a way to integrate some additional information into the form banner. So the hover element came to our mind. In test 26, we changed the content of this lower part of the banner, only visible when one moves with the mouse over it. In the space where prior only our account number had been shown, we now informed about tax deductibility, provided links to information about spending and about contact possibilities plus gave yet another payment possibility: 5€ via text message. This resulted in a certain amount of clicks on the additional information links so that the click rate of the test group improved by about 17%. However, only few of these eventually donated. Thus, the overall donation rate didn't change in test.

By changing the background colour of the hover element to blue, we aimed to put more attention to it (test 27). This did not really change the test results compared to the previous test. Because of this we started to wonder if the hover element would have any effect on reader's attention. Before, we had thought that the hover element supported the visibility of the banner in general. So now we wanted to evaluate this by testing banners with a hover against one without (test 29). An advantage of leaving the hover away was that the bank account now became visible independently of any mouse interaction. Also this test did not result in a change of donation rate. For the rest of the campaign we continued with non-hover banners beacause of the improved visibility of the account numbers.

test 26 , Saturday 30.11., 7:15 pm - Sunday 01.12., 5:30 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
hover_ctrl	2979400	4939	2699	50797.57	18.82	0.001658	0.0009059	0.54647	0.0170496	25.35
hover_var	2946200	4665	2525	49549.99	19.62	0.001583	0.0008570	0.54126	0.0168183	24.77
info_links	-	1023	72	1472.01	20.44	0.000000	0.0000000	0.07038	0.0000000	27.42

chi-squared-test donation/imp: p = 0,320062

test 27 , Sunday 01.12., 5:30 pm - Monday 02.12., 00:15 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
hover_ctrl	2754100	3760	1901	37246.49	19.59	0.001365	0.0006902	0.50559	0.0135240	29.11
hover_blue	2754900	3736	1873	35349.27	18.87	0.001356	0.0006799	0.50134	0.0128314	28.64
info-links	-	460	27	1000.00	37.04	0.000000	0.0000000	0.05870	0.0000000	23.81

chi-squared-test donation/imp: p = 0,992855

test 29 , Wednesday 04.12., 3:15 pm - Thursday 05.12., 11:00 am

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl_hover	2808400	3491	1777	34244.78	19.27	0.001243	0.0006327	0.50902	0.0121937	29.84
bnk_nohvover	2786500	3541	1748	34222.71	19.58	0.001271	0.0006273	0.49365	0.0122816	29.72

chi-squared-test donation/imp: p = 0,797848

Test 32: sticky on lower side of screen

Monday 09.12., 11:30 am - Monday 09.12., 5:00 pm

In the hope of gaining more attention of Wikipedia readers, we tested a modified position of the banner on the screen. Assuming that users attention while reading would focus more on the on the lower part of the monitor, we tried a position on the lower edge of the screen. However, the results suppose the opposite: the test group with the banner display on the lower edge of the screen had a 12% lower click rate. The overall donation rate decreased by 10%.

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
sticky_top	1539100	1562	700	12760.50	18.23	0.001015	0.0004548	0.44814	0.0082909	30.45
sticky_bottom	1428100	1277	585	10972.30	18.76	0.000894	0.0004096	0.45810	0.0076831	26.97

chi-squared-test visitors/imp: p = 0,0007806 chi-squared-test donations/imp: p = 0,061650

Test 33: donor comments on landing page

Monday 09.12., 10:00 pm - Tuesday 10.12., 4:00 pm

We still wonder why about 50% of the people filling out the form in the banner break up the process on the landing page and evenutally don't donate. After all, filling out the form on the banner indicates a clear intention to donate. To convince some of these 50% in this test we featured a box with the latest five donor comments on our landing page. In addition to the often very strong stories and compelling reasons to donate these comments entail, we tried to use them as a mechanism of social proof: to see that other people trust this site makes this site more trustworty for yourself. But unfortunately the comments didn't have a real influence and the donation rate didn't change during the campaign test.

Update: A similiar and simlutaneous test on wikipedia.de came to the same results.

Banner	Landing-Page	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	no_comments	2610300	2704	1179	22559.77	19.13	0.001036	0.0004517	0.43602	0.0086426	28.57
ctrl	comments	2552800	2709	1216	21889.62	18.00	0.001061	0.0004763	0.44887	0.0085747	31.27

chi-squared-test donation/visitor: p = 0,193106

Test 36: text and layout banner: short version in bigger font size

Wednesday 11.12., 5 pm - Tuesday 12.12., 00:30 pm

Here we tested a shorter version of the facts text which allows an increased size of letters. This WMF change in banner layout seems to get more people reading the banner text. With an increase in click rate the overall donation rate in our test was increased by 10%.

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
text_ctrl	2221500	2648	1244	26283.42	21.13	0.001192	0.0005600	0.46979	0.0118314	28.56
text_short_bold	2195500	2889	1355	28422.00	20.98	0.001316	0.0006172	0.46902	0.0129456	28.44

chi-squared-test donation/imp: p = 0,013206

--Tobias Schumann (WMDE) (talk) 14:07, 20 January 2014 (UTC)

Test 37 und 46: banner colors

Changes in banner colour very often had a profound impact on donation rates in campaign tests during the last years. One possible reason could be an increased awareness of the banner by readers. During the 2013 campaign however we saw contradictory test results concerning banner colours. The change from green to yellow on a banner on wikipedia.de (the German search portal for de.wikipedia.org) brought a significant increase in donation rate by the signal colour yellow (test 2: donation/imp +22%). We evaluated another change in colour - yellow against blue - in three further tests. While the first one showed a great success for blue (test 37: donation/imp +28%), the second one resulted in practically no difference between the colours (test 46: donation/imp: +3%). Likewise an identical test on wikipedia.de saw no obvious winner (test 4: donation/imp: -3%). It remains unclear if the first, very positive result really represents a factual effect of the colour blue or if the result is not trustworthy despite its statistical significance.

test 37, Tuesday 12.12., 11:30 am - 7:00 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
yellow	1565900	1780	764	15324.00	20.06	0.001137	0.0004879	0.42921	0.0097861	23.60
blue	1561200	2189	977	20816.09	21.31	0.001402	0.0006258	0.44632	0.0133334	29.84

chi-squared-test donation/imp: p = 0,000000

test 46, Tuesday 17.12., 7:10 pm - Wednesday 18.12., 3:30 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
yellow	2674600	2947	1296	28011.05	21.61	0.001102	0.0004846	0.43977	0.0104730	27.08
blue	2684200	2955	1339	27263.11	20.36	0.001101	0.0004988	0.45313	0.0101569	26.40

chi-squared-test donation/imp: p = 0,455724

Test 39: dialogue banner

Friday 13.12., 1:30 pm - 4:30 pm

In this test we tried a completey new approach. Our goal was to reach more Wikipedia readers by starting a dialogue with them. For this purpose we showed a banner which asks if a reader wants to help Wikipedia - without clarifiying what help that could be. Instead two buttons were displayed to allow for a decision to help or not to help Wikipedia. As the result of clicking on the not-help-button the banner disappeared. In the case of a desire to help the usual facts banner with the donation form included was displayed afterwards. Unfortunately, this new approach profoundly decreased the donation rate (donation/imp: -61%). Still, the test was instructive and perhaps shows possible further developments. Indeed, more readers reacted to the banner: 32% more people clicked on the button to help Wikipedia compared to the number of people which completed the donation form in the form banner.

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
facts_form	532500	0	594	275	5967.21	21.70	0.000000	0.001115	0.00000	0.0005164	0.00000	0.46296	0.0112060	23.79
ask_banner	585800	789	177	118	2335.11	19.79	0.001347	0.000302	0.22433	0.0002014	0.14956	0.66667	0.0039862	24.00

chi-squared-test donation/imp: p = 0

Test 47, 49, 57: donation progress bar

Overlooking the past years the 2013 fundraiser in his course has much more similarity with the campaign in 2011 than the one in 2012. In the latter, we succeeded relatively early in finding improvements to compensate and reverse the notorious decline of donations after the usually fulminant beginning. That's why we were able to show banners, stories and appeals of Wikip/medians, which gained much attention by readers, allowed for a gaze inside the world of Wikipedia, but did not convince readers to donate as much as other banners. Instead, in 2011 it took some time until the last campaign weeks before we found a way to uplift the donation numbers to bring the campaign to a successful end. It was the new 5€ banner in the final part of the campaing which resulted in a nearly dramatic increase of donations.

Likewise, the 2013 campaign followed a difficult course. With small improvements we had been stabilizing the revenue on a certain scale throughout the weeks, but that wouldn't suffice at all to reach the campaign goal. So we thought: Why shouldn't we illustrate this situation in the banner? We returned to the old concept of an animated donation barometer, which shows how many donations came in already, how much money is still missing and what time remains to complete the goal. The old barometer went through a revision and got a new look by our web designer. At first, we tested it in a 5€ banner because it's layout provides more space (test 47). Compared to the facts banner (with donation form), the donation rate improved profoundly by 83%. Knowing that the facts banner had outperformed the 5€ banner in previous tests, we now did our best to integrate the barometer in it's densely filled banner space and compared it in test 49 with the 5€ banner (with barometer). By this, the donation rate grew by further +24%. However, the real uplift was even higher because of the clearly lower donation average of the 5€ banner (amount/imp: +49%). A third and last test of the barometer focussed on it's description text of the remaining campaign time. On the last day of the fundraiser we tested "Today our campaign ends" against an automatic countdown of the last 24 hours. Here we can say: Keep it simple. Counting down minutes and hours decreased the donation rate by 20%.

With the development of the donation barometer, we succeeded in significantly rising the donation numbers in the last weeks to a sustaining higher level. Only by this key improvement of the 2013 campaign we were able to finally reach the campaign goal and finish in time right before new years eve.

test 47, Wednesday 18.12., 3:30 pm - Thursday 19.12., 10:30 am

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
5€_drop_thermo	2316000	25184	2666	2227	39153.21	17.58	0.010874	0.001151	0.10586	0.0009616	0.08843	0.83533	0.0169055	25.40
facts_form	2165400	0	2486	1135	21990.00	19.37	0.000000	0.001148	0.00000	0.0005242	0.00000	0.45656	0.0101552	27.72

chi-squared-test donation/imp: p = 0

test 49, Thursday 19.12., 5:30 pm - Friday 10:30 am

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
facts_form_thermo	1606600	0	3396	1992	43127.06	21.65	0.000000	0.002114	0.00000	0.0012399	0.00000	0.58657	0.0268437	23.79
5€_drop_thermo	1661000	16922	1925	1660	29984.78	18.06	0.010188	0.001159	0.11376	0.0009994	0.09810	0.86234	0.0180522	24.74

chi-squared-test donation/imp: p = 0

test 57, Monday 30.12., 11 pm - Tuesday 31.12., 1:45 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	1028100	2393	1597	35672.43	22.34	0.002328	0.0015534	0.66736	0.0346974	21.46
hour_countdown	1046000	2024	1302	27188.33	20.88	0.001935	0.0012447	0.64328	0.0259927	19.15

• chi-squared-test visitor/imp: p = 0
• chi-squared-test donation/visitor: p = 0,093128
• chi-squared-test donation/imp: p = 0

Test 48: layout and text of facts inside drop banner

Thursday 19.12., 11:45 am - 4:30 pm

For a short period during the fundraiser it appeared that the drop version of the banner would outperform the donation form banner. From last year we know that the facts text "inside" the drop also plays an important role. In this test we evaluated two versions of the facts text differing in layout and text length. A blue and shorter version came out ahead, but only with a minor advance (donation/click: +3,7%). Furthermore, the result was not statistically significant. That's why we assume that there is no real difference between the two versions of the facts text inside the drop banner.

Banner	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp	% Women
red_and_long	986300	9574	950	786	15047.00	19.14	0.009707	0.000963	0.09923	0.0007969	0.08210	0.82737	0.0152560	24.77
blue_and_short	1042000	9934	984	846	14543.54	17.19	0.009534	0.000944	0.09905	0.0008119	0.08516	0.85976	0.0139573	24.89

• chi-squared-test click/imp: p = 0,205996
• chi-squared-test donation/click: p = 0,597
• chi-squared-test donation/visitor: p = 0,049813
• chi-squared-test donation/imp: p = 0,706822

Test 50 and 54: landing page: WMDE vs. WP logo

During the fundraiser we tried to improve the conversion rate on the landing page in several ways due to a drop out rate of 50%. One assumption was that people got irritated by the WMDE logo on the landing page because they had not known the organisational background, yet. Suspicious by the missing Wikipedia logo they break up the donation process. So in two tests we replaced the WMDE logo by the Wikipedia puzzle ball, one time on the old landing page layout, the second time on the new one. We were really surprised that this had absolutely no effect on the landing page conversion.

test 50 , Friday 20.12., 1:00 pm - Saturday 21.12., 7:00 pm

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
WMDE_Logo	2997700	7056	4097	87600.11	21.38	0.002354	0.0013667	0.58064	0.0292224	21.83
WP_logo	3008900	7118	4143	88668.95	21.40	0.002366	0.0013769	0.58205	0.0294689	22.15

chi-squared-test donation/visitor: p = 0,86538749

Test 54, Friday 27.12., 00:15 pm - Saturday 28.12., 2:45 pm

Banner	Landing-Page	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	wmde_logo	3004100	6052	3618	75559.00	20.88	0.002015	0.0012044	0.59782	0.0251520	21.72
ctrl	wp_logo	3003600	6021	3576	77287.15	21.61	0.002005	0.0011906	0.59392	0.0257315	21.57

chi-squared-test donation/visitor: p = 0,66257838

Test 51: donation form: preselected amount

Saturday 21.12., 7 pm - Monday 23.12., 2 pm

In this test we simply analysed the effect of a preselected default amount in the donation form. It shows: No preselection of an amount has no real impact on the overall donation rate. On the other hand click rate (-15%) and landing page conversion (+20%) change on a statistically significant scale, but in such a manner that they equal each other out in their effect. We're puzzled by these results. One possible explanation could be that some users break up the process when they see a warn notification after not filling out an amount in the radio buttons and pressing the donation button. Perhaps these people just wanted to see more information before donating and because of the notification they think that pressing a donation button already initiates a financial transaction.

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	4206500	9061	5214	112115.67	21.50	0.002154	0.0012395	0.57543	0.0266530	22.75
no_default	4248100	7778	5393	116839.26	21.66	0.001831	0.0012695	0.69337	0.0275039	23.67

• chi-squared-test visitor/imp: p = 0
• chi-squared-test donation/visitor: p = 0
• chi-squared-test donation/imp: p = 0,217926

--Tobias Schumann (WMDE) (talk) 09:57, 9 April 2014 (UTC)

Tests on wikipedia.de

Wikipedia.de is a German search portal for content in de.wikipedia.org. During the fundraiser we also display banners on this site.

Test 1: with vs. without donation form in banner

Monday, 28.11.2013 - Wednesday, 30.11.2013

A lot of tests throughout the campaign documented the positive impact of integrating the donation form into the banner itself (here and here). Still, we didn't want to apply this change to wikipedia.de without retesting it here. Great to see: Also on this website the donation rate was clearly improved by using a form banner (donation/imp: +27%).

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
no_form	172291	634	137	3122.00	22.79	0.003680	0.0007952	0.21609	0.0181205	21.00
form	172128	307	174	4069.00	23.39	0.001784	0.0010109	0.56678	0.0236394	18.18

Test 2: banner color - green vs. yellow

Wednesday, 4.12.2013 - Saturday, 7.12.2013

During the 2013 campaign we ran several tests of banner colours. An evaluation summing up these tests you can find here.

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
green	290180	447	277	5817.00	21.00	0.001540	0.0009546	0.61969	0.0200462	22.43
yellow	396747	752	463	10122.70	21.86	0.001895	0.0011670	0.61569	0.0255142	21.23

Test 3: donor comments on landing page

Monday, 9.12.2013 - Sunday, 15.12.2013

Parallel to a similiar test on de.wikipedia.org we tried to improve the drop out rate by showing the newest donation comments on the landing page. Similiar to his twin, the test did not result in any change of the donation rate.

Banner	Landing- Page	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	no_comments	620875	936	584	13559.50	23.22	0.001508	0.0009406	0.62393	0.0218393	20.17
ctrl	with_comments	619731	998	621	17048.05	27.45	0.001610	0.0010020	0.62224	0.0275088	16.42

Test 4: banner color - yellow vs. blue

Sunday, 15.12.2013 - Tuesday, 19.12.2013

During the 2013 campaign we ran several tests of banner colours. An evaluation summing up these tests you can find here.

Banner	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
yellow	449946	647	419	13365.68	31.90	0.001438	0.0009312	0.64760	0.0297051	19.58
blue	450710	650	407	11335.00	27.85	0.001442	0.0009030	0.62615	0.0251492	16.92

--Tobias Schumann (WMDE) (talk) 11:48, 9 April 2014 (UTC)

Test 5: landing page layout: new vs. old

Thursday, 19.12.2013 - Tuesday, 31.12.2013

Here we tested a new, in layout and concept completely revised landing page. The detailled analysis of the test results is posted together with test results of the new landing page on de.wikipedia.org.

Banner	Landing- Page	Impressions	Visitors LP	Donations	Sum	Average	LP- Visitor/ Imp	Donation/ Imp	Donation/ LP-Visitor	Amount/ Imp	% Women
ctrl	old LP	1141958	3262	2321	67780.22	29.20	0.002856	0.0020325	0.71153	0.0593544	18.97
ctrl	new LP	1142043	3291	2451	68929.41	28.12	0.002882	0.0021462	0.74476	0.0603562	15.25

chi-squared-test donation/visitor: p = 0,002499

--Tobias Schumann (WMDE) (talk) 14:18, 20 January 2014 (UTC)

mobile Tests on m.de.wikipedia.org

During the 2013 campaign we made an important step forward in diversifying sources of donations by displaying banners on mobile wikipedia (m.de.wikipedia.org). Furthermore, we started testing in this new and interesting field and succeeded in some significant improvements. Due to the lower numbers of visits and clicks, tests needed a longer duration for trustable results. That's why we only accomlished six tests during the campaign.

Test 1: sticky function

Wednesday, 4.12.2013, 9:15 pm - Thursday, 5.12.2013, 9:15 pm UTC

Due to the sometimes suboptimal internet connection on mobile devices, loading of banners takes longer. Because of that, a lot of Wikipedia readers probalby don't see banners because they've already scrolled down on the website before the banner is being displayed. So we began testing on the mobile site by evaluating the sticky function. As expected the donation rate improved, however on a surprising level: +220%.

Banner	Impressions	Donations	Sum	Average	Donation/ Imp	Amount/ Imp
not_sticky	1510100	42	318.00	7.57	0.0000278	0.0002106
information links	-	0	0.00	0.00	0.0000000	0.0000000
sticky	1237500	108	705.00	6.53	0.0000873	0.0005697
information links	-	2	28.00	14.00	0.0000000	0.0000000

• p-value donation/impression = 0,00000

Test 2: text layout in drop down - facts text yellow vs. red

Saturday, 7.12.2013, 5:00 pm - Tuesday, 10.12.2013, 5:00 pm

On small sized smartphone screens we tried to gain more attention for the facts texts by changing the banner background inside the drop from yellow to red. However, this didn't effect the donation rate at all.

Banner	Impressions	Donations	Sum	Average	Donation/ Imp	Amount/ Imp
facts_yellow	4285600	279	1944.00	6.97	0.0000651	0.0004536
information links	-	0	0.00	0.00	0.0000000	0.0000000
facts_red	4203800	277	1741.00	6.29	0.0000659	0.0004141
information links	-	0	0.00	0.00	0.0000000	0.0000000

• p-value donation/impression = 0,88676786

Test 3 and 4: banner layout: button vs. link

This two tests focused on the layout of the mobile banner. Instead of a "donate now" button in the banner we tried a WMF orientated layout version with a blue link and the highlighted "If everbody...". By that the click rate droped down by about 70%, but the conversion inside the drop improved significantly by 294%. All in all the donation rate increased by about 12%. Unfortunately, between the two test groups a difference in letter size in the drop down text slipped in by the layout change. For the purpose of really trustworthy results, we repeated the test with similiar letter sizes in both test groups. The outcome gave us a double checked certainty to use the new layout - the test results were practically identical.

Friday, 10.12.2013, 8:15 pm - Monday, 13.12.2013, 3:15 pm UTC

Banner	Landing-Page	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp
button	L10h16	2677600	15522	320	186	1129.00	6.07	0.005797	0.000120	0.02062	0.0000695	0.01198	0.58125	0.0004216
info links		-	0	29	0	0.00	0.00	-	0.000000	0.00000	0.0000000	0.00000	0.00000	0.0000000
link	L10h16	2641000	4362	355	206	1286.00	6.24	0.001652	0.000134	0.08138	0.0000780	0.04723	0.58028	0.0004869
info links		-	0	21	0	0.00	0.00	-	0.000000	0.00000	0.0000000	0.00000	0.00000	0.0000000

p-value donation/impression = 0,25160119

Monday, 13.12.2013, 3:15 pm - Thursday, 16.12.2013, 2:15 pm UTC

Banner	Landing-Page	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp
button	L10h16	3600000	21112	511	300	2130.00	7.10	0.005864	0.000142	0.02420	0.0000833	0.01421	0.58708	0.0005917
info links		-	0	40	1	5.00	5.00	0.000000	0.000000	0.00000	0.0000000	0.00000	0.02500	0.0000000
link	L10h16	3652800	5922	546	347	2324.00	6.70	0.001621	0.000149	0.09220	0.0000950	0.05860	0.63553	0.0006362
info links		-	0	31	0	0.00	0.00	0.000000	0.000000	0.00000	0.0000000	0.00000	0.00000	0.0000000

p-value donation/impression = 0,09637155

Test 5: quantity donation amounts

Thursday, 16.12.2013, 2:15 pm - Monday, 20.12.2013, 14:15

From a lot of tests we know how important it is to find the optimal quantity and sum of selectable donation amounts in the donation form. A higher number of donation amounts to choose can increase usability but also complexity of the form. On small mobile screens this problem has to be readressed. So we began with testing four (3|5|10|20) against six (5|15|25|50|75|100) amounts. (Due to technical novelty of mobile banners we couldn't implement a free field for donation amounts, yet.) While tests on desktop banners showed an increased donation rate through more amount buttons in the form, in the current test the donation rate was not affected. A different relationship of usability and complexity on small screens may be an explanation. However, the modification still was a success. The form with six amounts performed better because of a higher donation average.

Banner	Landing-Page	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp
four_amounts	L10h16	3879100	74882	523	284	2020.00	7.11	0.019304	0.000135	0.00698	0.0000732	0.00379	0.54302	0.0005207
information links		-	-	61	2	15.00	7.50	-	0.000000	0.00000	0.0000000	0.00000	0.03279	0.0000000
six_amounts	L10h16	3870100	75166	536	285	2605.00	9.14	0.019422	0.000138	0.00713	0.0000736	0.00379	0.53172	0.0006731
information links		-	-	45	0	0.00	0.00	-	0.000000	0.00000	0.0000000	0.00000	0.00000	0.0000000

Test 6: text and highlight in drop down

Monday, 23.12.2013, 4:00 pm - Tuesday, 01.01.2014, 00:00 Uhr

Text and layout of the donation appeal inside the drop down were tested this time. We compared two versions of the facts text against each other. The test version featured an additional text highlighting. That's why the test is strictly speaking a little bit inaccurate, but given the few tests we could realize our main goal was to find improvements of the banner. Our test version really improved donation/click by 8% compared to the control banner. Unfortunately, the result is not statistically significant but the p-value is close to the threshold value.

Banner	Landing-Page	Impressions	Banner Clicks	Visitors LP	Donations	Sum	Average	Click Drop/ Imp	LP- Visitor/ Imp	Visitor/ Banner-Click	Donation/ Imp	Donation/ Click-Drop	Donation/ LP-Visitor	Amount/ Imp
ctrl	L10h16	10363700	199258	1291	735	7000.00	9.52	0.019227	0.000125	0.00648	0.0000709	0.00369	0.56933	0.0006754
information links		-	-	114	0	0.00	0.00	-	0.000000	0.00000	0.0000000	0.00000	0.00000	0.0000000
var	L10h16	10344900	199802	1251	791	8322.09	10.52	0.019314	0.000121	0.00626	0.0000765	0.00396	0.63229	0.0008045
information links		-	-	118	5	190.00	38.00	-	0.000000	0.00000	0.0000000	0.00000	0.04237	0.0000000

• p-value donation/click = 0,13141519 (chi-squared-test)

--Tobias Schumann (WMDE) (talk) 11:53, 5 May 2014 (UTC)