Fundraising 2010/Banner testing/Detailed

First test

Full stat report Discuss this test
For our first banner test, which is scheduled for August 19, I'd like to propose that we keep things relatively simple: one, perhaps two banners, and one or two landing pages. This is primarily a resourcing test to be sure that everyone knows who is responsible for which part of the work.

I'd like to propose the following:
Use two messages (both community submitted):

{{Sitename}}: Written by volunteers, supported by readers like you.

Donate today.

“One entry became 13 million. One language became 270. Help protect what we’ve created.”

Donate today.

I'm not wedded to these, but we need to make some decisions fairly quickly, since we need to get the banners and landing pages built. I purposely am proposing one that has a "reader supported" message, and one that has a "protect" message. We should model the landing pages to test these messages against providing no additional detail, so:

A landing page like wmf:Support_Wikipedia/en and
A landing page like wmf:Support_Wikipedia/en that also has language to match the banner that is paired to that landing page. I'm testing (in an incredibly simplistic style) whether adding language to explain the banner helps or hurts us when compared to a simple contribution box.

Thoughts? Philippe (WMF) 17:29, 18 August 2010 (UTC)[reply]

Like them both. The 2nd one made me think of using {{{NUMBEROFARTICLES}} to make it site specific but that may be something to test for another day? If we're putting it on Wikipedias I think we can do 16 Million now ;). I like the testing of the "more info", I love more info but it would be very interesting to see if that plays out as true in reality. Jalexander 23:23, 18 August 2010 (UTC)[reply]

These banners are project-specific. The first doesn't work with Commons, the second only works with Wikipedia. The landing pages are also Wikipedia-specific. --Yair rand 04:32, 19 August 2010 (UTC)[reply]

The landing pages are being worked on right now. :) And yeah, these banners won't work for all the projects, but they don't have to for the live fundraiser... we will target more specifically. For this test, we're mostly testing functionality. But I agree with your comments. and thanks for pointing that out. We should probably create a section for each of the banners that we are trying out and say whether they exclude projects. :) Philippe (WMF) 04:38, 19 August 2010 (UTC)[reply]

I don't know what do you want to test specifically, but please don't run English messages on non-English projects. If we want to test (almost) everything, conditions should be similar to those of the actual fundraising, so you need to run the centralnotices on all projects and to point to translated landing pages with chapters (the chapter fundraising agreement has espired, AFAIK, but most chapters will hopefully renew it). In this manner, we could just reuse some old, already translated messages and pages (e.g. [1] and [2]: you only have to remove the Wikipedia puzzle piece) redirecting wmf:2010_Landing_1/de to wmf:Global Support/de etc. (those landing pages are also often less Wikipedia-centric than wmf:Supporti Wikipedia and subpages).

If this is not possible, at least replace "written" with "edited" and "13 million" with "33 millions"[3] and "270 languages" with "280 languages" (see Special:Sitematrix; this should be because some languages have a Wiktionary or a Wikisource but not a Wikipedia), to make them work at least for all English-language projects. --Nemo 08:10, 19 August 2010 (UTC)[reply]

All very good suggestions, thank you! We'll get some changes made to reflect those tomorrow. This test will be english-only... we're testing mechanisms first, before we really get too deeply into message testing. Philippe (WMF) 09:02, 19 August 2010 (UTC)[reply]

A few comments on this test, mainly aesthetic:

The three-line banner is kind of off code-wise; the font is either too big or the banner box too small, or it needs more padding at the bottom (please, change the first and not make the banners bigger). I bring this up because it looks almost sloppily or hastily written up, something we do not want donors to think.
They're bland. I mean, the text is blue and the background grey. Even if we made the font colors slightly different, it'd look nicer. I mean, the words are good (I like the two chosen), but the actual design can definitely be improved, to be more visually appealing, maybe with an image, too.
The Commons banner says "Commons: Written by volunteers, supported by readers like you." Except, Commons is media-based, not text-based. Just a little odd to me.
Banner 2 says 33 million entries, but the donate page says 16 million articles at the top. Not sure which number means what exactly.

Again, sorry if this is the wrong place, as I'm not sure where or who to go to to leave comments about this test. These were just my first impressions. Hopefully, the kinks will be worked out before the global rollout later. fetchcomms☛ 22:02, 19 August 2010 (UTC)[reply]

Thanks, Fetchcomms: all very good points. As you guess, those are things that would be worked out. We haven't even begun to think about creative/art direction. Philippe (WMF) 06:55, 20 August 2010 (UTC)[reply]

Tracking

Test date	LP	CN	Banner text	Notes
2010-08-19	1	1	{{Sitename}}: Edited by volunteers, supported by readers like you.	includes text to explain banner. Using (utm_campaign=thursday_fundraiser) for tracking of our test banners.
2010-08-19	2	1B	{{Sitename}}: Edited by volunteers, supported by readers like you.	no explanatory text.
2010-08-19	3	2	“One entry became 33 million. One language became 270. Help protect what we’ve created.”	includes text to explain banner
2010-08-19	4	2B	“One entry became 33 million. One language became 270. Help protect what we’ve created.”	no explanatory text.

Report

Full stat report Discuss this test

Banner text	A: Sitename: Edited by volunteers, supported by readers like you.		B: One entry became 33 million. One language became 270. Help protect what we’ve created.
Same number of people saw both banners	622 clicked A (31% of total clicks)		1383 clicked B (69% of total clicks)
Landing page text	Text expanding on banner	No extra text	Test expanding on banner	No extra text
Clicks from each banner were directed 50/50 to each landing page	4 hit donate (.6% of A's clicks)	8 hit donate (1.3% of A's clicks)	8 hit donate (.58% of B's clicks)	10 hit donate (.72% of B's clicks)
Completed donations	1 actually donated (.16% of A's clicks)	2 actually donated (.32% of A's clicks)	3 actually donated (.22% of B's clicks)	2 actually donated (.14% of B's clicks)
Total donated (Which combo won?)	$35.00 (52% of A's donations)	$32.09 (48% of A's donations)	$55.00 (73% of B's donations)	$20.00 (27% of B's donations)
Which Banner won?	$67.09 (47% of total)		$75.00 (53% of total)

Hi. Thanks for this data. But, what about making longer tests? With the current numbers, we can't do serious stats about the best banners. And, can you give us any info about this week test? Thanks, Emijrp 10:56, 25 August 2010 (UTC)[reply]

Hey Emijrp, I replaced the data above with a consolidated version as well as a adding in a short form of our click data. Your right that the sample size was quite small which made it hard to see a lot of data though I think the banner click data does show some reasonable conclusions (Banner1 had only 43% of the clicks that Banner2 had which is significant). We're hoping to get better data both for this Thursdays test and as we expand the testing (to more banners, perhaps more time?). I'm creating a new page for more detailed data sets. I'll post some more data from last week as well as data from this week (hopefully very quickly Thursday evening/Friday morning). Jalexander 07:31, 26 August 2010 (UTC)[reply]

Second test

Full stat report Discuss this test
For the 2nd test we decided to do 2 separate A/B tests testing 2 new banners (both from the community) and 2 different landing page ideas. Each banner will be served 20% of the time (60% of the time it is no banner) and half of each banner will show Landing page A and half Landing page B so that each Banner/Landing page combo will be served 10% of the time.

Banners

The first banner (called testing3) can be seen here and says:

Help us change the world on the first line and
Support Free Knowledge on the second line.

The second banner (called testing4) can be seen here and says:

Help us change the world on the first line and
Donate today to support free knowledge on the second line

Landing Pages

Testing two different landing pages we have Landing page A (Landing 5) which is a plain and simple donation box without Jimmy Wales quote/donor quotes or anything else.

We also have Landing page B which is the standard landing page we used last week and recently located as Landing 2. This has the donation box on the left with a quote and picture of Jimmy on the right as well as a donor comments box.

Notes

One change to note from last week is that after comments made we switched to just the 2nd line being clickable (the line actually asking them to donate).
Important: This weeks set of banners is being displayed about 4 times more then the banners last week. This is because not only did we now double the set weights but they are also not competing against usability banners. This should be remembered when we look at the data.

Tracking

Test date	LP	CN	Banner text	Notes
2010-08-26	5	3	“Help us change the world Support Free Knowledge”	No explanatory text. Just a donate box and nothing more.
2010-08-26	2	3B	“Help us change the world Support Free Knowledge”	No explanatory text. Landing page includes Jimmy/Comments.
2010-08-26	5	4	“Help us change the world Donate today to support free knowledge”	No explanatory text. Just a donate box and nothing more.
2010-08-26	2	4B	“Help us change the world Donate today to support free knowledge”	No explanatory text. Landing page includes Jimmy/Comments.

Report

Full stat report Discuss this test

Banner text	A: Help us change the world / Support Free Knowledge		B: Help us change the world / Donate today to support free knowledge
Same number of people saw both banners	1052 clicked A (57% of total clicks)		807 clicked B (43% of total clicks)
Landing page text	Donor form only	form, Jimmy quote, donor quotes	Donor form only	form, Jimmy quote, donor space
Clicks from each banner were directed 50/50 to each landing page	8 hit donate (.76% of A's clicks)	9 hit donate (.86% of A's clicks)	2 hit donate (.25% of B's clicks)	3 hit donate (.37% of B's clicks)
Completed donations	0 actually donated (0% of A's clicks)	3 actually donated (.28% of A's clicks)	0 actually donated (0% of B's clicks)	1 actually donated (.12% of B's clicks)
Total donated (Which combo won?)	$0.00 (0% of A's donations)	$146.58 (100% of A's donations)	$0.00 (0% of B's donations)	$3.00 (100% of B's donations)
Which Banner won?	$146.58 (98% of total)		$3.00 (2% of total)

Third test

Full stat report Discuss this test
For our third test (September 2nd) we decided to do 5 different banners. 4 of them show to all English projects and 1 extra one for the English Wikipedia only.

We are running the percentages differently based on if the person is logged in or not.
- For logged in users each banner has a 5% chance of showing (so 25% chance of getting a banner on Wikipedia 20% for other projects).
- For Anonymous users each banner has a 10% chance of showing (so 50% chance of getting a banner on Wikipedia 60% for other projects).
All banners are going to the normal donation page that we've used before and you can see here

Banners

Our 1st banner (testing 5) can be seen here and says:

Free, Innovative, Non-profit. (1st line)
This is Wikipedia. (2nd line and linked to donation page)

Our 2nd banner (testing 6) can be seen here and is just:

Stay curious (1 line all linked to donation page)

Our 3rd banner (testing 7) can be seen here and says:

"If you make one gift this year, make it the gift of knowledge." -- Nelson Mandela (All on one line with only "knowledge" linked to the donation page)

Our 4th banner (testing 8) can be seen here and says:

Meta: Supported by readers like you. (all on one line with "supported by readers like you" linked to the donation page)

Our last banner (testing 9) is ONLY shown on the English Wikipedia, can be found here and says:

The 7th most visited website in the world might be the most important. (1st line)
Donate today (2nd line, linked to donation page)

Notes

We turned off the vector messages on English projects to try and have cleaner data.

Tracking

Test date	LP	CN	Banner text	Notes
2010-09-02	2	5	“Free, Innovative, Non-profit. This is Wikipedia.”	No explanatory text. Landing page includes Jimmy/Comments.
2010-09-02	2	5_anon	“Free, Innovative, Non-profit. This is Wikipedia.”	No explanatory text. Landing page includes Jimmy/Comments. (Anonymous users only)
2010-09-02	2	6	“Stay curious.”	No explanatory text. Landing page includes Jimmy/Comments.
2010-09-02	2	6_anon	“Stay curious.”	No explanatory text. Landing page includes Jimmy/Comments. (Anonymous users only)
2010-09-02	2	7	“"If you make one gift this year, make it the gift of knowledge." -- Nelson Mandela”	No explanatory text. Landing page includes Jimmy/Comments.
2010-09-02	2	7_anon	“"If you make one gift this year, make it the gift of knowledge." -- Nelson Mandela”	No explanatory text. Landing page includes Jimmy/Comments. (Anonymous users only)
2010-09-02	2	8	“{{SITENAME}}: Supported by readers like you.”	No explanatory text. Landing page includes Jimmy/Comments.
2010-09-02	2	8_anon	“{{SITENAME}}: Supported by readers like you.”	No explanatory text. Landing page includes Jimmy/Comments. (Anonymous users only)
2010-09-02	2	9	“The 7th most visited website in the world might be the most important. Donate today.”	No explanatory text. Landing page includes Jimmy/Comments. (English Wikipedia only)
2010-09-02	2	9_anon	“The 7th most visited website in the world might be the most important. Donate today.”	No explanatory text. Landing page includes Jimmy/Comments. (Anonymous users on the English Wikipedia only)

Report

Full stat report Discuss this test

Data for all users (Logged in and Anonymous) See here for separated data
Banner	Free, Innovative, Non-profit	Stay curious	The Gift of knowledge (Nelson Mandela)	Supported by readers like you	7th most visted = most important
Number who saw banner (impressions)	680346	687712	677690	677936	663880
Number who clicked banner (click through)	706	3037	416	375	539
Percentage who clicked banner after viewing	.1%	.44%	.06%	.05%	.08%
Percentage of total banner clicks	14%	60%	8%	7%	11%

Fourth test

Full stat report Discuss this test

Tracking

Test date	LP	CN	Banner text	Notes
2010-09-09		6	10	⧼Centralnotice-10-donate-text⧽	A knowledge revolution text, Jimmy and comments.
2010-09-09		6	10_anon	⧼Centralnotice-10-donate-text⧽	A knowledge revolution text, Jimmy and comments.
2010-09-09		6	11	⧼Centralnotice-11-donate-text⧽	A knowledge revolution text, Jimmy and comments.
2010-09-09		6	11_anon	⧼Centralnotice-11-donate-text⧽	A knowledge revolution text, Jimmy and comments.
2010-09-09		6	12	⧼Centralnotice-12-donate-text⧽	A knowledge revolution text, Jimmy and comments. (Wikipedia only)
2010-09-09		6	12_anon	⧼Centralnotice-12-donate-text⧽	A knowledge revolution text, Jimmy and comments. (Wikipedia only)
2010-09-09		6	13	⧼Centralnotice-13-donate-text⧽	A knowledge revolution text, Jimmy and comments.
2010-09-09		6	13_anon	⧼Centralnotice-13-donate-text⧽	A knowledge revolution text, Jimmy and comments.
2010-09-09		6	14	⧼Centralnotice-14-donate-text⧽	A knowledge revolution text, Jimmy and comments.
2010-09-09		6	14_anon	⧼Centralnotice-14-donate-text⧽	A knowledge revolution text, Jimmy and comments.
2010-09-09		6	15	⧼Centralnotice-15-donate-text⧽	A knowledge revolution text, Jimmy and comments.
2010-09-09		6	15_anon	⧼Centralnotice-15-donate-text⧽	A knowledge revolution text, Jimmy and comments.
2010-09-09		6	6	⧼Centralnotice-6-donate-text⧽	A knowledge revolution text, Jimmy and comments.
2010-09-09		6	6_anon	⧼Centralnotice-6-donate-text⧽	A knowledge revolution text, Jimmy and comments.

Report

Full stat report Discuss this test
Please note: All banners tested this week used the same landing page ( seen here ). It included the basic landing page we have used recently (with the picture of Jimmy Wales and a link to donor comments) with the addition of text about "The knowledge revolution" that is Wikipedia and it's sister sites.

Overall Click rates

	Banner 6 (SC) "Stay curious."	Banner 10(BC) "Be curious."	Banner 11(TOAW) "Think of a world without {{Sitename}} Keep free knowledge alive."	Banner 12(DN) "Wikipedia is the world's greatest repository of knowledge.^{[Donation needed]}" (Wikipedia only)	Banner 13(WFA) "Who funds {{Sitename}}? You do. Donate today."	Banner 14(WF) "Who funds {{Sitename}}? You do."	Banner 15(BCBB) "Be curious. Be bold."
Total banner clicks	3236	3826	1084	479	682	1224	3115
Percent of total clicks	24%	28%	8%	4%	5%	9%	23%

Impression to click comparisons

	Banner 6 (SC)	Banner 10(BC)	Banner 11(TOAW)	Banner 12(DN) (Wikipedia only)	Banner 13(WFA)	Banner 14(WF)	Banner 15(BCBB)
Impressions	727516	730043	726180	710188	734898	735222	730206
Banner Clicks	3095	3711	923	307	515	1064	2970
Conversion %%	0.43%	0.51%	0.13%	0.04%	0.07%	0.14%	0.41%

Donation Statistics

	Banner 6 (SC)	Banner 10(BC)	Banner 11(TOAW)	Banner 12(DN) (Wikipedia only)	Banner 13(WFA)	Banner 14(WF)	Banner 15(BCBB)
# who clicked donate	15	10	18	13	7	13	11
Completed donations	1	2	4	2	3	3	2
Conversion %	6.67%	20.00%	22.22%	15.38%	42.86%	23.08%	18.18%
$ received (USD)	$5.00	$85.00	$55.00	$98.39	$136.54	$75.00	$40.00