Talk:Fundraising 2010/Banner testing

Youtube link with banner

Is it possible for us to link the Foundation Youtube channel or videos on some of the banners? The number of views of the videos are lacking. This is also a good way to share information about the WP and the foundation. I know this idea comes really in a sudden, but I really hope we could work on this. --Diagramma Della Verita 12:57, 29 August 2010 (UTC)

I don't know if drawing people away from the donation page/Wikimedia sites is a good idea. I think after someone donates we should have a few social media-related buttons at the bottom, twitter, facebook, youtube, etc. --Dgultekin 16:38, 3 September 2010 (UTC)

This page will document the feedback that we received from the Thursday fundraisers.

See also m:Fundraising 2010/Banner testing

Testing comments Aug 19

What went well?

• Users loved the fact that the community was able to provide input on the banners.
• CentralNotice didn't break and the banners did not have any major CSS issues.
• We got stats!
  • More upcoming, Monday night or Tuesday Morning PST, James is scrubbing recently acquired squid data to usable form.

What did not work?

• "Commons: Written by volunteers, supported by readers like you." - When we're running the same banner on all of our projects... project-specific messages can have icky results.
• Issues with tracking (Open issues)
• Banners were tested before 3 PM... lets try to stick to schedule? :-) Alex had just gotten home and had no idea we were even testing yet - until he saw a banner up on the site.
• Landing page bump where we forgot to change a variable that led to people being sent to the wrong landing page. Noticed and fixed quickly but still mid-test example
• Stat bump afterwards posting some raw data with ip's still attached. Caught and oversighted but obviously need to watch for it.

What should be changed?

• Landing pages need to be updated eventually.
• Some users commented that the banners were not consistent with the landing pages. ("Banner 2 says 33 million entries, but the donate page says 16 million articles at the top.")
• There was a comment about the design of banner "2010_testing2/2B", the font being too large and that it needed more padding.
• We need more creative and eye catching banner designs.
• Some didn't like the fact that the banners included two links to the landing page. One for our main message "Wikipedia: Written by volunteers, supported by readers like you." and one for our donate now link.

What did we learn?

• Users want smaller banners.
• Was told that many users actually hide the banners with custom CSS in their user space.
  • This is also an issue that should be remembered for the fundraiser, usually at least on enWiki an admin will decide to activate a "gadget" to do this allowing anyone to hide them for the duration with a checkbox in their preferences. The gadget is not currently active.
• Banners (and possibly landing pages) should be less Wikipedia-centric
• The members of our community fundraising committee have a sense of humor

Testing comments Aug 26

What went well?

• basic stats got up fairly quickly and improved through the night as we found more data.
• Tomasz made a much more refined script to get the arrays for banner hits :)
• Test itself appears to have gone well

What did not work?

• Odd numbers, still not sure if it was lower viewers, horrible banners or missing numbers?
• Some of the stats went missing until Tomasz found them a couple hours later after poking around (including all hits from Banner 5 the test banner with only the donor form)

What should be changed?

• We need impression data!

What did we learn?

• Donor only form appears to be useless
• Difference between the two tested banners statistically minor
• Possible issue running into week long cookie that hides banner after people press "hide"

Testing comments Sep 2

What went well?

• The "Stay curious." banner preformed exceptionally well (with 3037 or so clicks).

What did not work?

• We had some issues with banner #7's spacing that were not present in a previous test on testwiki.
• Need to work on new stat presentations (have some options that are up working on others) Old styles not scaling horribly well.

What should be changed?

• We need to test, test, and test banners again. I'm thinking of possibly doing something with browsershots.
  • Need to get Testing plan set for next week. Probably done by Tuesday? Hopefully earlier.

What did we learn?

• Users are watching: When we had the bug with the Mandela banner people were asking about it in multiple channels very quickly including users from projects where no one clicked on the banner.
• Impression data (that we should be getting soon to fill out data charts and continue working on graphs) ended up being 2.9GB according to to Tomasz. Need the new system to scale, this won't work.

How many are being tested at a time?

If we are just measuring click through and not conversions, can't we test hundreds of banners per hour?

How many impressions do we need to get statistically significant measures of each? No more than a few thousand, isn't it?

Also, is there a master list somewhere of the top 100 banners sorted by click-through percentages? People being asked to come up with new banner ideas should be able to see that list so they can refine their ideas. 71.198.176.22 23:55, 6 September 2010 (UTC)

Yes, we could theoretically test hundreds per hour, but we're constrained by the time it takes to build banners. There aren't a top 100, because we haven't tested 100 yet :) So they're all in the top 100... Philippe (WMF) 23:11, 7 September 2010 (UTC)

How many per hour do you prefer to test? Testing four banners per day at 650,000 impressions each is a waste of time and effort -- those of us submitting banners would be able to use the clickthrough rates to refine our suggestions if there were more being tested. Why Other 01:57, 11 September 2010 (UTC)

Well, we tested about 7 last week, i believe... I haven't looked to be exactly sure... and we only tested for one hour. :) Anything more than that would risk community backlash, I think. But we definitely do want to accelerate the number that we test. We had to make sure that the technical infrastructure was in place and that the reporting was working before blowing through a ton of banners. Philippe (WMF) 03:36, 11 September 2010 (UTC)

Testing comments Sep 9

Add your thoughts/comments below!

Alternative banner designs

Would it be possible to test, or at least discuss, the banner alterations I posted at Talk:Fundraising_2010/Messages#Refresh_the_banner_design? I really think it could make a difference. Pretzels 23:31, 9 September 2010 (UTC)

Yes, absolutely. Some redesigns (including that one) will definitely be tested! But we wanted to be sure that we had the technology down before adding too many variables. Philippe (WMF) 23:31, 10 September 2010 (UTC)

Dollars on test 3

How come we don't have the dollar amounts on test 3? In the detailed stats I mean. Kaldari 01:40, 10 September 2010 (UTC)

Because there weren't enough to be statistically valid :) Philippe (WMF) 23:31, 10 September 2010 (UTC)

Some stats questions

• What is the desired metric? Is it clicks, conversion, contribution amount, ROI expressed in dollars/impression or dollars/click?
• Click-through rates are a reliable indicator of success, and conversion dollars ultimately matter, but almost all individual banners resulted in under 5 donations and few hit even $100 per test. Are those donations numbers statistically significant, especially since many banners with high click-through had low donations, and vice-versa? Ocaasi 04:49, 17 September 2010 (UTC)

Since this is a fundraiser, the desired metric is "dollars in the bank account" :)

No, they're not. They also weren't really expected to be, at this point. The first few tests - while gaining valuable information on the visceral reaction that people had to messaging - were geared toward making sure that we had the technology, systems, and people in place to run the type of campaign we're talking about. The technology is almost all tested, the campaigns have gotten more and more complex, and systems and processes have been defined from both the engineering end and the "fundraising" end.

Because of the short periods during which we are testing, there's some doubt as to whether we'll ever truly reach a statistically significant number of hits. We're also testing messages in other fora though; including focus groups and polling. That data will, of course, also be shared here. Philippe (WMF) 13:15, 17 September 2010 (UTC)

I asked, because, after the last test there were some fairly declarative keep/toss decisions about banners which seemed to make variable use of the metrics. One banner with high click-through was indicated as a toss; it had no donations. Another with low click-through had high donations, but still in the 'just a few' range. So I wasn't sure if the testing was rigorous enough yet to 'throw out' banners. Or will they be recycled for later, more comprehensive testing? Ocaasi 21:26, 19 September 2010 (UTC)

Some of it (Be curious) was just ones that we had tested a multiple times and always had that high click rate but never really performed in anything else. Be curious was especially disappointing along with all the curious motifs we used in past tests. This week we even gave it special treatment testing 1 step donation pages along with specially written landing page text and it still got very little action beyond the {{click. The others were a bit of a decision that it would be more helpful to fill testing spot with other, new, banners then with these older banners that hadn't really showed many good signs. Jalexander 22:22, 19 September 2010 (UTC)

Alright. That explanation was helpful, but I still have a question... The "One day people look will look back" banner had a nice click-through but no donations. It was tossed. I happened to write it. (I like to think I would have been curious anyway, though perhaps less persistent). It had a lot of support from the insiders. And seemed to test somewhat comparably. What was its deathknell? Will it ever test again? Ocaasi 23:56, 19 September 2010 (UTC)

Call to action testing.

Results showed that calls-to-action improved conversions (donations) but not click-through. Can we run a test comparing different phrases to refine that more. E.g. :

• Donate now.
• Donate today.
• Donate here.
• Please donate.
• Please donate now.
• Please donate today.
• Please donate here.
• Please consider donating.
• Support Wikipedia today.
• Support us today.
• Make a donation today.
• Make a donation to Wikipedia today.

etc., etc.

That kind of thing... And run it on one or two different banners (short/long, funny/serious). Thoughts? Ocaasi 04:30, 20 September 2010 (UTC)

1. I support Ocassi. Call for Action is a must in donation-banners. So the question is: what type of call for action is the most effective? I would add "Please consider" as a gentle approach. Till Mletzko (WMDE) 09:29, 20 September 2010 (UTC)
2. I'd also like to see which calls to action are most effective. We learned last week that a soft ask (eg. support wikipedia) was more effective than 'donate now' I'd --Deniz (WMF) 17:23, 22 September 2010 (UTC)

We might get statistically valid data on clicks, but none of the results for $ conversion should be treated as statistically significant at this point -- if we want really comparative data on actual fundraising efficacy, we'll need to run longer tests / try larger banners.--Eloquence 06:35, 23 September 2010 (UTC)

Testing comments October 26

What happened to the Portuguese tests? Was there really no donations (disappointing) or is it just that something is not complete? GoEThe 10:54, 29 October 2010 (UTC)

There was a stats problem, and it's taken us a couple of days to nail down. The stats for Banner 50 were rolled up, rather than separated out by language. We're having them rerun. Philippe (WMF) 18:28, 29 October 2010 (UTC)

The Spanish data is missing, why? Emijrp 12:41, 9 November 2010 (UTC)

Talk:Fundraising 2010/Messages/Language/pl

Hallo. We select the best banners of pl users. Could we test them? If not, they will be used by WMPL, ok? Przykuta 16:09, 29 October 2010 (UTC)

Hey Przukuta, please put the banners you want tested here in the Testing Queue. Thanks! --Deniz (WMF) 18:31, 29 October 2010 (UTC)

Done, thx Deniz. Przykuta 20:15, 29 October 2010 (UTC)

Ekhm not "Wikipedia jest za darmo. Jej serwery nie.", but "Wikipedia jest za darmo. Jej serwery nie. Dorzuć się!" - "Wikipedia is for free. Its servers are not. Chip in!" without "Chip in!"... :/ Huh, thx for this. Przykuta 15:12, 2 December 2010 (UTC)

Are any more banners going to be tested?

Is testing over? Everyone who submitted one of the vast majority of the banners which weren't tested should feel proud to have tried? 71.198.176.22 19:12, 19 November 2010 (UTC)

Banner testing is not over, throughout our campaign we will be trying new things. And everyone who has been involved in the process of making this fundraiser a reality should feel proud, it has been, and continues to be an incredible joint effort! --Deniz (WMF) 19:56, 19 November 2010 (UTC)

Great. When are the remaining vast majority of the submitted banners going to be tested? Ginger Conspiracy 01:23, 30 November 2010 (UTC)

We are continuing to test community submitted editor banners and appeals as they have been the only messages with similar click-through rates as the Jimmy Banner so far. We are asking editors to write a personal appeal for testing, and have received some fantastic submissions, if you are in the US you can see two new editor appeals from Sage and Abbas currently running :) -- Deniz (WMF) 19:39, 3 December 2010 (UTC)