Research talk:Measuring mission success

Clarification on the vocabulary ?

I do not think that the word metric is suitable for the description given by Denny in the wikisign oped. I would rather say it is some kind of representation of the measure of success. For me a metric has to be computable. -- jpfram

How many people speak which languages?

Is there a list of how many people speak which languages? I guess, a list of how many speakers there are per language could be used for a first approximation, but this would not cover e.g. if a group of speakers of language A is a subset of the speakers of language B, etc.

I.e. given all humans on earth, what would be the order of languages that should be supported in order to unlock content as fast as possible to most? (I am even struggling with formulating the question). --denny (talk) 17:03, 18 August 2014 (UTC)

• List of Wikipedias by speakers per article
• List of Wikipedias by language group
• List of Wikipedias by language family --Atlasowa (talk) 16:15, 19 August 2014 (UTC)

That's a decent starting point, but not exactly what I meant. Since more than half of the world's population is multilingual, most people are in the fortunate situation that they can choose between Wikipedia-articles in several languages. That is why it would be interesting to know the overlap of languages per speaker - how many people speak what combination of languages. I would be very happy about this kind of data! --denny (talk) 16:34, 19 August 2014 (UTC)

Hi Denny! See Page Views Per Country, Page Views Per Wikipedia Language, and Page Edits Per Country, Page Edits Per Wikipedia Language.

BTW, i think that building a multilingual Wikipedia search for www.wikipedia.org is amongst the most important and needed features for our readers. That is also what readers say when asked.

Last year, there was a short analysis on the changes in the pattern of language versions read per country, Who reads which Wikipedia? The WMF's surprising stats. I think this was due to changes in multilingual google search. Since you're working for google, you may check and confirm if the changing pattern coincides with any changes in multilingual google search? --Atlasowa (talk) 11:58, 25 August 2014 (UTC)

• 
  European countries and requested WP language versions, 2010.
• 
  Countries in Northern Africa and the Middle East and requested WP language versions, 2010.
• 
  Reasons chosen by readers to choose Wikipedia over other information sources like google (WMF Blog post)
• 
  Readers likely to use Wikipedia more if search was improved - by country. (Research:Wikipedia Readership Survey 2011/Results)
• 
  Readers very likely to use Wikipedia more if search was improved - by language (Research:Wikipedia Readership Survey 2011/Results)
• 
  Example for a kind of multilingual wikipedia search, screenshot of instapedia.com.
• 
  actual wikipedia search at www.wikipedia.org, screenshot of 2013

A first stab

Just a few ideas. The following sketch is extremely hackish and preliminary, and I doubt much of it will survive the necessary scrutiny as we proceed, but it might still be a helpful start.

• Let's assume that the metric is indeed in the form of a rectangular image with a single curve.
• Let's assume it is 1000 px time 1000 px.
• Then each pixel on the horizontal axis represents about (7b people/1000 px=) 7 million people.
• For each such group of people we need to establish how much of all knowledge is available to them. We can try to create an average over the 7 million, or take a representative individual. Let's do the latter.
• The representative individual could be the 'median' individual, the 'top' or 'bottom'. I doubt there would be a massive difference for that, so let's take the top.
• Which languages does the representative individual speak?
• What is their means to access knowledge from Wikimedia? Do they have free / expensive / any Web access, or access to Wikipedia content?
• How does this affect the line to draw for them?
• Based on WP:ALL, let's assume there should be 100M articles, and a logarithmic scale: then each order of magnitude makes about 111 pixels.
• Let's assume that quality matters as a factor for each article: a Good or Featured article might have a factor of 1, a stub a factor of .1, a decent article a .5
• How many articles of which quality are available to the individual based on their languages? Sum that up. That should lead to how many pixels to draw for that individual.
• Do that for all 1000 representative individuals.

Again, I see about a gazillion caveats here - and please feel free to make constructive counter-proposals, or refinements to the proposal. --denny (talk) 16:51, 19 August 2014 (UTC)

Empowerment vs. entertainment

Just a quick thought. Do we care whether readers are merely *entertained* by the content or ought they to be *empowered* by it. If we inadventently optimise for *entertainment*, would we reduce our ability to succeed at our mission? What does this suggest about how we think about measuring value-added? --Halfak (WMF) (talk) 18:42, 2 February 2015 (UTC)

I think that this might be dangerous as we might decide on the values the reader's are supposed to have. I would not mind to have it expressed as whether we fulfill the information need of the reader, but for us to decide whether that information need is legit (i.e. empowering) or frivolous (i.e. merely entertaining) sounds a lot like pushing some protestant work-ethic onto what a good reader is supposed to read when they come to an encyclopedia.

Examples: are articles about Simpsons-episodes entertaining or empowering (e.g. empowering me which episode to watch next)? Are articles about quantum physical effects on the first few milliseconds of the big bang empowering in any meaningful way, or merely entertaining my curiosity? An author who reads biographical articles of Wild West criminals or Wrestling storylines might feel inspired, and thus empowered, to write an excellent novella, a kid in India reading about the effect of climate change on penguins might not get any empowerement out of this. I would not want to be the one deciding between these two categories. --denny (talk) 18:43, 3 February 2015 (UTC)

So, more articles about porn actresses = more knowledge? And translating en:Category:American pornographic film actors by ethnic or national origin in 280 languages is our "mission"? I'll take the "dangerous" pushing of protestant work-ethic onto readers, thank you very much. --Atlasowa (talk) 15:27, 8 February 2015 (UTC)

Uhm, yes, sure, more articles about any topic do mean more knowledge, obviously. But I would argue that some knowledge is still more important than other - as you can see from Op-ed. So whereas translating the mentioned category is indeed part of our mission, at least I wouldn't rank it very high. On the other hand, no one can really tell volunteers how they should prioritize the work they offer in their free time, anyway.

But you are probably right in another point: with putting an ordering on knowledge, I already acknowledged that not all knowledge is equal. But I remain uncomfortable with having a binary and explicit separation of knowledge into two categories called "empowering" and "entertaining". But the difference is, probably, merely gradual: any ordering reflects a POV, too.

I think it is important to remember that we should be conscious when we categorize knowledge by its value. This discussion helped me with that, and also weakened my opposition to Aaron's proposal.--denny (talk) 16:52, 9 February 2015 (UTC)