Research talk:Media Viewer preference elicitation

Welcome!

Our goal here is to inform a discussion of Media Viewer with objective data. Please feel free to raise concerns about the study and it's description. Your participation in this process is invaluable and we appreciate your time and patience. --Halfak (WMF) (talk) 17:24, 18 July 2014 (UTC)

"View quickly" and "View all the details" sounds like very loaded language. I disabled MediaViewer because it was appreciably slower than the old default, and I can't be the only one. Mogism (talk) 18:00, 18 July 2014 (UTC)

Thank you for proposing this research. I believe it is long overdue, but am glad to see it is now being undertaken. As originally phrased, the introductory comment suggested that the Wikimedia Foundation team is the only entity with concerns about the needs of Wikimedia's readers. This suggestion, of course, is absurd; Wikimedians are generally united behind a vision of sharing knowledge, and are not in the business of forcing unwanted things on any audience. I hope my addition to the text to address that problem is acceptable. -Pete F (talk) 18:13, 18 July 2014 (UTC)

I'd like to keep the research page clear of discussion around Media Viewer and the arguments hashed in the RfC. There's a space for such discussions, but the introduction of a research manuscript is not it. Also, I am not on the team that created the MV UI. I had no involvement in any work on the system up to this point, yet I am here trying to make sense of the situation, so it turns out that some of your changes are inaccurate. It might be helpful to note that the academic we tends to refer to the authors of a study. I hope you'll appreciate my cleanup and trimming of your contributions and not take it as an affront to your participation here (which I value deeply). I'm trying to keep the report as to-the-point and non-political as possible. --Halfak (WMF) (talk) 18:48, 18 July 2014 (UTC)

Pete F: On second thought, I think that summarizing the concerns from the RfCs could add value and context (as you noted in your summary). I've added a section for them and re-added your comment. Can you help me flesh it out? --Halfak (WMF) (talk) 18:51, 18 July 2014 (UTC)

Thanks @Halfak (WMF):, your edit is an improvement. I've made one further change -- this eliminates a potentially divisive "us-vs.-them" from the intro section, and introduces the concern as a general one. Will that work for you? -Pete F (talk) 19:01, 18 July 2014 (UTC)

I like it, Pete. I try as best as I can to avoid saying "we" for the same reason. Keegan (WMF) (talk) 19:08, 18 July 2014 (UTC)

┌─────────────────────────────────┘
Agreed. Thanks guys. --Halfak (WMF) (talk) 19:55, 18 July 2014 (UTC)

Comments from Spinningspark

I have two issues to raise with this document:

1. The whole thrust of the research seems to be to find a way of circumventing the result of the recent RFC. That much is made explicit by the final sentence "Our goal is to identify acceptable thresholds to determine whether or not to disable Media Viewer based on the data we collected." I don't think that there is any way the Wikipedia editing community is going to accept that this research should override an RfC with very high participation.
2. The research is limited to the population of editors, but the largest group of Wikimedia users is the readers. The one thing that could justify WMF ignoring the wishes of editors is in order to improve the experience of readers. Yet this group is entirely omitted from the research. As I said in the RfC, I believe that the one thing readers mostly want on clicking an image is a larger version, not to be taken to a slide viewer. Many sites provide a magnified image onclick or onmouse rollover without leaving the substantive page. This IMO would be a much better solution than media viewer. The file page and other functions could still be accessed with a corner menu or drop-down. SpinningSpark 20:46, 18 July 2014 (UTC)

Without taking a stance on whether limiting research to logged-in users is a good idea, I want to address what I think is a major misconception here. There are about half a million users logged-in users interacting with the English Wikipedia on a given month. The number of editors who participated in the RfC was around a hundred (that is, about 0.02%). The number of logged-in users who knew about the RfC and could have participated but have chosen not to - or even the number of those who know at all that there are such things as RfCs and that they could participate in them - is by all likeliness still just a tiny fraction of the total. The group of editors who express their wishes through RfCs or any other active means (in other words, those participating in public discussions and site governance) is not at all the same as editors (or logged-in users) in general.

Also, without going into whether this RfC had very high participation (that has been discussed enough at other places), the attitude that data should never be allowed to override the opinion of a couple dozen community members saddens me a bit. --Tgr (WMF) (talk) 22:32, 18 July 2014 (UTC)

@Tgr (WMF): The numbers you are referring to have been addressed time and again, as well as directly below, which was in full view of your reply here. Once again, polls are used in all types of political research. For example, we don't have to get feedback from 360 million Americans to get an idea about how divided opinion is about a President. And once again, most editors simply don't care about the MV issue, obviously. If they did we would see more participation in RfC's, and more feedback, for and against, on the MV feedback page. Your 'argument' could be made towards the findings of any RfC, that it doesn't reflect the views of 'most editors'. So why do we even bother with RfC's in the first place? Hello?? Because like all other polls, they reflect a general overview of consensus. Once again, the RfC was consistent with the RfC currently being conducted at Commons, consistent with the overwhelming negative MV feedback and consistent with WMF's own statistics for English and German Wikipedia. (All linked to below.) All of this is consistent with the idea that MV was introduced with an array of bugs and faults, some of which have yet to be resolved. Is it really so difficult to ascertain that MV is by and large not wanted? If you guys had jobs in the private sector and you turned out a package like this you'd have all been fired log ago. Sorry to be so frank, but after having this fifth wheel viewer forced on everyone and then after having to listen to all the excuses it's really difficult to sit here with a painted smile on my face while we have to repeat all the things you ignored the first time. -- Gwillhickers (talk) 22:56, 18 July 2014 (UTC)

@Tgr (WMF): How many people are on the Multimedia team at WMF? Please express your answer in terms of a percentage of readers of Wikipedia. -Pete F (talk) 23:08, 18 July 2014 (UTC)

@Peteforsyth: I don't think this is the right place to debate under what conditions are RfCs a good tool of site governance. The point I was trying to make is that there is just as much to learn from studying the preference of logged-in users as from the same with anonymous users. There is absolutely no reason to think the RfC is indicative of the preference of all logged-in users. --Tgr (WMF) (talk) 00:20, 19 July 2014 (UTC)

OK -- but @Tgr (WMF):, you're the one who questioned the legitimacy of the RFC here. I'm happy to leave that aside, but just want to point out that there are big concerns about the conclusions your team has reached about what the millions and millions of Wikipedia readers need -- not to mention how other considerations factor in. If your point is "it's difficult to know what those millions want," don't you have to concede that you guys could be wrong about the conclusions you're trying to force on the world?

This research appears to be half of a concession. But the important part of the concession is dialing back the mistake while exploring how to fix it. We've been OK for the last decade, we've got a top 5 web site -- what on earth is the urgency of keeping this software enabled in the short term?? -Pete F (talk) 00:48, 19 July 2014 (UTC)

• @Peteforsyth and Tgr (WMF): -- Pete did a simple yet quite unique job of holding WMF's feet to the same fire where they roundly dismiss the RfC of June 2014. How can they make such an assertion when only a handful of individuals at WMF decided what millions of readers and thousands of editors will need and want? Esp since their "approval" is based on a relatively small number of casual and naive viewers, obviously unaware of all the bugs and faults, who were presented a slide show and then asked -- "did you find MV useful"? The RfC of June didn't occur in a vacuum. It is consistent with the RfC being conducted at Commons, consistent with the consensus on the MV feedback page, and consistent with their own statistics for English and German Wikipedia. Given all the faults of MV it shouldn't be anything amazing that MV has received such proportions of negative feedback. To be frank again, certain individuals on the WMF project team at this point seem to be engaged in nothing more than a turf war. They may have started out with the best of intentions but lost sight of that when faced with criticism. Disappointed. -- Gwillhickers (talk) 01:07, 19 July 2014 (UTC)

@Peteforsyth: again, the point I was trying to make is not about the legitimacy of the RfC but about its predictive power - the results of the RfC do not predict the preferences of editors at all. That's a factual claim which is easy to support by e.g. comparing opt-out data for users with different levels of engagement (I did that at some point and there was a 30x difference for optout rates between users who visited the site and users who had a huge amount of edits).

Whether that questions the legitimacy of the RfC is not something that can be decided by data, so I think discussing it here would just obstruct on-topic discussion. I am happy to rant about it some other place if you are interested :)

As for what readers want/need (the two are not necessarily the same - another interesting discussion that does not belong here), I do think that in lieu of further data the Foundation has a much stronger claim for being able to guess the answer. The interface was designed by UX experts (I know that it is not fashionable here to refer to expertise without substantiating it with multiple references to notable tertiary sources, but it is still a thing), it was tested with actual users, similar approaches have been taken by virtually all popular sites... I am by no means saying that is reason to be certain (even organizations with more expertise and experience in software development and far more resources make huge design blunders every once in a while), but if we had to decide with no chance of learning more about actual user preferences, I think leaving Media Viewer enabled as default would be the reasonable choice.

Fortunately, we can learn more, so we should. --Tgr (WMF) (talk) 01:41, 19 July 2014 (UTC)

Of course there's a big difference between experienced users and casual visitors. The casual ones might think it's just as terrible as the rest of us but they don't visit Wikipedia often enough to care, don't know the opt-out exists or can't find it.

This is exacerbated by the fact that the opt-out link seems to pop up and vanish at random. Anyone but the most determined might have looked for it at some point when it didn't exist and then gave up.

And frankly your smug "UX experts" comments are ridiculous. Windows 8 was designed by one of the best UX teams money could buy and MS has been backpedaling ever since. So could WMF please follow the lead of MS (!) and and start listening to their users? 93.104.189.190 19:47, 12 August 2014 (UTC)

Full Ack. User-centered design was state of the art in 1986. We should be moving in this direction aggressively. Of course, this does not excuse past/current mistakes. I hope we'll be able to count on you to engage with us via user-centered design processes in the future. --Halfak (WMF) (talk) 13:55, 13 August 2014 (UTC)

Hi Spinningspark: Thanks for your thoughtful comments, which are much appreciated. You make a good point that the original study proposal was "limited to the population of editors, but the largest group of Wikimedia users is the readers." We just discussed your recommendation to broaden the scope of this investigation and would be prepared to study both logged-out and logged-in users in this controlled experiment, if feasible without major development. Initially, we kept the scope of this proposed study very focused, because our analytics resources are quite limited at this time. But we are just as interested as you are in learning more about the preferences of both user groups, so we can make informed decisions about our next steps. So for now, I have revised the original proposal to include logged-out users as well, assuming it can be done practically in our time-frame. Note that this study would be a significant investment of time for our small multimedia team, so we would want to confirm that enough community members would be willing to base a decision on this new data before committing to undertake this study. Lastly, I encourage you to assume good faith on our part -- and think it possible that we are just as committed to the success of our movement as you are. So the purpose of this research proposal is to put our differences aside and try to focus on the needs of our users, based on objective observations of how they use this tool. I hope we can make this happen together in coming weeks. Thanks for your understanding. Fabrice Florin (WMF) (talk) 03:19, 19 July 2014 (UTC)

Comments from Gwillhickers

I have to agree with Spinningspark and others who have echoed what I have articulated before in two RfC's and on the Media Viewer feedback page. (1, 2(archived)) Indeed there is an ongoing attempt to circumvent the RfC of June 2014 by a few individuals at WMF. It was once explained to two WMF members that the RfC is a poll, and like any other poll, it reflects a greater picture. We don't have to get feedback from thousands of individual editors and readers, most of whom obviously don't care about the MV issue either way, to make an overall estimation about how MV has been received on English Wikipedia. Claiming that a large group of editors are not represented by the RfC is moot, as they simply don't care, as thousands of editors only log on occasionally to do general fixes, cleanup and spot editing to subject text. The consensus of the RfC was consistent with the RfC at Commons and with the MV feedback page itself. All of these forums were further consistent with WMF's own statistics which reveal that MV is not wanted or needed on English (and German) Wikipedia. All of it ignored! As MV was introduced with a number of faults, it's not difficult to ascertain why there is overwhelming negative feedback for this slideshow. Yet after all of this has been outlined, project team members continue to fall back on what seems to be pat rhetoric, that the RfC doesn't represent most editors, as if any editor with average intelligence is going to approve of this thing after becoming aware of all its faults. The fact that MV was introduced with many faults, some of which have yet to be corrected, by itself lends credence to the findings and the validity of the RfC, which, again is consistent with their own feedback page and MV statistics. The fact that this has all been ignored more than reveals that their 'reasoning' has been quite selective and suggests that there is simply a perceived turf war going on at WMF, where they seem to have contempt for the idea of introducing checks and balances to their ambivalent authority over all of Wikipedia. Arbitrators would do well to keep this perspective in mind when they are evaluating the excuses made by individuals on the WMF project team as to why, at this late date, they are refusing to make any concessions regarding the default status of MV on English Wikipedia. -- Gwillhickers (talk) 21:58, 18 July 2014 (UTC)

Hi Gwillhickers: Your comments here seem to be off-topic. The purpose of this page is to focus on the specific research study that is being proposed, not on the politics of the English RfC. We invite you to keep posting your personal views on the RfC pages, but please leave politics out of this discussion. Thanks for your understanding. Fabrice Florin (WMF) (talk) 17:37, 19 July 2014 (UTC)

He's welcome to post whatever he likes. If you have a problem with that, go and find yourself a job elsewhere. You'd be doing as a big favour.

Commment from Adam Cuerden: I like Media Viewer, but...

Can we make it really, really easy to turn on and off, as in, as easy as watchlisting a page? Just press one button, at the top of every page, to turn it on with no reload, hit it again to turn it off? This lets me use it whenever it suits me, but when I'm doing stuff it's just going to get in the way of, I can just turn it off, and be perfectly able to turn it right back on again after.

It might also be good to have the ability for pages to turn it off, such as en:Wikipedia:Featured picture candidates, where checking the file page is pretty much required, as it needs a full review.

Ideally, I'd like to see turning it on and off for non-logged in people work as well, presumably by using cookies or the like as an alternative to setting.

This might be slightly difficult, but I'm pretty sure that this would make a good rollout model for more than just this sort of thing. I think that a little button to turn it on and off would make people far, far more forgiving of any hiccups with the launch model - the WMF, unfortunately, is a little prone to launching new features on full rollouts, instead of just making opt-in very, very easy for the initial launch - and it tends to put people off what, with a slightly more careful launch, would be hailed as innovative new features. Adam Cuerden (talk) 23:31, 18 July 2014 (UTC)

@Adam Cuerden: I am happy to discuss this at the product talk page, but we should try to focus the discussion on this page to the research on Media Viewer, not on Media Viewer in general or how it should have been launched. --Tgr (WMF) (talk) 00:11, 19 July 2014 (UTC)

Dealing with a botched launch: This research plan isn't the way.

That said, the current research plan is a terrible idea. Frankly, you get one shot at initially launching something; if that launch you eat up the community's good will, you have to win it back. Turn it back on without a lot of careful work, and the metaphorical pitchforks and torches are bound to come out.

Make it really easy to turn on. Have the button take up the top 60 pixels of the webpage, with a request to "try out the new, improved Media Viewer! Click here!" But the moment you go mucking with people's preferences and overriding an RfC, you're going to really upset people, and that's going to make it far, far, far harder to ever be able to move forwards.

Another possibility: We have galleries on Wikipedia. Perhaps that might make for a useful relaunch chance. If, for example Media Viewer allowed galleries to be displayed as a slideshow - started by clicking a small, right-aligned button just above the gallery, for example, that new functionality would be, I think, very widely accepted, while getting you a new chance to launch. New functionality is generally accepted much more readily than changes to old functionality, and once people get used to it, it can be progressively rolled out from there. Adam Cuerden (talk) 23:31, 18 July 2014 (UTC)

Yup. Doing research like this was a really good idea before designing or launching the software. And if you have any hope of moving forward with the software, it would be a really good idea to do it now (with extensive input on methodology from outside WMF), but that research should only be done after the software is disabled by default. -Pete F (talk) 23:33, 18 July 2014 (UTC)

@Adam Cuerden and Peteforsyth:. It appears we're being led around in circles. More research? What have they been doing all these weeks? We already have the well articulated experiences revealed in two RfC's, the MV feedback page (1, 2(archived)) -- all consistent with WMF's own statistics -- all consistent with the many bugs and faults. Yet they still refuse to make any acknowledgements. e.g. Tgr's comment: "the results of the RfC do not predict the preferences of editors at all." 'At all?' At this point I think it's safe to assume that they have no intention of making any acknowledgments or concessions whatsoever. Given the very low approval among editors on English Wikipedia I recently asked @Fabrice Florin (WMF): why the WMF project team was still set on making MV the default for English/editors, and the only reason he offered was "offering different user experiences for editors and readers can be confusing." -- yet at the same time he commented that "our readers are a lot smarter than you imply". Now we have Tgr's comment: "That's a factual claim which is easy to support by e.g. comparing opt-out data for users with different levels of engagement" -- WMF's own statistics have already revealed the consensus for English and German Wikipedia. i.e. Most Editors overall.

WMF statistics for English Wikipedia: Usefulness by role
% Readers who find it useful: -- 37% (63% did not find it useful)
% Editors who find it useful: -- 21% (79% did not find it useful)
% Frequent Editors who find it useful: -- 16% (84% did not find it useful)

Bear in mind that when we speak of English WP editors, it should be understood that we are referring to the greater bulk of editors in all of Wikipedia. i.e.The numbers of articles and users in English Wikipedia dwarf the numbers in all other Wikipedias, and accordingly the number of English WP editors (active users) are in proportion. Bear in mind also that many (most?) of the articles in non English Wikpedias were copy/translated from English Wikipeida by editors at English WP. i.e.Since most Editors at English WP did not approve we are referring to most editors overall. i.e.10% of New York City's population is five times as large as 50% of 'Smithville's' population, so I wouldn't be swayed when they refer to percentages without qualifying the quantities involved. At this point if various individuals at WMF are still insisting that they need to do even more research I can only wonder if they've been paying attention and/or they simply don't care about outside opinion. I think this is a fair assessment given their continual denial of their own feedback and statistics, not to mention the categorical dismissal of the Wikipedia RfC process. -- Gwillhickers (talk) 17:05, 19 July 2014 (UTC)

Hi @Adam Cuerden, Peteforsyth, and Gwillhickers:. As proposed above, we invite you to focus your comments on this page to discuss the proposed research study, rather than your political views on what the WMF should or shouldn't do with Media Viewer. As a rule of thumb, we try to base our product decisions on user data -- and this new research can help us all gain a better understanding of actual user behavior, so we can plan our next steps for this feature. To that end, we invite community members who have experience with this type of research to chime in about the proposed methodology, along with any practical suggestions for improving the study (not the product, which can be discussed here).

What is the broader context of this research?

The intro section reads (as of 21 July):

This data will inform our understanding of users' preferences and help support a decision about whether or not to keep it enabled by default for logged-in and/or logged-out editors.

What other information will go into that decision? What research will produce that other information?

To date, it seems to me that there have been a number of questionable assumptions underlying the development of the Media Viewer. For instance:

• Wikimedia's strategic goals have not (apparently) been a part of the reasoning that produced the software. Specifically, I have seen no consideration of whether or how the Media Viewer will increase or decrease participation levels (i.e., recruitment of new contributors, or retention of existing contributors). This seems like a major oversight, and I believe it is entirely unrelated to the preferences of readers in how they consume media.
• Are rights of copyright holders, and the subjects of photos, adequately reflected in what the Media Viewer portrays? (The minimum standard would be legal compliance; but ideally, we would aim much higher, and base software decisions on standards more like respect, compassion, or collegial behavior)
• If some small percentage of files (say, 2%) are displayed in a way that is far from ideal (e.g., a very tall file, or a file with a high level of detail), would we expect that to noticeably impact whether or not readers prefer the software to the alternative? Is this a significant consideration nevertheless? If it is, do we consider it acceptable to have broken software implemneted by default while it is fixed?
• If creating software that matches readers' preferences involves a tradeoff of disrupting editors' workflow, is that an acceptable tradeoff? Need a software solutoin create that division, to begin with, and should it be considered "out of beta" if that division still exists?

Given the many faults of the process leading us up to this point, I am not confident that this research, coming from the same team that made these (and many other) mistakes and has resisted acknowledging them, will do any good. It seems more likely to add noise to an already noisy state of affairs. -Pete F (talk) 23:38, 21 July 2014 (UTC)

Hey Pete F. The pot shots at the Media Viewer team are getting a little old. I got involved and sought out feedback from you and others who participated in the RfCs to make sure that any follow-up research that we do would serve to inform future discussions. I'd advise you to spend your energy focusing on what can be done now as opposed to the (obviously good-faith -- to me anyway) mistakes that the MV team made in the past.

With that out of the way, it looks like the discussion you are trying to start here is a mixture of movement strategy, design/display bugs and objective issues we might address. I'd like to skip all but the potential to build improved understandings in this research study.

Now, to actually address the bit of your post that I think is relevant to this study: Is there an acceptable trade-off in reader vs. editor needs? Or, maybe more objectively we might ask how MV hampers editors and/or helps readers. I propose that any workflow change -- even amazingly good ones -- will have an immediate, negative effect on the productive work of regular users of old workflows while they come to understand and adopt the new UX. So, I'd hypothesize that, if we were to measure editor productivity in the "Media Viewer" condition of the study, we'd be likely to see a minor decrease followed by a recovery period.

Hypothetical productivity dip. A hypothetical productivity dip is plotted over time after a workflow change is deployed. Note the sharp decline followed by a steady recovery.

I propose that, if we do decide to observe editor productivity changes around the deployment of Media Viewer, we'll need to build statistics on the amplitude and duration of such a dip in order to understand the difference between a dips caused by a good workflow change and those caused by bad workflow changes. --Halfak (WMF) (talk) 15:45, 22 July 2014 (UTC)

I don't think anything I've said can be accurately characterized as a "pot shot." There have been criticisms of the methodology of the MV team's reasoning up to this point, of the design of the survey, etc., and I have seen no substantial effort to engage with it. I of course recognize that the Multimedia team -- just like everyone else involved, as far as I can tell -- is engaging in good faith, and is trying to do good things. I have never said otherwise.

I disagree with your assessment of what is germane to the present decision. The final section of the page reads as follows:

We will invite community members to discuss this proposed study on the discussion page. Our goal is to identify acceptable thresholds to determine whether or not to disable Media Viewer based on the data we collected.

That is a substantial and ambitious outcome, and should be based on a pretty comprehensive exploration of the impacts of the software. It seems that WMF is intent on making this decision based solely on what readers and editors like; I strongly disagree that user preference is the correct determinant for such a major decision. You are entitled to your opinion, but you are not entitled to declare mine out-of-bounds. Furthermore, when you ask for comments, and then you find yourselves arguing not just with one or two, but every single person who chose to come and comment; and when the project lead then finds it necessary to post a 942 word clarification to a 661 word proposal...well...forgive me if I suggest the natural response might be something other than pointing the finger outside your office, at the community that your organization supposedly exists to support. -Pete F (talk) 19:02, 22 July 2014 (UTC)

Pete F, I don't mean to start an argument here, but I'd just like to point out that here we are again, discussing the behavior of the Multimedia Team and Fabrice. If the Multimedia Team and their behavior were the subject of the study, I wouldn't complain. However, Media Viewer is under study. In my response to your original post, I made an effort to expand on your thoughts about reader vs. editor needs. Here, I'd like to riff on your concern about measuring preferences.

What is a preference? As a behavioral scientist, I don't really believe in them. I'd much rather watch someone act (e.g. use the UI) than ask them for their opinion (which can be tied up in political battles, imagined future needs and other strong confounds). However, I suspect that preference is a decent proxy for "what works for me". Just as with the RfC, we would be measuring preferences in the study proposed here, but unlike the RfC, we have the opportunity to gather much wider input -- and for behavioralists like me, we aren't really asking for someone's opinion as much as we are asking them to act. Users will show us which media workflow works for them by explicitly deciding to continue using one or the other. By surfacing the choice more clearly and asking users to decide, we both:

• gather a bunch of data that would help us pick a default setting that minimizes the need to adjust one's preferences
• make it really easy for those who disagree with the majority to have it their way

It's not perfect and I agree that it would be better to have done this analysis beforehand, but here we are. We have no time machine, so let's make the best of it. --Halfak (WMF) (talk) 19:23, 22 July 2014 (UTC)

@Halfak (WMF): OK. Let me try to summarize the points where I think (?) you and I might agree:

1. Knowing what interface users consider to "work for themselves," with other variables controlled for, would be useful information.
2. We do not yet (reliably) have that information -- either from the surveys done to date, or from the way the "disable" feature has been implemented to date, or from the RfC.

My central contention is that in the absence of such significant information, we should not be varying from the interface that we know has worked to build Wikipedia to date (even if imperfect), or from the decision-making processes that Wikipedia considers legitimate (even if imperfect). The software should be rolled back to its previous state while information is gathered to make a better decision.

An area that will not be addressed by this study (and I acknowledge, this is not, in itself, a problem with the study):

• A great many of the "people who disagree with the majority" (and this is true regardless of which position turns out to be the majority) disagree not out of a matter of personal preference, but due to beliefs about how we should present Wikimedia to our audience. Those beliefs have to do with a lot of things (e.g., what aligns with readers' expectations based on other web sites; what supports the strategic goals of the Wikimedia movement; what adequately honors the contributions of photographers and subjects of photos; etc.) So, the concerns of those people are not addressed by an individual ability to reverse the default position. The choice of a default has a very broad impact. If it didn't, I don't think English Wikipedians, or WMF staff, would be arguing so passionately about what the default will be.

Regarding "let's make the best of it":

• I think you will find that many of us are very happy to do that if the WMF rolls the default setting back to where it was in May 2014, and for many years prior to that, as strongly requested by a process that has up until now been considered legitimate and binding, while we discuss how to plot an effective path forward. If you want to include a very easy "here's how to re-enable the feature" button somewhere, that's probably fine. But the longer you delay that decision to re-enable, in my view, the more you dig yourselves into a hole in which it is impossible to build a coalition that can "make the best of it" and effectively plot a path forward. And the more damage you do to the willingness of the volunteer base that creates Wikipedia to engage with the project at all. Sowing the seeds of hostility and division should never be an acceptable trade-off when enabling a new software feature. We know that these things damage editorial participation. But by boldly changing a major feature of the site, and then ignoring the strongly negative reaction of its editorial community, that is exactly what WMF has done -- and continues to do. It is not possible for you to mitigate the impact of those decisions on tens of thousands of people, by arguing with me. But it might be possible for you to mitigate the damage by rolling back the default setting. -Pete F (talk) 19:56, 22 July 2014 (UTC)

┌─────────────────────────────────┘
Pete F, I understand your complaints about the current status of deployed Media Viewer. I have no power to change things though. I'm not even on the Multimedia Team. All that I can do is try to add some knowledge to the conversation. If you want to talk to the decision makers -- well, they won't be looking anywhere near these pages until we complete a study that adds some insights. I don't mean to discourage you from discussing your concerns about the practice of releasing community software and I certainly appreciate your continued discussion of this research study. However, I do suggest that this particular talk page is the wrong place to raise concerns about "the WMF" and that continuing to do so only delays our best shot at having a more informed conversation about this and future deployments. --Halfak (WMF) (talk) 20:20, 22 July 2014 (UTC)

@Halfak (WMF): I understand. (Though since @Fabrice Florin (WMF): has posted on this page, it's seemed reasonable to expect that at least some decision-makers are paying attention to it.)

I do think (some version of) this study would yield useful results, but I think (a) the study as submitted has problems, and (b) the costs of the study outweighs the benefits. If you choose to go ahead with the study, though, even in the absence of the community support you sought, that's your decision. If the reasons you didn't generate support are not interesting to you, there's not much I can do about that. -Pete F (talk) 20:30, 22 July 2014 (UTC)

Good point. Fabrice has some substantial say in whether Media Viewer remains deployed or not. However, I think that you'll find him and others interested in discussing the status of Media Viewer at mw:Talk:Multimedia/Media_Viewer and en:Wikipedia_talk:Media_Viewer.

However you've raised some concerns that I'd like to discuss. You said "the study as submitted has problems". What problems? Let's work those out! I'm not sure what you're referring to with regards to costs. If it is social capital, I think you'll find that I agree. If it is developer/analyst time and energy -- well, the devs were already going to add the options panel anyway. Really the only additional costs are (1) my time for engaging in this conversation and helping the MV team put together a study plan and (2) the dev time to add data recording to the options panel. For (1), I'm still on the fence whether this is worthwhile. It's a substantial amount of work. For (2), that's super easy to do on top of the current work. --Halfak (WMF) (talk) 21:27, 22 July 2014 (UTC)

Thank you, that's helpful info -- I didn't realize that en:WT:Media Viewer even existed, and from the looks of the amount of discussion there, I'm not the only one. mw:Talk:Multimedia/Media Viewer has not seemed like a place where people get listened to, but I do watch it.

Yes, I do see the major costs associated with MV as being social capital. The (potential) social costs are widely varied, and include an environment of increased hostility, reduced conversion of readers to editors, time spent in endless discussion as opposed to building an encyclopedia, loss of a sense of affiliation, etc. etc. Social costs are also, of course, difficult to measure, and I worry that to an organization that (rightly) values measurement, it might look as though they don't exist. But "difficult to measure" and "doesn't exist" are two very different things.

Substantive problems I see with the proposal:

• Summary of points from RfC seems really, really weird -- not at all reflective of my understanding of the key points
• Intro states that the results of the study will "help support" a decision, but doesn't say the results will "determine" a decision. In my view, things like strategic goals and treating photographers and models with dignity and respect would be important things to also inform a decision. However, in the conclusion, it says the study will "determine" whether to have MV enabled or disabled by default. Which is it? "help inform" or "determine"? If it's determine, why are these other factors being ignored, and what kinds of conclusion will support what determination? Will that be determined only after the data is in? Isn't that backwards?
• As has been pointed out above, the wording of the two choices presented to the viewer is ridiculously biased toward the WMF's preferred outcome (much like the survey, and the inadequate summary of the RFC's findings)

-Pete F (talk) 00:48, 23 July 2014 (UTC)

Hey Pete F, allow me to address your points in sequence.

Social capital

Regretfully, these things are not my call. However, I see generating new knowledge as useful regardless of the social context. While I'm concerned about external discussions and what this whole process means for future staff/volunteer relations, those concerns do not affect the nature of the subject at hand. We'll make better decisions as a group if we're more informed.

"Summary of points from RfC seems really, really weird ..."

Fine. I've already invited you to help me flesh them out. So Fix It! If you're not game to help fix them, then I'd at least expect you to explain what's so "weird" them that it required you to say "really" twice.

"Intro states that the results of the study will "help support" a decision ..."

New knowledge is useful for helping people make a decision. In the context of this study, I'm not very concerned about the decision-making process. I'm more concerned about informing decision-makers. Informed decision-makers should be desirable no matter the decision process.

Wording of options

Now we're talking. I agree that the wording might be confusing. I see that as less problematic since users will be able to test out the different workflows before settling on their workflow of choice, but it should still be as clear as we can manage. Do you have a proposal for improved wording?

--Halfak (WMF) (talk) 14:51, 23 July 2014 (UTC)

"We'll make better decisions...if we're more informed"

Absolutely -- but if and only if that information is rigorously generated.

"So fix it"

This goes to the core of my biggest concern about the current state of the WMF. It's important to the success of the organization that it learn to understand how its universe works. But thus far, in at least some cases, it has demontrated a poor ability to understand. (The very fact that WMF was surprised to find this feature become controversial is one of many bits of evidence for that.) Interpreting an RfC is hard work, and while I appreciate your willingness to hear my take, that is a large piece of work that I will not do (especially not for free -- this is getting into the kind of work I do for pay) on behalf of WMF, in support of a project that I believe should be put out to pasture. I think WMF should get it right, but the overall cost of not getting it right is not (from my perspective) so high -- [if it were to get it wrong,] WMF [would] simply continues to undermine its own credibility in developing and marketing this feature. [From my perspective, that outcome is not bad enough to motivate me to do free work.]

"I'm not very concerned about the decision-making process"

Valid point. In that case, I suggest removing a couple sentences that will likely cause confusion down the road. Maybe we agree that the proposal should focus on the merits of the study itself, and placing it in context should happen elsewhere.

"Users will be able to test..." and "Do you have a proposal for improved wording"

You may know users' general behavior better than me; but when I encounter a question like this, especially in the middle of a process I've done many times before (like clicking into an image on a Wikipedia article), I typically make a decision very quickly, so I can get back to doing whatever it is I'm trying to do (like learn about a sports team, or whatever). I don't sit there fiddling around with different ways I can view the photo of the team. So I would expect that the phrasing that informs my very quick decision would be very important. My hunch is that a lot of readers are like me. This, of course, is another hypothesis that could be tested...and maybe it has been. On "improved wording," see above -- I think this is the job of those designing and presenting the proposal, not those commenting on it. -Pete F (talk) 16:14, 23 July 2014 (UTC)

┌─────────────────────────────────┘
Hey Pete F. You're preaching to the choir[1]. My business is the development and execution of rigorous studies and my area of interest is the efficient and productive function of Wikipedia (and similar phenomena). I was a volunteer for 6 years before I started working for the WMF and I still spend a large portion of my free time continuing my volunteer work. In my role as a research scientist, I often publish studies that show that what we're trying isn't working -- because I value truth and transparency over my own job security.

Now it seems that again you've regressed to bashing "the WMF" rather than engaging in productive conversation about the material artifact of this talk page. I've welcomed you participate in my work. I've iterated on the project description and discussed the methods with you. While I have been critical of some of your changes, I've supported others and I've been generally agreeable despite your negative and unproductive tone -- finding ways to work with you rather than shutting you out. If this is not a good example of WMF staff being part of the community, then I've got to admit that I have no idea what you expect. If you're here to help me run a rigorous study, then good. Let's get on task. However, if you're here to lambaste "the WMF" and you can't possibly see me as an ally, then I suggest you take your fight elsewhere because I don't wish to participate in it. --Halfak (WMF) (talk) 17:13, 23 July 2014 (UTC)

@Halfak (WMF):, I'm not sure why this comes across as bashing -- but I think I maybe found the sentence that gives that impression above, and [added a little in square brackets] that might clarify the point, and hopefully remove that impression. I'm trying to explain why I won't do a bunch of free labor, not bash the WMF. The WMF is necessary to our movement, and I don't have any intention of bashing it as an entity. But I do think pointing out problems in the things it does is very important, for much the same reasons you describe above.

I hope you have not gotten the impression that I have any ill will toward you or any of your colleagues. I think some very bad decisions have been made here -- and I understand that for the most part they are not yours -- but I do not see bad people here. Maybe I should not assume that my good will toward you as people is obvious. I think sometimes my effort to be clear comes across as an intent to disparage -- which is not the case. I respect you as people, I respect that you do good work, I respect that your intentions are good. I am sorry that I have not made that sufficiently clear.

Also, I want to point out -- I was originally invited here to discuss the study, and that's what I've been trying to do. It's a different thing to ask somebody to help design the study. It's not wrong to ask somebody to do that for free, but I don't think it's fair for you to expect me to do it. I'm happy to bring this discussion to a close -- I think you have given me a good opportunity to express the issues I see, and I appreciate your attention, and your efforts to keep the discussion productive in spite of all the noise (and I'm certainly responsible for some of that noise). I hope the things I've pointed out are of some use to you, and I hope you don't take me bowing out as an act of protest. I feel that I have done what I was invited here to do, and I never agreed to do more. -Pete F (talk) 17:47, 23 July 2014 (UTC)

Research Clarifications

Media Viewer daily survey results by English users in June and July 2014

Media Viewer overall survey results by English users in last 2 weeks

Media Viewer graph of opt-in and opt-out events on the English Wikipedia, from June 27 to July 20

To make sure we're all on the same page, here are some clarifications about the research we have done so far, which we hope can provide more context about the many numbers cited on talk pages such as this one.

The multimedia team ran a user survey during the last phase of feature development for Media Viewer, from April 10 to July 8, 2014. The main goal of that survey was to collect feedback from a wide range of users, so that we could improve Media Viewer based on their comments. This turned out to be a valuable tool for that purpose, as it enabled us to hear from casual editors and readers, who are typically not comfortable posting on talk pages. During this period, we collected 18,199 responses in 8 different languages, and hand-coded thousands of comments to identify and prioritize over 50 key issues, as shown in this survey breakdown.

During the survey, we also tracked the percentage of users who found Media Viewer useful, and used this as a temporary approval metric, to get a sense of what people thought of the tool. For the first few months, we saw high approval rates in 60-70% range from thousands of users -- especially French, Portuguese and Spanish users. We also observed a gradual increase in approval rates, as we fixed more bugs and developed more features requested by users (for example, Hungarian approval started at 42%, and ended at 62%).

Given these high approval rates in most languages we tested in, it seemed reasonable to enable Media Viewer by default on the English Wikipedia, where over 14k users had been testing the feature in beta since November 2013, with favorable responses. Initially, we were surprised that English survey responses showed significantly lower approval rates at launch. But after a few weeks, we noted that English results also kept improving over time: total cumulative approval from English users grew from 28% to 36% (as shown in this dashboard, while average daily responses increased from 23% daily approval (6/4, after deployment) to 47% daily approval (7/8, the last day responses were actively invited), as shown in this graph (see thumbnail to the right). While daily response rates are comparatively low, the increase is consistent over time: many users found Media Viewer more useful as new features were developed and as they became familiar with the tool.

In the last two weeks from June 24 to July 8, more English users found Media Viewer useful (50%) than not (39%), based on 1,291 responses for that period, as shown in the graph to the right and on this dashboard. It's also worth noting that on average, readers found Media Viewer more useful (62%) than editors (39%) during these last 2 weeks (see linked dashboards).

We ended all surveys on July 8, because we had collected enough user feedback to inform feature development. We had never intended for these surveys to be a long-term metric for Media Viewer -- particularly since they were optional, making them prone to self-selection bias. Though it is likely that English approval rates would have kept growing as they did in other languages, we wanted a more reliable method for tracking the usefulness of this tool going forward.

On June 27, we started collecting opt-in and opt-out rates instead, as a more objective metric to track the adoption of this tool: you can see the live results on this global dashboard and on on this English Wikipedia dashboard (see also thumbnail to the right). These first results show that many more logged-out users disable Media Viewer than logged-in users: this is not surprising, if you consider that we serve about ~500 million readers versus ~85k active editors. We also note a 26% decline in opt-outs since we started tracking this metric, with fewer logged-out users disabling over time, as shown in this dashboard. And it's worth noting that a large portion of logged-out users opt-in as well as opt-out. In coming weeks, we hope to provide more analysis about these preliminary observations, as well as present them as a percentage of active users, as outlined in this ticket. A preliminary analysis of total registered users who disabled Media Viewer in their preferences suggests that they represent about 0.34% of total users who touched a page since Media Viewer launched on the English Wikipedia.

While this new opt-in/out metric shows promise for tracking user adoption trends over time, we believe that an A/B test as proposed for this study would provide a very useful measure of user preferences about Media Viewer, which can inform our decisions about default settings for this tool. As User:Halfak (WMF) points out in the proposal, setting either the Media View or the File Page as the default result of a file click can strongly affect the results of preference elicitation: we expect to get more reliable data by randomly splitting default preferences between users (50/50) by the time that they first click on an image before asking for their their preference.

I hope that this overview of our research so far is useful -- and can clarify why we think this new study can help us all better understand what our users think of Media Viewer. Going forward, User:Tgr will lead the discussion of our next steps for this research, but I wanted to provide this context, along with my perspective of why this study seems important, from a product management standpoint. Fabrice Florin (WMF) (talk) 01:55, 22 July 2014 (UTC)

To clarify, 26% decline in opt-outs refers to the daily number of optouts. Also, the 0.34% optout ratio is relative to users with a recent user_touched entry, which does not mean they touched a page (I have been spreading that misconception for a while; sorry about that). It pretty much includes all logged-in users who visited the site recently. --Tgr (WMF) (talk) 23:15, 22 July 2014 (UTC)

To clarify, the survey doesn't give "approval rates" and didn't ask about "usefulness". See the survey talk page for details. --Nemo 12:39, 10 October 2014 (UTC)

How to determine the winning design; UI factors affecting clickthrough and opt-out rates

The overall proposal looks sensible to me. I would advise the team to make it more explicit under which conditions one of the two options would be considered a winner (it's not very clear from the current write-up what would determine the superiority of one of the two designs).

I also wanted to raise some concerns on how the UI and UX design will impact user behavior and the validity of the data we collect.

The proposed new positioning for the options button in MediaViewer (test) is substantially more salient than the corresponding button in the default view (control). On top of this, MediaViewer has a much smaller number of UI elements and links than the default view (where it's much easier for the user not to see the options button). As a result of these two factors, I expect a higher proportion of users will find and click on the options button in test compared to control.

Why is this problematic?

1. more people will find the cog button and click on it in MediaViewer than in the control condition
2. among those who expand and decide to try the opt out only a fraction will be able to re-opt in
3. as a result we will take the initial baseline opt-out rate of MediaViewer as an indication that users like it less, where in fact this is because of the saliency of the control.

Should this happen, the results would be invalid.

In order for the two conditions to be comparable, the saliency and positioning of the cog button should be almost identical in each condition and it should be equally easy for users to switch back and forth between the two conditions, i.e. re-opt in if after they opt out if they don't like the user experience.

I don't think we can easily modify the positioning of the options button in control, but having the Viewing Options panel pre-expanded on the very first user impression of the page in each condition would mitigate this problem. Also, we should track the rate at which people re-opt in in each condition and not only track opt-outs (it's not clear if this is the plan when reading the proposal).

--DarTar (talk) 14:58, 25 July 2014 (UTC)

@Fabrice Florin (WMF) and Tgr (WMF): ^^ --EpochFail (talk) 18:09, 28 July 2014 (UTC)

Sorry about the lack of responses on this page. There is a lot of discussion going on what direction the development/research on Media Viewer should take, and it's still not certain whether this study will be part of it, hence the inactivity.

To answer the specific question, the plan is to 1) move up the optout menu to the cog icon for everyone 2) pre-expand the menu for people who participate in the study. I'll make the description more clear. --Tgr (WMF) (talk) 19:23, 28 July 2014 (UTC)

Postponed

Hi all,

this study is being postponed for now. We are reassessing many of our UX assumptions, and the optout panel design might get significantly changed in the process. Since the study depends on implementing that panel, it won't happen for a while (at a wild guess I would say a month or two).

Thanks to everyone who contributed so far; we do intend to follow through with this, but in the short term, small-scale usability testing is a more important priority. --Tgr (WMF) (talk) 21:09, 30 July 2014 (UTC)

Thanks @Tgr (WMF):. It would be nice to know more about the reasoning behind this decision -- and, as always, to know why mere user preference is considered to have such central importance to this project, rather than furtherance of the Wikimedia strategic goals that we worked so hard to establish a few years ago. (I realize this may not be a question you can personally answer, but I am surprised that nobody from the organization has come forward with an answer at this late date.)

I am glad that I did not take @Halfak (WMF): up on his request to "get on task", which would meant putting aside other things to take on a good deal of work, without pay, to meet an organizational timeline that apparently didn't mean anything after all. -Pete F (talk) 01:43, 31 July 2014 (UTC)

Tgr (WMF) and anyone else, I find it painfully unsurprising that this project has been dead for four months. The WMF's handling of Media Viewer has been unrelentingly awful.

While I'm here I would like to point out that the Mingle card on this is severely defective. It equally tracks initial opt-in or opt-outs, but then it details special treatment only for re-enabling Media Viewer and counting only the opt-ins. It does not detail equal handling for someone who re-enables the File page and counting those opt-outs. This may just be a drafting error on the cards and the programmers were supposed to handle both directions identically, but I don't know whether to laugh or rage that the card has TWO opt-in counters only ONE out-out counter. If this project ever goes anywhere, please fix the Mingle card to specify identical treatment in both directions. Alsee (talk) 22:43, 19 November 2014 (UTC)

We dropped this since no one seemed to really care about it and most RfC participants already made up their minds one way or another. (For reference, the resting place of the task after the Phabricator migration is phab:T77682.) I forgot to mention that here; sorry about that.

We are counting opt-ins and opt-outs (the research project would have been about creating a setup with a control group so the power of defaults can be eliminated); I'm not sure I understand your objection, but there is not much room for creativity in how they are tracked - increase opt-in counter when someone opts in, increase opt-out counter when someone opts out.

FWIW Commons was both opt-out and opt-in for a while, which makes a comparison possible (although a poor one since MediaViewer had less features when it was in the opt-out phase, and opting out is easier than opting in): about 1560 users have opted out between May 15 and Aug 20, and 1212 users have opted in since then (a 97- and a 112-day period, respectively). The query used is in phab:T71363. --Tgr (WMF) (talk) 23:36, 10 December 2014 (UTC)