Research:Article feedback/Usability

Article Feedback v5	Data & Metrics	Stage 1: Design (December 2011 - March 2012)	Stage 2: Placement (March 2012 - April 2012)	Stage 3: Impact on engagement (April 2012 - May 2012)
WP:AFT5 (Talk) Feature requirements Dashboards Feedback Feedback moderation	Overview Article samples Feature data Clicktracking data	Volume analysis Quality assessment Reader survey / Team survey Usability testing	Volume analysis Quality assessment	Conversions and newcomer quality Final tests Quality assessment Research report (2012 Q4) Moderation tools usability study

Overview

The Wikimedia Foundation, in collaboration with editors of the English Wikipedia, is developing a tool to enable readers to contribute productively to building the encyclopedia. To that end, we started development of a new version of the Article Feedback Tool (known as AFTv5) in October 2011.

Once the AFTv5 feedback forms were deployed, we ran a small usability study to test each design of the AFTv5 feedback form, and see what could be improved. A number of hypothetical scenarios were drawn up, two for Option 1 and one each for Options 2 and 3, and four users were invited to run through them and provide feedback on how the designs could be improved. These scenarios can be found in the Appendix. The four users were recruited through Usertesting.com, a user testing site used regularly by the Foundation, which also provide video screencasts for each option. The data submitted was reviewed by Oliver Keyes, Fabrice Florin and Howie Fung.

A primary goal of usability studies such as this one are to help the WMF product team better understand how typical users view new products under development, by watching screencasts of their experience, and hearing their first hand impressions. This provides a unique perspective on user expectations and behaviors, and helps team members refine the user interface design and requirements. For this short test, four users tested the Article Feedback forms for about 12 minutes each, on average. Here are the video screencasts of each test:

• Option 1 screencast (positive)
• Option 1 screencast (negative)
• Option 2 screencast
• Option 3 screencast

Highlights of user reactions include:

• Users generally found the feedback process pretty clear
• Users generally expected the feedback form to be at the bottom of the page
• The feedback form appeared hard to find, unless you were at the bottom of the page
• The feedback button was only noticed by some users and none clicked on it
• Some users were confused to see both the feedback form and the feedback button at the bottom of the page
• Some users were confused by the current implementation of the "yes/no" buttons, requesting different prompts
• An email notifications system was requested, so people know what is done with their edit
• Wikipedia's markup language seemed confusing and did not engage users to want to edit
• Some users expected that the Wikimedia Foundation was involved in the editing.

Option 1

The first form readers were asked to comment on, “Option 1”, asked “Did you find what you were looking for?”, with “yes” and “no” buttons, and contained a non-mandatory comments box; readers were only required to use either the check boxes or the free text field, not both.

In the positive scenario (see the the subject screencast), the subject understood that it is a feedback form to improve quality and correct incorrect information, or include information that isn't there. It was not readily apparent who the feedback was forwarded to: he assumed the author of the page, or the last person to edit it. The form was easy to use, but he expected the "yes" and "no" buttons to go away after one of them was clicked, to indicate that the question had been answered. He also suggests adding more questions, or a prompt for more feedback. The user expected a more general question than "did you find what you were looking for" (e.g. "share your thoughts on this article"), thought that the Wikimedia Foundation was in some way involved, and that editors would be emailed about his feedback, and believed that you have to have an account to edit. This usability test of the positive scenario was conducted on Dec. 5th, 2011, using a first prototype of this new feedback form. The sample article used for this prototype was abridged, to make it easier for users to find the feedback form.

The subjectfor the negative scenario (see the screencast) could not initially identify the feedback form as the place to go to report missing information, and did not initially identify it from the top of the page - scrolling to the bottom, past references and such did not occur to him until prompted. The presence of both the docked feedback tab and the form confused him when they were both shown together at the bottom of the page. When asked what he thought the feedback form did, he assumed it would either let him add some comments or allow edits to be made directly, once passed through a "clearinghouse" in which edits were checked. He found the "transparency" phrase underneath the box unclear, and missed the "yes" and "no" buttons (it is worth noting that, for screens that are of smaller resolutions, the box does not resize, and so these buttons are forced on to a different line from the preceding question). He was also confused by the markup language; he identified it as a markup language, comparing it to HTML, but did not know how to use it.

The subject also noted that "the main thing that would have helped me would be some explanation of how to use the particular markup language in Wikipedia". This usability test of the negative scenario -- and subsequent Option 2 and 3 tests below -- were conducted on Feb. 15th, 2012, using more advanced versions of the new feedback forms on the English Wikipedia. They also included a feedback button in the lower right corner of the browser window, enabling users to open the feedback forms above the fold (but they did not click on that button). An unabridged version of the sample article was used for these tests.

Option 2

The second form that was tested, “Option 2”, clearly indicated what sort of feedback is being looked for for by subdividing the form into “suggestion”, “praise”, “problem”, and “question”. A reader is invited to select one of those tabs (“suggestion” is checked by default) and then enter text into the free text box. The subject, given the negative scenario (see the screencast), was slightly confused due to the instructions given, and would like a clearer way to add information, not just provide feedback on it. The purpose of the form and who feedback is passed on to were correctly identified, although the subject did think the form could be used to make an edit as well as to leave feedback. It was thought that the "praise" button would or should include a thumbs-up icon. The subject would appreciate having a way to receive email notifications about what is done with the edit, and finds wikimarkup too confusing; "I was not expecting the page to be so technical". The initial reaction to being presented with markup was to assume she had broken something. The subject found adding feedback to be simple, and much more useful than a simple thumbs-up or thumbs down

Option 3

The last form, “Option 3”, asked readers “Did you find what you were looking for?” and contained a non-mandatory text box. Like the existing system, it includes a five-star rating system. The subject, who was given the negative scenario (see the screencast), easily identified the purpose of the feedback form and where the feedback goes. She did not see the five-star rating system or identify it as something that needed to be filled in: instead, she simply added a comment and posted her feedback, without rating the article. The purpose of the Call to Action was easily worked out, as was the process for submitting feedback; she suggests, however, that we make the form more prominent, saying "Maybe add a logo at the top of the page, in case one does not scroll down to the bottom of the page, one would not see the leave feedback option". The process was generally considered clear, although she questioned "How do you know if [content is] right?" after being informed anyone could edit.

Conclusions

Here are some general observations from this short usability test:

• Overall, users found the feedback process pretty clear
• Users generally expected the feedback form to be at the bottom of the page
• The feedback form appears hard to find, unless you are at the bottom of the page
• The feedback button was only noticed by some users and none clicked on it
• Some users were confused to see both the feedback form and the feedback button at the bottom of the page
• Some users were confused by the current implementation of the "yes/no" buttons, requesting different prompts
• An email notifications system was requested, so people know what is done with their edit
• Wikipedia's markup language seemed confusing and did not engage users to want to edit
• Some users expected that the Wikimedia Foundation was involved in the editing.

Appendix

Here are the text prompts that were given to the four users in this study, to guide them through the test.

Negative scenario

1. Please take 30 seconds to skim the article. Please look for information about the Golden-crowned sparrow's song -- especially audio or video clips to let you hear what that bird's song sounds like.
2. Now scroll to the bottom of the page. You'll notice a box entitled "Help improve this article". What do you think this box is for? (If you do not see this box, please enable Javascript on your browser).
3. Where do you think this feedback goes?
4. Please answer the question in the box and post your answer by clicking on "Post your Feedback".
5. You will now notice another box with the words "Did you know you can edit this page?" What do you think this box does?
6. Please click on "Edit this page." Is this what you were expecting? If not, what were you expecting?
7. Was anything unclear in the process of providing feedback on this article? How could this process be improved?

Positive scenario

1. Please take 30 seconds to skim the article. Please look for the number of notes in the Golden-crowned sparrow's song.
2. Now scroll to the bottom of the page. You'll notice a box entitled "Help improve this article". What do you think this box is for? (If you do not see this box, please enable Javascript on your browser).
3. Where do you think this feedback goes?
4. Please answer the question in the box and post your answer by clicking on "Post your Feedback".
5. You will now notice another box with the words "Did you know you can edit this page?" What do you think this box does?
6. Please click on "Edit this page." Is this what you were expecting? If not, what were you expecting?
7. Was anything unclear in the process of providing feedback on this article? How could this process be improved?