Research:Section Translation Usability Testing (Bengali Wikipedia)

Contact

Wikimedia Foundation

Duration: 2021-February – 2021-March

This page documents a completed research project.

Many Wikipedia contributors edit articles via mobile devices. Section Translation brings translation support to mobile device editors, and this project provided usability testing as the tool became available in the first wiki, Bengali

Background

We observe high mobile usage across many language versions of Wikipedia, including relatively smaller wikis such as Bengali, Malayalam, and Tagalog. Currently, translation of Wikipedia articles is supported for desktop editors via Content Translation, which has been used to create more than 800k articles. Section Translation brings many of the capabilities of Content Translation to mobile editors, and allows contributors to add individual sections of articles by translating and editing content from another language version of Wikipedia. This project provided usability testing of Section Translation as soon as it was available for Bengali editors to expand articles in their language wiki by translating sections from various source language wikis.

Prior to this project in late 2020, when Section Translation was still in development, but a basic workflow became available on a test server, we collected early feedback. In anticipation of 2021 deployments in real wikis, we reached out to a small group of experienced Content Translation users for feedback: Test server early feedback report.

Research goals

The primary goal of this project was to evaluate the general usability of the Section Translation experience and workflow, in order to inform subsequent product decisions, including prioritization of improvements. In particular, we wanted to know what the main points of friction were that may in part or full prevent editors from making successful contributions. Relatedly, where is better support needed through the process?

The project sought to collect data on the following aspects of the process:

Selecting an article and article section to translate
Reviewing section contents in the source/target language and initiating the translation
Reviewing, editing, and adding to the machine translation output generated by the tool
Completing the translation and publishing

Research approach

Methodology

We used moderated remote research sessions to test key aspects of the current Section Translation mobile experience. (Although the experience is in ongoing development and some aspects may change, for reference, it can be accessed on the Bengali Wikipedia here. Because the experience remains in flux, when relevant this report provides screenshots and descriptions of functionality alongside discussion of some results.) All sessions were guided by a detailed protocol and involved four main components. Each session began with a short interview. Following, participants shared their screen and used the think-aloud technique while translating an article section on the Bengali Wikipedia. Following observation of their use of Section Translation, participants were asked to complete four Likert scale ratings, and then the session wrapped up with a brief interview period. An interim summary was provided to the Language Team after roughly half of the sessions had been completed.

Participants and data

Participants were recruited from the existing community of Bengali editors through the use of community pump announcements, direct recruitment messages to contributor talk pages, and referrals. Interested participants responded by completing a screener questionnaire, which was followed by an invite to select a session date/time. A total of seven sessions were conducted through the use of a consecutive interpreter, and two were conducted in English. Most participants resided in Bangladesh at the time of the study, with the exception of one who resided in India and one based in Europe. Seven of the participants reported editing the Bengali Wikipedia on a daily basis; one edited on a weekly basis, and the other had more intermittent editing activity. All participants had experience editing from a mobile phone, and the majority edit multiple times each week from a mobile device. All but two participants regularly use translation to contribute to Wikipedia, with frequency ranging from daily to weekly to monthly. At the time of their sessions, three participants had previously tried Section Translation, whereas six were experiencing it for the first time. All participants joined the session and used Section Translation from a mobile phone. All sessions except one were recorded (audio and screenshare), and the process resulted in a total of approximately 8.5 hours of video. From these videos (and notes for one participant) we confirmed tracking of key metrics of interest, and extracted additional data points. These additional data points were coded for thematic analysis. The results section presents these themes alongside the key metrics we tracked during sessions.

Results

Please continue reading for the key takeaways from this research. For detailed discussion of results, please refer to the full report.

Key takeaways, recommendations, and next steps

Top 10 takeaways

Translators want to search for specific articles - Without the ability to translate articles of specific interests they perceive as new translation opportunities, editors face a near hard stop with Section Translation.
Editors’ workflows begin at the source wikis - Regarding Section Translation entry points, translators generally begin their work at a source wiki where they are browsing and searching for articles that match topical interests.
Section selection was relatively seamless - Overall, section selection presented fewer points of friction when compared to article section.
Section selections must provide a ‘core’ content part of the article - In order to present a compelling translation opportunity, missing sections must pertain to core content of the article and topic. Editors were less likely to feel compelled to translate closing sections perceived as peripheral, such as ‘references’ and ‘external links’.
Interactions with the quick tutorial were very brief - Few if any questions arise for editors at this point in the process, so the frequency with which these are shown can be reduced to avoid annoyance.
Aside from time required to edit MT outputs, main points of friction were related to input, links, and numbers. Bengali editors want Bing Translator support. - Numbers in particular were noted to almost never translate, and Bing is frequently preferred over Google for providing semantically- and syntactically-superior outputs.
Experienced editors want a source editor option - These editors feel slowed using visual editor, and are also looking for advanced options and integrations with Wikidata in one case.
The sentence-by-sentence flow doesn’t work for all editors - Editors were divided according to whether they had a strong preference for translating at the paragraph or sentence level. Some listed it as one of their most liked parts about the editing process, whereas for others it made their list of top dislikes.
The majority of participants published successfully - Participants said they could work faster with everything in a single tab with Section Translation. Still, editors described publication as one step in a larger process, followed most immediately by reviewing and making further revisions.
Most participants wanted to immediately correct section ordering - Editors liked getting confirmation that their section had been added, but immediately wanted to correct the ordering. This was not an easy copy/paste task on a phone.

Recommendations

Ten key recommendations include:

Support search to identify a section/article to translate
Improve section mapping
Improve suggestions
Support Bing Machine Translation
Adjust when to show the quick tutorial
Ensure that core content sections are available when suggesting articles
Improve published section ordering by implementing a first best guess and providing a mobile-friendly way of adjusting the order of sections
Improve discoverability of Section Translation, including entry points at common source wikis
Discuss the complexity and relative priority of providing source editor for Content and Section Translation
Continue exploring ways of supporting both translators who prefer working at the sentence and paragraph level

A Phabricator ticket is being used to track various lines of work stemming from research findings. T279064 can be used to view and track ongoing work, including tickets related to some of the items in the list of recommendations above.

Next steps

As the number of wikis where Section Translation is available increases, subsequent testing should be used to understand the specific needs of different editing communities that span linguistic, cultural, and geographic divides. Such research should be timed to correspond with the completion of additional development efforts stemming from the recommendations from earlier reports, such as those noted above. Follow-up testing should also specifically attempt to validate these design and technical improvements.