Training modules/Feedback/Pilot testing

Drafting/discussion spaces

Discuss design, accessibility, and formatting of modules

Previous research

Background

As part of the Wikimedia Foundation Support and Safety team's work on the development of Training Modules (Annual Plan 2016–17, Program 2), the team set up a pilot test of the modules for review by targeted volunteers. This follows previous surveys of four groups: Checkusers and oversighters across various Wikimedia Projects, non-functionary community members, academics in the field of online harassment and community, and industry professionals in the field of trust and safety.

This pilot test ran from February 27 through March 12.

Setup and goals

For this pilot test, the team sought to gather more in depth feedback from those who may ultimately take the final modules. To that end, five rough regional areas were defined and a number of community members selected from each of them.

The target was to have 3–5 volunteers take the survey from each of the five regions. Invitations were sent out to various prospective testers, as well as through other contacts at the Foundation's Global Reach team.

In all, 31 people expressed an interest in taking the pilot survey, and 21 ultimately completed it. (We only requested feedback on the "Keeping events safe" module if those taking the survey expressed interest in event organisation. Of those who took the survey, 14 gave feedback on that module.)

Report layout

This report is split into multiple parts, covering various aspects of the pilot test undertaken by the 21 respondents. All feedback has been anonymised.

Sampling discusses how participants were invited to take part in the survey, breaks down the demographic information requested of participants during the pilot survey. This includes account age, gender, native language, and on which projects they edited.

The general feedback section covers general comments and opinions on the modules as wholes, including comments on navigation of the platform, glitches, and any other comments provided at the end of the survey.

Feedback by section splits comments and participant responses to Likert scales^[1] by module, then by section. For each section, there is a table giving the averages of participant responses to these Likert scales, followed by its overall ranking compared to other sections within the parent module for ease of comparison. After this, the long-form feedback is summarised and split into three categories: Content-related, translation- or culture-related, and technical- or design-related.

Opinions were requested of interactive elements ("Test yourself", multiple choice review questions, and "What would you do?", scenario-based thinking points) separately using different questions. Text feedback on these was not categorised.

Each text response is labelled to indicate whether or not it was "actioned" in March 2017. Those indicated "actioned" were implemented into the final version. "Not actioned" were, for whatever reason, not actionable or were not possible – this is explained in each bullet. "Not yet actioned" points are still being processed or investigated. "No action required" indicates that the comment was general (or praise) and as such weren't actionable.

Limitations and compromises

In some cases, groups were combined in order to protect anonymity (for example, where a group would otherwise only contain one person).
Where the question was requesting input on a negative statement, the final "score" for this opinion was inversed. This allows for a more intuitive comparison with other data points as it, in theory, reflects positive rather than negative sentiment.

Sampling

Please note: These regions are very loosely categorised, and primarily the grouping were selected for an even spread of participants.

	Approached	Signed up	Completed	%
Region 1 (North America, Australasia, Europe, etc.)	10	11	8	72.7%
Region 2 (Latin and Central America)	10	5	4	80%
Region 3 (South Asia)	9	10	5	50%
Region 4 (Africa, Middle East, East Asia)	12	5	4	80%
Totals	41	31	21	67.7%

Demographic breakdown

Account age

Gender

* "Other" was bundled into "female" to protect the identities of those selecting the former option.

Native languages

Seven people^[2] (33.3%) stated that they spoke English as a native language. Of the others:

Spanish	5^[3]	Hindi	1	Punjabi	1
Arabic	1^[2]	Korean	1	Siswati	1
Catalan	1^[3]	Malayalam	1	Tamil	1
German	1	Odia	1

Projects edited

Respondents could select more than one option.

General feedback

Design and navigation

Actioned: The text overall was considered quite long, and images to break it up were recommended. The slide format may have played a factor in this. Images have been added to some slides to help with this.
Not actioned: One respondent had problems with navigation because the arrow keys are used to change slide while the participant uses them to scroll with. This is an issue with use of older browsers with the dashboard.
Not actioned: A double click on the "next page" button will browse more than one page forward. This is currently intentional.
Not actioned: On the left side there is not enough space in some browsers. This is a technical issue.
Not actioned: One respondent suggested the inclusion of tutorial videos for sharing experiences, saying these are the only one things missing. Currently there is no bandwidth to implement this suggestion.
Not yet actioned: Further clarification on for which group of users each module was designed was requested.
Not yet actioned: The use of a small, italic font was criticised for being hard to read. This is a technical issue that will need further work.
Not yet actioned: One person requested the training in PDF format.
No action required: A number of respondents praised the design and navigation overall. One said that the modules' flow was "seamless" and that, despite the length, they were interesting to go through.

Glitches

Actioned: Many respondents were confused by the lack of available answers to "What would you do?" questions. In those cases, it would be useful to give at least some guidelines of what acceptable answers would be. We are creating an area on Meta-Wiki for people to discuss what these answers might look like.
Actioned: There were a couple of topics that seemed out of place or in the incorrect order.
Actioned: The term "SuSa" needed to be explained before it was used.
Actioned: There were dead or broken links throughout.
Actioned: Many reported that the training needs some copyediting to address typos, unclear references, and repetition.
Not actioned: Some respondents wanted more multiple choice questions. We discussed this and, since the modules cover content that in some cases is not covered elsewhere, we did not want for them to make concrete "correct" answers in these cases.
Not actioned: There was an issue with a menu in which a translation was missing. This was a technical issue which requires more work than we can currently provide.

Other comments and miscellaneous feedback

Actioned: Several requested more interactivity in the modules to encourage people to better take in the information. More questions were added to the modules and an area for their discussion drafted.
Actioned: There were a number of typos and grammatical errors throughout the modules.
Not actioned: The banner on each module slide, reading "You are logged out. You will not receive credit for completing this module unless you log in." was confusing to one respondent as it implied these were required assignments. This is not the case; this is a holdover from the rest of the Programs and Events dashboard which does have "required" trainings used by those involved in programs and events. We will work on ways to make this more clear.
Not actioned: One person again suggested that the amount of text makes the modules dull. They suggested using video for some of the modules. As previously stated this is something that there currently is no bandwidth to undertake.
Not yet actioned: It was suggested that the modules make the target group clear – the juxtaposition of advanced advice and basic information seemed counterintuitive. It was suggested that some phrasing be clarified to make clear which information was targeted at certain groups of users with more or less experience than others.
Not yet actioned: The lists of resources were praised, but seemed brief; an on-wiki master list that could be linked to from the survey was suggested instead.
Not yet actioned: One respondent liked when the modules started with a definition of the problem, since some modules assume that the user is familiar with topics. They suggested that every module should begin with a "Basics" section. We will investigate ways to "ease in" those taking the modules, though both top-level modules already begin with a "Basics" section.
Not yet actioned: Some were keen to see the modules be translated, though at least one was concerned that some issues covered by the modules were specifically related to the English Wikipedia. Resources are also geared toward the United States and other English-speaking countries and communities.
No action required: Several had overall praise for module content and layout.
No action required: There was general praise for the interactive elements of the modules such as the multiple-choice questions.

Responses with most variance

Key

E	Keeping events safe	HR	Handling reports
HF	Fundamentals	HCC	Closing cases
HO	Other forms of harassment	N	Overall navigation and design
HC	Communication best practices

S.D. = Standard deviation. Red text indicates the slide was a "Test yourself" or "What would you do?" question. A dagger (†) indicates the figure is an "inverse" (as in, is the opposite of the actual report). See "Limitations and compromises" for more information.

Account age

			2-4 years	4-8 years	10+ years	S.D.
E	After the event	The layout is clear and easy to navigate	2.000	4.000	2.286	1.082
E	During the event	The tone is too formal†	2.667	1.500	3.571	1.038
E	What would you do?: Repeat offenders	It is easy for me to identify where the problem is	4.333	2.500	2.857	0.972
HO	Off-wiki harassment	The tone is too formal†	3.333	2.857	1.545	0.926
E	What would you do?: Board member behavior	The language used here is clear	3.667	2.500	4.286	0.907
HCC	What would you do?: Closing a non-actionable case	The question ties into the preceding section well	2.333	3.714	4.000	0.891
E	What would you do?: Dealing with the public	The situation seems realistic	3.667	2.250	3.857	0.878
HO	Off-wiki harassment	The length of the section is ideal	2.667	3.143	4.364	0.875
E	What would you do?: Dealing with the public	The language used here is clear	4.000	2.500	2.571	0.846
HO	Test yourself: Image-related abuse	The question is suitable in relation to the module content	3.000	2.429	4.091	0.845

Gender

			Male	Female/Other	S.D.
E	What would you do?: Board member behavior	It is easy for me to identify where the problem is	3.500	4.750	0.884
E	Before the event	The length of the section is ideal	4.000	2.750	0.884
HCC	Test yourself: Challenge question	The question is suitable in relation to the module content	2.563	3.800	0.875
E	What would you do?: User group members	The question ties into the preceding section well	2.800	4.000	0.849
HR	Investigating reports	The tone is too formal†	3.000	1.800	0.849
E	What would you do?: Dealing with the public	It is easy for me to find a solution to the problem	3.300	4.500	0.849
HCC	Test yourself: Closing an actionable case	The solution to this question is obvious to me	3.188	2.000	0.840
HCC	Closing cases	The length of the section is ideal	3.375	2.200	0.831
HR	Test yourself: Writing a good reply	It is easy for me to identify where the problem is	2.938	1.800	0.804
HCC	Test yourself: Challenge question	The solution to this question is obvious to me	3.125	2.000	0.795

Language

			English	Other	S.D.
HO	Off-wiki harassment	The language contains too many technical terms or abbreviations which are hard to understand	1.500	3.333	1.296
HF	Immediate action	The language contains too many technical terms or abbreviations which are hard to understand	1.833	3.600	1.249
HO	Handling personal information	The language contains too many technical terms or abbreviations which are hard to understand	1.833	3.333	1.061
HC	Providing support and advice	The language contains too many technical terms or abbreviations which are hard to understand	1.833	3.200	0.966
E	During the event	The tone is too formal	2.250	3.600	0.955
E	What would you do?: Board member behavior	The situation is similar to one I have experienced	3.750	2.400	0.955
HO	Off-wiki harassment	The tone is too formal	1.833	3.133	0.919
HR	What would you do?: What counts as 'actionable'?	The situation is similar to one I have experienced	4.167	2.933	0.872
HR	Investigating reports	The tone is too formal	1.833	3.000	0.825
HF	Immediate action	The tone is too formal	2.167	3.333	0.825

Region

			R1	R2	R3	R4	S.D.
E	During the event	The language contains too many technical terms or abbreviations which are hard to understand†	2.600	2.000	1.667	4.250	1.136
E	During the event	The tone is too formal†	3.400	3.000	1.000	3.250	1.075
E	Basics / Situations you might encounter	The language contains too many technical terms or abbreviations which are hard to understand†	3.600	3.000	1.667	4.250	1.059
HCC	Closing cases	The layout is clear and easy to navigate	3.600	4.000	2.000	4.000	1.033
E	What would you do?: User group members	The situation is similar to one I have experienced	3.400	4.000	4.333	1.750	1.031
E	What would you do?: User group members	The question ties into the preceding section well	2.400	3.000	4.333	1.750	1.020
E	Basics / Situations you might encounter	This section would be easy to translate	2.125	3.250	4.400	2.750	0.975
E	What would you do?: Repeat offenders	It is easy for me to identify where the problem is	3.400	4.500	2.333	2.750	0.960
HCC	Test yourself: Challenge question	The solution to this question is obvious to me	2.125	3.000	4.200	2.500	0.935
E	After the event	The language contains too many technical terms or abbreviations which are hard to understand†	4.400	3.500	2.333	4.000	0.915

Feedback by section

Keeping events safe (E1-6)

E1/2: Basics / Situations you might encounter

	Average rating	Rank within Events module
The layout is clear and easy to navigate	4.500	1
This section would be easy to translate	3.429	4
The length of the section is ideal	3.500	3
Too many technical terms (inverse)	3.286	4=
The tone is too formal (inverse)	3.286	3

Content

Actioned: One respondent indicated they would like to see another link to a similar topic to supplement the harassment survey cited here.
Actioned: Some of the language needed clarification, for example to define "expectations" and terms like "locally" or "globally" banned.
Not actioned: One respondent requested more links. The suggestion was too vague to really action.
Not actioned: One person requested that the "Situations you might encounter" page have risks listed by colour or in some other way to indicate the level of the risk they afford. This would require more technical involvement that there is currently no bandwidth for.

Translation/culture

Actioned: One respondent said the text in this section seemed too long. (Another respondent noted this would make translation difficult.)

Technical/design

Not yet actioned: The use of a large "next page" button and a smaller "previous page", formatted as a link, makes for asymmetry in the design.

E3: Before the event

	Average rating	Rank within Events module
The layout is clear and easy to navigate	4.357	2=
This section would be easy to translate	3.642	2=
The length of the section is ideal	3.214	4=
Too many technical terms (inverse)	3.714	1=
The tone is too formal (inverse)	3.214	4

Content

Actioned: One person noted there were too many abbreviations in this section.
Actioned: Some of the language used was considered unclear.
Actioned: There was a placeholder link left in by accident.
Not yet actioned: One respondent suggested that it would be helpful to work with visualisations.

Technical/design

Not actioned: The flow of the slides felt odd to one respondent, who noted that a slide seemed to be out of place.

E3 Q: What would you do? - Repeat offenders

	Average rating	Rank within all WWYDs
It is easy for me to find a solution to the problem	3.357	7
The situation is similar to one I have experienced	3.071	5
The situation seems realistic	4.071	1
The language used here is clear	3.714	7
It is easy for me to identify where the problem is	3.500	7
The question ties into the preceding section well	3.357	6=

Actioned: One respondent said the scenario was too vague because specious reports on noticeboards are very common. Even substantive ones may have no clear connection to event safety.
Actioned: One respondent was confused that this had no options with which to answer the question.
Actioned: The reference to the English Wikipedia's administrative noticeboard may not have been clear to people planning the event.
Not yet actioned: Another respondent agreed, suggesting the problem was too simplistic; in reality, there are usually many more variables to consider.

E4: During the event

	Average rating	Rank within Events module
The layout is clear and easy to navigate	4.214	4=
This section would be easy to translate	3.000	5
The length of the section is ideal	3.214	4=
Too many technical terms (inverse)	3.357	3
The tone is too formal (inverse)	2.786	5

Content

Actioned: One respondent said that writing up incident reports was a strange choice of emphasis for the beginning of a section in a module focused on live events. They explained that writing a report in the middle of an event could be time-consuming and conspicuous.
Actioned: The entire "Who may report?" section seemed conflicting to to one respondent, as there seemed to be a lack of clarity with respect to bystander reports. This may have also impacted on translatability.
Actioned: There were a few typos: "aleviated" should have been "alleviated"; "Establish they feel comfortable speaking to you" was missing an "if".
Actioned: One respondent asked for a list of interactive questions with answers.
Not actioned: One person said that it would be great to have something like first aid training here, as people panic or may not know what to do in such situations. We discussed this, and as a team we considered this outside of what we can really provide.

Translation/culture

Actioned: "Do the math" was noted as a slang term.

Technical/design

Actioned: The use of bold on words beginning the bullet points was praised as helping to guide the reader's eye.

E4 Q: What would you do? - Dealing with the public

	Average rating	Rank within all WWYDs
It is easy for me to find a solution to the problem	3.643	3
The situation is similar to one I have experienced	2.857	6
The situation seems realistic	3.571	7
The language used here is clear	3.857	4=
It is easy for me to identify where the problem is	3.929	1
The question ties into the preceding section well	3.357	6=

Actioned: One respondent didn't like this problem as it didn't apply to all cultures.
Actioned: One respondent was confused that this had no options with which to answer the question.
Actioned: One participant liked the inclusion of a message explaining the reason of the exercise, which was not present in the other exercises.
Actioned: The scenario was a bit unexpected to one person, in that most of the preceding discussion presumes that the reported party is a participant in the same event.

E5: After the event

	Average rating	Rank within Events module
The layout is clear and easy to navigate	4.214	4=
This section would be easy to translate	4.000	1
The length of the section is ideal	4.000	1=
Too many technical terms (inverse)	3.286	4=
The tone is too formal (inverse)	3.714	1

Content

Actioned: Typo: "issuew were hanled" → "issues were handled"
Not actioned: One person thought that if there were already meetups and groups with whom one could share experiences from events, this would help. The Wikimedia Foundation already requests this sort of thing from groups and meetups.

E6: Things to think about

	Average rating	Rank within Events module
The layout is clear and easy to navigate	4.357	2=
This section would be easy to translate	3.643	2=
The length of the section is ideal	4.000	1=
Too many technical terms (inverse)	3.714	1=
The tone is too formal (inverse)	3.500	2

Content

Actioned: Some of the paragraphs in this section were argued to be generally too long.
Not actioned: One respondent suggested that this section (or an area on meta-wiki) should collect harassment experiences in order to learn from them. We weren't sure of a way to do this while also keeping privacy and security intact.

Translation/culture

Actioned: The phrase "in-situ" was noted as being hard to translate.

E6 Q: What would you do? - User group members, Board member behavior

	Average rating	Rank within all WWYDs
It is easy for me to find a solution to the problem	3.429	4=
The situation is similar to one I have experienced	3.143	4
The situation seems realistic	3.929	5
The language used here is clear	4.000	2=
It is easy for me to identify where the problem is	3.714	4=
The question ties into the preceding section well	3.786	3

	Average rating	Rank within all WWYDs
It is easy for me to find a solution to the problem	3.357	4=
The situation is similar to one I have experienced	2.786	7
The situation seems realistic	3.857	5=
The language used here is clear	3.857	4=
It is easy for me to identify where the problem is	3.643	6
The question ties into the preceding section well	3.643	5

Actioned: Respondents were again confused about the lack of responses for the "what would you do" questions here.
Actioned: One participant liked the inclusion of a message explaining the reason of the exercise, which was not present in the other exercises.

Dealing with Online Harassment I: Fundamentals (H1-2)

H1: Basics

	Average rating	Rank within Harassment module
The layout is clear and easy to navigate	4.524	2=
This section would be easy to translate	3.476	7=
The length of the section is ideal	3.857	3=
Too many technical terms (inverse)	3.476	3
The tone is too formal (inverse)	3.143	9=

Content

Actioned: The claim that women are targets of harassment more frequently than men, and that this impacted Wikipedia, was cautioned against because of the lack of evidence or causality. It was argued that this could "distract" from the training unless sources were provided.
Actioned: One respondent praised the use of sources for describing harassment as a problem, noting that this was important because some people may not have a clear and concise delimitation of the problem.
Actioned: There was some repetition of information.
Actioned: Some terms (such as outing and doxing) would benefit from links.
Not actioned: One person argued that emailing emergency often isn't enough. This wasn't changed as it is still the advice we give in these situations.
Not yet actioned: A respondent said that this section is is too abstract, and did not make clear where the line is between freedom of speech and harassment.

Translation/culture

Actioned: One person said that the entire section is "clunky and too formalised for anyone to follow it in thoroughness", explaining that it would be difficult or frustrating for many non-native English speakers to follow the "meandering" nature of the text. Effort was made to improve this.
Actioned: The word "functionaries" was noted as one that is difficult to translate.
No action required: It was requested that, where no translation exists, the whole training ought to fall back to English rather than splicing other languages in.

Technical/design

Not actioned: The use of hyperlinks to support the meaning of terms was seen as "interesting" but not something that works for everyone.
Not actioned: A more visual representation (i.e. images) was recommended by one respondent. It is not clear which images would be suitable in this section, however.
Not actioned: One respondent requested a table of contents on each page. This is not currently possible with the dashboard.

H2: Immediate action

	Average rating	Rank within Harassment module
The layout is clear and easy to navigate	4.524	2=
This section would be easy to translate	3.333	10=
The length of the section is ideal	3.857	3=
Too many technical terms (inverse)	2.905	12
The tone is too formal (inverse)	3.000	12

Content

Actioned: Typo: "You might already be familiar with these actions, as they are fairly commonly used by by local administrators or functionaries."
Actioned: The use of the term "sysop" and "admin" wasn't consistent. Other words were used interchangeably as well, such as "policy" and "criteria".
Actioned: One person was pleased to see that the small wiki monitoring team was mentioned here.
Actioned: Some terms here, like "range blocks", "IP-hopping", and "revision deletion", could be better explained with examples.
Actioned: One person was confused by the order of "revdel" and "suppression". In the section where "revdel" is explained, it states "it doesn't reach the criteria for suppression", when suppression hasn't been defined yet.
Actioned: The term functionaries was used often without being properly defined.
Not actioned: One person opined that experienced users probably already know all of this already. This wasn't actioned if only because these are realistically going to be read by a variety of people and it makes sense to develop a "baseline" level of technical knowledge for those taking the training.

Translation/culture

Actioned: The language was criticised as being "convoluted" in places where simpler English would have been better. Effort was made to improve this.
Actioned: Some concepts may be difficult to translate. (No details were provided here.) Some of these terms were replaced or removed.

Technical/design

Actioned: There was a colon out of place on one of the slides. (Reported by several.)
Actioned: The "glossary" link was broken.
Not actioned: Images were suggested because reading text for so long was considered "boring" after a few slides. (This was recommended by a number of people.) It wasn't clear which images could be added to illustrate these concepts.
Not actioned: The "Done" button wasn't working for one respondent. We think this is likely a browser issue.
Not actioned: One person suggested that the "glossary" idea might work better as a simple "Terms used" at the bottom of each page. This would be a great thing to add, but technically this is difficult and there isn't currently bandwidth to implement this.

Dealing with Online Harassment II: Other forms (H3-5)

H3: Handling personal information

	Average rating	Rank within Harassment module
The layout is clear and easy to navigate	4.714	1
This section would be easy to translate	3.333	10=
The length of the section is ideal	3.810	7=
Too many technical terms (inverse)	3.095	11
The tone is too formal (inverse)	3.095	11

Content

Actioned: One person thought that the advice on using IRC could be seen as contradictory or confusing.
Actioned: The terms "IRC" and "IP address" should be better defined or linked.
Actioned: The use of "children's ages" as an example of personal information was confusing, since this might sometimes be valid information in some articles. Instead it was suggested that the examples be less ambiguous or case dependent.
Actioned: Some points, such as "treat the actual PII as serious if it was posted by accident", were buried within the rest of the section and as such lost their emphasis.
Actioned: A visualisation would be helpful to help contextualise the Streisand effect.
Actioned: The outcomes of "doxing" were noted as too severe and therefore not especially realistic.
Not actioned: One respondent noted a lack of emphasis where it is due. This wasn't really actionable as it was too vague.
Not yet actioned: One person suggested that this section's advice conflicted with the Wikimedia Foundation Legal team's statement on the English Wikipedia regarding paid editing.

Translation/culture

Actioned: The word "outing" would be easier to translate.
Actioned: One person noted that not all countries have "street addresses". "Home address" was suggested instead.
Actioned: Some internet-related terms or jargon may be difficult to translate. Some of these were unavoidable, but some were removed or replaced.

H4: Image-based problems

	Average rating	Rank within Harassment module
The layout is clear and easy to navigate	4.476	5
This section would be easy to translate	3.476	7=
The length of the section is ideal	3.857	3=
Too many technical terms (inverse)	3.190	8=
The tone is too formal (inverse)	3.143	9=

Content

Actioned: One person thought that examples should be sparing in these sections, so as not to give people ideas should they wish to abuse the system.
Actioned: Typos: "Wikimedia Commons has of identifiable people a guideline around images of identifiable people which can be useful in situations like this." (This was reported by several people.)
Actioned: The Wikimedia Foundation Legal team's meta page implies that the Legal team handles abuse reports made through the legal-reports@ email address. (This was more an issue with that page rather than with these modules.)
Not actioned: Some of the links take people to more advanced topics, which might make them miss the slides in-between. This means a "back" link would be ideal but difficult to implement.

Technical/design

Not yet actioned: The italic text on top of some slides was difficult for one respondent to read. This requires a technical solution that we're looking into.

H4 Q: Test yourself - Image-related abuse

	Average rating	Rank within all TY questions
The solution to this question is obvious to me	3.857	4
The situation is similar to one I have experienced	2.857	5
The situation seems realistic	4.190	5
The language used here is clear	3.667	5
It is easy for me to identify where the problem is	4.048	5
The question is suitable in relation to the module content	3.952	5

Actioned: The scenario sounded too complicated to some, and was not presented as well as it could be.
Actioned: One respondent argued that recommending they look into legal action against the offending site sounded too aggressive, especially if the user is supposed to try to reach an agreement with the site first. It was suggested that this be kept until after the site is spoken to first.
Actioned: One respondent had to look up what "Tencent QQ" was. We replaced this with a generic term.
Not actioned: For one person, the "next section" link worked even when the answer was incorrect. This wasn't replicable so is likely a browser issue.
Not actioned: One person thought this was more an editing conflict than a clear case of harassment. This feedback was taken on board but it wasn't immediately clear how to incorporate into the question.
Not actioned: The situation is not relatable, said one, if you are not an intermediate or advanced user. Simplification was attempted – this question is tricky, though it is not totally outside the realms of possibility.
No action required: One respondent had praise for the explanations for this question's answer (and the wrong answers).

H5: Off-wiki harassment

	Average rating	Rank within Harassment module
The layout is clear and easy to navigate	4.429	6
This section would be easy to translate	3.571	5
The length of the section is ideal	3.810	7=
Too many technical terms (inverse)	3.190	8=
The tone is too formal (inverse)	3.238	6=

Content

Actioned: One respondent considered the section beginning "some ways you may be able to investigate the identity of the harasser include" to be dangerous. They continued that clear cut lines between doxxing and investigating harassers must be specifically drawn.
Actioned: Consistency was urged between the use of "doxing" and "outing".
Actioned: One person wanted to broaden the "revenge porn" bullet point to include similar behaviour (like common internet shock images, memes, etc.) They also noted that reports to other sites are often ineffectual.
Not actioned: One respondent commented that it was a little shocking to read and learn about how to chase one's own information off-wiki. This feedback was a little too vague to action.
Not actioned: One respondent was a bit concerned about who "your team" might be. They weren't sure that any group in particular would be appropriate or happy to take this on as a regular task as it has proven incredibly difficult to them in the past. We tried to make clear that the
Not actioned: One person wanted more advice for removing harassment from social media, and said that helpful links would be good. Since there are just so many social media networks that would need to be included in here, we decided to stick with general tips and advice.
No action required: A respondent had praise for this section, stating that they had seen every one of the things detailed here at least once and that the advice on how to help in these complicated situations is pretty good.

Dealing with Online Harassment III: Communication best practices (H6–7)

H6: Communication

	Average rating	Rank within Harassment module
The layout is clear and easy to navigate	4.381	7=
This section would be easy to translate	3.524	6
The length of the section is ideal	3.952	1
Too many technical terms (inverse)	3.524	2
The tone is too formal (inverse)	3.238	6=

Content

Actioned: Typo: "Contact tusers". (Reported by many.)
No action required: One user explicitly called out that this section was a lot clearer than the others. Others were happy to see this content and advice published in one place, and one other person liked the emphasis on empathy.
No action required: Slide 3 ("Keeping yourself safe") was called out for praise by one person.

H7: Providing support and advice

	Average rating	Rank within Harassment module
The layout is clear and easy to navigate	4.524	2=
This section would be easy to translate	3.190	12
The length of the section is ideal	3.762	10
Too many technical terms (inverse)	3.190	8=
The tone is too formal (inverse)	3.429	1=

Content

Actioned: The term "functionary" was used here without any clarification as to what it actually means. The use of consistent terms was urged across all modules.
Actioned: The term "Trust and Safety" is used interchangeably with "SuSa" and "Support and Safety Team". (This was reported by a number of people.)
Actioned: The section on "Terms of Use violations" must link to the terms of use themselves.

Translation/culture

Actioned: Some links are valid only in English-speaking countries, which one person said could be a little depressing for non residents in both countries. (This was reported by more than one person.)
Actioned: One respondent argued the US-centric nature of this section might make it difficult to translate.

H7 Q: Test yourself - Providing support and advice

	Average rating	Rank within all TY questions
The solution to this question is obvious to me	4.095	3
The situation is similar to one I have experienced	3.143	4
The situation seems realistic	4.286	3
The language used here is clear	4.190	3
It is easy for me to identify where the problem is	4.333	2
The question is suitable in relation to the module content	4.381	1=

Actioned: One person suggested rephrasing "though we found your report valid" in the "correct" answer. The word "valid" might be misinterpreted as "accurate".
Not actioned: The question's responses seemed to some to be too long. We left them mostly as-is since the layout of the dashboard doesn't really allow for much variation right now.

Dealing with Online Harassment IV: Handling reports (H8–9)

H8: Handling reports

	Average rating	Rank within Harassment module
The layout is clear and easy to navigate	4.381	7=
This section would be easy to translate	3.667	2=
The length of the section is ideal	3.857	3=
Too many technical terms (inverse)	3.333	6=
The tone is too formal (inverse)	3.429	1=

Content

Actioned: "Give concrete details, and stick to them" must specify that it is only about timings.
Actioned: One respondent said that the phrasing "being transparent and open" is not clear and will not be easily getting across to most.
Actioned: A number of respondents considered this section clear and easy to read compared to others.
Not actioned: One person said that parts of this section seem redundant, and that it repeated things that were already established in previous sections. While we did remove a lot of redundancy, some of this is deliberately left in in case someone wants to complete this module first.

Technical/design

Actioned: The words "be prompt" weren't bold, like the other points.
Not actioned: One person said that scrolling with the mouse wasn't working in this section. It looks like this is probably an issue with an older browser.

H8 Q: Test yourself - Writing a good reply

	Average rating	Rank within all TY questions
The solution to this question is obvious to me	4.429	1
The situation is similar to one I have experienced	3.905	1
The situation seems realistic	4.333	2
The language used here is clear	4.286	1=
It is easy for me to identify where the problem is	4.381	1
The question is suitable in relation to the module content	4.381	1=

Actioned: The question wasn't at the end of the section, which one person recommended.

H9: Investigating reports

	Average rating	Rank within Harassment module
The layout is clear and easy to navigate	4.381	7=
This section would be easy to translate	3.476	7=
The length of the section is ideal	3.381	12
Too many technical terms (inverse)	3.381	4=
The tone is too formal (inverse)	3.333	5

Content

Actioned: A respondent noted that the section is unclear about what type of problems should be directed to community noticeboards, and that an example or two will be helpful.
Actioned: Discussion of "dossiers" is slightly unclear, and these aren't sufficiently explained.

Technical/design

Actioned: The link to "joe jobs" was broken, as reported by many. This term was however noted as being colloquial, and was replaced.

H9 Q: What would you do? - What counts as 'actionable'?

	Average rating	Rank within all WWYDs
It is easy for me to find a solution to the problem	3.667	2
The situation is similar to one I have experienced	3.286	2=
The situation seems realistic	3.905	4
The language used here is clear	3.762	6
It is easy for me to identify where the problem is	3.714	4=
The question ties into the preceding section well	3.762	4

Actioned: This question was in an incorrect part of the module.

Actioned: Numerous respondents were confused at the lack of a clickable answer to these situational questions.

Dealing with Online Harassment V: Closing cases (H10–12)

H10: Closing cases

	Average rating	Rank within Harassment module
The layout is clear and easy to navigate	4.333	10=
This section would be easy to translate	3.667	2=
The length of the section is ideal	3.810	7=
Too many technical terms (inverse)	3.381	4=
The tone is too formal (inverse)	3.429	1=

Content

Actioned: One person urged consistency with pronouns.
Actioned: A respondent argued that the text in this section is far too long.
Not actioned: Terms like "steward" may not be easy to understand for those new to the projects and should be better explained in these modules. The term's definition was improved and a link added, but it is unfortunately not a word we can really replace.

H10 Q: Test yourself - "Closing an actionable case" and "Challenge question: closing a case that involves advanced user rights"

	Average rating	Rank within all TY questions
The solution to this question is obvious to me	4.143	2
The situation is similar to one I have experienced	3.619	2
The situation seems realistic	4.476	1
The language used here is clear	4.143	4
It is easy for me to identify where the problem is	4.238	3
The question is suitable in relation to the module content	4.286	3

	Average rating	Rank within all TY questions
The solution to this question is obvious to me	3.619	5
The situation is similar to one I have experienced	3.286	3
The situation seems realistic	4.238	4
The language used here is clear	4.286	1=
It is easy for me to identify where the problem is	4.143	4
The question is suitable in relation to the module content	4.190	4

H10 Q: What would you do? - "Closing a non-actionable case"

	Average rating	Rank within all WWYDs
It is easy for me to find a solution to the problem	3.381	6
The situation is similar to one I have experienced	3.476	1
The situation seems realistic	4.000	2
The language used here is clear	4.000	2=
It is easy for me to identify where the problem is	3.905	2
The question ties into the preceding section well	4.048	1=

Actioned: One person had serious issues with the "Challenge question: closing a case that involves advanced user rights", as it seemed to have a problematic answer. They argued that a local block will will leave plenty of room for abuse of privileges. Regardless of whether there is a global block or not, there seemed to be no sense in making any sanctions that risk making the harassment worse before the entire matter of advanced user rights is resolved. Even if there was no outside action being taken, the more logical step is to perform the local block after that decision is taken.
Actioned: The question does not contain any answers, nor a place to input your thoughts. (This was reported by several people.)
Actioned: One person didn't quite agree with the "correct" answer in the "advanced rights" case – taking local action against a steward with global access before the WMF investigation has occurred may lead to them harassing the reporter or accessing personal information on other projects. This was improved by converting the question into a "What would you do"-style question instead, with no "correct" answer.
Actioned: The question mentions a "point 3" that doesn't seem to exist. (This was reported by several people.)

H11: Reporting out

	Average rating	Rank within Harassment module
The layout is clear and easy to navigate	4.333	10=
This section would be easy to translate	3.619	4
The length of the section is ideal	3.714	11
Too many technical terms (inverse)	3.333	6
The tone is too formal (inverse)	3.190	8

Content

Actioned: The text was again criticised for being too long.

H11 Q: What would you do?: Answering questions about a case

	Average rating	Rank within all WWYDs
It is easy for me to find a solution to the problem	3.714	1
The situation is similar to one I have experienced	3.286	2=
The situation seems realistic	3.857	5=
The language used here is clear	4.048	1
It is easy for me to identify where the problem is	3.857	3
The question ties into the preceding section well	4.048	1=

Actioned: The question does not contain any answers, nor a place to input your thoughts. (This was reported by several people.)
Actioned: The month block was originally attributed to the incorrect person, which made the question difficult to understand.

H12: After a case

	Average rating	Rank within Harassment module
The layout is clear and easy to navigate	4.283	12
This section would be easy to translate	3.667	1
The length of the section is ideal	3.900	2
Too many technical terms (inverse)	3.667	1
The tone is too formal (inverse)	3.421	4

Content

Actioned: It was suggested that the word "punished" should be changed to "sanctioned" or something similar.
Not actioned: One person suggested that mindmapping for further solutions could be helpful. It wasn't really clear what was meant to be implied here.
No action required: The debrief information was praised as something that should be done more often.

References and notes

↑ A Likert scale is a research technique involving assessing participants' responses to statement on a scale of "Strongly disagree" to "Strongly agree". There were five steps on this scale in this pilot survey, which were converted to numerals for analysis. Learn more on Wikipedia.
↑ ^a ^b The one Arabic speaker also speaks English natively, so is included in both categories.
↑ ^a ^b The one Catalan speaker also speaks Spanish natively, so is included in both categories.

[1] A Likert scale is a research technique involving assessing participants' responses to statement on a scale of "Strongly disagree" to "Strongly agree". There were five steps on this scale in this pilot survey, which were converted to numerals for analysis. Learn more on Wikipedia.

[:0-2] The one Arabic speaker also speaks English natively, so is included in both categories.

[:1-3] The one Catalan speaker also speaks Spanish natively, so is included in both categories.

[1]

[2]

[3]