Collaboration/Flow satisfaction survey/Report

In September 2016, the Wikimedia Foundation organized a survey around Flow, a modern discussion and collaboration system. Flow provides structured discussions and supports both wikitext and visual editing modes. A small number of wikis are using Flow pages by default for all talk pages, instead of unstructured wikitext-based discussion pages. Some other wikis allow individual users to convert their own user talk pages to Flow (Beta feature) or use Flow on community pages (help desks, village pumps...).

Summary

The survey, led by the Technical Collaboration team, was initially distributed to users and communities that had first-hand experience using both discussion systems. Participation was anonymous and open to anyone; all questions were optional. 583 responses from about 20 identified Wikimedia projects were recorded.

The main reasons for editors to prefer Flow discussions are:

• a structured-by-software discussion system which allows watching topics instead of whole pages,
• dedicated notifications when a reply is received or a topic is really appreciated,
• a clear design for adding new topics and editing your own messages, particularly appreciated by people who work with newbies,
• a better integration and use on mobile,
• mentioning (pinging), thanking and replying to other users is easier than on unstructured wikitext pages.

The main reasons for editors to prefer unstructured wikitext discussion pages are:

• finding specific discussions on unstructured wikitext discussion pages can be done with MediaWiki's search feature, while even elementary search features are still missing in Flow,

• watching changes and patrolling vandalism in unstructured wikitext discussion pages can be done with MediaWiki's features to watch article content, while Flow has several shortcomings that make watching and undoing changes more complicated,

• some users prefer the compact view of unstructured wikitext discussion pages. They find that Flow discussions leave a lot of white space and force too much scrolling,

• some users prefer the flexibility provided by unstructured wikitext discussion pages, which begin as blank pages, in contrast with the guided behavior of structured Flow discussions.

According to the survey data, daily and weekly contributors tend to prefer unstructured wikitext discussion pages, while Flow is the preferred system among editors who contribute to talk pages less frequently. This data reflects two key aspects of Flow in its current state:

• Flow is already a useful tool to engage and meet the expectations of new or casual editors who are familiar with other online discussion systems, but who are not familiar with MediaWiki's unique, unstructured wikitext talk pages and the social conventions that Wikimedia communities have built around discussions on talk pages (use of colons, bullet lists, discussion templates; signature to add...),

• however, Flow's missing features (for watching, searching, moderating, splitting...) and lack of user-experience maturity become more evident for regular users.

The survey shows that unstructured wikitext discussions have a solid support next to Flow at this point. 52% of participants expressed their preference for unstructured wikitext talk pages, next to 38% for Flow and 10% were indifferent. In free-forms, most editors somewhat preferring unstructured wikitext discussions or with no preference, point to Flow weaknesses that, if solved, would make them use the new discussion system more.

Free-form answers also show that Flow development continues to be a hot topic. There is a group of experienced and vocal editors who strongly oppose the development of Flow in principle. There are also experienced editors who ask for continued development of Flow, fixing the missing features, and deploying it in their wikis.

Improvements

Better support for checking the history of a conversation and having a way to search on a Flow board are the most requested features. These improvements would not have a big impact on Flow’s visual aspect, so they may be balanced by more visible changes, like providing a setting to compact conversations, collapsing indentations or having intelligible URLs.

The design could be reviewed to give every user a possibility to have something close to the current unstructured wikitext discussions pages' layout. Flow’s Notifications are appreciated but get some critics. Watchlist inputs also have critics. Both could be reviewed to avoid “the noisy effect”.

Satisfaction with particular functions

The survey asked users to evaluate how well both talk page systems perform particular functions.

Comparison of Flow and unstructured wikitext discussion pages functions

Fig. 1 - Favorite tool for similar actions, in percent.

Users were asked to compare how well Flow and unstructured wikitext discussions pages perform across five typical talk page functions:

• reading talk page discussions,

• understanding talk page discussions,

• writing and editing in talk page discussions,

• engaging with other users on talk pages,

• summarizing discussions on talk pages.

In general, users preferences in this breakdown line up with their overall tool preferences: a majority of users prefer unstructured wikitext discussions pages and prefer it more strongly, but Flow makes a strong showing as well. That comparison is not always relevant: unstructured wikitext discussions pages are blank pages allowing combinations of features such as discussing, polling, drafting and more. Flow has been design to be a discussion system, which could be used to build other tools (at the moment, Flow is mostly used on user talk pages).

Flow feature satisfaction

Fig. 2 - Appreciation of Flow features, in percent.

In this section, people were asked to say how useful they find Flow for particular talk page activities, absent any comparison to unstructured wikitext discussions pages.

• adding a new topic or comment in talk page discussions,

• editing a message in talk page discussions,

• following the layout of Flow discussions,

• resolving conversations,

• summarizing discussions

• watching and receiving notifications about individual topics (rather than entire talk pages).

Evaluation of Flow tools

50% or more of respondents find Flow "completely" or "somewhat" useful for all of the tasks.

The three tasks where Flow gets the most support are:

• watching a section instead of a whole page (68% satisfaction).

• adding a new topic to a Flow page (61% satisfaction). Flow provides dedicated tools for adding new topics to a discussion which, feedback indicates, even experienced users recognize as making the task easier, especially for newcomers.

• editing a message in talk page discussions (57% satisfaction). Because users don’t need to open a whole page to fix something in one comment.

While strong integration of Flow and Notifications is clearly an area where Flow shines, it's worth noting feedback in the free-form fields concerning the high number of notifications received by people who follow Flow boards. That Flow expands the volume of Notifications on wikis where it's implemented is a known fact. Any project seeking to expand Flow usage will do well to consider this and to provide new means for users to manage what Notifications they receive.

Of respondents who said that Flow is “Not at all useful” for the tasks in question, a majority (60%) are users who overall “strongly prefer" unstructured wikitext discussion pages. More people who "strongly prefer wikitext" appreciate Flow's features than vice versa, which makes sense if Flow users are in general less experienced.

In free-text answers, many respondents complain about the design of Flow pages (“... leaves much more space that lengthens and drowns the discussion”; "looks like a blog"). Other commenters point to Flow's superiority for mobile use ("Mobile support!!! It's already much better in Flow than in the old talk pages.") and archiving ("Archiving systems are not required anymore").

Flow functions for engaging with other users

Fig. 3 - In regards to engaging other users, to what extent are you satisfied with the following features of Flow.

Helping users to engage with one another is another area where Flow finds strong support. Thanking, mentioning and replying to users all get +50% satisfaction. Respondents show the strongest satisfaction with Flow's Thank feature (32% "very satisfied").

Evaluation of Flow discussions functions

Fig. 4 - Possibly useful aspects of Flow

To gauge how well Flow achieves a number of its design goals, users were asked to agree or disagree with a series of eight propositions:

• writing and editing with Flow is easy and intuitive,

• the layout of Flow discussions is easy to read and understand,

• watching topics on Flow discussions is easy,

• following individual topics on Flow through notifications is more useful than following an entire page,

• Flow makes discussions easier to summarize,

• Flow makes it easier to mention other participants in the discussion,

• Flow makes it easier to mark the conversation as resolved,

• with Flow, the interactions with other users are easier than with wikitext (easier to mention, thank, respond, watch).

It's worth noting that users who "strongly agree" and "strongly disagree" are more or less evenly divided. But total approval for most of these eight key functions is near or over 50%. A couple of results stand out:

• there is an overall appreciation of Notifications related to Flow, as seen in the popularity of both "Watching topics on flow" and Following individual topics on Flow through notifications." These same 2 value propositions point to another thing users like about Flow, the "structured" part of "structured discussions" —they like following individual topics. This is one of Flow's core ideas,

• mentioning other users is considered the least easy feature and should be investigated.

What Flow improvements do users want most?

Respondents were asked to rank the importance of improvements to Flow that users have requested in the past. Here is the list, as ordered by survey respondents:

1. better support for checking the history of a conversation, display diffs, and clearing vandalism,
2. search and find specific topics,
3. being able to move topics to a different page would help,
4. filter or categorize topics to make it easy to focus,
5. view all my Flow activity at once on a single page.

Better support for checking the history of a conversation, its changes, and clearing vandalism would be helpful for most users. On free-form fields, users have shared examples, like the lack of similarities with the current history system (lack of comparisons for the whole activity on a page between two moments using diffs, no way to undo things easily), the difficulties of doing some particular actions (delete contents from your own page, difficulties for sysops or oversighters to perform maintenance actions), etc.

Search was a virtual tie with better history support. In free-text comments, users clarified that they want to search Flow comments overall and they want to be able to search a single Flow page for a specific comment.

Managing and organizing conversations by moving topics among pages was a close third. In the comments, users asked for ways to split, merge, move or relocate topics or parts of topics.

Conclusion concerning particular functions

When users are comparing unstructured wikitext discussion pages and Flow for the same activities, there is a preference for unstructured wikitext talk pages. But when users are judging specific features, Flow has more support: strong support from people who like the tool and meaningful support from people who prefer unstructured wikitext discussion pages. Users who don’t have a preferred system tend to appreciate Flow’s ease of use.

That feature support notwithstanding, Flow has a lot of room to improve. Creating a search function, providing better history support and enabling users to better manage and organize topics will be first steps towards winning over Flow skeptics and strengthening the approval of existing fans.

Overall satisfaction

The two discussion systems are quite different; one purpose of the survey was to know which system people prefer. With some limitations, the data can provide useful information about attitudes.

Overall satisfaction based on frequency of contributions

Fig. 5 - Favorite tool by editing experience, grouped, in percents. "Wikitext" refers to unstructured wikitext discussions pages.

Looking at tool preference through the lens of frequency of contribution is revealing: it's clear that Flow appeals strongly to users who edit less frequently. Among users who edit once per month, 58% prefer Flow, compared to 24% for unstructured wikitext discussion pages. By contrast, among those who edit every day, almost 40% prefer unstructured wikitext discussion pages compared to 26% for Flow.

The same pattern can be seen when the axis of the analysis is reversed: among editors who prefer Flow, 40% are infrequent users (editing once a month or less). By contrast, among users who favor unstructured wikitext discussion pages, only 17% are infrequent users.

That said, it’s worth noting that Flow has many adherents among frequent users: of those who edit once a week, 32% favor Flow.

The fact that new users favor flow would seem to be consistent with other research showing that beginners are able to do more tasks using Flow than unstructured wikitext discussion pages.

Overall satisfaction based on project where Flow is mainly used

Fig. 6 - Preferred tool by wiki with more than 40 answers recorded, in percent. Wikis with less than 40 answers have been grouped in the "all other wikis" bar.

Unstructured wikitext discussion pages is favored by a similar margin on mediawiki.org, the biggest wiki where Flow is the default discussion system. Wikis that appreciate Flow and had more than 40 responses are Wikidata (over 70% support), French Wikipedia and Chinese Wikipedia (~40% support). As a comparison, users of English Wikipedia show the strongest support for unstructured wikitext talk pages, at about 70%, based on 24 answers.

Overall satisfaction based on first year of contribution

The largest cohort of respondents started to contribute in 2005 and 2006. 209 respondents started to contribute before 2009, 198 after 2009 (included). However data are not conclusive, because of the bias related to first year of contribution, which is not meaningful compared to experience. Users who contribute frequently gain experience and comfort with unstructured wikitext discussion pages quickly enough so that a survey question based on starting year doesn't show a pattern based on the first year of contribution.

Conclusion concerning satisfaction

Users who contribute frequently tend to prefer the unstructured wikitext discussion page editor. Users who contribute less regularly tend to prefer Flow.

However, these preferences are not absolute. Among people who contribute at least once a day, 34% say they value both systems, and 27% prefer Flow. The difference between daily users who prefer unstructured wikitext discussion pages and daily users who prefer Flow is only 12%. That difference decreases to 6% among users who contribute weekly (31% for Flow, 37% for unstructured wikitext discussions).

In free form comments, the majority of users without a strong opinion for unstructured wikitext talk pages (from strongly or somewhat supporting Flow to indifferent or somewhat preferring unstructured wikitext discussion pages) tend to point to Flow weaknesses and suggestions for improvement. However, a majority of editors expressing a strong preference for unstructured wikitext talk pages just ask to remove Flow.

Recommendations

Based on the result of this survey, the Technical Collaboration team recommends:

• to continue the development of Flow, focusing on major missing features and design issues, as a natural complement to the editing experience provided by the visual editor and as part of our effort to onboard and retain new editors successfully,

• to focus the development on the needs reported by the communities that are already using Flow or that are willing to adopt Flow,

• to focus first on the use cases where Flow is a viable alternative already today, notably user talk pages, which are the main channel through which new and casual users engage with other editors, and where other editors reach out to them,

• to offer Flow to communities that are looking forward to adopting it.

About the survey

Distribution

The survey was open to anyone. The survey was distributed to all public wikis using Flow in order to reach users who use Flow and can compare it knowledgeably to unstructured wikitext talk pages. A message was sent to users that use or used Flow (including people who disabled it) on wikis where Flow is enabled as a Beta feature, manually activated on user talk pages or used as the default talk page system. A message also was sent to main community hubs on wikis where Flow is used on a few pages as a trial. The invitation was also sent to some public Wikimedia mailing-lists or other public hubs, and shared by users to others. Any user who has seen the survey’s link could participate, even if they never used Flow. One answer was allowed by IP address.

Distribution has been done by messages batches, with an invitation posted in the wiki language, when possible. A few days after the beginning of the distribution, while no message has been sent to Mediawiki.org users, 17.47% of recorded answers were from that wiki. Due to the fact that Flow is the default discussion system on that wiki, it was not possible to identify which talk pages were used by really active users. Hence, a decision had been made not to deliver messages on Mediawiki.org user talk pages, just on community pages.

Limitations

These are the main limitations registered in this survey:

• this survey has been promoted to existing users. However, Flow is being developed equally for new users and those who are not here yet,

• this survey doesn’t claim to take into account a representative sample of Flow users or Wikimedia users, since participation was open to anyone:
  • regular users and active users at that time are more likely to be over-represented versus casual users.
  • a self-selection based on high interest for/against Flow is also expected.
  • the survey was promoted by some users seeking like-minded participation.
  • people that stopped using or never used Wikimedia wikis due to the barrier to entry of unstructured talk pages are not surveyed.

See also

• Raw data from the 2016 Flow satisfaction survey