Research:Collapsed vs uncollapsed section view on mobile web

Created
04:30, 16 June 2016 (UTC)
Duration:  2016-02 – 2016-06
This page documents a completed research project.


Animated screenshot of mobile Wikipedia showing a reader scrolling past the (uncollapsed) lead section to open the first collapsed section with a tap on the section heading

The mobile web version of Wikimedia projects currently shows all top-level sections below the lead section collapsed (when displaying a page such as https://en.m.wikipedia.org/wiki/Barack_Obama on a phone-size display). Users can tap on a section heading to show the content, and to collapse it again.

To examine the tradeoffs of this solution and inform future product decisions, we ran an experiment in December 2015/January 2016 (setup: phab:T120292), where 0.05% of mobile web users were shown all pages with every section expanded on initial load, instrumented alongside a control group of 0.05% that kept seeing the standard view where all sections all initially collapsed. (For the most part this was not an A/B test in a strict sense, because we are not comparing the same metric for both groups.) It was a followup to phab:T118041, which preceded that experiment.

This project is about the analysis of this data with respect to the main question of how readers use Wikipedia differently in these two settings and which one might be preferable. Some other data from the same instrumentation has also been published as part of Research:Which parts of an article do readers read and of a March 2016 tech talk.

Methods

edit

We collected data with an EventLogging instrumentation detailed at Schema:MobileWebSectionUsage.

Policy, Ethics and Human Subjects Research

edit

Data was collected and stored in accordance with the Wikimedia Foundation's privacy policy and data retention guidelines. No IP addresses were obtained.

Results

edit

A detailed work log with results is at phab:T128536. Below is an overview over the most important results:

 
Pageview session duration (90th percentile, hourly) on mobile Wikipedia with sections initially collapsed vs expanded, during one week

Readers in the test group (sections expanded) tend to stay longer on the page

edit
More detail: phab:T128536#2350536

The pageview session duration (defined as the time between opening the page and the last recorded action by the reader) in the test control group was considerably longer than in the control group, both for the 90th percentile (see chart) and the 75th percentile.

Readers in the test group tend to spend more time reading, and less time navigating

edit
More detail: phab:T128536#2383899

The additional time readers spent in the test (uncollapsed) condition could, for example, simply be the result of needing longer to navigate to the desired part of the page. We decided to look at the time a reader spent reading vs. navigating on a page. A timespan between events that are 5 seconds or more apart is assumed to consist of reading, and if subsequent events are 4 seconds or less apart, they are assumed to form part of a continued navigating activity. (I.e. the time the reader spends on a page is split into alternating periods of reading and navigating, with navigating periods ending with a scroll/open event that is not followed by another one within 4 seconds.) The result does not support this concern: The takeaway is that readers in the test condition spend considerably more time reading, and somewhat less time navigating.

Time spent reading a particular page:

condition 90th percentile 75th percentile median
test (expanded) 190 sec 53 sec 6sec
control (collapsed) 146 sec 38 sec 5sec

Time spent navigating a particular page:

condition 90th percentile 75th percentile median
test (expanded) 4 sec 1 sec 0 sec
control (collapsed) 6 sec 3 sec 0 sec
 
Histogram of the number of sections scrolled into view in the test group vs. sections opened in the control group

Readers in the test group tend to scroll more sections into view than readers in the control group open

edit
More detail: phab:T128536#2384068

This comparison is somewhat apples-to-oranges (for example because opening a section is normally a deliberate decision based on the content of its headline, whereas - almost per definition - the reader doesn't yet know what a section is about when it is scrolled into view). With all that in mind though, it is still reasonable to interpret this result as a strong indication that readers in the test group are more engaged with the content below the (always expanded) lead section.

Readers in the test group tend to stay shorter on the page than readers using the Android Wikipedia app

edit
More detail: phab:T128536#2384336

We also compared the above results about pageview session duration for the test group with existing data for the Android app (which has a similar setup with no sections collapsed). With some caveats about daily/weekly variations and differing ways to measure (the web data does not count the time after the last section open/scroll event when the user is still reading the page before leaving it), it looks like app readers tend to stay on a page longer than readers on mobile web with sections uncollapsed (30-40 seconds at the 75th percentile, 10-20 seconds at the 90th percentile). Interpretations of this result will need to consider that the app offers a floating table of contents for easier navigation, and that app users tend to be more engaged readers to begin with.

References

edit