Building tools for diverse users. Lessons learned from the Technical Wishes project

The PDF version of this paper is on Commons: White Paper Technical Wishes 2018

Intro

Over the past years, WMDE’s Technical Wishes team has been working on new and improved tools for the diverse communities and users of the Wikimedia projects, and has developed methods and approaches for building software in a collaborative environment. The Revision Slider, the Advanced Search interface and the improved Wikidiff2 algorithm are examples for software developed by the Technical Wishes team.

Software development for Wikimedia is not only a technical task, but a challenge on several levels: How can we ensure that the perspectives and requirements of diverse users shape our development work? Which workflows, what kind of research is needed? How do we define a product? How do we work together with all involved parties? These questions have been relevant for our work from the start: We’ve been successful, we have failed, and we have learned a lot, leading us to iterate and improve our workflows and methods with every new project started.

This paper shares our biggest lessons learned. We hope that it will be helpful for other software teams, other open-source projects and everyone who wants to learn more about developing software for an environment as collaborative as the Wikimedia projects.

Technical Wishes: A brief history

A brief timeline of the Technical Wishes project

From the beginning, the Technical Wishes team has been looking for solutions for problems that surfaced on German Wikipedia but are relevant for more wikis, in order to support the international Wikimedia movement. The Technical Wishes project originated from a survey called Technical Wishes on German Wikipedia, set up by User:Raymond at the end of 2013. Initial work on requests from the wishlist started in 2014, and a dedicated development team including product management was created in 2015. The survey has been conducted twice since 2013: in 2015 and in 2017. Each time, the number of participants has doubled, thanks to our knowledge building, changes in advertisement, and a growing recognition of the project within the community.

In addition to the survey, we have run workshops in different German-speaking cities in 2015, 2016 and 2017, and we have presented and discussed our products and learnings at conferences, including Wikimania, Hackathons, Developer Summit, Wikimedia Conference, DiversityCon and the German-speaking WikiCon.

Even though it all started with a survey, the Technical Wishes project always has been more than just that: It’s an approach to collect and prioritize technical needs in a collaborative way and to support diverse groups of users with technology. In this sense, the core feature of Technical Wishes isn’t the survey, it’s the community-centered process we apply – and this means profound research, transparent and continuous communication, close collaboration with the communities, as well as iterations, constant improvement, learning from failures and, last but not least, finding sustainable, unhacky solutions.

The survey part of the Technical Wishes approach was copied twice: The Wikimedia Foundation’s Community Tech team is running the International Community Wishlist on an annual basis; and a Developer Wishlist has been conducted by a technical contributor in 2017.

Community-centered software development: Our lessons learned

Collect and discuss problems collaboratively, in a survey or another format.

The Technical Wishes project is not limited to the format of the wishlist survey. It became more and more obvious that the success of the project doesn’t depend on the format we’re using to collect and discuss user needs, but comes down to our collaborative way of working and to our focus on research and transparent communication. As an alternative to the wishlist format, we have already tried out a workshop series around a specific problem, and we are striving to experiment with new formats in the future.

Experiences with the wishlist format

Rough timeline of a Technical Wishes survey

User needs collected in a workshop in Cologne, 2016

In general, a survey comes with trade-offs (probably like every other format). On the one hand, it is a great chance to focus on ideas and needs for a limited amount of time. Wishes can be discussed, shaped and merged by the help of many people at once. On German Wikipedia, many contributors take part in refining and discussing proposals together. Because of this, the survey is also a great tool to learn about solutions that already exist. It’s common that people submit wishes and then learn that what they were looking for is already possible, thanks to other contributors discussing their proposal.

Having one big central survey allows you to use tools like central notice to reach a large number of people. Central notice especially helped to get in people who are usually not involved in meta or technical discussions. The wishlist survey on German Wikipedia – similar to the international wishlist on Meta – is open for submissions around all topics and areas. This is an advantage, as diverse users have different problems, and it allows people to submit whatever is most urgent to them.

On the other hand, this openness also increases the number of proposals, which later makes it hard for people to vote on their favourite submissions. Reading through more than 200 proposals is very time-consuming and can be exhausting, especially if you don’t understand all of them, as some of the requests can be very specific, or touch areas you usually don’t work in.

Experiences with the workshop format

A different format we tried out to address technical needs was a workshop series. In 2016, we conducted events in three cities in Germany, with focus on a specific problem area: specialized searches for non-experts. We collected and discussed use cases and needs with participants, designed first ideas for a possible solution in working groups, and came out with a rough product idea based on the insights we had gathered together. The workshop format worked great, and was even better than the survey approach when it came to explaining workflows and problems. But it also excluded people who didn’t want to meet in real life.

Comparing formats

Both formats have their strengths and challenges – and this would be the case for any other formats we have investigated so far. In the end, it is not so much the format that matters, but the way of working together: Collecting and discussing problems, requirements and limitations collaboratively and “translating” from non-tech to tech and vice versa – be it in in person or online.

We plan to continue testing new formats in the future. Since all formats have their pros and cons, it might be helpful to combine different approaches or further evolve existing ones.

Focus on problems, not on solutions.

Talking about technical needs is difficult. When people face problems in their workflows, it’s only natural that they think about possible solutions first. This is what often happens when contributors describe their technical requests. They tend to present their own idea for a solution instead of laying out the issues and needs behind the request. Approaching technical issues with a problem-oriented mindset is something we as a development team needed to learn as well.

In short, understanding the underlying needs of a feature request helps us to find the best possible solution. It allows us to come up with alternative ways to tackle a problem that are more sustainable or work better in the context of diverse users and communities. For instance, we want to find solutions independent of wiki and script in order to serve the movement as a whole, and it’s quite likely that a specific feature proposal only works for one wiki, even though the underlying problem exists in all communities. On top of that, thinking about solutions first restricts us to the limits of the existing software and to solutions we already know.

The consequence for our process is that we take a lot of time to understand the underlying problems users have before we start planning any feature. This includes lots of research, conversations with users about their proposals, and targeted feedback rounds. We explore which user groups are affected by the issue, in which context the problem occurs, which alternative solutions or workarounds for the problem already exist and what side effects could emerge from using a specific solution and from solving the issue in general.

Besides, a crucial part of it is to help contributors write their proposals in the first place. For example, we guide users through the submission process with a template and a help page and we offer real-life-workshops where we define and phrase requests together with community members.

An example of how we took this learning to heart was the development of the Advanced Search feature. It began with workshops with Wikipedians around the topic of specialized searching. Back then, we didn’t know yet what kind of product we would end up developing. Instead, we discussed users’ needs and problems in depth. A product idea was then developed together, based on these needs.

Cater to diverse users throughout your process.

Our development process in a nutshell.

At Wikimedia, software development needs to take into account that we have a diverse user base: tech-savvy users, newcomers, long-term contributors, users with different areas of experience and with different backgrounds, languages and requirements. We are building software for more than 800 wikis with more than 280 languages and several different writing systems. In 2018, the team came up with a list of criteria to address diverse users’ needs in the development work. There is a lot more that could be done, but the list is a start and reflects what the team can do within the current resources. How software development can serve a diverse community of users best was also the topic of a session conducted together with a contributor from Arabic Wikipedia at the Wikimedia Diversity Conference in Sweden (session slides).

Diversity has to be kept in mind over the entire development cycle from the initial collection of ideas to the deployment of products. So what does that mean for the product development process?

Research

As described before, we are investing quite some time in learning about the underlying problems users have before the concrete planning a new feature starts. In terms of Wikimedia's diverse user base, this means for instance investigating if the problem exists in different wikis, and if these wikis already have found solutions to deal with it. The extensive research phase leads to a higher workload before the development of a product can start, but it gives us greater confidence to deploy to such a multitude of wikis and makes the products more usable and sustainable.

Feedback & iteration

Building software for a diverse user base also means transparency about what we’re planning to do, and thus giving users a chance to comment if the idea works for them or not. That’s why when we have an idea for a solution, we present it in a first feedback round, either as a testable prototype or as a design mockup. Further feedback rounds might follow throughout the development process.

A feedback round is time-consuming to prepare, to conduct and to analyse the results, so it’s critical to plan enough time and resources. Especially in the first 1-2 days, when the majority of comments pours in, both the communications team and the product manager need to be available to react to questions and unclarities. After a feedback round, it is important to let participants know when to expect a summary of the results and a decision on next steps.

Feedback collected at WikiCon 2017, Leipzig

At the moment, we are conducting feedback rounds both in English on Meta, and in German on German Wikipedia. Our ambition is to get a balanced and representative participation from the diverse users of our products. E.g. it is important to not only receive feedback from tech-savvy users, or long-term contributors from the biggest wikis, but to engage people from diverse experience levels, communities and language projects. If participation in a feedback round seems to be imbalanced, we put more effort in reaching out to more people via additional communication channels.

Though we are conducting onwiki feedback rounds and try to gather feedback from a diverse range of participants, it is still likely that we miss some groups and have biases: e.g. towards people with a background in tech, users who are active in meta discussions or users who feel comfortable expressing their thoughts in English. To mitigate these effects, we attend local and international events to gather additional feedback, insights, comments and further ideas.

On top of that, conducting user tests with people who have never edited a wiki before or have recently joined the Wikimedia projects is crucial to find out if new features also work for newcomers and non-Wikimedians.

Feedback is key in software development. You can have the best intention to develop useful software, but without insights from users and especially a diverse range of users, this is almost impossible.

Deployment

Our team introduced the concept of a “small release”: This means releasing a beta feature on German Wikipedia (where the wish originated from), on at least one wiki that uses a right-to-left (RTL) language system, that we have actively reached out to beforehand, and on any other wiki with a community request to have the feature enabled early. For example, Advanced Search was requested by the Arabic, Hungarian and Persian wiki communities. Aside from the benefit that the most obvious bugs in a feature are fixed before it becomes a beta feature for a much broader audience, this approach enables us to react to feedback from diverse users early on. For example, with the early deployment to a right-to-left wiki, bugs that are related to the writing direction can be detected in an early stage.

When it comes to releasing a product as a default feature, we repeat the first deploy to the same small set of wikis and wait for feedback before deploying to the rest of the wikis.

To announce deployments internationally, using different channels is key to best reach different people: multilingual onwiki messages, Tech News and Tech Showcase, mailing lists, Facebook, and updates to the project and help pages themselves. Extensions have dedicated multilingual help pages on MediaWiki.org. To make reading and translation less time-consuming, it makes sense to explain the workflow of a new feature with images, and to use as few words as possible (example).

Make communication a priority.

The barriers for effective collaboration with contributors on technical topics are usually quite high: The minority of wiki contributors are tech enthusiasts, or are engineers themselves. Most of our users contribute content, not code, or they contribute their time to maintain, patrol or curate the content. On top of that, development teams rarely have the resources to get in touch with editors in their local wikis, and discussions take place on central projects like Meta or MediaWiki.org, sometimes even in Phabricator, and in English. Even for people who can, for example, watch movies in English, it can still be very hard and time-consuming to talk about technical projects in English, depending on the complexity and technical level of a conversation.

Participants of the 2017 German-speaking Technical Wishes survey by year of registration

Participants of the 2017 German-speaking Technical Wishes survey by edit count

The benefits of communicating with people in their first language

Although we have a network of tech ambassadors, who are interested in sharing technical news in their local communities, actually engaging with editors around their workflows and needs in their language, or gathering feedback on proposals for planned developments goes beyond the news aspect, and requires a lot of work and time. We’re able to do this in the German-speaking community, since our project originated in the German-speaking community and our staff in communications are native German speakers. Our engagement there clearly shows how much we (both users and builders of the software) can gain from being able to communicate in people’s first language. Not having to leave the home wiki to comment on a proposal or an idea, and not having to use a foreign language lowers the barrier for participation significantly, and increases and diversifies the group of people who are giving feedback. In our 2017 survey, for example, participants weren’t only long-time contributors; a good number of people had joined German Wikipedia fairly recently. Similarly, it wasn’t only power editors who took part. Nearly one third of the participants had less than 1000 edits.

Be a stable contact

But still, it requires some efforts to convince non-technical users that their feedback is important and valuable. Getting people’s attention for a technical topic is usually hard – as for many, this might be the most scary or boring thing to discuss. One aspect we consider crucial for participation is providing stable contacts. The barrier to participate is much lower if you already know who is going to answer to your question or feedback, and if previous experience has shown that your interactions are well received and appreciated.

Being a stable contact means you need to get to know the people, the culture of a wiki and the various communication channels. It’s important to actually do things together, for example organise workshops, or conduct a survey with the help of many. It’s crucial to be transparent and honest about what can and what can’t be done. Try to be present at real life gatherings from local meetups to big events such as Wikimania, and always let people know that their feedback in fact matters a lot. All this helps build stable and long-lasting relationships in a collaborative environment.

Build bridges

In summary, being inclusive and building bridges is of utmost importance: From technical to non-technical language, from English to German or other languages and vice versa, from power user platforms like Phabricator to environments that more people are used to. Communication is key in tech.

Focus on MediaWiki instead of gadgets, tools, bots.

Wikimedia projects run on the MediaWiki software, but are enriched by a variety of gadgets, scripts, tools and bots. When it comes to improving contributors’ workflows, any of these technical options could be used. The Technical Wishes team decided to set a strong focus on MediaWiki software development instead of gadgets, tools or bots, because building MediaWiki core features or extensions requires a high amount of coordination, communication and continuous workload. This is what can be handled more easily by a full-time team. Besides, as we address diverse users’ needs in the MediaWiki software itself, our products can be used by all language and project communities. Last but not least, by improving MediaWiki, we hope to reduce technical debt in the project.

As a drawback to this approach, the development time for building a feature in MediaWiki is significantly longer than for programming tools or gadgets.

Support volunteer developers.

Volunteering at Wikimedia can take many forms, and one of them is participating in software development. We believe that the technical contributions and ideas of many are crucial for a collaborative open source project. Therefore, we have set up formats to specifically support volunteer developers and strengthen technical collaboration:

The Technical Advice IRC Meeting is a weekly office hour in an IRC channel, where people can ask any questions to experienced full-time MediaWiki developers. While this started as part of Technical Wishes, it was quickly supported by the whole Tech Team of Wikimedia Deutschland, and is now also supported by engineering colleagues at the WMF. Feedback we’ve received has shown that the Technical Advice IRC Meeting helps volunteer developers to solve problems in their projects and to learn more about Wikimedia’s technical environment.

Another project was tested this year: For a period of three months, we supported a team of three volunteer developers in their development of one of our products. In contrast to other mentoring programs, this project was specifically designed to create a team experience, with the goal of learning how to support each other. Although this format was quite time consuming, we were very happy with the results: Participants reported they got a deeper understanding of what it means to write production code and why it makes sense to do code reviews. Even better: This project had shown them how much value code reviews bring, even if one doesn’t have much development experience yet. Other learnings included mastering the code collaboration tool gerrit, how to set up a test environment and how to create sample data. While supporting a volunteer team is not good for saving time, it seems to be a great way of transmitting tacit knowledge.

Product Stories

With each of our products we learned new lessons. Here are three examples to give a better understanding of how community-centered software development is an evolving, never-ending process in which every product has its own character, pitfalls, and learnings.

An interface for specialized searches: Advanced Search

Screenshot of the Advanced Search

Where did it come from: 3-part workshop series in 2016 in the German-speaking community
What is it: Makes the hidden existing search options accessible for everyone through a form on the Special:Search page.
Who worked on it: WMDE Technical Wishes team, supported by WMF’s Discovery team
Where is it at: Since November 2018, the extension is enabled on all wikis.
more info

The Advanced Search feature showed us that formats other than the survey can lead to useful product ideas, since it was the first project that didn’t originate from the wishlist survey, but instead came out of a workshop series in three different German cities. The goal was to make the search features that already existed on Wikimedia wikis accessible for everyone, without requiring any knowledge of search syntax.

The special challenge of this project was that it didn’t target just one user group but everyone, from Wikipedia readers, to experienced contributors without knowledge of advanced searching, to very advanced users with highly specialized usages. This meant there were contradicting needs between different groups of users. To make sure to address this, our user experience specialists had to be highly involved in all stages of the development cycle and parts of the user interface had to be reworked multiple times to find a balance.

Although the team had only committed to improving the search interface, simply discussing the project with the WMF’s Discovery team already resulted in improvements on the search backend itself. Not only were bugs found during feedback rounds and then fixed by the team, but because of use cases that emerged in the research phase, the Discovery team developed the subpageof keyword and deployed a deepcategory keyword that allows users to not only look for pages in a specific category, but also in its subcategories.

Screenshot of the new edit conflict interface

A better interface for edit conflicts

Where did it come from: Wish #1 on the 2015 German-speaking Technical Wishes survey
What is it: Provides a two-column view for the edit conflict resolution page. It highlights differences between the editor's and the conflicting changes as an easier way to resolve the conflict, with less scrolling.
Who works on it: WMDE Technical Wishes team
Where is it at: Since May 2017, the extension was a beta feature on all wikis. Although it performed slightly better than the default resolution screen, we felt we could do better. That’s why we developed a newer user interface, which is a beta feature on all wikis since November 2018.
more info

In this product, we came to the conclusion that we would need to iterate upon the first solution we had developed. What had happened was that we had taken time during the research phase to work out two different possible solutions and collect lots of feedback. Users clearly favored one option, which we then started to build. However, it turned out that one of the two choices was overlooked as a result of a few conceptual mistakes: a) The two mock-ups varied in their level of realism with one being a hand-drawn sketch while the other a realistic mock-up. b) One of the mock-ups was easier to understand for non-tech-savvy users. c) The two approaches were not presented to the user in a randomized order.

The effect was that even though the solution we first developed improved the edit conflict page, it was just a small improvement. For example one originally highly enthusiastic user reported: “The more I use it the less I like it. I’m not understanding the edit conflicts any better, and I am not faster resolving them either”.

Based on user feedback and user tests we developed a prototype of the other option we had originally presented, which new tests proved to be better than our first choice. This resulted in our decision of stepping back and implementing this alternative interface, and brought us the new experience of replacing a beta feature with a new beta feature.

Another important learning was that since edit conflicts are rare, hard to reproduce and occur in stressful situations, testing of the interface under real conditions was very difficult. We solved that by developing a simulation page that allowed anyone to test the functionality in an easy and safe manner.

Screenshot of the improved diff page

Show changes in moved paragraphs

Where did it come from: Wish #2 on the 2015 German-speaking Technical Wishes survey
What is it: Marking paragraphs as moved, and highlighting changes that were made within, in order to avoid missing changes in the diff view.
Who worked on it: WMDE Technical Wishes team and WMF Desktop + Mobile Web team (for the frontend work of the mobile version)
Where is it at: In August 2018, the changes were deployed to all wikis.
more info

Contrary to the projects described before, this wish led us to work on existing backend code. The code for the diffing algorithm (Wikidiff2) was quite old and complicated, and we were the first ones in years who really touched it. While working on the improvement, we found an independent old bug, that would have made our effort useless in most cases and would have surfaced more often due to our changes, so we had to fix this problem simultaneously to the implementation of the actual feature. This lead to a complicated situation in communication, since two different changes on the same component had to be deployed within short time.

Working on the diffing algorithm is special in the way that this is an optimization problem and thus can never be optimal. We had to find the right threshold that improves the situation for many cases, while it worsens the situation for as few cases as possible. This resulted in mixed feedback, since some users were very happy about the improvements while a smaller number of users were disappointed by the regressions.

One important lesson we learned here was to use a test setup to be able to test the effect of a change against a bigger number of random example diffs. This allowed us to test variants of a code change to find a good threshold for optimization, and it helped us communicate with clear numbers.

Open questions and challenges

While the general approach of our working mode turned out to be very successful, there are a few challenges and drawbacks that we have identified.

So far we have mainly been focusing on the results of the wishlist survey. As the wishlist allows submissions in any possible area, the projects that evolve out of the wishlist are touching different areas as well, from all over the code base and feature space. Doing many things in different areas means that we need to dive into new topics all the time, and coordinate with different teams and people on each new project. On the downside, this is much less efficient than working in a specific area only. On the upside, the team is getting more and more experience in multiple areas of MediaWiki.
Due to the number of open wishlist projects and limited resources we are sometimes not able to solve a problem at its core but need to limit ourselves to easier solutions. For example, a good solution for one wish might have been to change how notifying users works as a whole in MediaWiki, but we could only add new notifications to the existing system instead.
Our projects often result in new MediaWiki extensions, and extensions require long-term maintenance. With the growing amount of extensions we own, the amount of maintenance work increases. This results in having less and less resources available for the development of new features.
Sometimes, when we start to work on improving existing code, we realize that parts of it haven’t been touched for years. This can lead into immediate maintenance, and slows down the development process. This happened, for instance, during our work on the wish Show changes in moved paragraphs, where we had to improve the Wikidiff2 algorithm in general.
Our processes – including the multiple feedback rounds and deployments – require parallel work on several unrelated projects in different project stages at once. This demands a high level of multitasking from everyone involved.
Over the past years, there were two independent big surveys collecting technical wishes: one on German Wikipedia for the German-speaking communities and one international (mostly) English-speaking survey on Meta. Not only does this pose a form of redundancy, it also opens questions about fairness among the communities. Why is the German-speaking community the only community to have a separate survey, and why are English and German native speaking people preferred in the way that they can discuss wishes in their first language?
Similar to that, it also seems unfair that people of other Wikipedias don’t have a direct contact to a development team for their issues, which would allow them to get involved earlier and more intensively into the development, like it is the case on German Wikipedia.

Conclusion

Since the first Technical Wishes survey in 2013, we’ve learned a lot about community-centered software development. In the past five years, we’ve continuously reflected on our methods, and we’ve iterated and improved them to find the best ways for how we can build software for the Wikimedia projects. Among all that we’ve learned, these are our main lessons:

It’s secondary whether the ideas for your products come from a survey. What’s more important is that whichever format you use to collect user needs makes it possible to collect and discuss problems collaboratively.
To find the best possible solution for an issue, focus on the problem, not on suggested solutions. This means making room for research, conversations with users and targeted feedback rounds.
If you have a diverse range of users like the Wikimedia projects do, integrate them into your development cycle, e.g. by trying to understand different users’ needs in research and feedback rounds. Building software for a diverse range of users is almost impossible without their insights.
Make communication a priority and actively work to lower the barriers for collaboration with contributors on technical topics.
As a full-time team, we set a strong focus on MediaWiki software development. This allows us to address diverse users’ needs in the MediaWiki software itself, so that our products are usable by all language and project communities.
Wikimedia’s technical environment is very complex. That’s why support formats that help others get an overview, navigate the code and find the right contacts are crucial.

Hopefully, our insights, lessons learned, and observations, can help others to better understand the challenges when developing software for collaborative projects, or to get some ideas to improve processes and methods. We also hope it helps contributors to see how important their voices are in the development process. We have already learned a lot from working on Technical Wishes, and we are sure we will continue to learn something new every single day. We’re excited to tackle our challenges and open questions in the years to come.

This project wouldn’t be possible without the many cool people from different language projects, from German Wikipedia, and from teams and people at the Wikimedia Foundation. In every project we do, there are so many people involved, from developers to translators, tech ambassadors, beta testers, people who answered questions and many many more. Thank you all so much!

We are looking forward to an exciting year 2019, with more projects to come. As always, feedback, comments and questions are very welcome.

WMDE Technical Wishes/Approach/Lessons learned from the Technical Wishes project