Grants:Project/Rapid/Sophivorus/Excerpts

My goal with this project is to give excerpts a chance to become useful and widespread in the English, German, Portuguese, French, Italian and Spanish Wikipedias.

Background and rationale

Content reuse is a key practice in knowledge management and Wikipedia. In one way or another, it has been with us since the beginning, from basic features such as links and templates, to entire projects such as Commons and Wikidata. Reusing content has several important advantages:

Fosters collaboration by channeling contributors to the same resources, rather than having them work in parallel
Reduces maintenance by avoiding duplicate content that needs to be updated independently
Improves content quality by encouraging editors to merge and reuse related content, rather than having multiple versions in various stages of development

Selective transclusion is an underused and underappreciated form of content reuse. Although it has been available for years and is used sparingly on some wikis, it hasn't gained the widespread use and popularity of other forms of content reuse. One reason is that so far it has been difficult to use: it requires dealing with #lsth parser functions, <section> and <includeonly> tags, and has annoying problems with unwanted elements being transcluded (hatnotes, amboxes, infoboxes, etc). Another reason is the lack of awareness and documentation. Thus, selective transclusion remains an advanced and obscure technique, rather than the simple and widespread technique that it could become.

But there's hope! The relatively recent Excerpt module can transclude any section of any article and leave out any unwanted elements without having to add confusing <section> or <includeonly> tags. Two years ago I wrapped the module in a easy-to-use Excerpt template and started using it on the Spanish Wikipedia. Slowly I improved both the module and the template, created documentation, examples, screenshots and tracking categories. Now over 700 articles and 100 annexes use it on the Spanish Wikipedia, often multiple times, and almost every day a new article or annex joins. Totally unexpected users and uses appeared, such as reuse of population tables, sports results and TV episodes, plus the amazing excerpt trees developed.

Thus, I believe selective transclusion, when done with the Excerpt module wrapped in a easy-to-use template, has the potential of becoming a useful and even popular feature. This project is about achieving this potential.

Project Plan

Activities

Tell us how you'll carry out your project. What will you and other organizers spend your time doing?

During the first month, I'll focus on the Spanish and English wikis only (the two languages I'm fluent). My main tasks will be:

Improve the existing documentation. I already created the basic Spanish and English documentation, but many improvements are still possible. For example, I intend to create short and clear videos showing how to add excerpts to the Spanish and English wikis, as well as step-by-step instructions with screenshots. Working in Spanish and English simultaneously will help me develop documentation that is easy to translate, before moving on to the other languages.
Manually replace the Template:Transcluded section for excerpts in the 1500+ articles that use it in the English Wikipedia. The template only inserts a hatnote saying that the content below is transcluded. The transclusion itself is done manually using the raw #lsth parser function, right after the template. This is a very crude and obscure method, compared with the Template:Excerpt that inserts both a hatnote and the transclusion itself, and does it using the Excerpt module, which results in cleaner output without having to add <section> or <includeonly> tags to the transcluded article. Replacing one template for the other is often a simple process (I did it on a few articles already) but the amount of articles makes it a very time-consuming task. However, developing a semi-automated solution is not worth it because the English Wikipedia is the only big wiki using this method. Once all the replacements are done, the old template can become a redirect to the new. Plus, adding excerpts to 1500 articles will increase exposure of the excerpt technique and template significantly, since every excerpt includes a hatnote that links to the documentation about excerpts. This ensures that every excerpt spreads knowledge about them further (this effect can be seen by the considerable increase in the traffic of the Spanish documentation when a link to the documentation was added to the hatnotes).
Manually add excerpts to some top articles of the Spanish and English wikis to increase exposure
Develop at least one excerpt tree in the English Wikipedia to demonstrate the concept and potential to interested users (the Spanish Wikipedia already has several)

During the second month, I'll focus more on the German, Portuguese, French and Italian wikis. My main tasks will be:

Create the basic resources (module, template, documentation, categories, videos, screenshots, etc)
Leave a message on the appropriate area of each wiki sharing the technique and conveying the potential
Manually add excerpts to some high-traffic articles
Develop a user script to facilitate the spread of excerpts into new wikis. Recently I discovered a simple way to add valuable excerpts to thousands of articles, but being so many articles, a semi-automated solution becomes a necessity. Furthermore, I'll try to develop a tool that is generic enough so that other users in other wikis beyond my scope can use it to introduce excerpts. I have ample experience developing user scripts, gadgets and other code (1, 2, 3). The user script will essentially:
- Use regular expressions to search for articles with sections that contain just a link to a main article
- Go to the first article and automatically check if the linked article has a decent-sized lead section worth transcluding (many articles have very small or non-existent lead sections)
- If a decent lead exists, replace the link for an excerpt
- Issue a warning if the excerpt causes any errors that need to be manually fixed (mainly when transcluding lead sections that contain references defined elsewhere)
- Jump to the next article
Use the user script to add valuable excerpts to hundreds or thousands of more minor articles

How will you let others in your community know about your project (please provide links to where relevant communities have been notified of your proposal, and to any other relevant community discussions)? Why are you targeting a specific audience?

Post in the Spanish café and English village pump
Post in the talk pages of the existing relevant resources in the English Wikipedia: Template:Excerpt, Module:Excerpt, Wikipedia:Excerpts and Wikipedia:Transclusion
Post in the talk pages of the existing relevant resources in the Spanish Wikipedia: Plantilla:Extracto, Módulo:Excerpt and Wikipedia:Extractos
Direct link in the most relevant resources: Wikipedia:Extractos, Plantilla:Extracto, Wikipedia:Excerpts and Template:Excerpt
Notify users that have shown interest in excerpts, selective transclusion, content reuse and related ideas: Betelgeuse1960, IIM 78, Leoncastro, Alvaro Marques Hijazo, Matu100, Marcelo, Scann

What will you have done at the end of your project? How will you follow-up with people that are involved with your project?

I will have developed all the resources required by excerpts in six languages
I will have added hundreds if not thousands of valuable excerpts to six wikis
I will leave a user script for other users in other wikis to make use of
I will follow up through the talk pages of the relevant resources created on the six wikis

Impact

How will you know if the project is successful and you've met your goals?

All the relevant resources will be available in English, German, Portuguese, Italian, French and Spanish (which can be easily proved with links to the resources)
Hundreds or thousands of smaller articles will be using excerpts in the English, German, Portuguese, Italian, French and Spanish wikis (which can be easily proved with tracking categories)
Some high-traffic articles will be using excerpts in the English, German, Portuguese, Italian, French and Spanish wikis (which can be easily proved via massviews)
At least one excerpt tree will be developed in the English Wikipedia (which can be easily proved like in the Spanish Wikipedia)
A fully internationalized and generic user script for spreading excerpts into new wikis will be made available (which can be easily proved with a link to the software)

Resources

What resources do you have? Include information on who is the organizing the project, what they will do, and if you will receive support from anywhere else (in-kind donations or additional funding).

My computer, skills and time. I will not receive any support from anywhere else.

What resources do you need? For your funding request, list bullet points for each expense:

USD 1000 for one month developing resources in Spanish and English, adding excerpts to the English Wikipedia and developing at least one excerpt tree
USD 1000 for one month developing resources in German, Portuguese, French and Italian, adding hundreds or thousands of excerpts in six wikis using a semi-automated user script developed for the purpose, and manually adding excerpts to a few high-traffic articles

Endorsements

Support Excerpts are a great feature - easy to use, massive number of potential usages, and powered by a sophisticated piece of Lua code. In my opinion, they should be getting used a lot more often than they currently are, especially on the English-language Wikipedia. This grant is a bargain in terms of the increased editor productivity it will provide, on various Wikipedias. Yaron Koren (talk) 19:41, 12 December 2019 (UTC)
Support The Excerpt template is one of the best ideas I've seen to improve Wikipedia in recent years. As it is well argued, its use will make us strive to give coherence to the entries of the main articles that will then be reused as summaries in some more. Its generalization will result in a good articulation of our information. Bravo for the initiative. Betelgeuse1960 (talk) 20:07, 13 December 2019 (UTC)
Support As the main author of en:Module:Excerpt, I write here in response to its talk notice. Naturally, I favour transclusions and am pleased to see other editors taking an interest. On enwiki, an unclosed RfC generally supported greater use of excerpts in portals. However, I should warn that they have not been universally welcomed. Most of the pages on which Module:Excerpt was deployed (via its simple wrapper templates) were either reverted to old versions or deleted completely. The matter is now the subject of an ArbCom case, which fortunately has not yet expanded into the area of the software. I wish you all the best and would be happy to continue providing technical assistance but you should be aware that, on enwiki at least, this development has proven unexpectedly controversial. Certes (talk) 23:13, 13 December 2019 (UTC)
- @Certes: Wow, thanks a lot for that info and links! I read the controversy and it looks like it was triggered mostly by a lack of clarity about what the transclusions would entail for the portals, and then was greatly exacerbated by the past conflicts between two involved users. I will take this precedent as a warning about the importance of being clear in my edit summaries and similar places, but I see no reason why that controversy should carry over to the edits this grant proposes, especially considering that transclusions are already being done in thousands of articles on the English Wikipedia (with plain #lsth parser functions). Anyway, thanks again for the warning and support! Sophivorus (talk) 23:41, 14 December 2019 (UTC)
Support This sounds like a great idea. It also allows for information to be better updated in the main core article, eliminating some redundancy that tends to happen and working towards more cohesion between articles that might not necessarily talk about the same but share similar bits and pieces of information. It helps to understand where the current gaps of knowledge are. I don't know if this is possible in the current version of the tool but something that I think would be cool to have is that the tool should actually let you select which paragraphs or section you want to make an excerpt from, but I guess that's a bit like asking a rocket to the moon when MediaWiki seems to still be in the steam age. In any case, I support this application. Scann (talk) 00:28, 7 March 2020 (UTC)
- @Scann: Actually the template already allows to transclude specific sections (see the template documentation). As to specific paragraphs, that's currently not possible but totally doable by enhancing the Excerpt module, so I'll try to implement it if this grant goes through. Thanks for the support! Sophivorus (talk) 01:42, 7 March 2020 (UTC)
- @Scann: It looks like the Excerpt module already supported cherry-picking paragraphs, so adding the functionality to the template was easy and I just did! Sophivorus (talk) 14:04, 20 March 2020 (UTC)
  - Oh cool! Thanks! I'll have to check that later on then :) enjoy the quarantine! Scann (talk) 15:00, 20 March 2020 (UTC)
Support I'm not familiar enough with rapid grants to know whether the proposed budget, etc. is reasonable, but I heartily support the idea of excerpts. They seem like an innovative way to reduce the workload needed to improve/update duplicate content. Sdkb (talk) 06:26, 13 March 2020 (UTC) (brought here from the EN-WP Village Pump posting)