Community Wishlist Survey 2022/Bots and gadgets/A bot or gadget to publish public Git repo to a gadget or user script
A bot or gadget to publish public Git repo to a gadget or user script
- Problem: It is hard to develop JavaScript scripts and gadgets within MediaWiki. No build tools (especially transpilation), no way to push out changes to many files in one go. Also no review tools (like pull requests).
- Proposed solution: Provide a publishing tool that would effectively publish a script from e.g. GitHub or GitLab to one of Wikipedia sites. I would assume that the code repository would need to be public (to avoid authentication). On the other hand the tool would publish as the user using it. So the tool would need to have Wikimedia OAuth or use Bot passwords. The publishing tool would be setup using:
- Some title of the gadget.
- Source and destination URL(s).
- This could be just one JS. Like https://example.com/build/replylinks.release.js → https://en.wikipedia.org/wiki/User:Nux/replylinks.js
- It could also be JS and CSS. Each with its own destination.
- Would be nice if the tool would have some transforms and e.g. would know that for "https://github.com/Eccenux/wiki-SearchBox/blob/master/SearchBox.js" it actually needs to download "https://raw.githubusercontent.com/Eccenux/wiki-SearchBox/master/SearchBox.js".
- JS and CSS itself would not be transformed in any way in this process.
- Additional info like e.g. default publish message (e.g. providing source URL in the description but also attribution).
- Note that I assume the person creating the publisher do not have to be the author of the script.
- Once the gadget-publisher is set up, one can publish with one click (optionally adding info to the default publish message to e.g. add version info).
- Who would benefit: Gadget developers and their users
- More comments: So at the time of writing Gadgets only support ES5. It is nice that some work on ES2015 is starting, but you know this is kind of all figured out in the frontend world. I think it would all just be easier if wiki developers would use things most other frontend devs use and just quickly publish to Wikipedia. It shouldn't be too hard. The hardest part would probably be making a decent GUI and auth. Other then that you just download one text file and push the text to wiki.
- Phabricator tickets: (related)
- phab:T36958: User-level gadget repositories
- phab:T71445: Implement a sane code-review process for MediaWiki JS/CSS pages on Wikimedia sites.
- Proposer: Nux (talk) 00:22, 19 January 2022 (UTC)
Discussion
Come to think of it probably more then one file per gadget would need to be published in some cases. For example build.js (could be a result of Babel transpilation or simple Gulp concatenation) and build.css (maybe built with Less or Sass). And probably some meta like name/title and maybe a project description. Default publish message would be a good thing too. --Nux (talk) 00:48, 19 January 2022 (UTC)
- This task probably wants ES6 to be fully supported in gadgets based on the motivation, which is probably phab:T75714 (and phab:T178356?), so that might be a good separate wish. Izno (talk) 05:03, 19 January 2022 (UTC)
- A proposed bot could easily implement babel to transpile ES6 code to ES5 Ed6767 (talk) 10:53, 19 January 2022 (UTC)
- meh. I don't think naive ES6 is critical, babel and rollup or web pack or whatever can take care of that just as part of the build process.. What I do think would be pretty cool is if we had some sort of NPM package and json standard that would allow you to map dist/ paths to pages on a wiki and then would automatically publish it for you (if you have the credentials). —TheDJ (talk • contribs) 09:33, 19 January 2022 (UTC)
- Yes, something like that. But not with JSON as I would prefer a GUI. That would make this more independent and easier to extend. When you have some description standard then adding new fields and references is getting complicated and progresses slowly. Also there really is no need for the tool to be published just by the author of the script. I think it would be good if anyone could update the script. Nux (talk) 11:17, 19 January 2022 (UTC)
- This request seems to mix easy publishing and a request for ES2015 and I really think those are two separate things that should not be in one request. "but you know this is kind of all figured out in the frontend world" Cool and all, but as long as there is no PHP implementation of an ES2015 parser, that's irrelevant to us, since we are NOT frontend and NOT nodejs, and shelling out to other languages for essential functionality is problematic (see parsoid which was in nodejs but is now in php again, because it was too problematic). —TheDJ (talk • contribs) 09:41, 19 January 2022 (UTC)
- I'm proposing easy publishing instead of tools for building stuff. And I would argue that gadgets are from frontend world. I did some changes in popups a few years back and it was... well a bit of a headache. It would be so much easier if I could just fork a repository for plwiki and the changes I need. Maybe add a pull request for some of them. Then just compile and click to publish. Nux (talk) 11:24, 19 January 2022 (UTC)
- @Nux I agree with TheDJ and others that the wording of the proposal seems to imply you want (a) ES2015+ support in gadgets/user scripts and (b) essentially a deploy script from a GitHub repo to the wiki. I think focusing on just (a) would be more appropriate. For (b), it might be difficult to come up with a standardized system that works for everyone. Many gadgets such as MoreMenu and Twinkle are written in ES2015+ and transpiled it to ES5 as a build step. That build step can vary based on developer preference. Some like GruntJS, others like Webpack, etc. A configurable deploy script, with a CLI interface (give it your BotPassword credentials, the JS file and the wiki page to deploy it to) is something I think we could make, and quite easily at that. But I don't think it's good to mix both ideas of build steps and deployment, since this varies.
- Do you think you could reword this proposal to focus on either (a) native ES2015+ support in gadgets/user scripts or (b) a configurable deploy script (that does not include build steps like ES5 transpiling)? MusikAnimal (WMF) (talk) 18:58, 20 January 2022 (UTC)
- @MusikAnimal (WMF) OK, I re-worded the problem description. ES2015+ was just an example of things that are hard because of how MW works. I also added some more info in the Proposed solution part. Let me know if you think it is still not clear yet. --Nux (talk) 22:25, 20 January 2022 (UTC)
- This looks great, thank you! The problem statement I think is more clear now. We can definitely deliver on the deploy script. Rethinking this, as a CLI tool (pass in BotPasswords credentials) or even a Toolforge tool (where you login with OAuth), we should be able to offer build steps as options, such as ES5 transpiling. We can iterate on it together :) Thanks for putting all this work into your proposal! I'm going to mark it for translation now. MusikAnimal (WMF) (talk) 05:03, 21 January 2022 (UTC)
- I also trimmed down the proposal a tiny bit to aid translators (hope this is okay!), and removed the mentions of phab:T75714 and phab:T178356. These are going to happen anyway, and are about ES2015+ support in gadgets (which is what I originally thought you were talking about in your proposal). Anyways again, your problem statement is clear, and that's all that matters. Together we'll come up with the best solution! MusikAnimal (WMF) (talk) 05:30, 21 January 2022 (UTC)
- This looks great, thank you! The problem statement I think is more clear now. We can definitely deliver on the deploy script. Rethinking this, as a CLI tool (pass in BotPasswords credentials) or even a Toolforge tool (where you login with OAuth), we should be able to offer build steps as options, such as ES5 transpiling. We can iterate on it together :) Thanks for putting all this work into your proposal! I'm going to mark it for translation now. MusikAnimal (WMF) (talk) 05:03, 21 January 2022 (UTC)
- @MusikAnimal (WMF) OK, I re-worded the problem description. ES2015+ was just an example of things that are hard because of how MW works. I also added some more info in the Proposed solution part. Let me know if you think it is still not clear yet. --Nux (talk) 22:25, 20 January 2022 (UTC)
- I'm proposing easy publishing instead of tools for building stuff. And I would argue that gadgets are from frontend world. I did some changes in popups a few years back and it was... well a bit of a headache. It would be so much easier if I could just fork a repository for plwiki and the changes I need. Maybe add a pull request for some of them. Then just compile and click to publish. Nux (talk) 11:24, 19 January 2022 (UTC)
Voting
- Support Qwerfjkl (talk) 22:30, 28 January 2022 (UTC)
- Support Izno (talk) 22:48, 28 January 2022 (UTC)
- Support JakobVoss (talk) 07:25, 29 January 2022 (UTC)
- Support Aca (talk) 12:22, 29 January 2022 (UTC)
- Support Douglasfugazi (talk) 21:08, 29 January 2022 (UTC)
- Support Thingofme (talk) 02:22, 30 January 2022 (UTC)
- Strong support FWIW, the Bulgarian Wikipedia has been using a custom Python daemon to synchronize its JS/CSS, the Module NS, and the spam and title blocklists to Git for more than 3 years, which allows us to do code review in GitHub (e.g. for JS/CSS). There's a lot to desire of the codebase (I was still quite a Python newbie when I wrote most of it) and there are some outstanding bugs, but it does the job. References: synchronization daemon code, JS/CSS repo, Module NS repo.
— Luchesar • T/C 11:02, 30 January 2022 (UTC)- I should've added that part of the complexity of our solution is due to the requirement to support continuous (as opposed to “on demand”) bidirectional (i.e. also Wiki->Git) synchronization, as some of our interface admins didn't like the idea of GitHub repos and requested the bot to handle gracefully the local JS/CSS edits in Wikipedia itself. Preventing and resolving possible edit conflicts between the live Wikipedia code and the GitHub repos turned out to be quite the task, but the bot seems to handle this well so far.
— Luchesar • T/C 11:16, 30 January 2022 (UTC)
- I should've added that part of the complexity of our solution is due to the requirement to support continuous (as opposed to “on demand”) bidirectional (i.e. also Wiki->Git) synchronization, as some of our interface admins didn't like the idea of GitHub repos and requested the bot to handle gracefully the local JS/CSS edits in Wikipedia itself. Preventing and resolving possible edit conflicts between the live Wikipedia code and the GitHub repos turned out to be quite the task, but the bot seems to handle this well so far.
- Support --Nux (talk) 13:16, 30 January 2022 (UTC)
- Support Titore (talk) 17:13, 30 January 2022 (UTC)
- Support Nw520 (talk) 22:53, 30 January 2022 (UTC)
- Support Daniuu (talk) 23:53, 30 January 2022 (UTC)
- Support This would make my workflow far easier, and I suspect the same is true of almost everyone who writes userscripts. JPxG (talk) 00:30, 31 January 2022 (UTC)
- Support Matma Rex (talk) 16:31, 31 January 2022 (UTC)
- Support Krzysiek 123456789 (talk) 08:42, 1 February 2022 (UTC)
- Support Msz2001 (talk) 10:49, 1 February 2022 (UTC)
- Support Alain Artivalys (talk) 13:16, 1 February 2022 (UTC)
- Support Rdrozd (talk) 17:53, 2 February 2022 (UTC)
- Support DannyS712 (talk) 03:06, 3 February 2022 (UTC)
- Support – Ammarpad (talk) 15:51, 3 February 2022 (UTC)
- Support AtlasDuane (talk) 18:38, 3 February 2022 (UTC)
- Support ·addshore· talk to me! 22:42, 4 February 2022 (UTC)
- Support paul2520 (talk) 18:54, 5 February 2022 (UTC)
- Support —Thanks for the fish! talk•contrib (he/him) 21:23, 5 February 2022 (UTC)
- Support Vulp❯❯❯here! 03:43, 6 February 2022 (UTC)
- Support Ayumu Ozaki (talk) 05:04, 6 February 2022 (UTC)
- Support ~Cybularny Speak? 20:10, 7 February 2022 (UTC)
- Support ~~~~
User:1234qwer1234qwer4 (talk) 22:28, 7 February 2022 (UTC) - Support Quiddity (talk) 07:26, 10 February 2022 (UTC)
- Support Helder 22:00, 10 February 2022 (UTC)
- Support Yair rand (talk) 23:47, 10 February 2022 (UTC)
- Support --evrifaessa ❯❯❯ talk 15:49, 11 February 2022 (UTC)