Future of Toolserver

Toolserver

This box: view · talk · edit

Toolserver closed down in 2014. The future of Toolserver was uncertain; as of September 2012, it's sure that the service will be discontinued in 2014-12-31 at the latest, but on 2013-11-11 the admins wrote all users encouraging them to delete their data and promising to disable all accounts (and hence tools) on 2014-01-06. Feedback is needed to ensure Wikimedia projects will keep having Toolserver vital services in the future; some Toolserver users added notices to their web tools to inform all wikimedians.

Overview

The Wikimedia Toolserver is a cluster of servers operated by Wikimedia Deutschland e. V., (not the Wikimedia Foundation). The situation was notified by DaB. (TS system administrator):

It is not possible for me to run the toolserver for another year with the current hardware – you all know why. For this reason I
will request a change of the budget at the general meeting at November, so there will be a vote about. If this vote should fail (and we get no money for new hardware), I am going to retire from my job as root at 30. December 2012.

Erik (WMF) clarified that it's WMF's decision, on its end, to discontinue services to Toolserver at some point:

However, for our part, we will not continue to support the current arrangement (DB replication, hosting in our data-center, etc.) indefinitely. The timeline we've discussed with Wikimedia Germany is roughly as follows:
Wind down new account creation on toolserver by Q2 of 2013 calendar year

Decommission toolserver by December 2013

Pavel (WMDE CEO) ensured:

Wikimedia Deutschland will make all necessary investments to keep the Toolserver up and running, and we will use the next year to discuss together with the Toolserver users and community a way to transition into WikiLabs. And I can only urge you all to take active part in the design and development of Wikilabs, so that it will provide for your needs.

Delphine (WMDE treasurer) asked a document to explain #Why Toolserver is important and what #Toolserver needs:

Hi Andre,
> I think we should help DaB to make a case why toolserver is important. We
> should list the projects depending on toolserver and how, and all the other
> things that are important. [...]
> With a document like that we can make a stronger case for WMDE to reserve
> budget, or have them apply for a separate grant to the foundation.
Thanks for this constructive approach. I believe this would be indeed the best way to go forward on this issue and would help the General Assembly of Wikimedia Deutschland make the right decision about this.

Why Toolserver is important

List tools and services you find most important, with a short explanation. Most of them should have a page on tswiki at some point.

Usage stats (to be checked and improved)
- An average of over a thousand database queries per second, each providing crucial info to wikimedians.
  - For comparison, Wikimedia Labs has served over 40 millions DB queries (or rather, DB connections) in total, as of January 2015.
- 3,4 millions hits per day, sometimes 12.
  - Update: Tool Labs has about 200 thousands per day as of June 2013.[1]
- Unknown number of web tools users.
- An average of 40 logged in shell users in any moment.
Wiki Loves Monuments: All the tooling around this really need toolserver.
Commons: commons has a lot of toolserver tools more or less integrated in the interface, and there are support tools running.
GLAM: a lot of glam-related work is done on toolservers
Small tools for wiki support: all the bots archiving talkpages, create administrative pages automatically etc, etc...
Steward support: there are a lot of tools (also used generally) that really make cross-wiki abuse detection and other stewardwork much easier. (SWMT support too relies on toolserver)
Research projects: use ToolServer to retrieve data (SQL queries and APIs ^[1])
Wikipedia analysis applications (e.g.: Manypedia^[2] and WikiTrip^[3])
Supporting enthusiastic users to run their own analysis queries and tasks
ScanCat is used in Spanish Offline Wikimedia project by Wikimedia Venezuela Chapter
It allowed using Proprietary Software (sometimes there is no open source software or it is suffering from lots of drawbacks)
MANY (if not all) Wikisources strongly rely on bots running from toolservers for its core activities and for statistics.
fiwiki: fiwiki has of toolserver tools more or less integrated in the interface, and there are supporttools running.
cs wikis have replacements for special pages which have been turned off.
Mapping on wikipedias when clicking on geographical co-ordinates: GeoTemplates often use GeoHack tools, because some cartographic viewers and GeoTemplates don't allow #expr such as 0.3124*{scale}
Connectivity analysis tools are used by many chapters. Several years of experience show that the tool allows improving connectivity of wikipedia articles, while interrupts in toolserver operation lead to all basic connectivity indicators becoming worse.
Wikipedia:Contributor copyright investigations is dependent upon Contribution Surveyor to complete its work.
Wikipedia:Request an account is dependent upon the Account Creation Interface to facilitate account creation requests on the English Wikipedia.
Reference generator enables streamlined creation of standardized citations in a way not otherwise possible.

Toolserver needs

See also (German) w:de:User:DaB./MV-TS-Antrag

Replacements
New hardware
- ...
- ...
Other
- WMF not switching it off?
- ...

Needs for a Toolserver replacement

Some tools already live outside Toolserver; Erik (WMF) asked Toolserver users to list on Toolserver features wanted in Tool Labs the requirements for their services to continue on the new service that he wants to replace Toolserver, known as Tool Labs (sub-project of Wikimedia Labs). Tool Labs is currently being designed and built by Marc Pelletier (WMF).

Request for help to all toolserver users

The following has been sent out to all active and expired accounts on the toolserver (on March 7th 2013). To make Tool Labs fit your tools or the tools you use, please add them and their technical dependencies to MediaWiki:Toolserver/List of Tools.

Hello!

As you might know, Wikimedia Foundation is building a cloud-based infrastructure (Labs/Tool Labs) that – in the long run – will be a replacement for the toolserver. Don't worry! The toolserver will be not just be switched off. WMF and WMDE would like to support you as well as possible when it comes to migrating tools. There will be enough time for this process. We will offer different forms of support which you will hear of soon.

Right now, Tool Labs is not ready. WMF staff, mainly Marc Pelletier is building it. You might have seen the general list of needed and wanted features. For many of you crucial features are database replication and user databases. Both are upcoming features in the nearer future.

Your tools and their dependencies are taken into account to build the new infrastructure. This is why I am asking you for help: Please provide information about your tools! I started an incomplete list at (MediaWiki:Toolserver/List of Tools, based on the Wiki and Jira). Personally I was really impressed about its length and diversity – how cool! Several people said there was so much missing in this list – so please help me to complete it!

Here is what I'm asking you to do:

Please check if your tools are on the list (correctly).
Please fill in your software dependencies, data dependencies, use patterns (are they running continously? or webservices? Batch runs? etc.), the license.
If you have not given your software an explicit license, please note that only free software can migrate to Labs. Consider putting your stuff under a free license.
If you have already migrated your bot or tool to Labs and it is in the list, please say so in the last column "status".
If your toolserver account has expired and/or your tool is not active on the toolserver right now, please add it to the second table at the bottom of the page. In the last column write that it is not running currently and if you want to revive it. If you are considering to revive it, please fill in all the details, too, so that your needs will be taken into account.
Please follow the toolserver-announce list for more information to come.

Thanks for your help! If you have any questions, please don't hesitate to ask me, to ask on toolserver-l and/or on labs-l.

Cheers, Silke.

December 2013

As of December 2013, there is wide consensus that Labs is not sufficient to fit the tools' needs and that Toolserver is decaying beyond reasonable.[2]

Notes

[1] ttp://gizmodo.com/5942168/the-wikipedia-gender-divide-visualized

[2] ttp://manypedia.com

[3] ttp://sonetlab.fbk.eu/wikitrip

[1]

[2]

[3]