Toolhub/Progress reports/2021-10-01
Report on activities in the Toolhub project for the week ending 2021-10-01.
Kubernetes deployment progress
editLast week ended with a successful deployment into the "eqiad" Kubernetes cluster. At that point the application was able to fully load without crashing and users with the correct level of production access could setup HTTP proxy access to test the application.
This week Giuseppe Lavagetto and Kunal Mehta created the necessary LVS load balancer, DNS records, and Varnish configuration to make https://toolhub.wikimedia.org route to the Toolhub deployment in the eqiad cluster. This allowed Bryan to continue with testing the OAuth authentication for Toolhub and the crawler's operation from inside the Foundation's production network.
The OAuth fixes done in T291447 were shown to be correct once the requests were seen by metawiki to be originating from toolhub.wikimedia.org. Bryan and others have been able to use OAuth to login to the pre-release service.
The next test was adding a toolinfo.json URL via the Toolhub UI and then waiting for the crawler component to run. Adding the URL worked as expected, but the crawler is failing to fetch data as hoped. When the crawler attempts to fetch the contents of https://toolhub.toolforge.org/toolinfo.json it fails with a ConnectTimeout error. This is a connection that we know must be made through the url-downloader HTTP proxy due to network configuration, but the stack traces were not clear if that was happening or if the application was trying to directly contact the Toolforge HTTP server. Originally Bryan thought that an http_proxy environment setting was all that would be needed to configure the application to use the proxy for this URL. Deeper investigation revealed that actually an https_proxy environment setting was needed. The python code (and honestly probably most other clients) will use the *_proxy settings matching the protocol of the requested URL, http_proxy for http://... URLs and https_proxy for https://... URLs. This configuration change changed the stacktrace generated by the failure slightly, but it has not turned out to resolve the basic issue.
Metawiki help documents
editBryan made quite a bit of progress this week on adding end user help documentation to the Toolhub pages on metawiki. The prior Toolhub landing page describing the Foundation project to build the software has been moved to Toolhub/About (thanks Quiddity!) and a new landing page has been built focused on user help. Work will continue on this help documentation, well... forever, but also certainly next week.
Planning launch announcements
editBirgit has been working on our draft announcement plan and timeline. We do not have final dates yet due to the remaining crawler bugs, but we do now have a sold start on a list of places we want to make sure to post an announcement and a set of folks who will be helping do that.
Wrap up
editOne step forward! One step to the side. :/ We are inching closer to a public release, but still have some technical challenges to work out before we can be confident in the launch date. The amount of time this is taking is a bit frustrating, but we know that we only get one chance at making a first impression with the larger community and we do not want to waste that with a buggy product.