Toolhub/Progress reports/2020-11-20

Report on activities in the Toolhub project for the week ending 2020-11-20.

No report for week of 2020-11-13

No report was published for the week of 2020-11-13. That week was a short week for Wikimedia Foundation staff.

Toolinfo read API

Tracked in Phabricator:
Task T264811 resolved

/api/tools - Paginated, sortable list of all toolinfo records
/api/tools/{id}/ - toolinfo record

Toolinfo.json crawler and status API

Tracked in Phabricator:
Task T264810

A basic web crawler driven by the registered toolinfo.json URLs has been implemented in gerrit:641559. The crawler is currently manually runnable using poetry run ./manage.py crawl in the developer's environment. Each crawl run's outcome is documented in a set of models which are exposed via the API for inspection:

/api/crawler/runs/ - list of known crawler runs
/api/crawler/runs/{id}/ - single crawler run
/api/crawler/runs/{run_id}/urls/ - list of all urls crawled in a run
/api/crawler/runs/{run_id}/urls/{id}/ - info for a specific URL crawled in a run

Some follow up work is desired to improve the crawler and the stored information, but we have a working system!

Locale switching in Vue frontend

Tracked in Phabricator:
Task T264815

Srishti has added preliminary support for locale selection in the Vue frontend app using the vue-i18n plugin and a custom locale switcher UI element. She plans to continue to build on this with RTL testing and improvements in the coming week. We expect to start deeper integration with TWN in early December.

Improved error responses from API

Tracked in Phabricator:
Task T267432 resolved

We have forked the drf-friendly-errors-2 library to add support for newer versions of Django Rest Framework. To keep things moving, we are currently installing our forked version but we submitted our changes upstream for inclusion in a future release.

Errors from the API will now return results which include error codes to help API consumers detect and respond to API errors with a slightly easier error determination method than message string processing.

Miscellaneous changes

Various other changes that landed in the repo in the last 2 weeks:

Fixed a typo in the GPLv3 header used in Python source files
Added a Makefile target to run black on the developer's git repo
"Crawler" prefix removed from names of models in the toolhub.apps.crawler app
Date sorting fixed in /add-remove-tools screen
phab:T267261 Upgraded to drf-spectacular 0.11.0 and switched to using it's @extend_schema_view decorator to add localizable descriptions to API endpoints

Wrap up

Even with the vacation time in the last 2 weeks, we are making good progress. The implementation of the crawler and toolinfo storage and their related APIs unblocks a lot of UI work. Both the Django and Vue systems are ready to start on the process of integrating with TWN for message translation, which could start happening as early as next week (although the first week of December is more likely). We are also very close to the point of having enough functionality to make demoing with others a useful process.

We are also starting to evaluate our actual progress vs our announced plans for the quarter. Audit logs, TWN integration, and OAuth for API consumers all remain to be done. With only 6 calendar weeks, and functionally 4 working weeks left in the quarter we are cautiously optimistic that all of these remaining components will be started. It is unlikely that they will all be completed however. We should be able to make better estimates in the next 2 weeks about what will and will not be possible in the remaining timebox.