Content Partnerships Hub/Software/Wikimedia Metric Tools

Wikimedia Metric Tools

22 February Meeting

This document is Wikimedia Sverige’s compilation of the input gathered during the online meeting on 22 February; it does not claim to give a full reflection of everything that was said during the meeting. For the full documentation and notes, see this link.

Outcomes

Quality of metrics

• Flawed metrics are better than no metrics, but it is a large issue that metrics disappear, that metric tools break, that metrics are flawed or that they are not explained (through documentation) or understood. It seems unprofessional from an external point of view, and is cumbersome when you are funded as a Wikimedian in Residence (WiR).
• Institutions, including those funding WiR positions,  are increasingly building in Wikimedia metrics and statistics in their annual plans, which makes it critical that the metrics exist. The fact that Baglama2, in this specific example, went down at the end of the year was disastrous as it is just in the middle of reporting time. It is crucial to have the metrics for reports as well as applications for funding.
• We need a workflow for identification, planning and execution of the tools selected (through the Content Partnership Hub or otherwise), but first we need to define urgencies as compared to priorities. When is something an emergency, and for whom? Our needs are different.
• The lack of tools that reliably fulfill the needs of the movement is in itself a hindrance for technical innovation (such as providing roundtripping), which is a major obstacle in the ability of the movement to provide knowledge as a service.

Selection of metrics

• We need to give a closer thought on what kind of metrics we should collect, as GLAM collaborations deepen and evolve.
• To compare and contextualize, we need standardization and documentation of the metrics. To measure impact, both quantitative and qualitative analysis is needed, but without documentation, it is hard to add qualitative analysis to the quantitative metrics.
• It is better to have basic, simple, stable and realistic tools than ideal tools.
• If we use wishlists or similar processes for determining tool needs/priorities, we need to make sure that the views collected represent the views of the global Wikimedia movement, including its affiliates, as well as the global GLAM communities and institutions, not just the ‘usual suspects’.
• It is unclear how WMF wants to have feedback, as well as how they make their decisions. What is the formal process for WMF to adopting a tool or its function/need?

Availability of metrics

• With the introduction of Structured Data on Commons we promised to our GLAM partners a future where they could more easily query the content on Commons. When Wikimedia Commons Query Service (WCQS) was officially launched, it required being logged in as a Wikimedia user, which was a surprise to many accustomed to Wikidata as a freely usable SPARQL endpoint not requiring any authentication. An authenticated SPARQL endpoint is also contrary to the premise of linked open data, and makes federated queries unusable. This was disappointing for the partners and it is difficult for us to explain how this restriction is in line with our values which are reflected by our otherwise open platforms.
• If we want GLAM institutions to use this kind of tools themselves and to integrate the tools into their own operations, we need to assure them that they will exist and work. That is, something in line with what Service Level Agreements (SLAs) provide in the contexts which GLAMs are familiar with.

Maintenance of metric tools

• Key tools cannot be maintained by just one person.
• What is the process for determining which the Key tools are and for revising such a list?
• What is the protocol for abandoning a previously prioritized tool (to maintain or develop further) in order to deal with an emergency with another tool? See previous note about when something is an emergency and for whom.
• We need to recommend and adopt best practices when it comes to the development of  the most widely used tools, in terms of accessibility, localization, platform dependency etc. These tools cannot rely on a small set of users or developers.
• Documentation and documentation protocols are needed for maintenance.

The Community Programs  team at the WMF requests that needs be described at this link in order to be more easily brought to attention during the upcoming annual planning process.

For the Wikimedia Sverige tool survey referenced in the meeting see this link.

For the GLAM manifesto (currently a draft) from the Wikimedians in Residence Exchange Network, see this link.