Grants:Project/Susannaanas/GLAMpipe

This project is funded by a Project Grant

proposal

people

timeline & progress

finances

midpoint report

final report

statusselected

Project Grants

GLAMpipe

summaryGLAMpipe is a Swiss knife from Finland for content and data manipulation and upload to Wikimedia projects.

targetWikimedia Commons, Wikidata

type of granttools and software

amount5 087 €

grantee• Artturimatias• Susannaanas• Zache

advisor• John Cummings• Navino Evans

contact• susannaanasgmail.com

this project needs...

volunteer

affiliate

advisor

created on13:30, 6 October 2016 (UTC)

Friendly space expectations

Project idea

What is the problem you're trying to solve?

Content imports to Wikimedia Commons and data imports to Wikidata are always complex. They require sorting out messy (meta)data, using a combination of tools and technologies, working your way through practices and policies, and finally creating a software tool for the import. And the work is always done on a case-by-case basis.

What is your solution?

To facilitate these workflows, Wikimedia Finland has developed a tool that can read data from different sources, manipulate it in a number of manners, and export it to a variety of repositories.

GLAMpipe is an extensible, open source web-application intended for cultural metadata manipulation. The workflow is based on documents and nodes instead of plain tables. Nodes are like building blocks of data flow. They can act as a data source, they can split, combine, create wikitext or process data other ways, and nodes can export data to files or web services like Wikimedia Commons or Wikidata.

GLAMpipe is a tool for power users. Compared to Pattypan, it serves more complex use cases. Compared to GLAM-Wiki Toolset, it offers tools to work with the data, and makes import and export very easy. It gives the user the power of bots without the need to code.

Project goals

When successfully launched and deployed, the tool can unlock vast amounts of data and content. It gives data-savvy users, such as GLAM professionals, the ability to upload content to Wikimedia Commons, or data to Wikidata.

It will give the users an opportunity of reusing the earlier data manipulation solutions created by the community. The programmable nodes are like customizable legos for each step of the import process for data or files.

1. With this project we want to focus on bringing data to Wikidata: creating a node for preparing data, and another one for saving prepared data to Wikidata.
2. Secondly, we focus on the distribution: delivering the tool in different delivery packages - as an online tool and a more powerful desktop application.
3. As a third point, we work with a group of test users to make the interface easy to use for GLAM professionals and other data partners. We work with them by tracking issues in the code repository, collect evaluations with a survey and meet online to discuss the experiences and to develop solutions.
4. The process will produce tutorial videos and documentation for using the application and developing additions to it. The aim is to have the tool adopted by the GLAM-Wiki community.

GLAMpipe has been developed in 2016 with the support of the Finnish Ministry of Education and Culture. This project will help it be adopted by the key GLAM-Wiki contributors, and extend it’s usage to Wikidata. The plan to develop the tool extends beyond the grant period. More capabilities can be developed in further project extensions based on needs.

• 
• 
• 
• 

Project plan

Activities

General framework development

An online version of GLAMpipe will be created that is able to use OAuth login. Currently GLAMpipe is intended for desktop installation. However, installation is quite technical and it is not suited for non-technical users. Initial online version exists, but it requires more work in order to be reliable. A regular web location will be established in Wikimedia Labs, for example.

Wikidata node

New nodes for transforming data for Wikidata will be created. The transformation node needs to take into account the triplet data structure, be able to map data to Wikidata and format appropriately. The Wikidata export node takes transformed data hash as parameter, makes sanity checks for it and saves it to Wikidata. The export node checks that there are no duplicate values and merges statement to existing ones if possible. The export node is made using a Widar-like OAuth interface and plans to share its code.

Co-development and community building

To measure the expectations of our test users and their satisfaction with the tool we will carry out a survey. The survey can be complemented with additional data further along the lifecycle of the project.

We will also arrange 2 online meetings with the test users to go through their feedback and solve design issues together.

The GitHub repository will serve as the platform for user support and communicating about the issues that arise during testing.

A community site will be established in the context of GLAM outreach and documentation.

Outreach and learning

A set of instructional videos will be produced to illustrate the use of GLAMpipe for different use case scenarios.

There will be 3 blog posts: in Wikimedia Finland blog, in Wikidata blog (by permission) and one other. GLAMpipe’s process will be publicized in the Wikidata + GLAM Facebook group and the Wikidata mailing list.

GLAMpipe will be presented in several Finnish and 1 international events during the grant period.

Reporting and documentation

The grant report will be created bearing in mind any possible learning outcomes, and published possibly in other formats and environments.

Schedule

The project is scheduled to run 6 months from January 2017 to June 2017. The due date of the project will be postponed to 30 September 2017. – Susanna Ånäs (Susannaanas) (talk) 16:19, 13 June 2017 (UTC)

Budget

Issue	Ari	Kimmo	Susanna	Amount	Sum	In-kind hours	In-kind sum
General development
Online version development	16 h			16 h	368 €	16 h	368 €
Installation in the server environment	8 h	8 h		16 h	368 €	6 h	138 €
Wikidata import node
Data transformation design		32 h		32 h	736 €
User interaction design and testing			16 h	16 h	368 €
Node development	32 h			32 h	736 €	12 h	276 €
Wikidata export module		32 h		32 h	736 €
Co-development
Test user survey			8 h	8 h	184 €
Online discussions	6 h	6 h	6 h	18 h	414 €
Community discussion and support	8 h	8 h		16 h	368 €
Outreach and learning
Instructional videos			8 h	8 h	184 €
Publicity (blog posts, social media)	4 h	4 h	4 h	12 h	276 €	2 h	46 €
Travel to Wikimedia hackathon		x			250 €
Accommodation at Wikimedia hackathon		x		3 x 79 €	237 €
Reporting, documentation
Reporting	4 h	4 h	10 h	18 h	414 €
Documentation of core and node creation	12 h	6 h		18 h	414 €	6 h	138 €
Total	90 h	100 h	52 h		6 053 €
In-kind	42 h				966 €
Applied sum					5 087 €
hourly salary	23 € / h

The budget table has been updated 11.10.2016. Part of Ari's contribution was made into an in-kind contribution, since he can work on those tasks as part of his day job.

Additional support

We request to be eligible for travel funding, if suitable events emerge for presenting the tool.

We will ask for permission to host the tool in Wikimedia Labs. While we do not expect to require specific help, we hope to make note of the possibility.

Community engagement

A small group of early adopters is our initial community. We will develop and enhance the functionality of the tool in collaboration with them. They do test uploads and we review their experiences together, both through issue reporting in GitHub and in joint video sessions.

Interested volunteers may take one of the following roles:

• Developers can review and create custom nodes.
• GLAM partners can learn to use the tool for actual uploads. If they do not feel capable, they can team up with a Wikimedian, or a technical member of their staff.
• Wikimedians working with GLAM imports can join the test users.

Sustainability

The application is an extensible, open source web-application. The source code is maintained in an open repository. Clear documentation of the code aids contributors to step up.

The tool is distributed on several platforms: as an online tool and a more powerful desktop version. This increases the ease of use and widens the user base.

The modifiable nodes can be altered or created by the users. With the help of the documentation contributions from the users will increase.

The tool takes advantage and reuses existing third party components.

Measures of success

Successful contributions made with GLAMpipe will form the basis of metrics for the project. Another indicator is the growth in the number of the users of the tool.

To measure the success in responding to the community needs, we will create an online survey. The key indicators are the ease of use, efficiency and comparison to alternatives.

Number of edits made with GLAMpipe	A growing trend, numbers to be negotiated
Change in the number of users	200%
Ease of use	Over 60% positive
Efficiency	Over 60% positive
Comparison to alternatives	Over 60% positive

Get involved

Participants

Kimmo Virtanen

Kimmo has administered the Finnish Wikipedia for several years. He develops solutions to integrate Wikidata in Wikipedia while taking care of the everyday technological framework of Wikipedia in Finland. While Kimmo is a Wikimedia Finland board member currently, the term is changing at the end of the year.

Ari Häyrinen

Ari worked for Wikimedia Finland as a technical specialist in 2014–15, and he has continued to conclude the work in the GLAMpipe project. His dissertation from the University of Jyväskylä summed up the use of open source tools and open data in the cultural sector. He works with GLAM metadata in his day job.

Susanna Ånäs

Susanna has been an active member of the GLAM-Wiki community for several years, pushing forward tools development for GLAM uploads and advocating the use of Wikidata. She promotes Wikimedia for historical mapping and as a component of Digital Humanities research. She administered Wikimedia Finland in 2014–15.

Community notification

• GLAM partnerships list cultural-partners@wikimedia.ch
• GLAM-Wiki Global Facebook group
• Wikimedia Commons Village pump
• GLAM + Wikidata Facebook group
• AvoinGLAM Facebook group (Finnish)
• Wikidata - Avoin kulttuuridata Facebook group(Finnish)
• Wikidata list wikidata@lists.wikimedia.org
• Wikidata project chat

Endorsements

Do you think this project should be selected for a Project Grant? Please add your name and rationale for endorsing this project below! (Other constructive feedback is welcome on the discussion page).

1. This is a cool idea! Indeed we have also been discussing the need for such tool. -Vojtěch Dostál (talk) 20:27, 12 October 2016 (UTC)
2. Great idea! It would be very useful for us at WMAR :) Mauricio V. Genta (talk) 20:32, 12 October 2016 (UTC)
3. This sounds like a fantastic development, and something i would definitely use to share National Library of Wales data. Jason.nlw (talk) 20:53, 12 October 2016 (UTC)
4. Sounds like a useful tool. - PKM (talk) 21:19, 12 October 2016 (UTC)
5. Really interesting and useful tool. --Marcok (talk) 21:44, 12 October 2016 (UTC)
6. The tool sounds great. Having a tool that easy to use then writing your own bot is great. ChristianKl (talk) 22:28, 12 October 2016 (UTC)
7. Well-designed tools to support content contribution are sorely needed. This proposal looks detailed and actionable. Jtmorgan (talk) 22:29, 12 October 2016 (UTC)
8. Cool! It sounds to be very useful. Uomovariabile (talk) 08:12, 13 October 2016 (UTC)
9. Looks like it could be really useful for the GLAM community! --Magnus Manske (talk) 09:14, 13 October 2016 (UTC)
10. This sounds like a really great and relevant tool 85jesse (talk) 09:35, 13 October 2016 (UTC)
11. Might become a very useful tool. Wesalius (talk) 11:27, 13 October 2016 (UTC)
12. Mass upload of content from external organisations is either very time consuming, requires a huge amount of technical knowledge or both. This project has the potential for many more people to be able import content from external organisations and websites avaiable under a suitable license.
    Additionally if this tool is able to make importing data from external sources into Wikidata easier this would be an enormous help. Currently the process for importing data using automated tools is incredibly time consuming, requires a lot of knowledge and trial and error.
    In short the potential is huge John Cummings (talk) 19:29, 16 October 2016 (UTC)
13. A very reasonable amount of funding for a project that has also received financial support from outside our movement, which is great. See my input on the talk page. Endorse per John Cummings. Spinster (talk) 09:40, 18 October 2016 (UTC)
14. Slowking4 (talk) 02:48, 20 October 2016 (UTC)