Grants:IEG/Wikimaps Warper 2.0/Final

This project is funded by an Individual Engagement Grant

Welcome to this project's final report! This report shares the outcomes, impact and learnings from the Individual Engagement Grantee's 6-month project.

Part 1: The Project

Summary

In a few short sentences, give the main highlights of what happened with your project. Please include a few key outcomes or learnings from your project in bullet points, for readers who may not make it all the way through your report.

• Development of the API (Tim Waters)
• GLAM upload and metadata refinement tool GLAMpipe, additions for the maps workflow (Ari Häyrinen)
• Proposal for Wikimedia Commons integration (Albin Larsson)
• Wikimania workshop (Susanna Ånäs and Albin Larsson)

Methods and activities

What did you do in project?

Please list and describe the activities you've undertaken during this grant. Since you already told us about the setup and first 3 months of activities in your midpoint report, feel free to link back to those sections to give your readers the background, rather than repeating yourself here, and mostly focus on what's happened since your midpoint report in this section.

In the latter part of the project most of the time went into development on one hand, and into advocacy and learning on the other.

See the midpoint report for the project setup.

The Wikimaps Warper & API

Tim Waters worked on the API and the Warper adding new features. Tim kept an extensive on wiki development log listing the main items of work undertaken, and used GitHub Issues to track tasks within the IEG milestone.

Warper Enhancements

Apart from the API, some enhancements were added to the Wikimaps Warper. See the development log for more details. Some highlights include:

• Making the warper work with a renewed Map template in Wikimedia Commons.
• Imports - users will be notified if maps within a category haven't got a Map template.
• Custom Basemaps - users can add their own custom basemaps to aid in warping the maps.
• Caching map and mosaic WMS and Tile requests to make the service much more resilient and faster.

New Warper API

The main body of work by Tim Waters was on the development of a new API for use by third party external applications. See the development log for more details. Generally speaking existing functionality was rolled into a new service and code. The main highlights of this work are:

• New API Documentation http://warper.wmflabs.org/api/v1/
• OAuth (Wikimedia Commons) auth support
• New token based authentication
• JSON-API format support
• API controller tests
• Combined geosearch and index calls so that text and geospatial queries can work as one call
• Ensured API worked with HTTPS

API Integration

Tim Waters wrote a proof of concept 3rd part JS app to connect to the Warper API via OAuth and do basic authenticated and non authenticated calls. See warper_oauth_token_auth_demo github Code

Albin Larsson worked on API examples and demos not only to provide the developer community with working examples and boilerplates, but also to detect features missing from within the API, inconsistency, bugs etc.

The demo/testing process emerged into two standalone repositories:

Maps UserScript Boilerplate (code)

The boilerplate was created for writing Commons UserScripts / gadgets, that take advantage of the Warper API. The boilerplate provides developers with a starting point when they want to work with Warper data. They can directly start working with the georeferenced data without writing any calls to the API. It also handles MediaWiki OAuth, which can often seem complicated to implement.

This boilerplate contains code to:

• Always run your code on pages with the Map template within the File namespace.
• Retrieve existing map data from the Warper.
• Authenticate a user using MediaWiki OAuth.
• Import a map to the Warper.
• Validate user tokens.

Warped Maps to IFrames (code, tool labs)

This tool was proposed by an attendee at the Wikimania workshop. It allows people to easily embed Warped maps externally through an iframe element. This makes it possible for nontechnical users to embed georeferenced content in blogs or other websites.

GLAMpipe

Ari Häyrinen produced the first public version of the GLAMpipe tool, a tool for working with media metadata and uploads. In the Wikimaps Warper 2.0 project he created the structure that allows writing interactive nodes to GLAMpipe. Nodes can import data, transform it, download files, upload files and data and display data in various ways. He created a draft version of a node for georeferencing, where one can go through images in a dataset and add control points using a reference map.

In addition, one can now fetch data from any MapWarper instance, manipulate data, convert MapWarper control points and upload files to Wikimedia Commons. The GLAMpipe tool is in an early alpha state, and the development will continue after gathering the first impressions from the GLAM community.

Community and learning

Wikimaps workflow was experimented with in a project in the Digital Humanities Hackathon arranged in Helsinki. The Historical Street View hack used old maps and aerial images to create a historical map of an area that had since been completely rebuilt. The resulting map was saved in OpenHistoricalMap and used to geolocate the old images, which were then sent to Mapillary for merging into a historical street view. See the blog post.

Susanna Ånäs and Albin Larsson organized a workshop on historical maps in Wikimania and participated in the hackathon. The workshop was given a prominent time on Saturday afternoon, and there were 18 listed participants in addition to the 3 presenters. Vahur Puik, an advisor in the project, also took part in arranging the workshop. Here are the workshop slides.

In the Second Swiss Open Cultural Data Hackathon Wikimaps project was introduced in a workshop. In an additional playful hack, the plans were to bring to life the historical map that was created as part of the Historical Street View project. The map would be imported to Cities: Skylines, a commercial moddable city simulation game. Within the scope of the hackathon, we were able to produce a series of images of unexpected erratic road constructions. As soon as there is a possibility, we will try it again, seriously. Feel free to join! See the images on Twitter.

Outcomes and impact

Outcomes

What are the results of your project?

Please discuss the outcomes of your experiments or pilot, telling us what you created or changed (organized, built, grew, etc) as a result of your project.

The Warper has been made accessible through the API

It is now possible for any developer to use warping in their projects. The API also makes it possible to create several different interfaces that access the data and functionalities of the warper.

Serving GLAMs with batch processes

New functionalities were built into the existing and the new Warper code. Uploading and working with batches has been enhanced, which will help GLAM organizations to contribute material to Wikimedia Commons. The GLAMpipe tool will also facilitate creating large donations with quality metadata.

Support in Wikimedia Commons for accessing and using Warper data

New workflows are made possible when the Warper data can be accesses from Wikimedia Commons. Users will not need to leave Commons when they work with historical maps.

Addressing new audiences, strengthening ties with existing ones

Hackathons and events are important meeting points of users, creators and re-users of historical data. Wikimaps was present in 5 events during the spring: The #Hack4FI hackathon and the DHH16 hackathon, both in Helsinki, Wikimania, The 2nd Swiss Open Cultural Data hackathon and the Digital Humanities 2016 conference. This will help achieving a global accordance of the necessity of the workflows.

Progress towards stated goals

Please use the below table to:

1. List each of your original measures of success (your targets) from your project plan.
2. List the actual outcome that was achieved.
3. Explain how your outcome compares with the original target. Did you reach your targets? Why or why not?

Planned measure of success (include numeric target, if applicable)	Actual result	Explanation
The Warper code will be refactored. User interface is separated from computation to allow more versatile use of the software. Operations inside the code are also separated to make possible developing individual parts separately. The API will be enhanced to be the primary means of operating the software.	We met our goal. Warper code was refactored and new tested code was created for a newly documented API. Additionally code was written to integrate with the new API. Now third party applications can (and have been) successfully be created and interact with the backend software.	We reached our targets. We had good planning of tasks, a clear vision of what we were wanting and documented progress as we went. We used GitHub Issues and Milestones and wrote a monthly development log. We also had a well attended project meeting every month.
New functions will be added. Mass import of already rectified maps will come handy to GLAM users and can be made part of maps uploads. Category imports will be enabled. Developing the new functionalities will start before the project.	We met our goal. The new functionalities are included in the current, revised version of the software. Importing maps were made easier. GLAM users can import whole category of map images into the Warper. We fixed several issues with Imports of maps for GLAM users. We also gave better feedback to users importing maps, and made integrating with the new Map Template on Commons better.	We reached our targets. We had good planning of tasks, a clear vision of what we were wanting and documented progress as we went. It was made easier as the features had already begun before the IEG project started.
After this stage the project will be ready for the next phase where user interfaces will be remade. We decided to do this in steps because of time constraints, and to assure that the backend is ready when the front end is made.	There needs to be an additional effort to conclude the design phase based on the outcome of this project before production. We will make an extension request for this.	The decision to break up the process to consecutive pieces is a good means to keep a non-commercial project at check. Committing to only one stage at a time ensures the prerequisites for the next phase are covered. The downside is that it makes the process much slower and increases the bureaucratic overhead. When communicating about the project new people are found and ideas can be tested.
Wikimedia Finland has an ongoing project for processing metadata for Commons uploads. Warper upload will be made an option in it.	The GLAMpipe tool has been released an an early alpha release. Several map related functionalities have been added.	While the tool is functional, we need to continue developing especially the user interface. The project will share design principles with the Wikimaps Warper tool, so design efforts made for the Wikimaps Warper will benefit the GLAMpipe project as well.
We are looking for Commons volunteers to participate in Commons integrations, coding gadgets and templates.	There are continuously less Wikimedia Commons contributors than what we would wish for.	The deployment of structured data for Commons will change the required skillset of contributors, and we need to respond to that in the next stages.
Linking with Wikidata will be a future direction.	Structured data for Wikimedia Commons has not yet been addressed.	The advent of structured data for Wikimedia Commons may change most tools dramatically, as they have earlier been handling text, and will now deal with data. We wish to address the shift in the proposed design extension.
Design decisions will be based on earlier work in the Wikimaps project, and updated to reflect development in sister software such as the OSM iD editor.	Design of the future tool has deliberately not been advanced within this project.	We propose to make it a separate phase, including user research, implementation plan, css style guides and design guidelines for further projects.

Think back to your overall project goals. Do you feel you achieved your goals? Why or why not?

The primary goal has been established: to make the Warper usable primarily through the API. All other additional goals have been reached to varying degrees.

Global Metrics

We are trying to understand the overall outcomes of the work being funded across all grantees. In addition to the measures of success for your specific program (in above section), please use the table below to let us know how your project contributed to the "Global Metrics." We know that not all projects will have results for each type of metric, so feel free to put "0" as often as necessary.

1. Next to each metric, list the actual numerical outcome achieved through this project.
2. Where necessary, explain the context behind your outcome. For example, if you were funded for a research project which resulted in 0 new images, your explanation might be "This project focused solely on participation and articles written/improved, the goal was not to collect images."

For more information and a sample, see Global Metrics.

Metric	Achieved outcome	Explanation
1. Number of active editors involved	not applicable	The project focus is in making changes to the tool backend, and regular editors cannot take advantage of it before the interface is built. This would be the next stage of the project.
2. Number of new editors	not applicable	There are new people who are interested in working with maps as well as participating in the development.
3. Number of individuals involved	8 advisors 18 workshop participants in Wikimania approx 15 workshop participants in the Second Swiss Open Cultural Data hackathon At least 2 prominent future developers
4. Number of new images/media added to Wikimedia articles/pages	pending	This phase of the project is not providing content to Wikimedia projects. The pilot cases, which have not been successfully carried out yet, would provide such input. (Pilot cases being test sets from the British Library and a set of Carelian maps from Linked Data Finland). Both partners have been contacted for continuing with the plans.
5. Number of articles added or improved on Wikimedia projects	not applicable	The workflow will only produce improved articles when the tool and resulting content have been made accessible from Wikimedia production. This will require collaboration with Wikimedia Foundation tech.
6. Absolute value of bytes added to or deleted from Wikimedia projects	not applicable	not known

Learning question

Did your work increase the motivation of contributors, and how do you know?: We still cannot display the potential of the workflow efficiently, since the new API does not have a user interface that could be used by the general public. There is continued enthusiasm in a maps workflow (maps, images, linked data, places). This was evident in events during the summer: The Wikimania workshop, the Second Swiss Open Cultural Data Hackathon and the Digital Humanities 2016 conference, where the Wikimaps project has been presented.

Indicators of impact

Do you see any indication that your project has had impact towards Wikimedia's strategic priorities? We've provided 3 options below for the strategic priorities that IEG projects are mostly likely to impact. Select one or more that you think are relevant and share any measures of success you have that point to this impact. You might also consider any other kinds of impact you had not anticipated when you planned this project.

Option B: How did you improve quality on one or more Wikimedia projects?

The overall goal of this project is to provide a workflow for bringing in and managing historical map documents and gathering data from them for further use in Wikimedia projects. This will include annotated, zoomable historical maps in Wikipedia articles and data about historical places to be stored in Wikidata. The data can not only be displayed but also used as a medium for annotating further content, drawing historical maps or computing with historical geographic data.

This phase alone does not yet provide that advantage. The interfaces need to be developed and the workflow put together, connecting to different environments.

Project resources

Please provide links to all public, online documents and other artifacts that you created during the course of this project. Examples include: meeting notes, participant lists, photos or graphics uploaded to Wikimedia Commons, template messages sent to participants, wiki pages, social media (Facebook groups, Twitter accounts), datasets, surveys, questionnaires, code repositories... If possible, include a brief summary with each link.

Project log in Commons

• Warper monthly development log Development Log

Repositories

• The Wikimaps Warper - repository for the actual Warper API
• warper_oauth_token_auth_demo - repo for Proof of Concept 3rd party authentication and API use
• Warped to iframe - repository for the Warped to iframe tool
• Maps UserScript Boilerplate - repository for the gadget/userscript boilerplate

API Documentation

• live API documentation
• GitHub link

All blog posts

• Wikimania Blog post in Finnish
• Wikimaps Blog - Warper 2.0 IEG Blog Post
• Wikimaps Blog - historical mapping apps in Berlin and Leeds
• Thinkwhere (Tim Waters)- IEG & Wikimaps announcement and plan
• Wikmaps Blog - Introducing Metapipe (GLAMpipe)

• A pending Wikimaps blog post / newsletter

Workshop in Wikimania

• 
• 
• 
• 
• 
• 
• 

• Workshop slides

Swiss Open Cultural Data Hackathon

• Event page
• Presentation (currently not linked from the event page)
• Documentation at Infoclio web pages

Project public folder

• The public folder includes meeting memos, UI sketches etc.
• Presentation includes information about the current Warper, and future ideas – what is it that this could do.

Poll

• All Our Ideas: Wikimaps Warper Here it's possible to vote which enhancement are more important

Learning

The best thing about trying something new is that you learn from it. We want to follow in your footsteps and learn along with you, and we want to know that you took enough risks in your project to have learned something really interesting! Think about what recommendations you have for others who may follow in your footsteps, and use the below sections to describe what worked and what didn’t.

What worked well

What did you try that was successful and you'd recommend others do? To help spread successful strategies so that they can be of use to others in the movement, rather than writing lots of text here, we'd like you to share your finding in the form of a link to a learning pattern.

"Break a project into steps (and hope that you will find continued support)"

We will find the right the way to formulate the new ideas into this existing learning pattern: Achievable goals

• It is good to break your long-term vision into smaller, even independent units. Working with one unit before embarking on another one ensures that there are no pending tasks that prevent you from proceeding.
• With smaller, well-defined tasks it is easier to keep the project schedule together. Creating tasks that you know you can handle also helps in keeping the amount of work reasonable.
• When you create small independent tools, it is more likely that they are dependent on communicating with other existing software. This increases their further usability and applicability.
• An intermittent project timetable forces to reflect on the work done and roll feedback into the next phases.

Notes from the project:

• We had regular project meetings with the team and with our IEG Advisor. This was useful to keep us all updated and allowed feedback to flow.

For warper:

• We wrote clear objectives and aims for the development during the grant application stage
• These objectives and aims were translated into Github Issues, or tasks before we begun.
• Issues were then added into a special IEG milestone, and some were deemed "out of scope".
• Tasks were completed and progress logged and reported during monthly meeting
• We didn't go over time and we didn't give ourselves too much work.
• During development we got feedback from users as to what they wanted first, and prioritized that first.

What didn’t work

What did you try that you learned didn't work? What would you think about doing differently in the future? Please list these as short bullet points.

• The mentor (advisor) community we gathered is a great group, and we would like to continue work with them. We tried to arrange one meeting with everyone at the same time, to be economic with spending time, but that did not work out. Having more flexible ways to meet would be needed. Meetings consume a lot of project energy, and they should be very targeted and punctual gatherings, or alternatively very casual chats every now and then in small settings. The formats can be further investigated, to come round ill-functioning conferencing systems, low participation or large heterogenous groups. If knowledge is produced in small settings, there need to be efficient ways of distributing the knowledge and building upon previous work. Conversations from direct one-to-one chats or email discussions should be shared in a good way.
• The case studies with partners were almost completely not done, but they can be conducted at later phases. Partners have been contacted and will continue work with the project.
• It was difficult to recruit mentors initially - they had their own time and commitments.

Other recommendations

If you have additional recommendations or reflections that don’t fit into the above sections, please list them here.

Next steps and opportunities

Are there opportunities for future growth of this project, or new areas you have uncovered in the course of this grant that could be fruitful for more exploration (either by yourself, or others)? What ideas or suggestions do you have for future projects based on the work you’ve completed? Please list these as short bullet points.

• Create the user interface for Wikimaps Warper, and establish a site / hub / address for it and other tools added in the future
• Connect historical maps to Wikidata, store historical geodata in Wikidata
• Create workflows to insert annotated, layered historical maps on Wikipedias
• Support the development of OpenHistoricalMap, an OpenStreetMap for historical geodata

We are proposing a grant renewal on user interface design and user research.

Think your project needs renewed funding for another 6 months?

Part 2: The Grant

Finances

Actual spending

Please copy and paste the completed table from your project finances page. Check that you’ve listed the actual expenditures compared with what was originally planned. If there are differences between the planned and actual use of funds, please use the column provided to explain them.

Expense	Approved amount	Actual funds spent	Difference
Backend	4000	4000	0
Front end consultation	1000	1000	0
Integrations consultation	1000	1000	0
Coordination, design	1000	1000	0
Administrative costs > Participation in Wikimania	500	500	0
Total	7500	7500	0

Remaining funds

Do you have any unspent funds from the grant?

Please answer yes or no. If yes, list the amount you did not use and explain why.

• No

If you have unspent funds, they must be returned to WMF. Please see the instructions for returning unspent funds and indicate here if this is still in progress, or if this is already completed.

Documentation

Did you send documentation of all expenses paid with grant funds to grantsadminwikimedia.org, according to the guidelines here?

Please answer yes or no. If no, include an explanation.

• Yes

Confirmation of project status

Did you comply with the requirements specified by WMF in the grant agreement?

Please answer yes or no.

• Yes

Is your project completed?

Please answer yes or no.

• Yes, but we apply for renewal

Grantee reflection

We’d love to hear any thoughts you have on what this project has meant to you, or how the experience of being an IEGrantee has gone overall. Is there something that surprised you, or that you particularly enjoyed, or that you’ll do differently going forward as a result of the IEG experience? Please share it here!

The project went well overall, everyone exceeded themselves! This is a start of a community of skilled and devoted people, creating building blocks for other people interested in historical geographical materials to base their future contributions onto. Thank you Tim, Albin and Ari, and thank you Marti!