Download dataset with modified or uploaded file links from Commons, with their file names, M-ids, and metadata
Yes
Yes
Yes
01 🌟 Project creation
Provide OpenRefine with one or multiple Wikimedia Commons categories; OpenRefine then loads the (reconciled) file paths of all the files in these categories
No
Yes
Yes
10 🔺 Upload files
Upload new files to Wikimedia Commons
No
Yes
Yes
We may also decide to implement file upload in OpenRefine itself. It will certainly be possible as part of the Commons extension.
04 ➡️ Reconciliation / data extension
Retrieve Wikitext, split and cleaned per parameter and template, from existing Commons files which are reconciled in OpenRefine
No
Possibly?
Yes
07 👀 Check/preview/test upload
See an example preview of the structured data that will be (batch) added to newly uploaded Commons files
No
Possibly?
Yes
07 👀 Check/preview/test upload
See an example preview of the Wikitext (generated) infobox of files uploaded in batch
No
Possibly?
Yes
02 🚿 Data cleaning
Directly see thumbnails of media files while cleaning and editing their metadata
No
No
Yes
OpenRefine is very data-centric and does not natively support (direct) preview of thumbnails of files in its data operation screen. In a next version of the Commons extension we can, for instance, introduce a media-centric editing and viewing screen that allows end users to batch edit metadata of media files in a more visually oriented way.
05 📌 Metadata mapping
Map the user's dataset with a preset template / checklist of fields from Wikimedia Commons (e.g. Artwork, Book, Map...)
No
No
Yes
01 🌟 Project creation
IIIF support (retrieve and process images/files hosted on a IIIF service)
No
No
Possibly?
We're aware of this potential use case; it can be developed after deployment of core functionalities and after more research.
09 📚 Batch Wikitext edit
Edit Wikitext of existing Commons files
No
No
Possibly?
Wikimedia Commons editing and upload functionalities in OpenRefine focus on Structured Data first and foremost.
10 🔺 Upload files
Upload new files to an arbitrary Wikibase
No
No
Possibly?
We're aware of this potential use case; it can be developed after deployment of core functionalities and after more research. Members of the Wikibase stakeholder group have already indicated interest.
Please prepare a brief project update each month, in a format of your choice, to share progress and learnings with the community along the way. Submit the link below as you complete each update.
Started the hiring process for our two developers:
Two job openings were posted on OpenRefine's blog (1) (2). They were announced and promoted in the Wikiverse and on various international open source related job boards and in the Outreachy community.
We received a considerable number of applications for both positions, with people from 14 countries and 4 continents applying. (Most applicants discovered our vacancies either via the Outreachy or Wikimedia-tech networks.) We held interviews to get to know the most promising candidates.
Next steps on the hiring process for our two software developers:
We selected the two top candidates - one developer for Wikimedia-specific features and one developer for OpenRefine-specific features.
Contracts for both candidates are being finalized this month. As soon as the candidates signed their contracts, we'll announce their names :-)
Preparations on community engagement, testing and piloting of the features we'll develop
Sandra has met with Fiona and Giovanna of the Wikimedia Foundation's GLAM and Culture team, which advises this project. Together, we are assembling a longlist of active Wikimedians and GLAM staff who will be actively approached with the request whether and how they want to provide feedback and run tests during our development process.
Sandra is writing down a few example workflows in detail this month (e.g. for adding structured data to artwork-related files on Wikimedia Commons).
Product management preparations
We are preparing various platforms and structures for transparent management of this project: a GitHub project and two Phabricator projects/workboards (one about OpenRefine, one about reconciliation). We included the links to these on this page.
Our weekly development team meetings will take off in September. We will keep public notes here.
Eugene Egbe started working as a junior developer (contractor) for Wikimedia-specific features in this project. Eugene is experienced in Structured Data on Commons development, as he was also the developer behind the popular ISA Tool. For OpenRefine, Eugene develops the Wikimedia Commons reconciliation service and a batch upload tool.
Antonin has ported the EditGroups tool (which is already quite popular on Wikidata) to Wikimedia Commons: https://editgroups-commons.toolforge.org/. This makes it possible for Commons contributors to undo certain batch edits on Wikimedia Commons, including future 'faulty' batch edits by OpenRefine.
Community outreach:
There is now a landing page for OpenRefine on Wikimedia Commons: https://commons.wikimedia.org/wiki/Commons:OpenRefine. For now, this page will point to information about the development process. As features are deployed, the page will point to general information and documentation.
We are preparing work on upload functionalities in OpenRefine and are further researching the viability and necessary functionalities of a new, external (and generic) upload tool (which we included as optional/possible in our grant application). This external tool, if we decide to develop it, should be able to serve as a (more advanced) substitute to QuickStatements. From November 2021, we plan to do some design research (including user interviews) to explore such UX and functionalities in more depth. Lozana Rossenova will work on this.
Antonin has requested the creation of an OpenRefine 3.6 tag on Wikimedia Commons, which will be used to indicate any edits on Wikimedia Commons that will be done through OpenRefine. In this way, future edits done through the tool can be counted and measured.
We are preparing for development of an external batch upload tool, which will be more powerful and flexible than QuickStatements and which will allow OpenRefine users to edit and upload large batches of media files and metadata without the need to keep OpenRefine running on their computers. We are running a request for feedback for a representation format for Wikibase edits which will be processed by this new tool.
This month, Lozana Rossenova has started design research for file upload via OpenRefine, by doing a series of user interviews with prospective users of OpenRefine's batch upload functionalities. She has interviewed various GLAM staff, Wikimedia community members and Wikimedia chapter staff, to assess their basic needs related to batch uploading. Based on insights from these interviews, in December Lozana will design workflows and wireframes for batch uploading through OpenRefine.
We resumed development on the Wikimedia Commons reconciliation service this month. Eugene has worked on code cleanup, on improvements to the data extension service, and on support for more diverse formats of Commons file names. The data extension functionality in OpenRefine randomly produces error messages, possibly due to Toolforge instabilities, which we are investigating further.
Lozana analyzed and presented her design research to our team, which provided us with valuable insights on what the user's workflow will look like and how the needed features can be integrated step by step into OpenRefine.
Thanks to the increased clarity introduced via this design research, our team has taken some major architectural decisions:
We will deploy various Wikimedia Commons-specific features for OpenRefine (such as support for loading a dataset from Wikimedia Commons categories; parse infobox Wikitext; see Commons-specific previews of files before edit or upload) in a separate new piece of software: a Wikimedia Commons / media file extension for OpenRefine. This decision will make it possible for us to create dedicated user interfaces for such workflow steps. We want to build this extension in such a way that, in a future iteration, it will also be able to support batch file editing and batch file upload for arbitrary Wikibases. In this table (also included on the top of this page) you can see an overview of features which will be enabled by OpenRefine itself and/or by the Wikimedia Commons extension.
In our grant application, we (optionally) mentioned the possible development of an external, generic batch upload tool for Wikimedia Commons files, which would provide an alternative to QuickStatements. As a major benefit, such a tool will make it possible for end users to perform very large batch edits and uploads ‘in the background / in the cloud’ rather than via OpenRefine directly (for which they would need to keep their computer and OpenRefine running for very long periods of time). However, inclusion of very large batches of to be uploaded files (potentially several gigabytes at once, with varying file sizes and formats) offers a new challenge for such a tool. Therefore, we have decided to currently dedicate all our available development resources to deploy batch upload properly in OpenRefine itself. As we have learned from this process, we will decide upon the creation of a generic, separate batch upload tool after OpenRefine’s own file upload functionalities have been properly tested.
After conversations with the Wikimedia Foundation's GLAM team, we have decided to ask for a project extension with some additional funding, for which the rationale can be found in this subpage of our grant.
We started writing our midpoint report (which is due on January 14).
And our team took a short break to recharge and celebrate the end of the year. We wish everyone a happy and inspirational 2022!
We are organizing a community meetup on February 22, in which we will showcase how to batch edit Wikimedia Commons files using OpenRefine, and we'll talk about next steps.
We held a community meetup on February 22, 2022, which was attended by more than 40 people. Recording, slides and Etherpad with links and notes are available. We presented our work done so far, demonstrated SDC editing with OpenRefine, and talked about future development and the project extension that has been awarded for this grant.
Wireframes for the Wikimedia Commons upload workflow in OpenRefine (version 2, currently - March 2022 - under feedback).
UX design: Lozana has gathered feedback from our original interviewees on version 2 of our design wireframes, and is processing this feedback. (Interviewees were a mix of GLAM and volunteer contributors to Wikimedia Commons.)
We also prepared a short user survey for April 2022, asking Commons uploaders (1) how they would like to start batch editing and upload projects in OpenRefine and (2) how they want information templates (showing e.g. artwork, book or map focused metadata) to work on Wikimedia Commons.
As our February 22 community meetup was so well attended, we have started hosting monthly office hours. The first one took place on March 22. Next ones are listed on our info page on Wikimedia Commons.
On the development side, we continued working on support for starting an OpenRefine project by entering one or more Wikimedia Commons category/ies (GitHub issue #3), did bug fixes for GREL functionalities for parsing Wikitext (Pull Request #18), and continued on backend work inside Wikidata Toolkit and OpenRefine's code to make uploading files possible.
Analysis of user survey held in March-April 2022On the design side, Lozana processed feedback from various end users into a newer version of our upload-focused wireframes. She also organized, and summarized, a user survey (32 respondents) that asked more in depth about users' expectations on how they would start editing and uploading files via OpenRefine, and how they would like to deal with typical templates for media files (e.g. for photographs, artworks, books...). OpenRefine now also has a brand new project focused on UI/UX design, which includes (but is not limited to) design tasks focused on SDC features: https://github.com/orgs/OpenRefine/projects/1/views/1
The OpenRefine/SDC team spent three full days (April 25-27) together in person in Ghent, Belgium, for an intensive work sprint. We discussed the upcoming planning, design and technical tasks for this project in depth and made decisions around our upcoming timeline and priorities until end October. As a result, you can follow progress on the project on a new workboard on GitHub, which also contains a lot of detailed new tasks: https://github.com/orgs/OpenRefine/projects/2
During our work sprint in Ghent, we also held a workshop for Flemish GLAM staff on April 26, hosted by meemoo, Flemish Institute for Archives (see this tweet for a few photos). More than 20 people attended this session. After a two-hour introduction to OpenRefine, we demonstrated OpenRefine's new SDC editing features; Lozana presented designs for the upcoming upload functionalities. The audience was enthusiastic, and we received great feedback, including the request to look more deeply into IIIF support (GitHub issue). One participants suggested to generalize OpenRefine's upcoming media/image functionalities (including thumbnail views) to work with any image / media file database or DAMS (not only Wikimedia Commons), which is a request that we certainly would like to investigate in more depth.
The first file uploaded to Wikimedia Commons with OpenRefine: Johann Adam Klein - Cossacks Eat a Meal in the Field (1819), Thorvaldsens Museum E721,6We continued work on the Wikimedia Commons extension, more specifically on the feature to allow Wikimedians to load files from one or more Wikimedia Commons categories. Basic functionalities are now in place, and we are continuing work to implement a suggestion or auto-completion service
Video demo of Wikimedia Commons (structured data) editing with OpenRefine during Wikidata Lab XXXIV, 9 June 2022 (approx 1 hour 20 minutes).We released OpenRefine 3.6-beta2, which supports batch editing structured data of Wikimedia Commons files. We continue working on file upload functionalities in the upcoming version 3.7.
Work on the Wikimedia Commons extension has continued, finalizing a suggestion service when typing Wikimedia Commons categories to start an OpenRefine project. We also continued work on selecting multiple Commons categories and on parsing of the retrieved files when the new OpenRefine project is created.
As basic upload functionalities (although still rough) are available, we launched a call for test uploads, and immediately received several enthusiastic responses.
Lozana also organized and held five structured user tests in interview form, to help us discover the major UX hurdles in the current (still very rough) upload workflow, and to prioritize the upcoming work for file uploads in July-October.
Batch SDC editing in OpenRefine was demo'ed during Wikidata Lab XXXIV, organized by Wiki Movimento Brazil.
Summary of user tests in June 2022In OpenRefine's Wikimedia Commons extension, we implemented a feature that will display the fetched files' M-ids, and a column with each file's related categories.
Lozana published a report of last month's user tests, and created various follow up issues on GitHub. As a team, we also discussed prioritization of the last work before the end of this grant. We will focus on the following work, which we see as most impactful within the current grant period:
Finishing the Wikimedia Commons extension with its support for starting projects from Commons categories
Improving schema building, implementing the notion of schema templates - making sure that OpenRefine users can work with pre-defined data models (such as artwork, book, art photo) and build their own for sharing with others
Including thumbnail previews of files in OpenRefine's grid (project) view
OpenRefine 3.6.0 was released: the first official release which natively supports batch editing Wikimedia Commons files!
We started work on schema templates (mentioned above).
Sandra followed up with several test uploaders, and held info sessions with a group of American art librarians (ARLIS; focused on data modeling in SDC) and with New Zealand Wikimedians and GLAM staff (demo of file uploading in OpenRefine).
We continued development on OpenRefine's Wikimedia Commons extension, specifically on the feature where the user can retrieve M-ids and categories for each file upon loading a new project. We also performed code cleanup.
We also held an OpenRefine training in two parts for the Wiki Mentor Africa program. 100 people participated here. The sessions are available on YouTube (1) (2)
The OpenRefine team is preparing for the last sprint of work before this grant ends. During a small team meeting in Berlin, we discussed prioritization for the last two months.
We made great strides in including first support for an often-requested feature: schema templates, or the ability to select 'templates' for popular data models, such as digitized artworks, photos of 3D artworks, and books.
In OpenRefine's Wikimedia Commons extension, we continued work on code restructuring, retrieval of categories of requested files, and started work on 'category depth' fetching. We have also started working on some last user interface improvements there.
Lozana has started conducting a final round of user tests, with a focus on testing our new features for schema templates, and general functionality of the Commons extension.
The work on this project is finished! But I would like to ask for a two-week extension for submitting our final report. We are still in the process of collecting all expenses. Because it is the busy end of the year, OpenRefine's fiscal sponsor hasn't been able to collect a full financial report yet.
Feel free to contact me with any questions you may have! With kind regards, SFauconnier (talk) 17:55, 23 November 2022 (UTC)
Hello JTud (WMF), I wanted to ask an additional question. Two final invoices for this project have been submitted in November 2022 (as we have been working until the very end of October 2022 and a few invoices have arrived afterwards). I silently assumed that our 15-day reporting extension would allow this, but I now realize that that assumption is premature. Is it OK for us to include these invoices still, and hence submit a financial report that runs until November 30, 2022? Many thanks for your consideration, and with kind regards! SFauconnier (talk) 16:39, 12 December 2022 (UTC)
Hi SFauconnier, yes, it is fine to include the final invoices in your upcoming report. I have also amended your grant's end date to allow for the bills received/paid later than original October grant end date. Thanks for checking. -- JTud (WMF), Grants Administrator (talk) 17:34, 12 December 2022 (UTC)
This project is funded by a Project Grant
Yes
No
Possibly?
,_Thorvaldsens_Museum_E721,6.jpg)
