Wikimedia Meta-Wiki

WMDE Technical Wishes/Move files to Commons

< WMDE Technical Wishes

Correctly move files to Wikimedia Commons from other wikis

edit
Tracked in Phabricator:
Task T140462
 

The FileImporter and FileExporter implement a new functionality that allows moving files with all their original data intact, while documenting the move in the version history.


Background

Background

The wish

Being able to move files from Wikipedia to Commons including the version history and information about who moved the file was a wish from the 2013 survey of the German-speaking communities:

Files in Wikimedia Commons can be used by all wikis. That’s why files from local wikis should be transferred to the central repository if their license allows it. But the current transfer process has limits: The version history page isn’t kept on Commons, e.g. it doesn’t show who originally uploaded the file. Further, former revisions of the file have to be added separately to the file page.

Situation before

There were several tools that helped users move files from local wikis to Commons. But keeping all file information intact could not be done by tools, as this has to be programmed in the backend. This concerned the history of the file and of the file page, including information about who originally uploaded the file and who moved the file to Commons. As a workaround, people added this information manually, e.g. with the {{Original Upload Log}}.

How it works

edit

FileExporter is now a default feature on all wikis.


Custom categories can be calculated from the source URL or timestamp. If a custom message is provided for MediaWiki:fileimporter-post-import-revision-annotation, it will be included in the imported file info, with message parameter $1 set to the full URL of the source file, and $2 set to an ISO 8601 timestamp for when the import began. For example, the message "{{#invoke:SourceCategory|urlToDomainCategory|url=$1}}{{#invoke:MonthCategory|main|date=$2}}" might be substituted into the file info as "{{#invoke:SourceCategory|urlToDomainCategory|url=http://dev.wiki/wiki/File:Mittens.jpeg}}{{#invoke:MonthCategory|main|date=2020-07-06T11:21:49+00:00}}".

Workflow

edit

You need to be logged in in order to use the feature.

  •  
    1: On a local file page, click on the "Export to Wikimedia Commons" link.
  •  
    2: The FileImporter checks if the file can be imported and if replacements need to be made during the import. This step is based on the wiki's configuration file. Read more.
  •  
    3: A preview page on Commons shows you what will be imported. You can edit the file title and page information, and ask the FileImporter to clean up the source wiki in your name, by adding a template like NowCommons to the source file, or deleting the source file if you have admin rights. Clicking on “Import File” copies the file to Commons.
  •  
    4: The file is imported, including the name of the original uploader and name of the importer, the previous history of the file and the file page, and any changes that were made during import.

Limitations

edit

The focus of this functionality is on moving local files to Commons correctly in a technical way. What the functionality doesn’t do is:

  • automatically move files to Commons
  • move several files to Commons at the same time
  • match categories on local wikis with Commons categories
  • replace the necessary legal check regarding correct licenses etc.

User rights

edit
Action required right on Commons required right in the source wiki
Importing files right to upload files (logged-in user) auto-confirmed user
Reverting file imports on Commons admin
(because this action is equivalent to file deletion) 		/
Deleting files from the source wiki / admin rights

Without admin rights, you can add a template like {{nowCommons}}) to the local file during the import so that the wiki can do whichever cleanup process they decided on.

Technical implementation

edit
  • The functionality contains two MediaWiki extensions:
    • The FileExporter provides a link to Commons on the local wiki and hands over an URL to the extension FileImporter.
    • The FileImporter is a Commons extension which imports the file including all data when all prerequisites are met.
  • Technically, the files aren't "moved", but "copied".
  • JavaScript: The feature works without JavaScript. JavaScript elements might be added in a later version so that file name and file info can be changed on the same page.

Development

edit
  • To understand the requirements and the scope of the wish, several conversations with users took place.
  • The team presented the planned workflow for this feature here on the project page in March 2017. First mocks were presented in July 2017.
  • In March 2018, Commons users were asked if they consider it necessary to keep the Original Upload Log when using the new feature.
  • Since April 2018, users can test an alpha version of the feature.
  • In June 2018, a beta version of the feature was deployed to first wikis in order to test it in a smaller group first before making it a beta feature everywhere. This included, as always, one left-to-right wiki. With this small beta approach, we could take care of issues before rolling it out as a beta on all wikis. For example, we've added some features and fixed some bugs during the small beta phase.
  • In January 2019, a beta version of the feature was released to all remaining wikis, inviting everyone test and share their thoughts. This full beta feature already gave a good idea of what the default feature would look like in the future, even though development wasn't fully finished yet. The purpose of the beta phase is to find out how this feature works for anyone who might want to use it in the future.
  • Since September 2019, the FileExporter is a default feature on a few first wikis. Deployment on more wikis is planned for later this year.

Deployment roadmap

edit
FileExporter FileImporter
2018-06-12

Files from mediawiki.org and test2.wikipedia.org can be exported to test.wikipedia.org.

beta feature on mediawiki.org and test2.wikipedia.org

For this deployment, the link on the file page is labelled "Export to Test Wiki". To test the export on these wikis, you need to be an auto-confirmed user.

test.wikipedia.org
2018-06-25 beta feature on German, Farsi and Arabic Wikipedia (small beta deployment) Wikimedia Commons
2018-07-11 beta feature on sourceswiki (Wikisource for multilingual books)
2018-09-17 beta feature on Korean Wikipedia
2018-10-08 beta feature on Marathi Wikipedia
2019-01-16 beta feature on other wikis (full beta deployment) What's new in this release – some examples:
  • Log entries and tags are added to file imports. (read more)
  • The matching and replacement of templates was extended. (e.g. T194505, T198317, T198711)
  • If no config file exists, files can't be moved. (T199108)
  • If a user's file is moved to Commons, their Commons account is auto-created (T209346)
  • Bug fixes and smaller changes were made (e.g. T198321, T198681, T194642)
April - July 2019 several improvements. The feature now
  • shows on the import page which categories will be used
  • can match different spellings for the same parameter (T213955)
  • use language typing on template parameters if requested in the configuration (T198607)
  • has interwiki links working with all Wikimedia projects

and more

September 24, 2019 default feature on first wikis:
  • German Wikipedia
  • Korean Wikipedia
  • Marathi Wikipedia
  • Farsi Wikipedia
  • sourceswiki (Wikisource for multilingual books)
 Some examples of what’s new:
  • It is easier to add a template like NowCommons to the local file during import.
  • The local file can be deleted during import (requires admin rights in the source wiki).
  • All imported revisions are now tagged with “Imported with FileImporter”, and the revision that is created during import now has the tag with “Modified with FileImporter”. That makes it possible to detect incomplete imports, as well as build a comprehensive list of all edits made by FileImporter on the target wiki.

plus some bugfixes.

August 5, 2020 deployment of the FileExporter as a default feature on all remaining wikis 'Now Commons' template can be added to the file on the source Wiki by clicking the corresponding check box on the import page.
edit

FAQ

edit

Are all revisions imported, or only the latest revision of a file?

edit

All revisions are imported. (example)

Files with hidden file versions can't be imported, see #Can I export deleted files?.

Can I export deleted files?

edit

The FileExporter can't import

It can import files with other hidden parts. In this case, the following hidden parts are replaced:

It can import partially undeleted files. In this case, the deleted revisions/file versions (not marked as hidden nor otherwise listed in page/file history) are not imported.

How can I find files that were imported with FileExporter?

edit
  • In the revision history of a file, the import is shown as a null revision (i.e. an entry in the revision list that doesn’t actually change the page). If changes were made during import, an additional text revision is created.
  • The revision that is created during import is tagged with Modified with FileImporter. This way it can be filtered on the Recent Changes page.
  • For the import, an upload log entry and an import log entry are created and associated with the null revision and with the importing user, and timestamped with the import date. Having both the null revision and the upload log entry is the default for uploads, although it leads to "doubled" entries on the history page, see this example file and its log entries.
  • For (old) imported file revisions, upload log entries are created with the date and user of the original upload to make all file revisions appear in Commons' upload log. These revisions are also tagged with Imported with FileImporter. (an example file from the beta cluster)

This is true for files that were exported after January 16, 2019. Files that have been imported with FileExporter before this date don't have the tag or the log entries.

What if I want to import a file to Commons that has been imported before (without the history)?

edit

The decision if you import an already imported file again or not is up to you. The advantage: You don't lose the file and version history and have more transparency about who uploaded the file or edited the file description page. How to do this:

  • 1. If necessary undelete the local file in the wiki
  • 2. Delete the already imported file on Wikimedia Commons
  • 3. Import the file with FileExporter.
  • 4. Delete the local version.

Attention: Perform steps 2 and 3 in a timeframe of 10 minutes to avoid that the User:CommonsDelinker starts it work and removes the file usages from the wiki pages.

If the file page of the old imported file on Commons has been changed in the meantime, preserving both version histories can be tricky. You can try this:

  • 5. Undelete the file that you have deleted in step 1, to merge the version histories.
  • 6. Restore the file description on Commons and refine it.

This requires some manual work (example). It's up to you if you want to do this or not. There's a request on Phabricator for a follow-up project to build a tool that makes such imports easier.

There's a license missing in my wiki's configuration file. What do I do?

edit

Each wiki is responsible for their own configuration file. If you think your wiki's file is incomplete or wrong, it's recommended that you discuss it within your community and then change the configuration page.

What do I have to do after the import?

edit

This depends on the processes in your local wiki. One possibility is to make sure the old file name is not used any more, and then delete it. FileImporter supports this cleanup process by adding templates like NowCommons to the source file in your name, if you want it, and by allowing you to delete the source file during import, if you're an admin on the source wiki.

I'm an admin in my local wiki and I'd like to delete the original file directly from Commons. Can I do this?

edit

Yes, that’s possible. As an admin you can choose during the import if you want to have the file automatically removed in your name after the import or not.

I want the feature to be in my wiki's language, for example the name and description on Special:Preferences. How do I do that?

edit

Texts both for FileExporter and FileImporter can be translated. Any translation is very much appreciated!

Can I use the FileExporter to export files to other Wikimedia wikis than Commons?

edit

No, this is currently not possible. Requests on Phabricator:

Can files be moved back to the local wiki?

edit

There's a manual way to do this:

  • 1. Delete the file on Commons. This requires admin rights on Commons.
  • 2. Restore the original file on the local wiki. This requires admin rights on the local wiki.

There's a request on Phabricator to make this a built-in feature.

My source template has more parameters than the template on Commons. How can I transfer their contents?

edit

Example: The Information template on English Wikipedia has the parameters "Location" and "Additional_information" that the Information template on Commons doesn’t have.

Additional parameters are imported to the new file description page on Commons, but they are only shown in wikitext, not on the rendered page. You can manually put this wikitext into the right position in the Commons template.

There’s a request on Phabricator for a built-in feature which merges additional parameters from a source wiki into a Commons parameter.

My wiki allows different spellings for the same parameter. How can I make sure they both get replaced by the same Commons parameter?

edit

FileImporter can now match different spelling variants of a parameter if they have been defined in the configuration file.

Example: The Russian Information template allows the same parameter to be spelled |Описание= or |описание= (see configuration file). FileImporter matches both to |Description=.

How can I add categories during import?

edit

Categories on local wikis aren’t the same as on Commons. FileImporter does not match categories during import, so you have to manually change category names from your local wiki into the corresponding Commons categories.

The FileImporter does show which categories will be imported. Missing categories are shown as red links, giving an indication which categories need to be changed. To change them, you have to

  • click on “edit file info”,
  • change the categories in the wikitext.

Feedback

edit

Feedback is always welcome and needed to improve the feature! Please use the central feedback page on MediaWiki.org, in case you have any questions, comments, advice ... Thanks :-)

Retrieved from "https://meta.wikimedia.org/w/index.php?title=WMDE_Technical_Wishes/Move_files_to_Commons&oldid=21793373"