Grants:Project/Jayprakash12345/Improve documentation of MediaWiki maintenance scripts/Final

This project is funded by a Project Grant

proposal

people

timeline & progress

finances

midpoint report

final report

Report accepted

This report for a Project Grant approved in FY 2020-21 has been reviewed and accepted by the Wikimedia Foundation.

• To read the approved grant submission describing the plan for this project, please visit Grants:Project/Jayprakash12345/Improve documentation of MediaWiki maintenance scripts.
• You may still review or add to the discussion about this report on its talk page.
• You are welcome to email projectgrantswikimedia.org at any time if you have questions or concerns about this report.

Welcome to this project's final report! This report shares the outcomes, impact and learnings from the grantee's project.

Part 1: The Project

Summary

This project is about to improve the documentation of MediaWiki maintenance scripts on MediaWiki.org. There are almost 200+ maintenance scripts. Out of 200+, 125 scripts needed to document, and others were required to restructure. There were 42 scripts that only have a predefined template, and 43 scripts that only had predefined template + one-line descriptions.

MediaWiki maintenance scripts play a vital role in wiki maintenance. According to WikiApiary, There are 21647 standalone wikis and it is painful for the sysadmin to read documentation from the docstring of the script file before running the maintenance script. Lack of documentation can cost to site down for some time. MediaWiki.org is the primary source to provide these kinds of basic support. According to MassViews in 2021, There were 287 views per day on maintenance script pages. So it became important to document them on MediaWiki.org.

Under this project, 180 MediaWiki maintenance scripts have been documented in a structured way with Details, Options/Arguments, Usage, and Common issues section along with Category.

Apart from this, 13 categories have been created according to the nature of the scripts, and the landing main page has been rewritten along with their new sidebar.

Project Goals

• Creating a format/structure with minimum sections like Details, Usage, Parameter/Options, and Troubleshooting.

Originally only 124 scripts mentioned on grant proposal page for documentation but 180 MediaWiki maintenance scripts have been documented in a structured way. This structured documentation consists Details, Options/Arguments, Usage, and Common issues sections (Troubleshooting section renamed to Common issues; The scripts that don't require troubleshooting, don't have this section). And every 180 script also have category in it according to their nature.

• Creating templates for maintenance script documentations

Under this project, mw:Template:Maintenance scripts and mw:Template:Force option have been created. These templates are used in the scripts pages and main landing pages. These templates help to remove repeated work.

• Expanding scripts having an only predefined template

All 42 scripts mentioned on grant proposal page has been documented.

• Expanding scripts having predefined template + one-line description

Out of 43 scripts mentioned on grant proposal page, 40 scripts has been documented. In remaining three, two scripts are include files that don't require documentation as they are scripts to support other scrips.

• Expanding scripts having predefined template + one-line description + outdated template

Out of 8 scripts mentioned on grant proposal page, 6 scripts has been documented.

• Expanding/Improving other scripts

All 31 scripts mentioned on grant proposal page has been documented.

Project Impact

Targets

Planned measure of success (include numeric target, if applicable)	Actual result	Explanation
124 scripts	180 scripts	Originally 124 maintenance scripts were planned to document in grant but we go beyond that.

Grantee documented the below scripts with the following details.

S. No.	Script	Details	Parameter/Options	Usage	See also	Category
1	generateSchemaSql.php	Y	Y	Y	Y	Y
2	makeTestEdits.php	Y	Y	Y	Y	Y
3	findMissingFiles.php	Y	Y	Y	Y	Y
4	findDeprecated.php	Y	Y	Y	Y	Y
5	invalidateUserSessions.php	Y	Y	Y	Y	Y
6	jsparse.php	Y	Y	Y	Y	Y
7	validateRegistrationFile.php	Y	Y	Y	Y	Y
8	resetPageRandom.php	Y	Y	Y	Y	Y
9	dumpCategoriesAsRdf.php	Y	Y	Y	Y	Y
10	mysql.php	Y	Y	Y	Y	Y
11	view.php	Y	Y	Y	Y	Y
12	protect.php	Y	Y	Y	Y	Y
13	mctest.php	Y	Y	Y	Y	Y
14	orphans.php	Y	Y	Y	Y	Y
15	populateInterwiki.php	Y	Y	Y	Y	Y
16	purgeModuleDeps.php	Y	Y	Y	Y	Y
17	mcc.php	Y	Y	Y	Y	Y
18	addSite.php	Y	Y	Y	Y	Y
19	categoryChangesAsRdf.php	Y	Y	Y	Y	Y
20	deleteTag.php	Y	Y	Y	Y	Y
21	exportSites.php	Y	Y	Y	Y	Y
22	importSites.php	Y	Y	Y	Y	Y
23	refreshFileHeaders.php	Y	Y	Y	Y	Y
24	renameRestrictions.php	Y	Y	Y	Y	Y
25	initUserPreference.php	Y	Y	Y	Y	Y
26	cleanupRevActorPage.php	Y	Y	Y	Y	Y
27	checkBadRedirects.php	Y	Y	Y	Y	Y
28	findOrphanedFiles.php	Y	Y	Y	Y	Y
29	migrateFileRepoLayout.php	Y	Y	Y	Y	Y
30	copyFileBackend.php	Y	Y	Y	Y	Y
31	updateCredits.php	Y	Y	Y	Y	Y
32	manageForeignResources.php	Y	Y	Y	Y	Y
33	cleanupRemovedModules.php	Y	Y	Y	Y	Y
34	populateBacklinkNamespace.php	Y	Y	Y	Y	Y
35	populateRevisionSha1.php	Y	Y	Y	Y	Y
36	runBatchedQuery.php	Y	Y	N	Y	Y
37	syncFileBackend.php	Y	Y	Y	Y	Y
38	checkImages.php	Y	Y	Y	Y	Y
39	manageJobs.php	Y	Y	Y	Y	Y
40	mergeMessageFileList.php	Y	Y	Y	Y	Y
41	cleanupBlocks.php	Y	Y	Y	Y	Y
42	addRFCandPMIDInterwiki.php	Y	Y	Y	Y	Y
43	checkDependencies.php	Y	Y	Y	Y	Y
44	deduplicateArchiveRevId.php	Y	Y	Y	Y	Y
45	deleteAutoPatrolLogs.php	Y	Y	Y	Y	Y
46	populateIpChanges.php	Y	Y	Y	Y	Y
47	minify.php	Y	Y	Y	Y	Y
48	populatePPSortKey.php	Y	Y	Y	Y	Y
49	refreshExternallinksIndex.php	Y	Y	Y	Y	Y
50	checkLess.php	Y	Y	Y	Y	Y
51	deleteSelfExternals.php	Y	Y	Y	Y	Y
52	populateArchiveRevId.php	Y	Y	Y	Y	Y
53	updateArticleCount.php	Y	Y	Y	Y	Y
54	cleanupWatchlist.php	Y	Y	Y	Y	Y
55	deleteLocalPasswords.php	Y	Y	Y	Y	Y
56	populateParentId.php	Y	Y	Y	Y	Y
57	removeUnusedAccounts.php	Y	Y	Y	Y	Y
58	purgeExpiredWatchlistItems.php	Y	Y	Y	Y	Y
59	purgeExpiredUserrights.php	Y	Y	Y	Y	Y
60	updateExtensionJsonSchema.php	Y	Y	Y	Y	Y
61	getConfiguration.php	Y	Y	Y	Y	Y
62	renameDbPrefix.php	Y	Y	Y	Y	Y
63	updateRestrictions.php	Y	Y	Y	Y	Y
64	initEditCount.php	Y	Y	Y	Y	Y
65	patchSql.php	Y	Y	Y	Y	Y
66	convertUserOptions.php	Y	Y	Y	Y	Y
67	preprocessorFuzzTest.php	Y	Y	Y	Y	Y
68	fixExtLinksProtocolRelative.php	Y	Y	Y	Y	Y
69	dumpLinks.php	Y	Y	Y	Y	Y
70	generateLocalAutoload.php	Y	Y	Y	Y	Y
71	cleanupInvalidDbKeys.php	Y	Y	Y	Y	Y
72	getReplicaServer.php	Y	Y	Y	Y	Y
73	checkUsernames.php	Y	Y	Y	Y	Y
74	rebuildmessages.php	Y	Y	Y	Y	Y
75	formatInstallDoc.php	Y	Y	Y	Y	Y
76	showSiteStats.php	Y	Y	Y	Y	Y
77	rebuildImages.php	Y	Y	Y	Y	Y
78	deleteOrphanedRevisions.php	Y	Y	Y	Y	Y
79	importSiteScripts.php	Y	Y	Y	Y	Y
80	checkComposerLockUpToDate.php	Y	Y	Y	Y	Y
81	deleteDefaultMessages.php	Y	Y	Y	Y	Y
82	purgeChangedPages.php	Y	Y	Y	Y	Y
83	fixTimestamps.php	Y	Y	Y	Y	Y
84	migrateImageCommentTemp.php ‎	Y	Y	Y	Y	Y
85	tidyUpT39714.php	Y	Y	Y	Y	Y
86	uppercaseTitlesForUnicodeTransition.php	Y	Y	Y	Y	Y
87	fixDefaultJsonContentPages.php	Y	Y	Y	Y	Y
88	populateFilearchiveSha1.php	Y	Y	Y	Y	Y
89	populateImageSha1.php	Y	Y	Y	Y	Y
90	populateExternallinksIndex60.php	Y	Y	Y	Y	Y
91	populateLogSearch.php	Y	Y	Y	Y	Y
92	compareParserCache.php	Y	Y	Y	Y	Y
93	findHooks.php	Y	Y	Y	Y	Y
94	fileOpPerfTest.php	Y	Y	Y	Y	Y
95	getSlaveServer.php	Y	Y	Y	Y	Y
96	mwdoc-filter.php	Y	Y	Y	Y	Y
97	populateRecentChangesSource.php ‎	Y	Y	Y	Y	Y
98	populateRevisionLength.php	Y	Y	Y	Y	Y
99	populateContentTables.php	Y	Y	Y	Y	Y
100	pruneFileCache.php‎	Y	Y	Y	Y	Y
101	purgeChangedFiles.php	Y	Y	Y	Y	Y
102	populateLogUsertext.php	Y	Y	Y	Y	Y
103	clearInterwikiCache.php	Y	Y	Y	Y	Y
104	eraseArchivedFile.php	Y	Y	Y	Y	Y
105	deleteBatch.php	Y	Y	Y	Y	Y
106	refreshImageMetadata.php	Y	Y	Y	Y	Y
107	populateCategory.php	Y	Y	Y	Y	Y
108	nukePage.php	Y	Y	Y	Y	Y
109	updateDoubleWidthSearch.php	Y	Y	Y	Y	Y
110	dumpTextPass.php	Y	Y	Y	Y	Y
111	attachLatest.php‎	Y	Y	Y	Y	Y
112	fixUserRegistration.php	Y	Y	Y	Y	Y
113	updateCollation.php‎	Y	Y	Y	Y	Y
114	copyJobQueue.php‎	Y	Y	Y	Y	Y
115	wrapOldPasswords.php‎	Y	Y	Y	Y	Y
116	alltrans.php	Y	Y	Y	Y	Y
117	langmemusage.php	Y	Y	Y	Y	Y
118	dumpMessages.php	Y	Y	Y	Y	Y
119	digit2html.php	Y	Y	Y	Y	Y
120	benchmarkHooks.php	Y	Y	Y	Y	Y
121	benchmarkParse.php	Y	Y	Y	Y	Y
122	benchmarkPurge.php	Y	Y	Y	Y	Y
123	checkStorage.php	Y	Y	Y	Y	Y
124	moveToExternal.php	Y	Y	Y	Y	Y
125	lag.php	Y	Y	Y	Y	Y
126	dumpRev.php	Y	Y	Y	Y	Y
127	date-formats.php	Y	Y	Y	Y	Y
128	listVariants.php	Y	Y	Y	Y	Y
129	sql.php	Y	Y	Y	Y	Y
130	SqliteMaintenance.php	Y	Y	Y	Y	Y
131	sqlite.php	Y	Y	Y	Y	Y
132	sqlite.inc	Y	n/a	n/a	Y	Y
133	doMaintenance.php	Y	n/a	Y	Y	Y
134	createCommonPasswordCdb.php	Y	Y	Y	Y	Y
135	convertExtensionToRegistration.php ‎	Y	Y	Y	Y	Y
136	deleteArchivedFiles.php ‎	Y	Y	Y	Y	Y
137	deleteArchivedRevisions.php ‎	Y	Y	Y	Y	Y
138	deleteEqualMessages.php ‎	Y	Y	Y	Y	Y
139	deleteOldRevisions.php ‎	Y	Y	Y	Y	Y
140	deleteUserEmail.php‎	Y	Y	Y	Y	Y
141	fixMergeHistoryCorruption.php ‎	Y	Y	Y	Y	Y
142	getText.php‎	Y	Y	Y	Y	Y
143	fetchText.php	Y	Y	Y	Y	Y
144	findClasses.php	Y	Y	Y	Y	Y
145	emptyUserGroup.php ‎	Y	Y	Y	Y	Y
146	findMissingActors.php ‎	Y	Y	Y	Y	Y
147	getLagTimes.php	Y	Y	Y	Y	Y
148	generateConfigDefaultSettings.php ‎	Y	Y	Y	Y	Y
149	generateConfigNames.php‎	Y	Y	Y	Y	Y
150	generateConfigSchemaArray.php	Y	Y	Y	Y	Y
151	generateConfigSchemaYaml.php	Y	Y	Y	Y	Y
152	cleanupUsersWithNoId.php ‎	Y	Y	Y	Y	Y
153	createAndPromote.php	Y	Y	Y	Y	Y
154	cleanupUploadStash.php	Y	Y	Y	Y	Y
155	cleanupTitles.php	Y	Y	Y	Y	Y
156	cleanupSpam.php	Y	Y	Y	Y	Y
157	cleanupPreferences.php	Y	Y	Y	Y	Y
158	cleanupEmptyCategories.php	Y	Y	Y	Y	Y
159	cleanupCaps.php	Y	Y	Y	Y	Y
160	addChangeTag.php	Y	Y	Y	Y	Y
161	blockUsers.php ‎	Y	Y	Y	Y	Y
162	changePassword.php ‎	Y	Y	Y	Y	Y
163	createBotPassword.php ‎	Y	Y	Y	Y	Y
164	dumpUploads.php ‎	Y	Y	Y	Y	Y
165	invalidateBotPasswords.php ‎	Y	Y	Y	Y	Y
166	migrateActors.php ‎	Y	Y	Y	Y	Y
167	nukeNS.php	Y	Y	Y	Y	Y
168	pageExists.php ‎	Y	Y	Y	Y	Y
169	populateChangeTagDef.php ‎ ‎	Y	Y	Y	Y	Y
170	purgeExpiredBlocks.php‎	Y	Y	Y	Y	Y
171	purgeOldText.php	Y	Y	Y	Y	Y
172	purgePage.php‎	Y	Y	Y	Y	Y
173	reassignEdits.php	Y	Y	Y	Y	Y
174	rebuildall.php	Y	Y	Y	Y	Y
175	rebuildrecentchanges.php‎	Y	Y	Y	Y	Y
176	rebuildtextindex.php ‎ ‎	Y	Y	Y	Y	Y
177	removeInvalidEmails.php ‎ ‎	Y	Y	Y	Y	Y
178	recountCategories.php‎	Y	Y	Y	Y	Y
179	purgeMessageBlobStore.php	Y	Y	Y	Y	Y
180	purgeParserCache.php‎	Y	Y	Y	Y	Y

Story

When I got this grant approved, I got COVID infected along with my family at that time. That was hard time for me as this project slip for two months. But my project advisor Alex was so supportive that we started project with new energy.

There were many things that I did not think but I added that thing in scripts while documentation like using MW version template to show MediaWiki version in which the script was added.

Originally, I planned for 124 script as mentioned in the grant proposal. But I got so interested in this project that I continued the work till 180 maintenace script. We finnally finished the work with 180 maintenace. This number is too far from what I expected while starting the work.

It is weird for me but an experienced user endorsed my grant even after 9 months after from grant approved. This show success of this project.

Survey(s)

Project Pralekhan feedback was collected from 25 Feb 2022 to 5 March 2022 through Google Forms. We got 12 responses from Google Form. There are 6 responses with username (including 2 WMF staff). Please read the detailed compiled version at mw:Project Pralekhan/Survey Result.

Interesting outputs or outcomes

1. 91% of users are finding new structured documentation helpful. (A huge surprise success)
2. 75% users are satisfied with the usage section (Again, huge success as project documentation main goal was to demonstrate usage)
3. 33% users are not finding easy navigation between the scripts. (We improved mw:Manual:Maintenance scripts/List of scripts based on this feedback)
4. Use LimeSurvey ( LimeSurvey ) instead of Google Form for future surveys as Google Form is forbidden in China and many other countries.

Apart from this, grantee also took a Unconference session on this project at Wikimania 2021 to showcase and take feedback from community.

Other

Is there another way you would prefer to communicate the actual results of your project, as you understand them? You can do that here!

Methods and activities

• I have set up a local installation of MediaWiki using MediaWiki-Docker to get output of each script. (see mw:MediaWiki-Docker)
• I followed mw:Documentation/Technical documentation templates and suggestions and mw:Documentation/New technical writers created by SRodlund (WMF).
• I set up project page mw:Project Pralekhan to track the progress of every script.
• I took meeting call in every 20 days with project advisor Alex Paskulin.
• I used Zulip chat to communicate in between meetings.
• I took a Unconference session on this project at Wikimania 2021 to showcase and take feedback from community.
• I used Template:MW version to show MediaWiki version in which the script was added.
• I used mw:Template:Terminal for showing usage of script in terminal.
• I used mw:Template:Codesample for showing code snippets and file content.
• I viewed source code of a maintenance script in MediaWiki core repository for their option and arguments.
• I used wiki table on the script pages to show Options/Arguments, and which one are required or optional through mw:Template:Required and mw:Template:Optional.
• I used DynamicPageList MediaWiki extension on List of scripts. This ensure that user always see updated list of scripts. (Previously, it was hard-coded which does not update automatically)
• To improve discoverability of maintenance scripts
  • I made improvements on landing page
  • I categorized of scripts based on their nature
  • I interlinked the scripts in See also section

Project resources

• Manual:Maintenance scripts (Main landing page)
• Project Pralekhan (Project page)
• List of improved scripts
• Discovered bugs
• Created categories
• Survey Privacy Statement
• Survey Result
• Meeting notes: https://etherpad.wikimedia.org/p/VQrdm1hIcDXKtttvzS0A

Learning

What worked well

It is now good to see that so many scripts now have structured documentation. With great help from APaskulin (WMF), I have done 180 maintenance scripts. Everything going very well except the below challenges.

I find below learning pattern useful:

• Learning patterns/Writing durable documentation
• Learning patterns/Keeping documentation of discussions with team

What didn’t work

• Every script has its own use so creating the environment for every script is sometimes very stressful. I need to collect information and then have to replicate different-different situations for every script to run in the terminal.
• Some scripts have a bug (Like one I discovered during this project see phab:T291331) that makes documentation a little bit hard.
• I faced an issue with collecting feedback from visitors as it has some privacy concern that needs some follow-up with WMF's legal team. It was discussed with project advisor at phab:T295695 and we decided to not take feedback at wiki instead, used Google Form.

Other recommendations

If you have additional recommendations or reflections that don’t fit into the above sections, please list them here.

Next steps and opportunities

There are total of 200-250 maintenance scripts. Under this project, I have documented 180 maintenance scripts. This project already done lot of work and there is small amount of work remained. So I will be work on those as volunteer to document them. So there is nothing major for future's next steps.

Part 2: The Grant

Finances

Actual spending

Expense	Approved amount	Actual funds spent	Difference
Technical writer	8750	8750	0
Internet	200	240	-40
Contingency	400	261	139
Total	9350	9251	99

There is a minor $40 increase in the Internet bill as I forget to add tax on the $200 internet plan. There is an 18% of Goods and Services tax per Government of India. This amount has been adjusted through the contingency fund.

Although, It took 420 hours for the technical writer as he documented 180 scripts instead 124 scripts that were originally mentioned in the grant. But this is not being compensated through a contingency fund. It is just being mentioned here for any reference.

Remaining funds

Do you have any unspent funds from the grant?

Remaining funds have been used or will be used for other approved mission-aligned activities. This use has been requested in writing and approved by WMF.

• Yes, the remaining amount is 99 USD. Since, It is part of contingency so this was as a backup fund.

If you have unspent funds, they must be returned to WMF. Please see the instructions for returning unspent funds and indicate here if this is still in progress, or if this is already completed:

Documentation

Did you send documentation of all expenses paid with grant funds to grantsadminwikimedia.org, according to the guidelines here?

• Yes

Confirmation of project status

Did you comply with the requirements specified by WMF in the grant agreement?

• Yes, best in my knowledge.

Is your project completed?

• Yes, with major success.

Grantee reflection

It is a really very awesome experience for me to work on maintenance scripts. Even, I have been working on MediaWiki since early 2017 but only after working on this project, I came to know many other scripts as well that are very useful. I got Alex as my mentor. I can't express how she is good in nature. A great thank you, Alex.

The project got some delay from my end which I acknowledge. But this is the nature of human work. We can't expect to things go as well as planned. Working and giving shape to this project with help of Alex really made me proud. Thank you again Alex for your support and WMF for funding.-Jayprakash >>> Talk 19:08, 30 March 2022 (UTC)