Grants:Project/Rapid/Pine/Continuation of educational video and website series/Report

Report accepted

This report for a Rapid Grant approved in FY 2018-19 has been reviewed and accepted by the Wikimedia Foundation.

To read the approved grant submission describing the plan for this project, please visit Grants:Project/Rapid/Pine/Continuation of educational video and website series.
You may still comment on this report on its discussion page, or visit the discussion page to read the discussion about this report.
You are welcome to Email rapidgrants at wikimedia dot org at any time if you have questions or concerns about this report.

Goals

Did you meet your goals? Are you happy with how the project went?

Here are the goals as stated on the project page:

The goal of this $2000 mini-project is to create and complete a single module from start to finish, specifically the module regarding English Wikipedia references that was originally discussed here (link to original LearnWiki script outline). In addition to the output of the module content, this rapid grant will fund a valuable test of the process for creating a complete module, providing information about time and budget required, and the effectiveness of the module development process including communications with English Wikipedians.
Both the content of the module output and information from the testing of the process are outputs from this mini-project. The collection and reporting of information about the development process is very important.

There is good news and bad news here. In the big picture, I think that the good news outweighs the bad news.

I think that referencing was a good module to choose as a test. However, after getting into the details of explaining how referencing works, it became apparent that explaining referencing with an intermediate level of thoroughness requires more than a single short script and a single short video.
I am happy with the quality of the scripts, and I intend to continue this project by moving at least 2 of them through full production on volunteer time.
I continue to believe that "Additional materials" will be very important for the completed videos. Also, I think that organizing the video series in a way that is easy for newcomers to understand, in multiple languages, is valuable for the long term. I spent many hours developing the NavWiki project space on the Outreach Wiki during time not covered by the funds for this grant, and that space will eventually host or link to "Additional materials" in addition to linking to the finished videos which will be hosted on Commons.
The main bad news is that I underestimated the amount of time, and therefore the amount of budget, that would be necessary for the type and quality of production that I had in mind.
I consider this project to need further work, but in many ways I consider this project to be going well. Again, I intend to continue with it on volunteer time, at least through the production of 2 pilot videos and hopefully beyond.
There are reasons that I have not requested more WMF funding for this project, and I explain some of them later in this report.
As compared to the predecessor LearnWiki project, the margin of error here is much narrower, and in the big picture does not cause me great concern. I currently estimate that the margin of error is less than $10,000, which is much smaller than the error for LearnWiki which in hindsight feels like trying to create a BBC documentary with what would ordinarily be a seven-figure budget except with a budget of thousands instead of millions. The margin of error for the budget for this pilot project, in my opinion, is much more sane.

Outcome

Please report on your original project targets.

Target outcome	Achieved outcome	Explanation
200 views (in the longer term the video will likely get thousands of views if people find it to be helpful)	Needs additional work	The pilot videos need to be produced and published. After the videos are published, I believe that 200 views will be easily achievable.
Generally positive feedback to be left on relevant talk pages regarding the usefulness of the video for content contributors, and generally more positive feedback than negative feedback	Feedback so far has been positive regarding the project, but the project needs additional work.	Feedback regarding the project and the NavWiki space on the Outreach wiki has been generally positive. The videos need to be published so that the community can use them and review them.
Translation of the video subtitles into at least two additional languages, possibly by volunteers. Pine would consider this to be a significant sign that contributors in non-English languages feel that the videos are valuable. Pine may have enough skill with Spanish to do a Spanish translation, and if there is unused funding in the $2000 grant then Pine may consult with WMF regarding using that funding a the Spanish translation.	The videos need to be published first, but there has been community interest in creating subtitles for the finished videos.
"Organic" sharing of the video, beyond the sharing that Pine does	The videos need to be published, but I consider this goal to be achievable.	Organic sharing of the videos will likely indicates that the community finds the videos to be useful.

Learning

Projects do not always go according to plan. Sharing what you learned can help you and others plan similar projects in the future. Help the movement learn from your experience by answering the following questions:

What worked well?

The quality of script outputs were good.
Much larger quantity of script output than initially estimated.
The information that I received in response to my early announcements, which informed me of existing instructional videos that I did not realize existed, was very useful. I started the instructional video catalog page to track these videos. In particular, the Spanish video regarding creating references with VisualEditor was very well aligned with my plans for the video that I will produce in this project, and I got some good ideas from watching the Spanish language video. Also, having a catalog of existing videos (now located at outreach:NavWiki/External Resources, although I am considering renaming the page) helps to reduce duplication of effort and to share good ideas. Instructional videos for Wikimedia are not necessarily easy to find on Commons, and the quality of those videos and the currency of their information is not necessarily easy to determine simply by searching the Commons category tree and by searching Commons file names. (I later learned of the pages Video tutorials (on Meta) and Instructional videos (on Outreach), which appear not to be maintained. I plan to integrate those into the NavWiki page that I linked.)
My communications such as newsletters and emails were almost always received with no complaints.
The pilot nature of the project, the relatively narrow scope, and the relatively small budget were good choices for risk management of this project, and I do not regret starting this project. The small budget turned out to be a problem relative to the ambition for this project, but that does not mean that the project was a bad idea.

What did not work so well?

Topic chosen required more extensive explanation, and therefore more script writing, than initially estimated.
The $2,000 budget was exhausted prior to the beta 3 script revisions
The $2,000 budget was insufficient for production of all 30+ pages of script.
The initial approval for this grant from WMF took months longer than anticipated. However, once approval was granted, WMF staff were quick to respond to my communications, which I greatly appreciated.
I found it a bit confusing to have a somewhat carefully defined project concept and a somewhat demanding set of off-wiki instructions from WMF regarding the grant, and then after the signing of the grant agreement be given significant discretion by WMF regarding how to design and to execute the project including making modifications as I think best. I would prefer that the styles of management be more consistent from the beginning to the end of the grants process, or that this shift in management style be planned and communicated earlier. The management style can be very flexible, very firm, or something in between, and I can adapt to any of the above, but I would find consistency to be beneficial so that I know how to plan and what WMF wants from me, including what communications WMF wants from me.
Communications before, during, and after drafting the scripts were surprisingly time consuming. However, the scripts became better as the drafting process continued, due in part to very valuable comments from reviewers. In the future I would plan for more time to be spent on communications.

What would you do differently next time?

I am generally happy with how this project went. However, one lesson is not to underestimate the complexity of English Wikipedia workflows in general, and the time that is required to explain them with high quality at an an intermediate level of proficiency.
In hindsight I might have written more of the scenes as highly detailed outlines instead of a full word by word scripts. Moving from highly detailed outlines to full scripts could have waited until those scenes were selected for production.
I would not attempt a future iteration of this project with a similarly small number of estimated hours and a similarly small budget. Although I will need to wait to finish production and publication before I have final numbers for what this pilot project would cost at $30 an hour, I think that a reasonable budget for this type of project might be in the range of $5,000 to $10,000. The final numbers will be useful for estimating the cost of production of additional existing scripts and/or writing additional scripts.

By the way, I have an increasing appreciation for why full time instructional designers make approximately $80,000 per year plus benefits for this type of work. A project with this level of complexity is not trivial to execute even as a pilot, and good quality work requires many skills and a surprising amount of time to do successfully.

Finances

Grant funds spent

Please describe how much grant money you spent for approved expenses, and tell us what you spent it on.

Note: the total cost for the project to date exceeds the value of the grant. I consider the cost in excess of the grant to be a voluntary contribution and I do not expect reimbursement.


Item	Time	Amount
Phase 1: Early design and planning	2:20	$70.00
Phase 1: Initial communications, including creation of content on Meta talk pages	18:46	$563.00
Phase 1: Script draft alpha 1 and communications while this draft was being created	11:32	$346.00
Phase 2: Communications regarding draft alpha 1 and/or during the writing of alpha 2	13:04	$392.00
Phase 2: Script draft alpha 2 and/or beta 1	13:17	$398.50
Phase 2: Script drafting for the unexpected development of search strings in "cite"	2:21	$70.50
Phase 3: Design, project management, and planning issues	3:00	$90.00
Phase 3: script drafting	6:40	$200.00
Phase 3 communications, not including the NavWiki space on the Outreach wiki and not including reports	18:39	$559.50
Phase 3: final report and related communications	10:42	$321.00

Not funded this grant, but within the scope of the larger NavWiki "product" and completed on my own time, I spent the time shown below on creating the space on Outreach, which at the rate of $30 per hour would have cost the amount shown below. I am including the information here as a supplement to the information above.


Item	Time (not funded by grant)	Amount (not funded by grant)
Development of the NavWiki space on the Outreach wiki. The amount shown here is lower than it would have been if I needed to design this space from scratch. I already had planned significant elements of the design based on the predecessor LearnWiki project.	21:37	$648.50

I spent some time thinking about audio cues and music for the videos. This was not necessary for the limited scope of the pilot project but would be beneficial as a "nice to have" feature to unite the videos in the collection. Although this line item is not included in the scope of the grant, I am documenting this here to supplement the required information to document the volunteer time. This is an underestimate of the amount of time spent on these activities. I spent considerable informal time thinking about audio issues when I was doing non-project activities.


Item	Time (not funded by grant)	Amount (not funded by grant)
Audio cues and music for videos in the collection	2:00	$60.00

Remaining funds

Do you have any remaining grant funds?

Remaining funds have been used or will be used for other approved mission-aligned activities. This use has been requested in writing and approved by WMF.

Yes, a few cents that were earned as interest after the March 31, 2019 originally scheduled end date. I am willing to return this money to WMF, but the cost for staff time of processing would almost certainly exceed the value of the interest.

Anything else

Anything else you want to share about your project?

Overall I think that this was a good pilot project, although the budget was too small for my goals.

The decision to have a limited budget of $2,000 and a relatively narrow scope were good choices for risk management, although teaching referencing on Wikipedia turned out to be too time consuming to teach at an intermediate level of proficiency and with my desired level of quality on a $2000 budget. I have an increasingly good understanding of why instructional designers are paid relatively well and why developing high quality training for complex topics such as Wikipedia referencing is surprisingly time consuming.

While I would be happy to work on this project full time, and there is plenty of work that could be done including translations, for at least the near future I will continue this project without using WMF funding. Ideally, non-WMF funding would be available so that, if the community likes the pilot videos, I could work on this project full time without using WMF funding.

Even if WMF is willing to fund me to work full time on this project after producing the pilot videos, I would have questions about how I or anyone else can remain genuinely a part of the community if I am accepting a significant amount of WMF funding. Regarding social and content topics, I have concerns about WMF telling community members what to do or how to do it, which includes providing training directly or indirectly regarding content development, content management, and community policies and norms. I think that WMF-sponsored training regarding topics that are almost purely technical, for example "How to translate a page" or "How to sign an edit", are at lower risk from WMF involvement than training about social and content topics such as "Explanation of verifiable and reliable sources" or "How to interact with other contributors". I would like for this project to offer training regarding all of these types of topics, and I do not know how I could adequately protect myself, the community, and the project from undue WMF influence if WMF provides financial support for my time for this project, although I would be very interested to hear suggestions. There are some government agencies that are semi-independent of national governments, such as U.S. states which are semi-independent from the U.S. Federal government, and there might be a way to offer similar separation between this project and WMF but I would need to consider further how this might work.

I am generally happy with the NavWiki project pages on Meta.

The next significant steps are:

Write and publish another newsletter that provides a status update regarding the pages on the Outreach wiki and regarding production
Produce 2 pilot videos from the finished scripts
Write and publish another newsletter that announces the finished scripts