Grants talk:PEG/Retrospective 2009-2012

This is brilliant. Thank you, Kevin. –SJ talk  01:03, 14 March 2013 (UTC)

Is this a draft or the final version? Tony (talk) 01:28, 14 March 2013 (UTC)

This is the final version. Asaf Bartov (WMF Grants) talk 22:31, 21 March 2013 (UTC)

Diagrams / Axis

Thank you a lot for this report! May I ask you to update the diagrams to have a description on each axis? Especially in

it is unclear what the X axis means (I guess days... but 350 days for a decision would be awkward?). Thanks a lot. --Manuel Schneider^(bla) (+/-) 07:58, 14 March 2013 (UTC)

Hi Manuel - I thumbnailed the image you linked, I hope you don't mind. It was breaking tables in my browser. In the bar graph you linked, the Y-axis is time, measured in days. The x-axis separates out accepted and unaccepted grants by fiscal year. I'll slap a label on the y-axis and make some minor edits to the text to make this more obvious. (I separated accepted and unaccepted grants because they took distinctly different amounts of time to review.) It is worth noting that it is likely that many decisions were conveyed to applicants before they were posted on Meta, especially in the first year of the grant program, but I agree with you that the amount of time grant applicants had to wait in the first year of the program was quite awkward. Kevin (talk) 16:11, 14 March 2013 (UTC)

Thanks a lot Kevin. I hope it's worth the extra effort to make the diagrams more obvious, even though you're right that out of context it should be clear anyway. I just thought that it might be better if they can be understood on their own. --Manuel Schneider^(bla) (+/-) 19:34, 14 March 2013 (UTC)

Question on totals and percentages

A few times in the report, statements along the lines of "Grant applications totaled ~$1,376,772. A total of $1,280,299.05 of funding was approved for grants in the year, meaning that ~92.9% of all requested funding was approved." are made. Do these figures include grants which were withdrawn by the applicant after being advised that the grant would not be approved? Craig Franklin (talk) 13:13, 14 March 2013 (UTC).

They should, yes. Kevin (talk) 16:17, 14 March 2013 (UTC)

Fully protect grant applications after they have been funded

A changelog, seriously? Sounds 19th century. Why not FlaggedRevisions in the Grants namespace, or making sure that everything points to permalinks and that people understand that last revision is not necessarily the most accurate (OMG it's a wiki). --Nemo 11:22, 20 March 2013 (UTC)

FlaggedRevisions, as far as I know, has no way to turn it on/off for particular pages at particular points. If I am wrong about that then it would totally work and would be a superior solution. There are obviously a lot of legitimate reasons to edit grant applications before they have been funded. There aren't really legitimate reasons to edit a grant application after it has been funded - it serves as the application, not the final project report. So turning on FlaggedRevisions would create a lot of unnecessary work - there's no reason for someone to have to go through and manually approve a bunch of revisions to a grant that hasn't reached a decision yet.

In an ideal world, people would realize that changes might have been made to grant applications after they've been funded and would go through the history tab, but we don't live in an ideal world. There have been previous situations where grants have been edited after they have already been funded without that being noticed by various WMF staffers reviewing submitted grant reports. This is problematic, because one of the purposes of a grant report is accountability, and that is significantly undermined by unnoticed changes. Fully protecting a grant after it has been funded guarantees that type of situation will never happen again, and since it is such a low effort option that would guarantee that a previous actual problem would never occur, I think it is worthwhile. It's a small technical fix than can guard against the possibility of human error recurring.

There have been a large number of situations in the past where changes to grants have been approved (or allegedly approved) via email communications or phone calls that were either hard to retrieve or impossible to retrieve for WMF grant staff reviewing reports at later dates. This has resulted in confusion and disagreement between grantees and WMF about what the parameters of a grant were. Having an official/definitive log of approved changes would ensure that this would not happen in the future. An actual changelog, though pretty 1990's, is the easiest way to create such an official record that would definitively end this kind of disputes. Obviously this problem would be helped by other things as well such as ensuring all grant communications are archived, but given how easy this recommendation is to implement, I think it is more than worth it. If anyone comes up with a more technically elegant solution that felt less like 1996 that would solve the same problem that would obviously be preferable.

It's a bit late, so I probably won't get to all of your questions tonight, but thank you for posting them and please ask if any bit of my answers are unclear. If you can come up with a technically superior solution that would address the same problems, I would amend it in to the report. Kevin (talk) 08:50, 21 March 2013 (UTC)

Yes, you're wrong on FlaggedRevs. :) Unless I misunderstood what you mean, you can now see on mw: that on Extension/Manual/Help namespace FlaggedRevs gets active only after the first time a revision is marked "checked" (cf. mw:Special:UnreviewedPages), and "quality" flag is more or less what you'd need here. --Nemo 22:44, 21 March 2013 (UTC)

I was unaware FlaggedRevs had that functionality. That makes it an infinitely more elegant suggestion. Thanks for bringing it up, I'll update the text of the report in the near future. And I would highly suggest that WMF grant staff take this path - it's been an issue multiple times in the past, and even with the grant team having far better communications now than they used to, I think it is still likely to be an issue in the future if no action is taken to address it. Kevin (talk) 18:56, 5 April 2013 (UTC)

Administrative burden

«WMF staff should also consider rejecting grants that have unusually significant administrative burdens unless the grant is coming from an organization with a proven track record of managing significant administrative burdens.» How many can such grants be? What's this administrative burden composed of, concretely, and have you assessed whether it's all necessary or could be reduced? --Nemo 11:24, 20 March 2013 (UTC)

Grants that fall in to this category are not common, but when they occur, they can cause significant problems - and for that reason they are worth being aware of. For a direct example of the type of grant I was talking about, take a look at the section of my report that reviews WMCZ's mediagrant. The grant had been proposed with the assumption that a lot of the monotony and administrative burden involved in handling microgrant requests would be handled by a contractor hired for the grant, but that contractor was not approved. This resulted in a situation where the grant was approved, but WMCZ volunteers didn't have the ability to easily handle the administrative burden (in terms of things like getting grantees reimbursed on a timely basis, keeping track of all financial paperwork, etc) that the grant represented. This caused serious problems for WMCZ - in the words of a WMCZ member it almost destroyed the chapter - and was easily predictable ahead of time.

I didn't not assess in detail whether the work involved in this particular grant or others of this sort could be significantly reduced; I would imagine a lot of this sort of work could be reduced significantly, although figuring out how to do that might require a significant time investment for each grant which could pose an additional problem anyway. In the case of the WMCZ grant, I don't think that the administrative burden could have been reduced to a level that could have been easily handled by WMCZ without making really significant changes to the structure of the grant. Kevin (talk) 21:39, 5 April 2013 (UTC)

Avoid single points of failure (especially in communication)

I liked this simple suggestion of keeping at least one cc (plus a possible program-mailbox): it matches the organisational best practices adopted by some chapters, Wikimedia Conference 2011/Documentation/Volunteer management#Sharing knowledge: Tandems, and in my experience as WMIT treasurer (and in other organisations) the ability to transfer all the documents, information and communication to your successor is key for success. --Nemo 11:30, 20 March 2013 (UTC)

One of the most significant problems I ran in to in creating this report was tracking down emails (and other communications) from previous years. WMF has matured organizationally significantly within the last several years and it's definitely easier to find info w/r/t recent WMF stuff than it used to be, but it's still not simple. I'm glad that this is something that has already come up in chapter discussions, and even though this wasn't scoped towards chapters at all, hope that it is a problem that you and other chapter members are actively aware of and considering how best to address. I can imagine a lot of chapters three or four years down the road having matured organizationally from where they are now having this same sort of problem trying to retrieve earlier communications and knowledge. Kevin (talk) 21:43, 5 April 2013 (UTC)

Interlanguage communication

The suggestions in the «Take action to increase the visibility of the grants program» section, i.e. use Global message delivery or equivalent, CentralNotice and WikimediaAnnounce-l or other mailing lists, seem in contradiction with the «Language and cultural barriers are likely to in some ways limit the efficacy of the grants program» section: haven't all the obvious things like the notifications you suggest already been tried? --Nemo 11:38, 20 March 2013 (UTC)

No, they have not, actually. To my knowledge, GMD has never been used to inform about the Wikimedia Grants Program (it has been used for the recent IEG inauguration), nor has CentralNotice after the program's inception. Asaf Bartov (WMF Grants) talk 22:11, 21 March 2013 (UTC)

And another aspect on languages: Could we try to translate this report? It's interesting for more than only the english speaking community but the length and the absence of a summary concerning the contents raise the barrier for non native speakers. Alice Wiegand (talk) 22:23, 21 March 2013 (UTC)

The report does include a summary section. It is certainly possible to translate the report, but in commissioning the report I have not budgeted for paid translation, so it would be up to interested members of the community to do so, if there's interest. I can help by preparing and marking the report page for translation if I see any other expressions of interest. Asaf Bartov (WMF Grants) talk 22:30, 21 March 2013 (UTC)

I've read the report and I know that there is a summary section ;-) But if you try to only read this section you'll recognize that it's more a summary of the structure of the text than one of its content. That's what I've meant. Alice Wiegand (talk) 22:37, 21 March 2013 (UTC)

FYI

I've had an emergency situation come up that means that I'll be unable to participate in discussions here (or anything else Wikimedia related) for approximately the next week. I apologize and I'll be back as soon as I can, but this situation has to take priority. Kevin (talk) 22:40, 21 March 2013 (UTC)

The situation has been primarily resolved, so I will be significantly more responsive here now :) I hate to take off immediately after launch, but something came up that was unavoidably urgent. Kevin (talk) 23:21, 3 April 2013 (UTC)

Specific thanks, and thoughts on outreach

I'm thankful to Kevin for his work on this, and to my colleagues for commissioning it.

The section "Take action to increase the visibility of the grants program" strongly speaks to me; the Engineering Community Team has similar issues in trying to reach out to Wikimedia contributors regarding opportunities for training, input, etc. We're trying to follow these best practices. Now that we have a few years of past grantmaking, it would be great to point to specific model grants when we do our outreach, to help give contributors ideas for the kinds of endeavors that grants could facilitate. In my experience, having a few examples of that sort is helpful in persuading people that an opportunity they've never considered before is worth considering. :-)

I am also particularly interested in this insight:

enthusiasm and desire to do something among volunteers are strongest at the beginning of a project, and if major delays occur, that productive energy may be wasted instead of being successfully channeled into valuable programmatic work, especially if grant applicants are not kept up to date about the cause of any delays they experience.

Timely responsiveness makes a huge difference. Thanks for reminding us. Sharihareswara (WMF) (talk) 16:05, 24 March 2013 (UTC)

There's something a little ironic about me replying to this post something like eleven days after you made it, but life came up in a very significant way ;-) One thing that I know I talked to Asaf about, although I'm not sure I mentioned in the final draft of the retrospective, was the idea of pointing to specific model grants or at least outlines of how to perform common general types of grant activity. Kevin (talk) 21:48, 5 April 2013 (UTC)

SPoF

Thanks for the report, very useful, and I enjoyed reading it. Besides single point of failure in communication, I believe that it is very important to be aware of single points of failure in other aspects of project execution (for example, if project leader or otherwise key person leaves the project, will there be a suitable substitute to pick up and continue work?) -- Obradovic Goran (talk) 18:57, 11 April 2013 (UTC)