This report is about the process: what's working and what needs improving in the Individual Engagement Grants program, from the perspective of grantees who completed their projects from round 1 2013, and grant proposers and committee members who reviewed proposals in round 2 2013.
Grantees say the experience deepens their involvement in the community, and committee members expressed a changing sense of stewardship for the global movement.
Wow, it's deepened my involvement in Wikimedia tremendously, given me a great sense of confidence, helped validate my methods and improve my understanding of all phases of research, testing, and execution. I can't imagine having accomplished the same amount--not nearly--without it. -grantee
It gave me the opportunity to work in a more professional way to projects I really like, and in the end it gave me more expertise and experience and "hope" issues can be solved. It offered solutions, and it taught me that solutions can be built, if you work on them. I "used" the IEG for working on certain issues I knew from a long time, and it proved effective and useful. Overall, it deepen my already high involvement in Wikimedia world. -grantee
I found the whole experience enlightening and inspirational, while also making me think about how donation money should be used globally rather than locally (I have participated in two strategy-forming discussion rounds for my chapter, in 2011-2012 and in 2012-2013). -committee
Liked: The use of idealab and the evaluation process for new members -committee
I was glad that only minimal bureaucracy (if any) was involved. It was assuring that candidates were asked to make a detailed profile in public before the selection. -committee
In round 2, we divided proposals up into 4 types and committee members joined working groups to review each type. This appears to work well.
I was rather time-pressed IRL, so it was good that being able to concentrate on reviewing and commenting a particular type of proposal only and still my output was still something useful (I hope). -committee
Don't split reviewers into groups. I review everything anyway. However, do split proposals into groups. It's easier. -committee
I like that you can decide to review just one or all of the proposals, because you never know going in just how much time it will take you. Not all proposals are equal!!! -committee
The selection process involved asking people to sign up in the track of their expertise. This helped the members to concentrate on the areas of their choice instead of having to look into all project proposals at the same time. -committee
Proposal creation and submission works reasonably well for those who complete the submission process. However, it is worth noting that we're only hearing from those who were able to make it to this point in the process, and that the process is still a bit joyless even for these, so there is still some room for improvement.
The submission process isn't complicated, with a guide on meta about IEG, and prepared template for standard submission - proposer
Liked: Proposal template -proposer
I liked the simple, informal approach through the wiki, which also made it easy to do updates. -proposer
Easy to use, clear structure, sensible questions -proposer
100s of volunteer hours go into submitting and reviewing proposals during a round of IEG.
For all but 1 committee member, this workload was about what they'd expected. Nearly all proposers said they thought the time they spent was "about right," although 2 thought it was perhaps "too little time." Proposers whose submissions were funded say they spent about the same time as proposers whose submissions were not funded.
reviews are time consuming but that is expected -committee
Committee members were, as a group, a bit more satisfied with the overall review process than proposers. But most participants in both groups said they would return to IEG in the future, and would be likely to recommend it to others.
Proposers - even those who weren't ultimately funded - appreciated getting feedback on their ideas.
It was fair, and I liked those feedbacks most that were given in a respectful and encouraging manner -proposer
The review process, both the formal one with WMF IEG stuff, and the non formal, with helping comments from the community was constructive and it helped us to better understand the requirements, improve the proposal and the planned project. -proposer
The reviewers were all very thorough and asked a lot of questions, which really helped me solidify my proposal. -proposer
it was constructive in general. i agree with the reason why it was rejected, and it makes it still possible to come back with it when some factors have changed -proposer
Yes, I hope that I can improve my proposal based on the feedback and apply again. -proposer
Table 1: Average discussion activity, proposals selected and not selected (round 1 only).
Committee members who returned from round 1 reflected feeling more prepared in round 2, and noted that round 2 adjustments have been useful.
much less time spent on committee procedure development in round 2 which is a sign of maturation -committee
Overall, it felt more efficient for me. Individual proposal were discussed on the iegcom wiki and general/meta discussion were on the mailing list (as opposed to do almost everything on the mailing list), so I was able to concentrate more. -committee
More committee members, more fun! :) I was to use more of the IEG wiki in this round compared to the past round. In short, I had an easier time in scoring and reviewing the proposals this time probably because I am used to doing this. -committee
I worked for less amount of time in round 2 as i knew the process, criteria etc. -committee
Grantees report high satisfaction with the staff support they received in areas like grants administration, communications, and research. Proposers and committee members also highlighted appreciation in this area.
Winifred is great with prompt reminders and helps us keep track of everything going on. -grantee
An experienced driver knows the problems of the car by only hearing the sound of the engine, that's what the experienced staff means to the whole project. Not only provide the sense of security, but also the problem detector. -grantee
I gave the Technical support a "Very satisfied" because they gave us the opportunity to use Qualtrics, which proved a very powerful asset for our IEG project. -grantee
WMF staff is very responsive to questions and reachable through a number of channels. They also help grantees like me to promote my work using tools like the WMF blog. That's a really nice way to share what's going on. -grantee
We love Siko. -committee
I appreciated the level of support provided by WMF staff and how wonderful they were about answering my questions and helping me through the process. -proposer
what i really liked is how the staff helped in solving problems. I think my proposal is not following the rules and it was cool to see how this made people looking for solution, not withdraw me (a really fantastic sign). -proposer
Grantees all said they were satisfied or very satisfied with reporting requirements; more interestingly, their comments suggest they see the reporting structure as helpful for their own projects, rather than simply a requirement to fulfill in exchange for funds.
I like the lightweight nature of the monthly reports, and the ability to switch from a simple list of activities completed to a blog post or other outward facing piece. Midpoint and final reports are both tough, but after such a long time of working they really help to frame the experience and distill the core achievements and learnings. As always, IEG staff gives great clarity and focus to the process.-grantee
Monthly reports help me to fully review what I did last month, and make it easy to find the problems and how the project is progressing. Also Monthly reports help me to save many details that some of them I already forget when writing final report. The midpoint report and final reports provide chance to organize own experience, and a full view of the project.-grantee
I like that, using Siko's word, we could "choose our adventure" for the monthly report. This proved valuable, as we decided writing blog posts for the Wikimedia Blog, and this was useful and gave us attention and visibility. I also liked that in Final and Midterm reports we could write what we thought was going good, we were free to express our insights. Siko was always very attentive and responsive about our feedbacks, and that is rewarding per se. In the end, I enjoyed very much writing those reports, and I think they are good :-)-grantee
All round 1 grantees were offered funding to attend Wikimania, and those went said it was useful for themselves and their projects.
When you share your project, or hearing others' experience, it definitely extends your own thinking.’' -grantee
Well, Wikimania is always awesome, and this time, for the first time, I went there to "work". Going there sponsored by the IEG gave me a responsibility, so I tried to work on the project all the time, focusing less on the conference and more on the possible collaborations. -grantee
Basically, I appreciate the fact that I got to go! Events are amazing networking opportunities and they are the best way to jumpstart skill development in new areas--much harder to do remotely and solo. In person is great and refreshingly useful! -grantee
Suggestion: Offer wikimania scholarships to highly active committee members. FDC members get travel funds so this seems fair and would be rewarding for IEG members. -committee
Human support adds much value to the IEG process, and participants want us to continue to think creatively about how to do more in this regard.
IEG is where I would instinctually send our most talented and creative people to take the next step in the scale of their contributions, especially where project management is involved. There's a whole skillset around program/project management that we should continue to nurture and teach. It's not learned easily without patient and thorough help, and I very much appreciate the IEG program for being that mentor. -grantee
IEG staff is great and dedicated, but there is only a limited number of them. I think we need a way to scale the advisory and mentorship capacity for grantees who haven't yet been through the project management rodeo.-grantee
Well, maybe finding a effective way to "accelerate" projects (startup-like), and providing competence and technological infrastructure to grantees. Issues can emerge during the IEG, as solutions, insights, intuitions. Counting on a network of competences can be extremely useful for the project. -grantee
I think the reason why i got the grant is because i have experience in project writing and i think wikimedia need and have to make sure projects and ideas by people who are not experienced in project writing can be submitted and financed. I'd like eventually to tutor a submission; if i find a project i like, i would be willing to contribute to the application process and side-work (evaluation, monitoring, checking it). i think this would allow people without experience in project application to get the grant and it would provide more chances to a person who is unknown or less known within the community to have a chance to be selected. wikimedia is based on a geriatric hierarchy; the longer people know you, the better can be your position; I think tutoring (which can also work as a recommendation) can balance this and it can create opportunities also for new people and people who have not being active long in the movement. -proposer
Increase community/committee interaction with proposers/grantees
Grantees fell on a spectrum between neutral and very satisfied about the level of committee and other community support or involvement in their projects. This may be due to varying levels of committee and community involvement in each project.
I think the whole "How does the community react to your proposal" bit needs to be fleshed out. The typical proposer is not that well versed in what "the community" is or how to contact them and how to generate supporters and feedback. Lots of tips are needed here, beginning with emphasis on the title and first paragraph of the proposal. -committee
Compared with round 2, I think the proposals had more community feedback. -committee
People were interested in our IEG project, and that is very rewarding. I also think that they liked the outcome of the project, which is even more rewarding :-) -grantee
comments by "community members" are not always pleasant or even fair -proposer
Area to improve: Address proposers' queries early and precisely. And assist users whose grants were not approved and share the reasons with the community, so that they can come up with proposals that will more likely get approved. -committee
How difficult is it to open scoring up to the general public? People may be interesting on voting for proposals anonymously, though they won't comment on talkpages. -committee
Self-service tools and systems for matching people and problems with solutions can help grantees develop ideas, share skills, learn from past projects, measure outcomes, and more. These tools could help scale up human support, but they still have room for further development in order to meet our needs.
Make it easy to post an idea, and improve it as a proposal if many people like it (that's just what idea lab does). But maybe make it more easier. -grantee
Area to improve: get more feedback early in the proposal process to grant submitters and recruit more people to participate in the idealab -committee
I'm really encouraged by learning dissemination tools such as the Learning Patterns Library. I'm also anxiously waiting for WikiMetrics to be robust enough for me to use so I don't have to rely on database queries from research staff alone. -grantee
I think we needed more technology support both for the committee and for the proposers, and short of that limited our ability. -committee
Least liked: Lack of promises of tools proposals to integrate into the wiki. -committee
Please process the wiki integration of tools proposals and voting software concerns mentioned earlier. Thanks. ☺ -committee
let's get more committee members with expert knowledge especially developers -committee
How could WMF help me do my work better? Give me some software developer staff time for few months, and actually set some sister projects in the Priority list :-) -grantee
Tech proposals could be better regarding integration. If it's difficult to ask developer's time, we could ask for training of volunteer developers among proposers (to use WMF Labs and Wikimedia Gerrit, etc). Professional comments we got from WMF developers were invaluable, so I would hope to continue asking for these. -committee
Perhaps my biggest suggestion to improve IEG is to just make sure that more people know about it. It's been such an amazing and invaluable growth and learning experience, but I think many editors don't quite grasp its potential or that it's for them too. Programs that harness individual editor talents to broad impact and so important to our movement. -grantee
Suggestion: More outreach for IEG, especially among those working with open knowledge communities. -committee
I think we should make the split more visible in types of proposals (technical, community, and so forth), and also let people suggest new types (specific wikiproject subjects like trains, fossils, rivers, cultural heritage, or GLAM outreach, Geographical outreach (Global South), minority outreach (Women, people with no available internet at home, young/older people, and so forth). This will help draw reviewers in from specific parts of the community. -committee
Calls, text, and wikis all have their pros and cons.
Well, Hangouts and Skype are very useful, provided you don't have connection issues, in which case they are frustrating. I always had bandwidth issues, so this always drives me crazy :-D Moreover, English is not my native language and it's not really easy to speak it in front of people you don't actually see. But this is not a real issue, or something that can be easily solves if you all live in different continents. -grantee
I had a lot of trouble with the scoring process, and I felt it was frustrating to make comments on proposals and then check back for reactions. I also expected a chance to discuss proposals with other committee members, but I realize time differences are prohibitive for this. On the whole I felt I benefitted from all comments coming through the mailing list and onwiki. -committee
I wish a better and smoother system to allow avoiding COI by separating venues of discussions more easily, as what we did in R2 was rather ad hoc. -committee
I missed some requests posted on the talk page for some time. Maybe discussions should happen elsewhere. -proposer
We can take think more creatively about the proposal process, to take better advantage of Wikimedia's strengths.
I think creating and submitting a proposal at WMF is not different from creating and submitting one to any other foundation. this might be perceived by some as a sign of professionalization; personally i consider it just the emulation of well-known process and i expect the wikimedia community to do better and to do differently; because the wikimedia community is different. As in all proposal creation and submission the capacity of creating a good proposal (not a good idea but a good application) wins. I don't think this is what we necessarily want. -proposer
Allow proposals in my native language or honor if I translate the proposal into multiple languages. -proposer
Continue to consolidate and streamline pages/processes
As our pages and processes continue to develop over time, we can avoid the sprawl of information by occasionally taking a step back to consolidate.
As for structure, I have a feeling that relevant meta wiki pages are getting a bit too many, and that is making hard for us to keep track of which discussion was taking place where. We might want to consider merging some of them or creating something like a sitemap (just for navigation, not for providing quick information). -committee
I think the information for new grantees are complete. I think there is a little too much repetition. maybe the creative commons format with information divided into simplified and legal can be a good way of presenting them. we receive a lot (email with the information, legal text with the non legal information inside, guidelines online) and i find it a little too much. -proposer
I hope to see the review process shortened overall (and it seems already planned). -committee
I think a checklist for what we used for scoring should be used for each review and these should be in the back of the reviewer's head while reviewing and commenting. Only relevant and personal comments are needed in the early stages, I would hate to see templated comments on the talk pages of proposals in early stages. -committee
Tools proposals may need additional criteria: level of wiki integration -committee
I think the criterion need to be more specific for outreach proposals. Most of the outreach proposals not innovative are not getting through. We could save user times by doing that. -committee
demands for small grants were labeled "unrealistic." I won't request more (than the small amount I asked for) as this would be complicated with local tax and I really don't like to ask more for what I have in mind because it should be fun and not paid work. But in general it was an experience and I am likely to submit something in the next round. -proposer
I think the jury considers low budget and high budget a criteria; this is not bad but it is not an evaluation of the budget according to the actual costs of the activities; i think they would need some support to understand if a budget is proportionate (so low or high) or not. There is some background knowledge which can help the jury taking its decision. -proposer
As round 1 2013 grantees complete their 6 month projects, they were asked to take a brief survey to help us understand what's working well and what could be improved in the program. The findings from grantees are based on responses from those who took the survey in January 2014.
After decisions were announced for IEG's second round, we collected feedback from 2 surveys of participants in January 2014, via methods similar to those previously used. The findings from grant proposers (both selected and not-selected for an IEG) and committee members are based on responses from those who took these surveys in January 2014. Where appropriate, anonymized committee feedback from discussions on-wiki or on the mailing list has also be used.
Demographics of those surveyed are generally in-trend with overall makeup of IEG and with Wikimedia projects - proposers, for example, cited their home-wikis as being on Commons, Wikidata, and English, German, Hebrew, and Bengali Wikipedia, male respondents outnumbered female respondents, etc. About half of the proposers who responded to the survey had their project funded.
