Grants:Simple/Pilot

Pilot plan

Application form and process ready by 1 October 2015. Done
Applications will be submitted by 1 November 2015 for funding beginning in January 2016. Done
Decisions for the first applications will be made by 15 December, and grants will start 1 January. Done
Reporting format will be ready by ~~1 January 2016~~ 1 February 2016 (midpoint) and 1 May 2016 (final). Done
We will do an evaluation of the first round of applications ~~in February~~ in September 2016. By this time, some grantees will also be testing out the new reporting format. Done
A second evaluation will be done in September 2017, taking into account data from the second round of 12-month grants and the first round of completed grants.
We will continue to take applications throughout the year. In progress…

Applicant participants

The Simple APG funding option accepts applications throughout the year, but applicants requesting funding for January are required to submit applications by 1 November since there is an application freeze during November and December. This has resulted in the largest cluster of applications being reviewed during the months of November and December, with smaller groups of 2-3 applications submitted in March, May, and October. The timeline below illustrates grant terms and application cycles during the pilot period.

2016

Wikimedia Czech Republic - granted $44,900 of $44,900 requested.
Wikimedia District of Columbia - granted $63,827 of $63,827 reuqested.
Wikimedia Eesti - granted $64,900 of $67,100 requested.
Wikimedia Espana - granted $65,700 of $70,000 requested (in the form of two 6 month grants).
Wikimedia India - did not complete application, no longer eligible.
Wikimedia New York City - granted $74,000 of $85,000 requested.
Wikimedia Suomi - granted $25,600 of $25,600 requested.
Shared Knowledge - granted $31,900 of $52,200 requested, a legally incorporated organization.
Wiki Education Brazil User Group - $20,300 granted of $21,000 requested (for 6 months), not a legally incorporated organization.

2017

Wikimedia Czech Republic -$81,000 of $81,000 requested
Wikimedia Eesti - $68,800 of $68,800 requested
Wikimedia Espana - $65,000 of $65,000 requested
Wikimedia Indonesia -
Shared Knowledge - $31,100 of $47,000 requested
Wiki Education Brazil User Group - $32,870 of $32,870 requested (6 months)
Art+Feminism - $99,820 of $99,820 requested
Wikimedia Community Ireland -

Committee member participants

The committee is currently composed of eight active members, and there is one open position on the committee. These eight members include six male committee members and two female committee members, and committee members are located in seven different countries in Central and Eastern Europe, North America, Western Europe, South Asia, East Asia, and the Middle East. Committee members have experience with Wikipedia as well as several non-Wikipedia projects, including Wiktionary, Wikisource, and Commons. Committee members have experience as leaders or members in Wikimedia Chapters and Wikimedia User Groups, including geographic and thematic groups, and many years of collective experience in the Wikimedia Movement. Most committee members have served on another grants committee before joining the Simple APG Committee, including the Funds Dissemination Committee, the Individual Engagement Grants Committee, and the Grants Advisory Committee. Several have experience making grants or applying for grants as part of a chapter or user group.

The inaugural committee consisted of nine members:

Addis Wang, , 1 October 2015 - 31 July 2016
Anders Wennersten, , 1 October 2015 - 31 July 2016
Chinmayi S.K., 1 February 2016 - 31 January 2017
Ido Ivri, 1 October 2015 - 31 July 2016
Kirill Lokshin, , 1 October 2015 - 31 July 2016
Kiril Simeonevski, 1 October 2015 - 31 July 2016
Nataliia Tymkiv, 1 October 2015 - 31 July 2016
Pete Ekman, 1 October 2015 - 31 July 2016
Sydney Poore, 1 October 2015 - 31 July 2016

In August 2016, two members of the inaugural committee were offboarded due to inactivity (Pete Ekman, Kiril Simeonevski), and one resigned due to conflicting commitments (Nataliia Tymkiv). Two new members were inducted in order to replace them:

Nikola Kalchev, 1 August 2016 - 31 July 2017
Linar Khalitov, 1 August 2016 - 31 July 2017

In September 2016, this is the composition of the committee, with terms listed:

Addis Wang, 1 August 2016 - 31 January 2017 (serving since October 2015)
Anders Wennersten, 1 August 2016 - 31 January 2017 (serving since October 2015)
Kirill Lokshin, 1 August 2016 - 31 January 2017 (serving since October 2015)
Ido Ivri, 1 August 2016 - 31 January 2017 (serving since October 2015)
Sydney Poore, 1 August 2016 - 31 January 2017 (serving since October 2015)
Chinmayi S.K., 1 February 2016 - 31 January 2017
Nikola Kalchev, 1 August 2016 - 31 July 2017
Linar Khalitov, 1 August 2016 - 31 July 2017

Midpoint report

This report includes information from committee members, staff, and applicants during the first 12 months of the pilot period, from September 2015 to September 2016, including:

A report on how the program is meeting the needs of groups and organizations that need annual funding, including a summary of results through September 2016.
An analysis of what we learned about the application process from feedback from committee members, applicants, and staff.

Overall, applicants and committee members are satisfied with their experience, and the pilot appears to be meeting the needs it was designed to address. We have identified some specific areas for improvement in the application process, including the application forms on Meta and the process for writing committee recommendations. Many thanks to all applicants and committee members who provided feedback for the purposes of this evaluation.

Simple APG application process survey results

Applicants and committee members were surveyed about their experiences with the application process in January and July 2016. Survey results indicated that applicants and committee members are satisfied with their overall experience with the Simple APG application process, that the process is less time-consuming for them than the FDC process, that resources that are considered important by applicants need some improvement while resources considered important by committee members are adequate, and that staff and committee support is good or adequate.

Experience of applicants and committee members

Both applicant respondents and committee member respondents are positive about their overall experience with the Simple Annual Plan Grants funding option. 67% of applicant respondents agree or strongly agree that they are satisfied with their overall experience, while 33% neither agree nor disagree; 83% of committee members agree or strongly agree that they are satisfied with their overall experience on the committee and indicate that serving on the committee is a rewarding experience for them, while 16.67% neither agree nor disagree. Most applicants agree they have enough opportunities to improve upon and discuss their applications (100%), while fewer applicants agree that the committee's recommendations are timely and useful enough (56%). Most committee members agree they have enough opportunities to influence and engage in the decisionmaking process (86%), while fewer committee members agree that they have enough time to make decisions (71%).

Time spent on the application process

While the time spent on the application process reported by applicants and committee members is greatly reduced compared with the time reported by FDC applicants, applicants and committee members are still spending a significant amount of time on the application process.

“

Spending 30 hours on an application in one month does require some personal commitment, especially as we were all volunteers.

”

Applicants reported that they spent an average of 47 hours on their applications. 40% of applicant respondents indicated this was too much time, 40% indicated it was just right, 10% indicated it was not enough time, and 10% were unsure. Compare this with an average of 112 hours reported by applicants as spent on their FDC applications in 2015-2016 Round 1. Applicants also provided some useful information in their qualitative responses, to help us interpret this data. One applicant respondent noted that much of this time was spent on elements of “restructuring discussions and planning, which took (and are taking) lot of our time”. Another applicant respondent noted that “it's difficult to know if a work done is for the APG or simply for the development of the chapter”, and that “this is our first annual plan and grant, so I think that we spent more time than in a next process.”

Committee member respondents spent an average of 17 hours on the fall round of applications, which included four applicants. 57% of committee members responded that this amount of time was just right, 29% responded that it was not enough, and 14% of committee members reported that they were unsure. The number of hours spent is likely to increase significantly over the course of the year, as rolling applications are accepted and evaluated. Even so, committee members are likely to spend less than the average 120 hours spent by FDC members on reviewing applications for 2015-2016 Round 1 over the course of the year.

Resources

As may be seen from the detailed breakdown below, the highest quality resources are not yet aligned with the resources ranked as most important by applicant respondents; therefore, it may be advisable to find ways to improve the resources ranked as highest quality applicants. For example, collaboration on budget planning documents was ranked as lower quality than the other resources, but was ranked by 100% of applicants as important. Discussions with committee members on Meta, WMF's written decisions, and calls with WMF were all ranked important by 100% of applicants but had lower quality ratings than the other resources considered by applicants. Qualitative data reveals that the application forms and budget templates are a significant pain point for a number of applicants; therefore, this may also be a priority area for considering improvements.

In the qualitative feedback surrounding committee resources, committee members highlighted the need for better coordination of online discussions and the need for training materials for committee members. As may be seen from the detailed breakdown below, the highest quality resources are well-aligned with those ranked as important by committee members; therefore, committee resources may not require significant adjustments, although new resources that are not considered here may need to be developed.

Staff support and committee support

“

Efficient, clear, practical. I felt that the staff was clearly a trainer or instructor for me and the board was the one that made decision.

”

Staff support was rated good or excellent by 95% of respondents. 55% of applicant and committee respondents rated staff support as excellent, 40% rated staff support as good, and 5% (one respondent) rated staff support as poor. Qualitative feedback about staff support from committee member and applicant respondents focused on several ideas: (1) clear communication; (2) speed and responsiveness; (3) quality of feedback and analysis; (4) staff in a supportive role, ensuring applicants and committee members have the autonomy to make decisions.

Committee support was rated excellent by 13% of applicants respondents, as good by 37% of applicant respondents, and as fair by 50% of applicant respondents. No applicant respondents rated committee support as poor. One applicant respondent qualified that, “Rating is good not excellent, because their support compares to WMF staff and this is not fair for the committee.” Applicants had positive comments about the committee’s support, focusing on the committee’s attentiveness, engagement, and helpfulness.

Committee and applicant self-evaluation

Committee members rated their own performance less positively than applicant respondents rated their performance. 28.5% rated themselves as excellent, 28.5% rated themselves as good, 14% as fair, and 29% as poor. Committee members expressed a desire to improve their understanding of applicant backgrounds, and communicate more with applicants. They reported that they did a good job with providing quality evaluations and starting conversations with applicants, as well as sharing points of view with their fellow committee members.

No applicants rated themselves as excellent, 62% of applicants rated themselves as good, 25% as fair, and 13% as poor. In particular, applicants expressed a desire to improve their ability to manage the timing of the application process and the accuracy of the data and information they needed to participate. Applicants cited past experience with other grants programs as a factor that contributed to their satisfaction with their own performance as applicants. Committee members were not asked to evaluate applicant performance, but several responses in other areas of the survey pointed to some areas where committee members would like to see applicants improve, including in the quality of supporting documentation and the timeliness of applicant responses.

“

We need to improve what we know about ourselves.

”

The applicants

10 chapters and user groups participated in the pilot as applicants by September 2016.

By September 2016, a total of $327,000 US dollars has been granted to 7 grantees, with an average request of $59,000 US dollars and an average grant amount of $47,000, compared with an average grant amount of $227,614 in Round 1 2015-2016 of the FDC process.
On average, grantees received an increase of +120% over their most recent grant year, compared with an average increase of +23% in Round 1 2015-2016 of the FDC process.
By July 2016, grantees had met or exceeded targeted performance against global metrics. Some are prepared to enter the FDC process in the 2018 or 2019 grant years.
Applicants have made use of the flexible rolling application cycle, submitting applications throughout the calendar year, with most 12-month applications submitted in October and November.
4 of 7 grantees receive significant funding from a source other than WMF grants.
6 of 7 grantees received some funding for staff through their Simple APG by September 2016, and 4 of 7 were hiring staff for the first time.
3 o 7 grantees are located in the Central and Eastern Europe region, 2 of 7 grantees are located in Western Europe, 1 of 7 grantees is located in Central and South America, and 1 of 7 grantees are located in North America.

The committee

11 committee members have participated by September 2016; 8 are currently active in September 2016.

Volunteer committee members have a diversity of experience as program leaders, chapter leaders, user group members, and members of different Wikimedia committees.
Committee members use a combination of tools and methods borrowed from the FDC and IEGCom.
Committee members use eligibilty assessments and financial analysis provided by staff, as well as an early assessment tool adapted from the FDC process and a committee assessment report adapted from the IEG process.
Committee members disucuss applications on calls with fellow committee members, discussion pages on Meta, using the committee Email list, and through calls with applicants.
The committee has been able to reach a consensus in 100% of cases. WMF has approved 100% of the committee's recommendations. Before September 2016, the committee has not needed to use tools such as the "Gradients of Agreement" used by the FDC.
WMF staff has approved the committee's full recommendation for 100% of applications during the pilot period. Unlike the FDC, the committee submits their recommendations to WMF staff rather than the WMF Board of Trustees. This greatly reduces the amount of time and formality required for the approval process.

Challenges identified and what's working well in September 2016

What's working well for the committee in September 2016

Committee members reported a positive experience with working remotely together.
Committee members valued dedicated staff support, including quality analysis to assist in making decisions. Several committee members requested more of this.
The recruitment process was successful in recruiting a diverse committee, including committee members from different geographic regions and speaking different languages, with a diverse range of experience with programs, organizations, groups, and other committees in the Wikimedia movement.
Committee members have been mentoring applicants and grantees; grantees expressed appreciation for this approach or desired more mentorship.

What's working well for staff in September 2016

Diverse, engaged, and high-functioning committee is providing high quality review and supporting many aspects of the pilot program. From a staff perspective, this has been critical to the success of every aspect of this program. We have learned a lot (through our various grants programs) about ways to help committees work, but a lot of this depends on the people involved. Continuing to recruit such high quality committee members may become a challenge at some point, but right now the composition of the committee is a clear strength. We do know that significant staff time is required to support a high-funcitoning committee, and help committee members have a positive, engaging experience. Staff time is required both to facilitate committe discussions and to provide the committee with the information and resources they need to make decisions.
Applicants are highly engaged throughout the eligibility and application processes, and are working in partnership with WMF. Starting conversations early on in the eligibility process is an important opportunity for growing relationships, sharing a clear understanding of the process and deadlines, and creating shared goals for WMF and the applicants. Staff time is critical to this process, as providing one-on-one support takes time.
High quality administrative support and expertise in grantmaking provided by the CR Team at WMF is the backbone of the program. This is especially critical when our need to be inclusive creates challenging situations for our team (see challenges below).

Challenges identified by the committee in September 2016

The inaugural group of committee members were the first to serve on the Simple Annual Plan Grants Committee, and stepped into this role without a long history or a wealth of training materials. Materials and methods for training and orientation can be improved to increase the confidence of committee members and help build relevant skills. Committee members recommended more training be provided.
Committee member respondents were challenged with making good decisions within the timelines set for each application. Applications were reviewed in several rounds to accommodate a rolling application schedule, which created a particular challenge with managing committee workload. The rolling schedule has made it more difficult for committee members to plan in advance and ensure adequate engagement at the right times. Committee members recommended formalizing the program's procedures, finding a way to encourage more timely responses form applicants, increasing the time for writing the committee's recommendations, grouping applications into rounds, more engagement with the financial details of each proposal (including ways to facilitate discussions about this aspect of applications), making a plan to scale work as the number of applications increases, and more interactions between committee members and applicants. Relatedly, during the pilot round, many processes and timelines were defined on an ad hoc basis and processes and timelines were treated flexibly and adapted according to emerging needs. While this is a normal part of a new process, committee member respondents commented that the committee's work would benefit from a more stable process and timeline in the future.
Committee members identified challenges with tracking discussions with committee members and conversations with applicants. It was challenging to conduct effective conversations over Email, and to schedule meetings across multiple timezones. Committee members recommended more frequent face to face meetings, attempting to include all committee members in one call rather than splitting sessions, and a system to track conversations with applicants and committee members.
Several committee member responses commented that they would like to improve how engaged committee members were with the financial details of the applications. For example, a spreadsheet that is open to comments.

Challenges identified by staff in September 2016

Balancing diversity and inclusivity with inflexible organizational requirements from WMF, and the need for consistency and fairness across funding options. Creating a simple and flexible process that will work well for such a diverse pool of potential grantees is a major design challenge. We are working to increase the diversity of this pool of grantees by implementing broader eligibility requirements, providing more support, and making program requirements more flexible. At the same time, we are held to some rigid guidelines about how to whom we are able to send funds.
Balancing committee needs for time and structure and applicant needs for higher quality feedback, with applicant needs for speed and simplicity in the application process. The committee wants more time, higher quality resources, and more structure in the review process; while applicants want speed, flexibiltiy, and simplicity. Applicants also want higher quality feedback through the review process, without sacrificing speed, flexibiltiy, and simplicity. As staff, we're challenged with managing this polarity.
Balancing legitimate desires for growth (especially in staff to support volunteer work) with finding healthy paths for long term organization development and ensuring that growth leads to commensurate impact. We see a continued desire for significant growth in budget and staff from small volunteer-based organizations; while we are eager to support their development, we have also seen the negative effects of funding organizations that aren't yet prepared to execute larger budgets or supervise staff. It is an ongoing challenge to support the goals and growth of groups and organizations in this program, while working with groups and organizations to maintain the positive effects of this growth in the long term, and working as a funder to ensure that significant funding is leading to commensurate impact.
Planning for a future where many more applicants will be eligible for Simple APG funding. If we meet our goals in making this a more inclusive program with a more diverse applicant pool, and if diverse models of affiliation such as user groups and thematic organizations continue to grow in our movement, we can envision a future where the demand for grant money, committee time, and staff time through this program exceeds supply. What will we do?
Risk for a single point of failure in terms of PO support on the CR team. Right now there is a single PO supporting this program. It would be good to build out more support on the CR team to mitigate this concern, as the team stabilizes.

Conclusions

Is the program as currently designed adequately meeting the needs of committee members and applicants, identified through the Reimagining WMF Grants consultation?

Respondents identified the most difficult parts of the WMF grants process:
Respondents ranked applying for grants and reporting as the most difficult.	More information is needed to evaluate reporting. Preliminary data indicates that work is still needed to make the application process easier, particularly in reducing barriers to editing application forms on Meta.
Respondents identified two other priorities, other than speed and simplicity in the application process:
Achieving impact on the Wikimedia projects through grants.	Preliminary results indicate a group of high-performing grantees and qualitative feedback indicates that organizations feel the funding process is helping their organizations achieve more impact through grants.
Speed of the application process.	The time from application to a final decision is reduced from 3 months to 3-6 weeks. The time from eligibility request to decision is reduced from 6 months to 6-12 weeks.
The following resources were ranked as most important by respondents:
1. Connections to others	The funding option is designed to facilitate peer to peer sharing in several ways, but more information is needed to evaluate the effectiveness of connecting peers to one another through this funding option.
2. Budget and financial guidelines	Survey respondents indicated that they were satisfied with the guidance provided, but requested improvements to budget templates in their qualitative responses.
3. Online program resources	Survey respondents indicated that they were satisfied with online program resources that were part of the application process, which included the good practices guidelines on the eligibility page.
4. Specific feedback or coaching	Survey respondents indicated that they were satisfied with staff feedback that was part of the application process and would like improved committee feedback during the application process.

Midpoint recommendations based on what we have learned

This is what we should keep in the process:

Intensive one-on-one support from staff and peer-to-peer connections. Though costly, applicants and committee members have ranked one-to-one staff support highly and feedback and coaching, including more peer-to-peer connections, have been identified as a top priority by respondents in the Reimagining WMF Grants survey.
Inclusive eligibility requirements. By September 2016, we have seen limited participation by non-traditional APG applicants such as user groups or informal groups, but we expect participation to increase in the remainder of the pilot period; to encourage this, we will continue to maintain inclusive eligibility requirements and expand them if necessary.
Speedy application process. A speedy application process was identified as a top priority by applicants in the Reimagining WMF Grants survey, and was ranked as a higher priority than community participation in the grants process. (Speedy applications are not always compatible with rigorous participatory processes. Considering the importance of participatory decision-making in the WMF movement, this is likely to continue to be a polarity that needs to be managed.)

This is what we should keep in the process but improve:

Budget templates. While survey results indicated satisfaction with budget templates, qualitative feedback indicated that improvement is needed in this area.
Financial data available to the committee -- Some committee members expressed a desire to become more involved in this area as part of the review process, and indicated that more resources or analysis or training in this area could facilitate more engagement with applicant financials.
Process for writing committee recommendations -- Applicants expressed a desire for better quality feedback from the committee, and committee members expressed a need for more time or a better process for writing committee recommendations.

This is what we should change about the process:

Format of application forms. The format of the application forms on Meta was consistently identified as a pain point by applicants and staff.
Make forms available prior to deadlines. A few applicants expressed a desire for application and reporting forms to be finalized well before deadlines to help them prepare. This is something that is likely to improve as the funding option formalizes, but more attention can also be paid in this area. Interestingly, this has also been a pain point in the FDC process, as constantly changing forms have been stressful for applicants. We might aim to achieve a more stable application form for Simple APG sooner rather than later.

From FDC process:	A single application each year, that includes all of the applicant's programs, operations, and staff. A holistic view of the organization's impact, that considers operating costs and staff in light of ongoing organizational health and program results. A high quality community-driven participatory process for making decisions about grants.
From IEG:	A functional and efficient committee review process that takes place online, without the need for expensive in-person meetings. An emphasis on intensive one-on-one grantee support through regular check ins rather than an emphasis on formal reporting. An openness to intelligent experimentation.
From PEG:	Broadening eligibility requirements to include groups as well as formal organizations. Evaluating eligibility on a case by case basis, rather than as part of a complicated batch process. Greater flexibility in the timing and content of grant applications.
From TPS:	Simple application forms. A speedy process. Consistent budget guidelines.