Grants:APG/Proposals/2013-2014 round1/Wikimedia India/Staff proposal assessment

Wikimedia India received a funds allocation in the amount of 3,325,000 INR (equal to about US$53,000) in 2013-2014 Round 1. This funds allocation was recommended by the Funds Dissemination Committee on 15 November 2013 and approved by the WMF Board of Trustees on 31 December 2013.

Applicants should read this document to familiarise themselves with the assessment criteria; however, it should be edited only by FDC-related staff. The staff will use the document to set out their objective assessment of each funding request from eligible entities – specifically, to determine the extent to which plans align with the mission goals of the Wikimedia movement, and the extent to which the entity is well-placed to execute those plans effectively. The staff will base this report on proposals from entities and input from the community and other stakeholders (including, where required, deeper due-diligence on the entity's previous track record within the Wikimedia movement). The staff will provide completed reports to the FDC, which will then make a recommendation to the Board of Trustees on funding allocations. See a note from FDC staff explaining the process is more detail here: FDC portal/Note on staff proposal assessments 2013-14 Round 1.

Methodology behind the 2013-2014 Round 1 FDC staff assessments

The FDC staff assessments provide a snapshot of the staff review of Annual Plan Grant (FDC) proposals, and identify both strengths and concerns with the work proposed.

We did this through both portfolio analysis - looking across proposals and calibrating across different organizations - as well as assessing the context of each individual organization and its proposal. We did our best to be as nuanced as possible. For example, we understand that an organization that is beginning to formalize structures and bring on staff may have far fewer resources for learning and evaluation processes than an organization with a longer history and full-time staff. At the same time, a larger organization is also likely to have complex program structures and processes, that reflect in its use of metrics and measures.
We significantly increased the inputs to the staff assessment process from last year. They include:
- The proposals themselves, with the attached documents (strategic plans, annual plans, detailed budgets etc)
- Past and current reports (FDC quarterly progress reports, grant reports etc)
- A portfolio view of financials across proposals: FDC_portal/Proposals/2013-2014_round1/Financial_overview.
- Internal financial, evaluation and compliance inputs from the WMF Finance, Programs, Legal and Grantmaking teams.
We used a few key principles for the assessment, across all proposals. These are not new, and are detailed in the narrative and scoring sections of the assessment. They include:
- Impact on Wikimedia projects and the global movement. We looked for the rationale behind the work organizations do, aligning them to the Wikimedia strategic priorities and goals. We looked particularly for direct and indirect impact of this work on Wikimedia projects; for example, growth in contributors and content donation. Most importantly, we looked at whether the funds requested were justified in the context of current and potential impact of the proposed work.
- Organizational strategy, leadership and governance. We looked for evidence that an organization had a good understanding of its own context, and the needs, opportunities and challenges of the communities it seeks to serve or partner with. We looked for board, staff and volunteer leadership that is committed, effective and engaged with these communities and practising movement values of openness and transparency.
- Rates of growth and financial management. In this round, only two returning applicants asked for funds within the recommended maximum growth rate in movement resources (annual plan grants/FDC allocations) of 20%, which is itself a significant rate in many contexts. In addition, many organizations are underspending on previous grant allocations and/or have extensive reserves, which makes overall growth rates even higher. We appreciated organizations that were not spending for spending’s sake, and being careful and prudent about resources; we hope this continues. However, significant underspends (particularly on programs) across a number of organizations flag the challenges of budgeting, planning and program design. It may be that the most effective and impactful programs do not always need significant resources. We specifically looked at the nature of long-time underspends in the context of organizations with a history of overbudgetting and underspending.
- Program design, learning and evaluation. We recognize that the Wikimedia movement at large is still developing a more nuanced understanding of how we measure our impact, as volunteers and as organizations. We looked at each organization’s self-defined metrics and measures for success, both qualitative and quantitative. We searched for indicators that went beyond process (or outputs) and looked at intended outcomes and specific goals or targets. We recognized, however, that many organizations are in the process of establishing baselines for their work, and incorporating learning into program design and re-design. We appreciated organizations that were open about these challenges and were honest and reflective about their own capacities and systems.

Note: We shared our nearly final draft assessments with all FDC applicants a day ahead of publishing them, so that organizations had some lead time for sharing these with their Board and staff. We then took all responses into consideration where possible, making factual changes and clarifying our statements where necessary. We encouraged organizations to note all substantive comments and concerns on the Discussion pages of these assessments.

Proposal overview

Name of eligible entity	Wikimedia India
Summary staff assessment [substantial concerns: none/some/many]	Many concerns
Total amount proposed to the FDC for the upcoming year annual plan in currency requested	11,054,802 INR
Estimate of total amount proposed to the FDC for the upcoming year annual plan in US dollars	175,771 USD
Exchange rate used to calculate estimate in US dollars and date accessed (mm/dd/yy)	0.0159 US dollars to 1 Indian Rupee (calculated on 1 October 2013)
Date of proposal submission	1 October 2013
Adherence to proposal process [yes/no]	Yes
Adherence to legal, regulatory, and policy regulations [yes/no]	Yes
Adherence to US anti-terrorism laws [yes/no]	Yes

Financial summary

A detailed summary of the financial information from all proposals is here: financial overview document. Movement resources are defined as what the entity received through FDC allocations or grants or what the entity retained via payment processing in the prior funding cycle.

	Projected 2013	Proposed 2014	Proposed change (as a +/− %)
Movement resources*	Not confirmed.	11,054,802 INR	Not confirmed.
Overall budget	1,744,843 INR	1,763,648 INR	788%
Number of staff	0.5	3	500%

Annual plan

Overview of staff assessment

Strengths

What are the top strengths of the plan?

Success in Open Educational Resources (OER) work
High-potential region for significant global impact on Wikimedia projects; plan uses good contextual data to highlight opportunities
Language communities in India are gaining momentum and have some committed volunteers

Concerns

What are the top concerns with the plan?

Proposed amount is too high for context, with an overly ambitious plan
Focus on conferences and other events may not yield high impact
Too large a portion of the budget for operational costs
Governance and leadership challenges could impact decision-making and execution of plan

Strategic alignment

To what extent does the plan address the strategic goals and priorities of the movement?

The plan is somewhat unevenly aligned with the strategic priorities: 63% of the funding in this request is for four conferences and events (totaling US $47,774) and operational costs (office rent, equipment and furniture are budgeted at US $35,416, Governance, Legal and Insurance are budgeted at US$ 26,048, and Office administration is budgeted at US$ 1,506; the total for these costs is US$ 62,970). (See, expenses by program for more details
The Capacity Building and GLAM programs are aligned with Participation, with an ambitious goal of raising the number of active contributors in India by 50% (from 450 to 675). The proportion of funds requested for this is 16.5%.
WMIN has a detailed strategic planning document that is currently in draft form.

Context of the entity and community

To what extent does the context of the entity (e.g., community participation, fundraising context, sociopolitical environment) enable or inhibit the entity's potential for impact? To what extent does the entity's experience enable or inhibit the entity's potential for impact?

India is one of the global regions showing an increase in active editors, with a few language communities starting to gain traction and momentum. Most very active contributors have strong commitment to Wikimedia projects, although community participation in India is uneven, with some burnout of significant contributors.
Geographic coordination and coordination across online language or thematic communities can be difficult given India’s size, diversity and complexity.
WMIN has experience working with community members to host a national conference in 2011, and successful Wiki Loves Monuments competitions in the past two years.
The chapter Board or Executive Committee (EC) hired a (part-time) executive manager in late 2012; his work so far has been focused on administration.
The lack of public coordination and cooperation with a well-established organization like Centre for Internet and Society (CIS), which supported the chapter in its start-up phase, is surprising. WMIN’s plan does not address potential complementary work with CIS, which may be important given the complexity and scale of India.
Current governance and leadership challenges within the Executive Committee are concerning. A recent report noted that there was a lack of disclosure before the chapter elections around potential conflict of interest amongst three EC members. The response from the EC did not acknowledge the implications on the elections, but documented the need for creating a Conflict of Interest policy.

Feasibility and risks for the annual plan

How feasible is the annual plan? Specifically:

To what extent are the budget, staffing, availability of volunteers, and other resources realistic for achieving the anticipated outcomes?
Are the timelines realistic for the proposed initiatives and associated activities?
Are there acknowledged and/or unacknowledged risks in the proposal?

The amount requested from the FDC is significant in the Indian context, with a rate of growth in the last year in movement resources of 526%, and a rate of growth in its annual budget of 784%. (In the past two years WMIN received about US$ 44,840 in WMF grants; this includes a 2013 grant of US$ 28,166. In addition, in 2011, the chapter shared a grant with the Mumbai and Pune communities of US$ 40,000.)

Feasibility

WMIN is working in a region with a high potential for global impact, and an environment with many potential new contributors and readers as well as currently active volunteers could enable programmatic success.
The chapter has had past successes in contests (Wiki Loves Monuments work) and content donation (NCERT work), which are ideas with potential for impact.

Risks

WMIN proposes to increase staff from one part-time staff to three full-time staff. For a chapter that is moving from an all-volunteer structure to some full-time staffing, this is too ambitious in scope and scale at this point in time.
The plan concentrates too strongly on events with four national and regional conferences planned along with a number of wiki-academies and editing workshops. These events are high in cost and labor-intensive, and require high levels of volunteer energy and commitment, and do not have proven outcomes or impact on Wikimedia projects. The plan also looks at a number of new initiatives that are high-intensity projects and long-term investments, like digitisation.
The chapter has acknowledged the risk of board and volunteer burnout, but the scope of this plan seems to increase the risks of such burnout.
The chapter has not acknowledged the opportunities - and the need, given India’s diversity - of working in a more inclusive way across the country, and leveraging existing organizations and community groups. The Special Interest Groups (SIG) policy mentioned in the proposal may be a step in this direction, but WMIN has not been uniformly open to supporting multiple community-based groups. For example, the EC discouraged a group of MediaWiki developers from becoming a Wikimedia User Group (as the group originally intended) at the global movement level. While the EC then suggested the group become a SIG, this has not happened so far.
The chapter has applied for funds after activities have been completed, without budgeting for these activities in advance. WMIN has also reallocated underspent project funds without it being approved in the grants, including in its hire of the executive manager; reallocation approval was sought only after the proactive urging of the Grants staff.
The governance challenges of the EC have implications for its own internal decision-making as well as its relationship with the community, which has been seeking transparency and openness. This may have impact on the overall execution of the plan.

Summary of expert opinions and/or community commentary

Summary of the opinions of subject-matter experts on the plan

None requested.

Summary of community commentary on the plan

For detailed comments from the community, the FDC and FDC staff, as well as responses from the organizations themselves, please visit the proposal form discussion page. The summary offers a brief overview of the questions or comments from community members, and is not an endorsement of the views expressed on the proposal form discussion page.
A few community members asked a number of questions about the plan, including the kind of ‘lobbying’ required by the chapter to work with the government on content donation, the efficacy of working on a QRpedia type project rather than content donation with GLAM institutions, the relative usefulness of digitization projects and the nature of the Special Interest Groups (SIG) policy.
Concerns were expressed about the relatively high operational costs, the experience needed by the Board and current staff to manage a plan of this size and some of the measures used for quality.
A few community members agreed that wiki workshops in different cities were important.

Scores, comments and feedback

Scores provide only a preliminary assessment for consideration by the FDC. The rubric for scoring is below.

Dimensions	Criteria	Score (1–5)	Comments and feedback
Potential for impact against strategic priorities	(A) The plan addresses the Wikimedia movement strategic priorities of the Wikimedia movement (infrastructure, participation, quality, reach, and innovation)	3.00	Programs in this plan address strategic priorities, including Participation, but the rationale presented is unclear and targets may not be realistic.
	(B) TThe initiatives in the plan have the potential to lead to movement strategic priorities, if executed well	3.50	High potential for impact in India on many language projects, but significant investment in events is unlikely to lead to impact on global targets.
	(C) The initiatives in the plan have the potential to lead to significant impact, given context and amount of funds proposed	2.00	Programs are unlikely to lead to significant impact and the amount requested is too large for their context.
Ability to execute	(D) The entity has the resources, skills, and capacity (staff, volunteer, non-staff) for the plan to succeed	2.00	While there is an active and committed community in India, the staff and board do not have the capacity to execute this plan.
	(E) The entity has a record of success in similar initiatives	2.00	WMIN lacks experience in many areas of work described in this proposal.
	(F) The entity's leadership is effective, committed, and relatively stable	2.00	Serious concerns about leadership and governance include poor management of past grants, poor budgeting, ineffectiveness at working with other groups and their current response to governance and CoI concerns.
Efficient use of funds	(G) The funding requested is reasonable given the proposed initiatives	1.50	The plan is far too expensive given the costs of operating in India, the budget overemphasizes events and operational costs, budgeting for programs is not realistic.
Efficient use of funds	(H) The entity has a record of using funds efficiently and staying on budget	1.50	Very poor record of budgetary management and compliance with respect to past grants.
Quality of proposed measures of success	(I) Clear indicators of success are outlined in the plan	3.00	Metrics across most programs are focused on outputs not outcomes; however, the proposal includes some clear targets and uses contextual data to highlight opportunities.
	(J) A plan is in place to track the proposed metrics	3.00	Provides list of indicators to be tracked.
	(K) The entity has a feasible plan in place to track the proposed metrics	2.00	It will be challenging for WMIN to both implement the programs and track all of the indicators it has listed; capacity is not proven from past performance.
Potential to add knowledge and/or other benefits to the movement	(L) The initiatives in the plan, if successful, could be helpful and/or productively adapted elsewhere in the movement	3.00	Work with contests and content donation may have potential to be adapted if executed successfully; the focus on events is not innovative.
	(M) The entity is ready to share with the movement the lessons learned from the initiatives	1.50	Poor track record with reporting regularly; uneven collaboration with other groups in their region (no mention of CIS in this proposal); weak in discussing challenges.

Scoring rubric

Dimensions	Evaluation criteria	1 = Weak or no alignment with criterion	3 = Moderate alignment with criterion	5 = Strong alignment with criterion
Potential for impact against strategic priorities	(A) The plan addresses the Wikimedia movement strategic priorities	The plan will not directly address any strategic priorities	The plan will partially address strategic priorities or other goals that are closely related to the strategic priorities	The plan will directly address several strategic priorities
	(B) The initiatives have the potential to lead to the strategic priorities, if executed well	The initiatives have no clear connection to strategic priorities	Some or all of the initiatives have an unclear relationship to strategic priorities	The initiatives have been shown (by the entity or the movement) to lead to strategic priorities
	(C) The initiatives have the potential to lead to significant impact, given context and amount of funds proposed	The initiatives will have marginal impact in the local community and the broader Wikimedia community, given context and amount of funds proposed	The initiatives proposed may have a significant impact in the local community, given context and amount of funds proposed	The initiatives could have significant impact at scale, both in the entity's local community and in the broader movement, given context and amount of funds proposed
Ability to execute	(D) The entity has the resources, skills, and capacity (staff, volunteer, non-staff) for the plan to succeed	The entity has neither the number nor the amount of resources, skills, and capacity needed for success.	The entity has some relevant staff and volunteer capacity, but may be under-resourced for some initiatives	The entity is well-resourced with the relevant skills and capacity for success
	(E) The entity has a record of success in similar initiatives	The entity has had trouble succeeding in similar initiatives	The entity has not engaged in similar initiatives in the past	The entity has conducted similar initiatives in the past and has been successful in achieving desired results
	(F) The entity's leadership is effective, committed, and relatively stable	Leadership has been unstable (e.g., changing on an annual basis)	Current leadership is committed to the entity; some concerns exist around stability or effectiveness of leadership	Current and past leaders have demonstrated the ability to develop and execute on strategic plans; leadership team has been stable and transitions have been smooth
Efficient use of funds	(G) The funding requested is reasonable given the proposed initiatives	Based on FDC experience and relative to other proposals, the plan contains significant over- or under-budgeting, and/or has not thoroughly accounted for costs	Based on FDC experience and relative to other proposals, the budget is generally consistent with the initiatives; some initiatives are likely to be over- or under-budgeted	Based on FDC experience and relative to other proposals, the budget is reasonable, and rigorous cost projections have been presented
Efficient use of funds	(H) The entity has a record of using funds efficiently and staying on budget	The entity has a poor record of budgetary management: e.g., consistently and significantly below budget (such as 50% underspending), and drawing on organizational reserves due to overspending	The entity has a middling record of budgetary management; e.g., generally on-budget for its biggest initiatives and cost items, properly recording any over- and under-spending, and with plans in place for improvements	The entity has consistently strong record of financial management; e.g., consistently staying on or below budget (within ~5% of budget), and putting surplus funds towards the entity's and/or movement's goals
Quality of proposed measures of success	(I) Clear indicators of success are outlined in the plan	Initiatives have no associated indicators of success, or the proposed indicators are unrealistic and/or unmeasurable	Each initiative has indicators of success, but at least some indicators are unrealistic or are not clearly connected to the initiative	Each initiative has a realistic and carefully framed set of indicators of success associated with them, which are rigorous and time-bound
	(J) A system is in place to track the proposed metrics	The plan does not include methodology or timing for tracking and collecting metrics to measure progress against proposed indicators of success	A proposal to track progress against indicators exists, but has flawed methodology or unrealistic timelines and initiatives	The plan includes timing and tracking of proposed metrics, and additional initiatives (e.g., surveys, research) to track indicators
	(K) The entity has a feasible system to track the proposed metrics	No systems are in place to track metrics, or staff or volunteer capacity to manage metrics tracking	There is some overall capacity (e.g., staff, volunteers) to track and monitor metrics, but the entity does not clearly have the relevant skills to do this	High-quality systems, and the necessary skills and capacity to manage these systems, are in place
Potential to add knowledge and/or other benefits to the movement	(L) The initiatives, if successful, could be helpful and/or productively adaptedelsewhere in the movement	The initiatives would not be relevant to other movement groups	Some initiatives may be relevant to other movement groups	Many initiatives proposed are new or have seldom been tried elsewhere in the movement; if successful, other movement entities would benefit from doing similar initiatives
	(M) The entity is ready to share with the movement the lessons learned from the initiatives	The entity is isolated from other movement groups and rarely shares information; the entity's leaders and members seldom participate in movement-wide functions	The entity is somewhat active within the movement, and is developing more relationships with other movement entities with which it could share lessons learned	The entity has regular communication with other movement entities, is a regular participant in Wikimedia events, and has a track record of sharing with others

This form was created by: KLove (WMF) (talk) 00:43, 9 November 2013 (UTC)