Grants:APG/Proposals/2013-2014 round2/Wikimedia Norge/Staff proposal assessment

Wikimedia Norge received a funds allocation in the amount of NOK 1,295,000 (~US$215,000) in 2013-2014 Round 2. This funds allocation was recommended by the Funds Dissemination Committee on 24 May 2014 and approved by the WMF Board of Trustees on 28 June 2014.

Applicants should read this document to familiarise themselves with the assessment criteria; however, it should be edited only by FDC-related staff. The staff will use the document to set out their objective assessment of each funding request from eligible entities – specifically, to determine the extent to which plans align with the mission goals of the Wikimedia movement, and the extent to which the organization is well-placed to execute those plans effectively. The staff will base this report on proposals from entities and input from the community and other stakeholders (including, where required, deeper due-diligence on the organization's previous track record within the Wikimedia movement). The staff will provide completed reports to the FDC, which will then make a recommendation to the Board of Trustees on funding allocations.

Methodology behind the 2013-2014 Round 2 FDC staff assessments

The FDC staff assessments provide a snapshot of the staff review of Annual Plan Grant (FDC) proposals, and identify both strengths and concerns with the work proposed.

We did this through both portfolio analysis - looking across proposals and calibrating across different organizations - as well as assessing the context of each individual organization and its proposal.
Inputs into the staff assessment process this round include:
- The proposals themselves, with the attached documents (strategic plans, annual plans, detailed budgets, supplementary documents)
- Past and current reports (quarterly progress reports on annual plan grants, other grant reports, annual financial and narrative reports)
- A portfolio view of financials across proposals for both rounds this year Grants:APG/Proposals/2013-2014_round2/Financial_overview and Grants:APG/Proposals/2013-2014_round1/Financial_overview.
- Internal financial, program evaluation, and grants compliance inputs from the WMF Finance, Programs, Legal, and Grantmaking teams.
- Detailed comments and questions collected during the Community Review period, which are also summarized in these staff assessments.
We used a few key principles for the assessment, across all proposals. These are not new, and are detailed in the narrative and scoring sections of the assessment. They include:
- Impact on Wikimedia projects and the global movement, commensurate with funds requested. We looked for the rationale behind the work organizations do, aligning them to the Wikimedia strategic priorities and goals. We looked particularly for direct and indirect impact of this work on Wikimedia projects; for example, growth in contributors and content donation. Most importantly, we looked at whether the funds requested were justified in the context of current and potential impact of the proposed work.
- Organizational strategy, leadership, and governance. We looked for evidence that an organization had a good understanding of its own context, and the needs, opportunities and challenges of the communities it seeks to serve or partner with. We looked for board, staff and volunteer leadership that is committed, effective and engaged with these communities and practising movement values of openness and transparency.
- Rates of growth and financial sustainability. In this round, no applicants asked for funds within the recommended maximum growth rate in FDC funding of 20%, which is itself a significant rate in many contexts. The minimum growth rate in the FDC/movement funding proposed is 22.7% and the maximum proposed is 216%. Given the challenges faced by some organizations with raising funds externally, there are significant concerns about the long term financial sustainability of this level of dependence on movement resources in areas where other funding may be available.
- Program design, learning, and evaluation. We recognize and appreciate that organizations in this round aim to link their work directly to outcomes on the Wikimedia Projects they are targeting and to work toward aligning their high-level goals across programs; at the same time, this work does not always coincide with clear links between these high-level targets and the direct outcomes of program activities, which may make it difficult to understand the effectiveness of some program activities. We appreciate the good work that organizations have done in this area so far, and urge organizations to continue to work toward making the links between program activities, objectives, and high-level targets clear.
- Inclusion of exceptional organizations. Two organizations were included in this round that required exceptions. The Centre for Internet and Society is the first non-Wikimedia organization to participate in the FDC process, and was made eligible through a resolution by the WMF Board of Trustees. As a result, detailed attention was paid to the financial data presented by CIS to understand the work specific to the Wikimedia movement that will be funded by the FDC, in addition to understanding their broader budget context, and only this work was described in detail in the CIS proposal and analyzed in the staff assessment. Other than this, the process for assessing the CIS proposal has been the same as that for other proposals. The Wikimedia Foundation also participated in this round of the FDC process; at the FDC’s request we did not complete a staff assessment for the Wikimedia Foundation.

Note: We shared our nearly final draft assessments with all FDC applicants a day ahead of publishing them, so that organizations had some lead time for sharing these with their Board and staff. We then took all responses into consideration where possible, making factual changes and clarifying our statements where necessary. We encourage organizations to note all substantive comments and concerns on the discussion pages of these assessments.

Proposal overview

Name of eligible organization	Wikimedia Norge
Summary staff assessment [substantial concerns: none/some/many]	Significant concerns
Total amount proposed to the FDC for the upcoming year annual plan in currency requested	2,600,000 NOK
Estimate of total amount proposed to the FDC for the upcoming year annual plan in US dollars	432,120 USD
Exchange rate used to calculate estimate in US dollars and date accessed (mm/dd/yy)	0.16620, 1 NOK to USD (2014-03-01)
Date of proposal submission	2014-04-01
Adherence to proposal process [yes/no]	yes
Adherence to legal, regulatory, and policy regulations [yes/no]	yes
Adherence to US anti-terrorism laws [yes/no]	yes

Financial summary

A detailed summary of the financial information from all proposals is here: financial overview document.

	Projected 2013	Proposed 2014	Proposed change (as a +/− %)
FDC or WMF funding	US$136,576.18	US$432,120	216.4%
Overall budget	US$144,594	US$520,621.50	260%
Staff costs	US$81,438.00	US$259,687.50	61.1%

Annual plan

Overview of staff assessment

Strengths

What are the top strengths of the plan?

Wikimedia Norge (WMNO) has some ambitious high-level targets directly related to its online areas of focus, and these high-level targets are well-aligned with the strategic priorities of Increasing participation (tracking increases in the numbers of editors and active editors) and Improving quality (tracking increases in the number of articles).
WMNO has opportunities to collaborate with GLAM and other institutions in Norway, and is building on past work in this area, including its successful Wikipedian-in-Residence program.
WMNO has created a framework for understanding and reporting on progress against its strategic plan, which includes an evaluation matrix.

Concerns

What are the top concerns with the plan?

Funds requested are not commensurate with the likely outcomes of this proposal, and the potential for global impact is limited by the size and potential of the language communities served, even if program activities are very successful.
While WMNO has increased staff capacity and is increasing program activity, the impact of its activities on the Wikimedia Projects it serves are not yet known. The high-level targets in this plan may not be realistic or achievable based on past performance, and do not always draw a clear link between the direct results of WMNO’s program activities and the trends it is tracking.
WMNO’s current growth trajectory is not supported by a clear statement of need from the communities it serves, and may be unsustainable. While there are opportunities in its local context for raising funds from other sources, WMNO has not yet succeeded in leveraging these opportunities.

Strategic alignment

To what extent does the plan address the strategic priorities of the movement?

Alignment with the strategic priorities

WMNO’s proposal focuses on three program areas that may be well-aligned with the strategic priorities of Increasing participation and Improving quality: providing support to its community and a welcoming environment for new contributors, developing and managing GLAM partnerships (including Wikipedians in Residence), and an education program (including focused work on the Saami language Wikipedia). However, WMNO sometimes does not offer clear rationale for this strategic alignment in this proposal, nor does WMNO make clear how specific program activities will lead to impact on these strategic priorities.
WMNO is one of the few Wikimedia organizations with a part-time staff person devoted to gender gap work, although the proposal does not include a detailed strategy for this work, and we are interested to see how this work progresses.

Program objectives and measurable outcomes

WMNO’s program objectives are high-level targets with baselines that describe trends in the numbers and types of articles and editors on the Wikimedia Projects on which its work is focused (Bokmaal, Nynorsk, and Saami Wikipedias). Given the small size of the language communities and projects it focuses on, there is an implied link between WMNO’s program activities and this data, although WMNO may face challenges in isolating the results of its work from natural trends in these communities since no plan clearly links the direct results of its activities to these trends. A stronger contribution analysis would be possible if results are demonstrated consistently across the three languages; baseline data over past years may be helpful in showing a significant difference between each Wikipedia before and after WMNO’s interventions.
If plans are in place to measure outputs and outcomes directly related to program activities, those plans and targets are not articulated for most activities described in this proposal. When plans are articulated, they are not always clear or specific. For example, WMNO intends to use wikimetrics to evaluate the regional outreach activities. While wikimetrics may indeed be an effective tool for measuring outcomes and outputs of these activities, WMNO does not specify what it will be measuring with this tool or how it will tie those results to its high-level objectives. As a result, it is sometimes difficult to understand how the activities proposed will lead to these high-level objectives.
While WMNO’s baseline data for program objectives is not included the proposal form, please note that more details, including baselines and specific targets for progress during each quarter, are outlined in the evaluation matrix included with WMNO’s strategic planning document and its reports on its progress against the strategic plan.
Some proposed targets may not be realistic when considered against baseline data and current trends. For example, editorship on Bokmaal and Nynorsk Wikipedias is declining (from 1,749 editors in 2011 to 1,543 editors in 2013, and from 491 active editors in 2011 to 442 active editors in 2013 for Bokmaal Wikipedia and from 55 active editors in 2011 to 32 active editors in 2013 in Nynorsk Wikipedia) despite WMNO’s past programming in line with its strategy, and WMNO intends not only to stabilize this trend in both communities, but to reverse it with a significant increase to 2000 total editors and 1000 active editors in Bokmaal and 100 active editors in Nynorsk.

Strategic planning

WMNO has a strategic plan in place for 2011-15, which is now updated twice a year, and reports on the progress made toward its strategic goals according to an evaluation matrix with statistics related to its online areas of focus. As part of these updates, WMNO also produces a planning matrix divided into specific activity groups, to outline the roles and responsibilities related to program objectives.
WMNO is adapting the programs and strategies of other movement groups to its context; however, the rationale for doing this is not clearly provided in this proposal. We do not know how WMNO has determined that there is a need for the specific volunteer support programs it is implementing or how WMNO has determined that these programs will be successful in its context. For instance, if editing workshops have not been successful in recruiting new editors in other contexts, it is not clear why WMNO expects them to work in its own context.

Context of the organization and community

To what extent does the context of the organization (e.g., community participation, fundraising context, sociopolitical environment) enable or inhibit the organization's potential for impact? To what extent does the organization's experience enable or inhibit the organization's potential for impact?

Online focus and potential for global impact commensurate with requested resources

The potential for global impact is limited with respect to the significant amount of FDC funding requested. Programs targeting Bokmaal, Northern Saami, and Nynorsk language projects may be limited by the size of these Wikipedias and language communities. WMNO could have greater global impact if it clarifies how some of its work - like GLAM partnerships - can generate content that is more easily accessible and usable across the movement.
WMNO’s programs are generally more adaptive than innovative, as WMNO is following other program leaders in GLAM and education and is pursuing strategies for community support that are similar to those being run by other Wikimedia organizations.

Financial sustainability and proposed growth

WMNO is proposing growth in FDC funding from 822,000 NOK (~US$$136,576.18) to 2,600,000 NOK (~US$$432,120.00), or a growth rate of 216%. WMNO received a significant increase (about 164%) in funding beyond the 20% recommended guardrails during the 2012-2013 Round 2 funding cycle, making this a growth rate of 735% over two years, from the ~US$51,785.00 awarded in 2011-2012. WMNO is also proposing to grow its budget from 870,000 NOK (~US$144,594) projected spending to 3,132,500 NOK (~US$520,621.5) planned spending in 2014-15, which is a 260% proposed growth rate.
With the very significant increase in the last FDC funding cycle, WMNO has greatly expanded its programs this last year with three part time staff (1.5 full time positions) after some delays in implementing its plan. WMNO reports that growth has been healthy for the organization and that many opportunities for expanding its work are available (especially in the GLAM sector). However, WMNO is yet to establish the impact of its current programs on the Wikimedia sites and the impact of this rapid growth on the organization.
WMNO acknowledges high operating costs in Norway are an ongoing challenge, and has not yet successfully raised funds as planned in its region. This raises significant long-term concerns about financial sustainability.
At 1,562,500 NOK (~US$259,687.50), staff costs make up almost 50% of the proposed budget and operating costs are at 345,000 NOK (~US57,339) or 11% of the proposed budget. Some of the largest non-staff expenses in this proposal do not align with the proposal’s objectives. For example, the CATI survey work, film work, and the Wikipedia Academy at Tromso at 100,000 NOK (~US$16,620) are grouped with key activities like Wiki loves memories (100,000 NOK (~US$16,620)) and Saami-focused work (500,000 NOK (~US$83,100)) as some of the highest-cost program activities, while their strategic importance is not discussed in-depth in this proposal.

Feasibility and risks for the annual plan

How feasible is the annual plan? Specifically:

To what extent are the budget, staffing, availability of volunteers, and other resources realistic for achieving the anticipated outcomes?
Are the timelines realistic for the proposed initiatives and associated activities?
Are there acknowledged and/or unacknowledged risks in the proposal?

Feasibility strengths

WMNO is achieving success in GLAM partnerships, especially through its work with Wikipedians in Residence, in which it has exceeded its goals, and proposes to continue this work. When WMNO does not yet have experience in a program area, it seeks to build on the work of other Wikimedia organizations.
We appreciate that WMNO is demonstrating increasing capacity for evaluating its work and engaging with the broader movement, as shown by WMNO’s use of an evaluation matrix and increasing engagement with movement reporting.
WMNO is growing and thriving now that it has staff in place to expand its program activities; members of the board are motivated to achieve more and are enthusiastic about current results and future work.

Feasibility concerns

The rate of growth proposed in both total budget and FDC funding is not feasible.
While WMNO is increasing its capacity to evaluate and measure its work, this plan still lacks clear ways for WMNO to understand how effective its activities are and how the results of its activities are clearly linked to its high-level targets.
Some program targets are unrealistic, and the scope of some programs is too broad and ambitious. This is a special concern for an organization still establishing a track record of programmatic success in the movement.
WMNO has acknowledged difficulties engaging members and volunteers in its work in the past and WMNO will need to effectively engage the communities it serves for some of the proposed programs to succeed, but WMNO has not yet demonstrated a capacity to increase engagement.

Summary of expert opinions and/or community commentary

Summary of the opinions of subject-matter experts on the plan

No expert commentary or opinion was requested for this plan.

Summary of community commentary on the plan

Community engagement in the FDC process is relatively limited, perhaps because the FDC proposal form is in English and it is difficult for members of the Nynorsk, Bokmaal, and Saami communities to engage. We acknowledge this is a challenge for organizations with communities that work in languages other than English.

For detailed comments from the community, the FDC and FDC staff, as well as responses from the organizations themselves, please visit the proposal form discussion page. The summary offers a brief overview of the questions or comments from community members, and is not an endorsement of the views expressed on the proposal form discussion page.

A community member asks questions about WMNO’s recruitment process after receiving its first FDC grant and WMNO clarifies that the recruiting process was successful but that staffing plans did change as a result of the funding received.
A community member expresses concerns about WMNO’s not raising external funding commensurate with the opportunities in Norway and about community health; one community member answers these concerns.
A community member expresses concerns that WMNO may not be eligible to receive a particular government grant.

Scores, comments and feedback

Scores provide only a preliminary assessment for consideration by the FDC. The rubric for scoring is below.

Dimensions	Criteria	Score (1–5)	Comments and feedback
Potential for impact against strategic priorities	(A) The plan addresses the Wikimedia movement strategic priorities of the Wikimedia movement (infrastructure, participation, quality, reach, and innovation)	3.50	The plan addresses movement priorities, including Increasing participation and Improving quality, but the rationale for these links is not clear enough in this proposal.
	(B) The initiatives in the plan have the potential to lead to movement strategic priorities, if executed well	2.50	The potential for global impact is limited by the size of the language communities targeted in this proposal, particularly in the context of currently declining editing communities.
	(C) The initiatives in the plan have the potential to lead to significant impact, given context and amount of funds proposed	2.00	The focus of impact is primarily local while the amount requested is considerable.
Ability to execute	(D) The organization has the resources, skills, and capacity (staff, volunteer, non-staff) for the plan to succeed	2.50	WMNO is performing well now that more resources are in place; at the same time, a more solid track record is needed before a plan on this scale is proposed.
	(E) The organization has a record of success in similar initiatives	2.50	WMNO has shown some early success with institutional partnerships, but it needs to establish a firmer track record of success before proposing a growth in program activity at this scale.
	(F) The organization's leadership is effective, committed, and relatively stable	4.00	This organization's leadership is effective and dedicated, while still in the process of managing the transition from volunteer-led to staffed programs.
Efficient use of funds	(G) The funding requested is reasonable given the proposed initiatives	2.00	The proposed growth, and the amount of funds requested, are not commensurate with potential impact.
Efficient use of funds	(H) The organization has a record of using funds efficiently and staying on budget	3.00	WMNO has improved greatly in this area through recent performance on its current grant, but its history with previous grants has been uneven. A more consistent track record is needed before growth is proposed on this scale.
Quality of proposed measures of success	(I) Clear indicators of success are outlined in the plan	2.5	Based on the information presented in this proposal, it will be difficult to understand the direct outcomes of specific activities. While relevant trends on the online projects are tracked, the link between these trends and the proposed activities is not clear. As a result, WMNO may not know whether its programs are successful.
	(J) A plan is in place to track the proposed metrics	3.00	There is a clear plan in place to track relevant trends on the online projects, but WMNO may need a clearer plan to track direct outcomes of the proposed activities.
	(K) The organization has a feasible plan in place to track the proposed metrics	3.00	WMNO has the capacity to track what is being currently proposed.
Potential to add knowledge and/or other benefits to the movement	(L) The initiatives in the plan, if successful, could be helpful and/or productively adapted elsewhere in the movement	2.50	The programs and approaches in this proposal have been adapted from others in the movement, but more analysis is needed to understand their usefulness in the local context. WMNO is currently not proposing any new approaches to its work.
	(M) The organization is ready to share with the movement the lessons learned from the initiatives	3.00	While WMNO has improved in its reporting practices in its most recent reports on its current grant, its previous track record has been uneven. More consistency is needed to justify growth on this scale.

Scoring rubric

Dimensions	Evaluation criteria	1 = Weak or no alignment with criterion	3 = Moderate alignment with criterion	5 = Strong alignment with criterion
Potential for impact against strategic priorities	(A) The plan addresses the Wikimedia movement strategic priorities	The plan will not directly address any strategic priorities	The plan will partially address strategic priorities or other goals that are closely related to the strategic priorities	The plan will directly address several strategic priorities
	(B) The initiatives have the potential to lead to the strategic priorities, if executed well	The initiatives have no clear connection to strategic priorities	Some or all of the initiatives have an unclear relationship to strategic priorities	The initiatives have been shown (by the organization or the movement) to lead to strategic priorities
	(C) The initiatives have the potential to lead to significant impact, given context and amount of funds proposed	The initiatives will have marginal impact in the local community and the broader Wikimedia community, given context and amount of funds proposed	The initiatives proposed may have a significant impact in the local community, given context and amount of funds proposed	The initiatives could have significant impact at scale, both in the organization's local community and in the broader movement, given context and amount of funds proposed
Ability to execute	(D) The organization has the resources, skills, and capacity (staff, volunteer, non-staff) for the plan to succeed	The organization has neither the number nor the amount of resources, skills, and capacity needed for success.	The organization has some relevant staff and volunteer capacity, but may be under-resourced for some initiatives	The organization is well-resourced with the relevant skills and capacity for success
	(E) The organization has a record of success in similar initiatives	The organization has had trouble succeeding in similar initiatives	The organization has not engaged in similar initiatives in the past	The organization has conducted similar initiatives in the past and has been successful in achieving desired results
	(F) The organization's leadership is effective, committed, and relatively stable	Leadership has been unstable (e.g., changing on an annual basis)	Current leadership is committed to the organization; some concerns exist around stability or effectiveness of leadership	Current and past leaders have demonstrated the ability to develop and execute on strategic plans; leadership team has been stable and transitions have been smooth
Efficient use of funds	(G) The funding requested is reasonable given the proposed initiatives	Based on FDC experience and relative to other proposals, the plan contains significant over- or under-budgeting, and/or has not thoroughly accounted for costs	Based on FDC experience and relative to other proposals, the budget is generally consistent with the initiatives; some initiatives are likely to be over- or under-budgeted	Based on FDC experience and relative to other proposals, the budget is reasonable, and rigorous cost projections have been presented
Efficient use of funds	(H) The organization has a record of using funds efficiently and staying on budget	The organization has a poor record of budgetary management: e.g., consistently and significantly below budget (such as 50% underspending), and drawing on organizational reserves due to overspending	The organization has a middling record of budgetary management; e.g., generally on-budget for its biggest initiatives and cost items, properly recording any over- and under-spending, and with plans in place for improvements	The organization has consistently strong record of financial management; e.g., consistently staying on or below budget (within ~5% of budget), and putting surplus funds towards the organization's and/or movement's goals
Quality of proposed measures of success	(I) Clear indicators of success are outlined in the plan	Initiatives have no associated indicators of success, or the proposed indicators are unrealistic and/or unmeasurable	Each initiative has indicators of success, but at least some indicators are unrealistic or are not clearly connected to the initiative	Each initiative has a realistic and carefully framed set of indicators of success associated with them, which are rigorous and time-bound
	(J) A system is in place to track the proposed metrics	The plan does not include methodology or timing for tracking and collecting metrics to measure progress against proposed indicators of success	A proposal to track progress against indicators exists, but has flawed methodology or unrealistic timelines and initiatives	The plan includes timing and tracking of proposed metrics, and additional initiatives (e.g., surveys, research) to track indicators
	(K) The organization has a feasible system to track the proposed metrics	No systems are in place to track metrics, or staff or volunteer capacity to manage metrics tracking	There is some overall capacity (e.g., staff, volunteers) to track and monitor metrics, but the organization does not clearly have the relevant skills to do this	High-quality systems, and the necessary skills and capacity to manage these systems, are in place
Potential to add knowledge and/or other benefits to the movement	(L) The initiatives, if successful, could be helpful and/or productively adapted elsewhere in the movement	The initiatives would not be relevant to other movement groups	Some initiatives may be relevant to other movement groups	Many initiatives proposed are new or have seldom been tried elsewhere in the movement; if successful, other movement entities would benefit from doing similar initiatives
	(M) The entity is ready to share with the movement the lessons learned from the initiatives	The entity is isolated from other movement groups and rarely shares information; the entity's leaders and members seldom participate in movement-wide functions	The entity is somewhat active within the movement, and is developing more relationships with other movement entities with which it could share lessons learned	The entity has regular communication with other movement entities, is a regular participant in Wikimedia events, and has a track record of sharing with others

This form was created by: Winifred Olliff (FDC Support Team) talk 20:17, 8 May 2014 (UTC)