Grants:APG/Proposals/2017-2018 round 1/Wikimedia Serbia/Staff proposal assessment

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) A narrative assessment; (2) An assessment that scores each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview

Summary

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or other relevant funding	$111,931	$118,229	5.63%
Budget	$105,277	$118,229	12.3%
Staff (FTE)	4	4	0%

Overview

This section summarizes the themes that emerged from the proposal, which are explained in the detailed narrative section of this assessment.

Themes

• Improved strategic plan
• Improved program plan
• Qualitative achievements in education
• Still not an FDC chapter

This is a significant year for WMRS, as they have made great strides in creating and implementing an improved strategy in 2017, which led to the development of an improved program plan for 2018. In 2018, they are also building on important achievements in the area of education and continuing to lay a foundation for future GLAM work.

While this plan is a major improvement, it still reads as a list of subprograms grouped into themes, and it may be a challenge to evaluate and understand the results. Despite past recommendations, WMRS is not taking on a leadership role within their region. We still think WMRS would be more at home in the Simple APG program.

Staff proposal assessment narrative

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

WMRS has strong ties with educators and educational institutions in Serbia, and has utilized these connections to advance their work in the education sector. We are interested to see them focusing more on pursuing opportunities in the GLAM sector, and hope this will also lead to good results. In 2017, WMRS worked to improve opportunities in their environment by laying groundwork for these GLAM collaborations, and this will be an ongoing focus in 2018.

WMRS continues to maintain strong ties with the communities of contributors they work with in Serbia, but their lack of deliberate focus in improving diversity may mean that they are missing opportunities to work with underrepresented communities within the country. They also don't address gender gap or other diversity relevant themes in this plan.

WMRS is ideally located to play anactive regional collaboration role through the CEE network, but despite being the most well-funded chapter in this group, they have not taken a leadership role. They don’t have many plans in this area, aside from participation in regional contests.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed?

At the 2016-17 midpoint WMRS is on track to meet their targets, but we note that this upcoming year’s plan does not include several activities where significant achievements were made and targets for 2018 aren’t always aligned with the current year’s achievements. For example, in past years, WMRS pursued many programs that were focused on media acquisition and use in specific content areas, and generated impressive numbers of images in use (some of the best in the entire APG portfolio). Based on this, we think it likely that WMRS will meet their more modest target in this area this year, and we suppose that the reduction in this target is likely due to the change in program emphasis.

In the current year, WMRS is on track to meet their modest yet realistic target in terms of total number of pages with about 3,500 pages out of about 4,000 targeted, but they have included a very ambitious target for 2018 of more than 14,000 pages! Here are some thoughts about that:

• Given WMRS’s ongoing emphasis on education, we are confused to see them set a low target for pages created through the education program work (around 1,100, which is in line with the 2017 achievement), considering that they have already achieved more than 2,000 pages through the education program at the midpoint of 2017.
• We are confused to see a huge leap in pages targeted through the community support program. 2016 achievements were about 300 pages and 2017 achievements are 22 (could be low due to timing of activities like competitions) at the midpoint. It seems that WMRS expects a large number of images (7,000) and images in use (2,400) to be generated through the Animating the Community subprogram. It is not obvious how that will happen from reading this proposal, so we would welcome further explanation in this area to clarify this target.

GLAM is a newer area for WMRS, and so it is also tricky to understand how past results may lead to future impact. WMRS also did not report any specific qualitative results for this program that might give us an indication of how it is working. Results were very modest at just over 200 pages, and may not capture the work WMRS is doing to lay the groundwork for cooperations in this sector. We do note that their success in the education sector may be a positive indicator, as they extend their successful approach to partnerships to their work in GLAM.

WMRS continues to achieve important qualitative successes in the area of education, and this program generates the largest number of pages of any of their programs (over 2,000 at the 2017 midpoint). This year’s achievements include launching the new model of cooperation with the Faculty of Political Sciences where students receive credit for attending independent workshops. They are also further tailoring their work with faculties with expertise in specific fields, and emphasizing new elements such as Wikidata and continuing with successful approaches such as the Wiktionary work in education. WMRS is also expanding their work to partner with other NGOs in the education field. We believe this past success indicates they can do much better than the modest targets they set.

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed?

WMRS has built their staffing capacity over several years in the FDC process, and has not requested a staffing increase in the current plan. We like the way this organization’s staff functions without the need for an Executive Director, which is a model that is very successful for many high performing organizations in Simple APG. It works well for WMRS, and we don’t feel they need to change it. We like that staff are focused on programs, and also that WMRS has the expertise to effectively run their finances, which operate very well for an organization of their size.

We know that WMRS feels that the role of the communication person is an important capacity for their organization, and an area where they need the professional expertise that a full time communications person can provide. However, it may help if WMRS would better explain in future reports the added value of this role, and how it results in improved outcomes for the organization. Having a full time communications person with such a small staff is a unique model in the Wikimedia movement; other organizations could look to WMRS for leadership in this area. We are concerned about seeing this model continued by WMRS and adopted by other small chapters without having more information about how it is working.

Strategy and programs

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan?

We appreciate WMRS’s efforts to create a new strategic plan in 2017, which they have acknowledged was a significant undertaking. We like that they have also created a more strategic annual plan, that is informed by this document. We appreciate that this document includes various techniques for WMRS to analyze their working environment, and that it includes three areas that are focused on relevant long term outcomes rather than operations.

Programs

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

We greatly appreciate that WMRS has adopted a different approach to annual planning, that is in line with a more strategic approach, and we do think that their move in this direction is likely to result in increased impact in the long term. It is not yet clear whether it will do so in 2018.

While a significant improvement, there are still some issues with the program plan. For example, while there are three clear program groupings, it seems that these programs often consist of several subprograms that are not united by common objectives or a common theory of change. In this sense they are more like thematic groupings around topics (e.g. “GLAM”, “Education”, “Community”) than true programs. While this technique is common among Simple APG organizations, it does not reflect the sophisticated approach to program design or evaluation that is on par with what we expect of organizations in the FDC process.

We hope that the new plan sets WMRS to do a better job of evaluating their work than in the past, but we see a few issues in this area. For example, some programs include non-SMART goals, but some also include specific targets; yet the relationship between the two is not explained, and it is not always clear what WMRS hopes to learn from measuring these various aspects of their work. This is exacerbated by this sub-program approach, which is likely to make results difficult to interpret. This is definitely an area that WMRS should tackle within the next grant cycle.

Despite challenges with the program plan, we see that WMRS is poised to make impact in some important areas, and not all of it will be captured adequately by their targets. For example:

• We think it is very likely that WMRS will continue to make important gains in the field of education. They have put a lot of thought into explaining their different approaches in this area. We are interested to see their innovative work with accredited seminars being developed in 2018.
• GLAM is a more nascent program, and we need more time for observation to project whether or not it will be successful. However, we find some of the ideas presented in the proposal very intriguing, especially the way WMRS is looking at GLAM and decentralization, and devising long term strategies to make GLAM partnership work sustainable. This could be very important to achieving good results in this area in the long term!

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

WMRS is requesting a 12% increase over their current year’s projected spending. According to current projections, they are continuing to underspend significantly, though at a lesser margin than in some previous years. Their planned budget in 2018 is very similar to what they planned in 2017, with some adjustments that reflect changes in the program structure.

Summary of expert opinions (if applicable)

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

• Major strength
• Strength
• Neither a strength nor a concern
• Concern
• Major concern

Criterion	Assessment	Description
Program design
P1. Strategy	Strength	WMRS has put a lot of effort into creating a new strategic plan, and has created an annual plan in alignment with their strategic plan. This is a very positive step for the organization.
P2. Potential for impact at scale	Neither	We are very glad to see some improvements in the program structure, which we help will enable WMRS to achieve more impact in the long term.
P3. Objectives and evaluation methods	Neither	We also think this improved program structure will lead to more effective evaluation. We are looking forward to the results.
P4. Diversity	Concern	There are no attempts to prioritize diversity in this plan, and this is in contrast to work other organizations in APG are doing.
Organizational effectiveness
O1. Past results	Neither	Qualitative achievements in the area of education continue to be impressive, as in their large numbers of images in use achieved by the 2017 midpoint. Due to changes in the program structure, we are not sure how this performance will be linked to future success, but we look forward to observing. GLAM work is nascent, and we hope it leads to results in the future.
O2. Learning	Neither	WMRS is not making contributions to movement learning at a level we would expect from an APG organization. Their approach to evaluation and reporting is often confusing but seems to inform some of their decisions. We hope to see improvements in this area accompanying the new program planning approach.
O3. Improving movement practices	Neither	WMRS is not making contributions to improving movement practices at a level we would expect from an APG organization. While they are the most highly funded chapter in CEE, they have yet to take on a leadership role in this region. We hope that their increased focus will help them make that step.
O4. Community engagement	Strength	WMRS continues to be a volunteer-focused organization that involves volunteers in all of its programs at every level.
O5. Capacity	Neither	WMRS has worked to build a stable staff team and withstand changes (such as leave periods).
Budget
B1. Past budgeting and spending	Concern	WMRS continues to underspend.
B2. Budget is focused on impact	Neither	The new program plan is an improvement and will likely to lead to impact this year. We do, however, have some questions about how effectively staff positions are allocated across different areas.

This staff proposal assessment is the work of FDC staff and is submitted by: Morgan Jue (WMF) (talk) 01:34, 9 November 2017 (UTC)

Staff proposal assessment framework

• Major strength. This is something the organization does very well, and this is a strong indicator of future success.
• Strength. This is something that the organization does well, and this could indicate future success.
• Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
• Concern. This is something that the organization does not do well, and this could make funding the application a risk.
• Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion	Description
Program design
P1. Strategy	The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.
P2. Potential for impact at scale	Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested
P3. Evaluation methods	Programs include a plan for measuring results and ensuring learning, and employ effective evaluation tools and systems. Programs include SMART objectives, targets, and logic models.
P4. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Organizational effectiveness
O1. Past results	This organization has had success with similar programs or approaches in the past, and has effectively measured and documented the results of its past work.
O2. Learning	This organization is addressing risks and challenges effectively, is learning from and documenting its experiences, and is applying learning to improve its programs.
O3. Improving movement practices	This organization effectively shares learning about its work with the broader movement and beyond, and helps others in the movement achieve more impact.
O4. Community engagement	This organization effectively engages communities and volunteers in the planning and implementation of its work.
O5. Capacity	This organization has the resources and ability (for example, leadership, expertise, staff, experience managing funds) to do the plan proposed.
Budget
B1. Past budgeting and spending	This organization has a history of budgeting realistically and managing funds effectively in the past.
B2. Budget is focused on programmatic impact	Based on past performance and current plans, funds are allocated to programs and activities with corresponding potential for programmatic impact.