Grants:APG/Proposals/2015-2016 round1/Wikimedia Argentina/Staff proposal assessment/es

The staff proposal assessment is one of the inputs into the Funds Dissemination Committee (FDC) proposal review process, and is reviewed by the FDC in preparation for and during the in-person deliberations each round. The purpose of the staff proposal assessment is to offer the FDC an overview of the expert opinions of the FDC staff team about this annual plan grant proposal, and includes (1) An overview about all applications in the current round; (2) A narrative assessment; (3) An assessment that ranks each applicant according to specific criteria in program design, organizational effectiveness, and budgeting.

Overview for all assessments

High quality of proposals and annual plans

Proposals in this round have significantly improved in quality. Proposals are accompanied by detailed strategic plans, and some strategic plans include ways to monitor long term progress over time. With a few exceptions, proposals have been more focused and organized into 2-3 programs with common objectives. Most proposals have been more clearly focused on program work and have included less of an emphasis on operational details. Organizations have included logic models and other clear explanations (e.g. needs assessments and community surveys) to demonstrate why their work is needed and how proposed activities are likely to lead to results.

We appreciate the significant effort that has gone into these proposals, and also believe that the high quality of these proposals will aid the Funds Dissemination Committee in their understanding of each organization’s planned work. Furthermore, quality proposals with clear objectives will provide a strong foundation for each organization’s evaluation of their results in the coming year.

Better targets and effective storytelling

The inclusion of global metrics targets for each proposal has enabled a better understanding of the entire portfolio of Annual Plan Grant proposals in Round 1, and should help show the impact of Annual Plan Grant organizations as a group in a few key areas.

We know global metrics don’t tell the entire story of an organization’s impact. Organizations have included program-specific metrics that measure results that are specific to their work. We see organizations thinking about ways to understand the quality of the content contributed through their work, as well as ways the content is used by contributors (e.g. use rates for media), or readers (e.g. pageviews). Organizations are doing important work that is not yet reflected in global metrics, such as work in the areas of policy advocacy, building the MediaWiki community, and attempts to address community health. Organizations are striving to find good ways to show their achievements in these areas, and to link these achievements to online impact. We appreciate the different ways organizations are working to show the impact of their programs through metrics that are specific to their contexts, and we know this work will continue to be important moving forward.

Beyond the metrics, organizations are improving in their abilities to effectively tell stories and document lessons learned, both in terms of organizational learning and program learning. We have seen significant improvement from last year, with several organizations becoming systematic in the way they document and share their stories. We appreciate where organizations have highlighted important aspects of their work through effective storytelling.

Community health

This year, several organizations are using the lens of community health to structure their work with online contributors. Wikimedia Nederland is doing innovative work around training Arbitration Committee members and administrators. Amical Wikimedia, Wikimedia Nederland, and Wikimedia Israel are addressing online conflicts in their communities. We encourage them to share what they learn about this developing approach with the broader movement and Wikimedia Foundation, as we all stand to gain from these experiences.

Beyond this work, we have seen many organizations implement community surveys to gain a better understanding of the communities they work with and their specific characteristics and challenges. We see movement organizations as clear leaders in this area, since most have direct relationships with their communities and are well-equipped to address challenges that are specific to the geographies and languages they work with. We appreciate that learning how to address challenges in specific Wikimedia communities may be used to improve our understanding of community health in the broader Wikimedia context, which is one reason we see this work as so important.

Many organizations are looking beyond Wikipedia and Commons communities, to focus on other projects. Wikimedia Sverige and Wikimedia CH are both focusing on Wikidata this year, and Wikimedia Israel and Wikimedia Serbia are continuing their strong work on Wiktionary. A number of organizations are focusing on other projects like Wikisource and Wikivoyage, which may offer significant opportunities for improving content and increasing participation and reach.

Policy and legal work

Organizations are focusing on policy areas like copyright reform in a consistent and coordinated way. Wikimedia organizations already have a track record of participating in policy discussions. Some organizations, like Wikimedia Sverige, are seen by others as policy experts in their regions.

However, organizations do not yet have a strong track record with influencing policy, as it remains a nascent area of work. Plans focus on processes and mechanics of policy works rather than identifying specific and achievable policy goals.

Outside of policy work, organizations are also working on supporting contributors with legal expertise (e.g. Wikimedia CH and its work on copyright guidance for users on Commons).

In general, we see this policy work as well-aligned with Wikimedia Foundation’s strategies in these areas. We encourage organizations to continue to articulate how these policy goals benefit Wikimedia projects. We acknowledge that it is challenging to measure the outcomes of this work, and encourage further discussion.

Technology work

Organizations are continuing to request funding for technical work, with a focus on technology tools. In general, we have seen an improvement in plans for technology work and have seen these projects proposed on a smaller scale. Some organizations already have expertise in these areas, while others are working to build more expertise. Beyond Wikimedia Deutschland’s software development plans, Wikimedia Sverige is working with its community to address bug reporting, and Wikimedia Sverige and Wikimedia CH are building relevant tools to expand the use of Wikipedia (text-to-speech, Kiwix and Wikimini). Wikimedia UK is also continuing work on VLE (Virtual Learning Environment) and QRpedia. Wikimedia Israel is working on ways to support technical contributors to MediaWiki, and Wikimedia Ukraine is planning to support the development of gadgets relevant to its community.

Coordination among organizations

Organizations are continuing to coordinate with one another to improve movement practices and accomplish shared goals. Wikimedia Nederland, Wikimedia Österreich, Wikimedia Ukraine, and Amical stand out as high performers in documenting learning and improving movement practices, and Wikimedia Israel, Wikimedia Argentina, and Wikimedia Sverige, are making important contributions too.

Wikimedia Argentina and Wikimedia Ukraine are taking leadership roles in regional collaborations like Iberocoop and the CEE regional group. Organizations are collaborating across languages (e.g. Amical and Wikimedia Sverige), as well as within languages (e.g. DACH). A number of organizations are supporting international program groups and convenings, including Wikimedia Ukraine (Wiki Loves Earth), Wikimedia Nederland (GLAM conference), Wikimedia Österreich (Wikisource conference), Wikimedia Israel (Hackathon). Organizations in Europe are coordinating around EU advocacy issues.

Beyond technology tools, organizations like Wikimedia Österreich, Wikimedia Nederland, Wikimedia Sverige and Wikimedia Israel are working on tools for advocacy work and education work that can benefit the broader movement. Organizations are also engaging actively with the Community Engagement team at Wikimedia Foundation to coordinate with Wikimedia Foundation on movement learning.

Quality of budgets

Budgets in this round varied in quality. Some organizations are still struggling to include an appropriate amount of detail in their budget. We believe more conversation is necessary to agree on the level of detail needed in these budgets, and to establish shared norms for good practices in budgeting and financial reporting. WMF plans to provide more in-depth guidance for budgets in the future.

Wikimedia Sverige and Wikimedia Nederland have performed consistently in the area of making quality budgets and offering transparent financial reporting, and we’ve also seen significant improvement from smaller organizations like Amical and Wikimedia Serbia. Several organizations included staff time and costs for each program. This is a good practice, which can help organizations demonstrate how staff are contributing to different program areas, and tracking this closely will also help organizations understand how their resources are being allocated.

In some cases, however, we received budgets that included line large line items without sufficient detail for understanding the costs. This means that we could not draw strong conclusions about how budgets were weighted toward impact in many cases.

Finally, we noticed large amounts allocated to travel expenses, printing and promotional materials, and it is sometimes difficult to link these significant expenses to commensurate impact. Where expenses are significant, we would like to see clearer links to impact.

Pilot: one restricted grant request

The FDC makes general support grants to support programmatic and operational costs. For the first time, the FDC has received a request for a restricted grant, from Wikimedia Deutschland. We are piloting this approach to specifically support Wikimedia Deutschland’s software development focus. In a restricted grant, funds are allocated to specific line items in an organization’s budget.

Overview

Resumen

	Current (projected)	Upcoming (proposed)	Proposed change (as a +/- percentage)
FDC or PEG funding	$212,000.00	$241,614.00	14%
Budget	$212,000.00	$241,614.00	14%
Staff	3.5	3.5	0%

Overview of strengths and concerns

This section summarizes the strengths and concerns identified by staff, which are explained in the detailed narrative section of this assessment.

Strengths

• Strong focus on diversity (global south organization, focus on gender)
• Leadership in movement practices at a regional level (Iberocoop)
• Some approaches may be scalable (e.g. online training in education program)

Concerns

• Targets too low with respect to funding requested and number of staff
• Results improved recently, but past results do not correspond to past funding
• Plan is overly complex, will likely be difficult to track

Staff proposal assessment narrative

This section takes an in-depth look at this organization's past performance and current plans.

Context and effectiveness

This section takes a close look at this organization's context. Here are some questions we will consider:

Environment

How does this organization's environment (including its local context and its community) enable this plan to succeed and have impact?

• There are significant opportunities to expand reach to those not yet served by or contributing to the Wikimedia projects. We see potential for this both through Wikimedia Argentina’s focus at their national level, as well as in the Spanish-speaking world (through its regional work).
• Wikimedia Argentina operates in a large geographic area, and aims to expand its work beyond the capital of Buenos Aires. Wikimedia Argentina is operating in the context of the country government’s embrace of inclusive technology, which may enable opportunities for partnerships.
• Beyond focusing on Spanish Wikipedia, which is an important opportunity to expand the reach of the Wikimedia projects, Wikimedia Argentina works across several other Wikimedia projects, including Wikivoyage and Wikisource.

Past performance

Will this organization's past performance with program implementation enable this plan to succeed?

• Wikimedia Argentina has shown improved results in 2015, already exceeding its accomplishments of the past several years, which is a good indication that future results will improve.
• Wikimedia Argentina has shown stronger results in terms of outputs around participation (e.g. numbers of participants or new editors involved), while content-focused online outcomes (e.g. images in use, articles) have not been strong. It’s important to note that outputs related to participation may be a good indicator of Wikimedia Argentina’s expanding reach, which is particularly important in a context where there are significant opportunities for reaching new people. Yet, at this level of funding, those output indicators should be complemented by more evidence of significant online impact (e.g. content, contributors retained), or other scalable achievements.
• Wikimedia Argentina has been laying important foundations to build future work, and these accomplishments do not come through in the global metrics. While we know this work is important and that Wikimedia Argentina faces particular challenges in its working environment, which they have explained well, we are still concerned about the lack of significant online outcomes over the past several years of significant support.

Organizational effectiveness

Will this organization's overall effectiveness enable this plan to succeed?

• After a year of many staff transitions in 2014, Wikimedia Argentina has had a year of stable staff, having managed these transitions effectively.
• With the plan proposed, Wikimedia Argentina may struggle in evaluation, as there are an overwhelming number of metrics that they will need to track.
• Wikimedia Argentina has continues to lead in the Iberocoop region, contributing to improving movement practices among this key group of organizations.
• Wikimedia Argentina is documenting its learning, although in some cases, ineffective strategies, programs, and approaches are still being pursued: for example, Wikimedia Argentina’s overly complex approach to planning and the continuation of the Federalization approach, which has been adapted from a stand-alone program to a cross-cutting strategy.
• Wikimedia Argentina conducted a community survey to learn more about how to engage more effectively with its community.

Strategy and programs

This section takes a close look at this organization's programs. Here are some questions we will consider:

Strategy

Does this organization have a high-quality strategic plan in place, and are programs well-aligned with this strategic plan?

• Wikimedia Argentina has a new strategic plan, which includes a clear way to evaluate long term results. Among the four strategic program areas identified in this plan, participation and liberation of content are strongly aligned with movement strategic priorities.
• Wikimedia Argentina’s current plan is well-aligned with its strategic plan, as it incorporates these four strategic lines across its three programs (education, GLAM, community support).

Programs

Do proposed programs have specific, measurable, time-bound objectives with clear targets, and are program activities and results logically linked? Are there specific programs with a high potential for impact?

• Wikimedia Argentina has continued to prioritize gender and diversity in its work, incorporating diversity-relevant targets into its program objectives and doing specific program work toward these objectives.
• In addition to improving on results during the current year, Wikimedia Argentina is setting higher targets (particularly in GLAM). Targets that show online impact in the areas of content and online participation (e.g. contributor retention) are still not high with respect to the amount of funding requested, although Wikimedia Argentina is setting strong targets for involving more participants and editors in its work.
• We are pleased to see Wikimedia Argentina set more ambitious target for the upcoming year. Based on past performance, some of the proposed targets may not be realistic. For instance, Wikimedia Argentina has created or improved 617 articles so far in 2015, while their target for 2016 is 3,310. We realize that activities scheduled in the second half of the year may impact 2015 performance.
• Wikimedia Argentina’s education work is building momentum. If successful, the online training approach could result in impact on a national scale, or beyond.
• It will be important for Wikimedia Argentina to track students and teachers retained as part of this program, and Wikimedia Argentina is focusing on that now. It will also be important to link program activities to online impact. Given the expanding program, engaging more volunteers, scaling work, and building a platform for materials, may be too much for Wikimedia Argentina to take on.
• Of Wikimedia Argentina’s three programs, community support is currently projecting the smallest results, and has a large budget compared with other programs.

Budget

Is this plan and budget focused on the programs with the highest potential for online impact?

• Wikimedia Argentina is planning to increase its budget from $212,000 to $241,680, which is a significant increase at 14%.
• This increase is almost entirely funded through Annual Plan Grant funding, since opportunities for fundraising are rare in Argentina.
• We acknowledge the significant challenges of the devaluation of the Argentine Peso. At the same time, we need to evaluate the overall impact of each organization’s work with respect to the total amount of movement funding received. In this sense, while we understand that currency fluctuations are a large part of Wikimedia Argentina’s requested increase, we are still concerned about this significant increase in budget given our concerns about past performance.
• Aún cuando el programa más costoso de Wikimedia Argentina es el Soporte Comunitario por $37,923, este programa no es probable que sea el programa de mayor impacto, lo que genera preocupación en que el presupuesto no esté orientado hacia generar impacto.

Resumen de la opinión de expertos (si corresponde)

This section will summarize any expert opinions or other research.

N/A

Staff proposal assessment framework

This framework assesses annual plan grant proposals across the three dimensions of (1) Program design, (2) Organizational effectiveness, and (3) Budgeting. To complete the assessment, we identify whether each criterion is a strength or a concern:

• Major strength
• Strength
• Neither a strength nor a concern
• Concern
• Major concern

Criterion	Assessment	Description
Program design
P1. Strategy	Fortaleza	Wikimedia Argentina tiene ahora un plan estratégico de calidad que incluye formas para evaluar progreso de largo plazo, y algunos de sus programas están fuertemente alineados con su estrategia y con las prioridades del movimiento. La racionalidad para el alineamiento estratégico está claramente explicado.
P2. Potential for impact at scale	Ninguno	WMAR está planeando programas que tienen protencial para escalar regionalmente a través de recursos y entrenamientos en línea. La organización ha incluido algunos objetivos de resultados importantes para el 2016. WMAR tiene potencial para expandir su alcance al invitar a nuevos y activos editores en sus actividades. Al mismo tiempo, algunos resultados deseados son bajos, y el rendimiento pasado no es un indicador fuerte de impacto con escala, especialmente que corresponda con el monto solicitado.
P3. Objectives and evaluation methods	Preocupación	El plan complicado y extremadamente detallado de WMAR será un desafío de rastrear efectivamente. Continuamos luchando por entender el progreso de WMAR año a año, dado el enfoque inconsistente a la evaluación.
P4. Diversity	Fortaleza principal	La diversidad es una fortaleza principal para WMAR. WMAR aborda concienzudamente con contribuciones femeninas y minoritarias como parte de sus programas, e incluye objetivos relacionados a esas audiencias. Aún más, WMAR está trabajando con comunidades emergentes en el sur global y está enfocado en el contenido del sur global en su trabajo.
Organizational effectiveness
O1. Past results	Preocupación	A pesar de la inversión continuada, todavía no vemos que los resultados en línea sean adecuados con el financiamiento. Admitimos que WMAR ha hecho un trabajo fundacional importante con la educación y ahora con GLAM, pero todavía no lo hemos visto crecer en resultados cuantificables más allá de los participantes. Vemos que la organización está mejorando sus resultados y los motivamos a continuar en esta dirección.
O2. Learning	Ninguno	WMAR está documentando su aprendizaje correctamente. Todavía no vemos evidencia de que WMAR esté aplicando aprendizaje efectivamente. Por ejemplo, a veces vemos a WMAR continuar persiguiendo programas y formatos que en el pasado no han sido efectivos (e.g. Federalización, diseño de programas demasiado complejos).
O3. Improving movement practices	Fortaleza	WMAR continua liderando el movimiento de colaboración latinoamericano a través de Iberocoop.
O4. Community engagement	Fortalezas	WMAR está haciendo su primera encuesta comunitaria, la cual fue de alta calidad. La organización comprometió a la comunidad en el desarrollo de sus planes estratégicos y anuales.
O5. Capacity	Preocupación	Estamos preocupados por la factibilidad de este complicado plan, particularmente dada la evaluación requerida para completarlo satisfactoriamente.
Budget
B1. Past budgeting and spending	Ninguno	WMAR ha preparado presupuestos en forma efectiva en el pasado, y lo está gastando enfocadamente, aunque los resultados pasados no son adecuados al financiamiento pasado. Grandes cantidades han sido entregadas para viajes y merchandise.
B2. Budget is focused on impact	Preocupación	Los resultados propuestos no es adecuado al financiamiento. El número de personas es alto relativo al impacto propuesto.

This staff proposal assessment is the work of FDC staff and is submitted by: KLove (WMF) (talk) 21:53, 9 November 2015 (UTC)

Staff proposal assessment framework

• Major strength. This is something the organization does very well, and this is a strong indicator of future success.
• Strength. This is something that the organization does well, and this could indicate future success.
• Neither a strength nor a concern. This is something that does not indicate future success or make funding the application a risk, or aspects of this criterion conflict.
• Concern. This is something that the organization does not do well, and this could make funding the application a risk.
• Major concern. This is an area where the organization is not strong, and this could make funding the application a serious risk.

Criterion	Description
Program design
P1. Strategy	The organization has a quality strategic plan in place, programs are aligned with this strategy, and this strategy is aligned with online impact.
P2. Potential for impact at scale	Programs could lead to significant online impact at scale, and corresponding to the amount of funds requested.
P3. Métodos de Evaluación	Los programas incluyen un plan para medir los resultados y asegurar aprendizaje, y emplear herramientas y sistemas de evaluación en forma efectiva. Los programas incluyen objetivos y modelos lógicos.
P4. Diversity	Programs will expand the participation in and reach of the Wikimedia movement, especially in parts of the world or among groups that are not currently well-served.
Efectividad organizacional
O1. Resultados anteriores	Esta organización ha tenido éxito con programas o enfoques similares en el pasado, y ha medido y documentado efectivamente el resultado de su trabajo pasado.
O3. Aprendizaje	Esta organización está haciendo frente a los riesgos y a los cambios en forma efectiva, está aprendiendo y documentando sus experiencias, y está aplicando lo aprendido para mejorar sus programas.
O4. Mejorando prácticas del movimiento	Esta organización comparte efectivamente el aprendizaje sobre su trabajo con el movimiento en general y más allá, y ayuda a otros en el movimiento a obtener mayor impacto.
O5. Compromiso comunitario	Esta organización atrae comunidades y voluntarios en forma efectiva en la planeación e implementación de su trabajo.
O6. Capacidad	Esta organización tiene los recursos y la capacidad (por ejemplo liderazgo, experticia, personal, experiencia en el manejo de fondos) para ejecutar el plan propuesto.
Presupuesto
B1. Presupuestos anteriores y gasto	Esta organización tiene una historia de preparar presupuestos realistamente y de manejar fondos en forma efectiva en el pasado.
B2. El presupuesto está enfocado en impacto programático	Basado en el desempeño anterior y los planes actuales, los fondos están asignados a programas y actividades con potencial para tener impacto programático.