Learning and Evaluation/Archive/Share Space/Questions

Take me back to the main Share Space page! no, instead take me back to the main PE&D page!

Archives

Welcome to the Program Evaluation & Design Questions & Answers Forum! This is a space where you can ask anything and everything about program evaluation and design. Here, professionals in the field of program evaluation, program leaders and community members can answer your questions. Please ask and reply in any language you see fit - this is a multilingual project.

Also, be bold and respond to others questions, and feel free to ask for additional help once your question has been responded to!

Please add new questions to the bottom of this page

Where should I report my activities as a WiR funded by my organization internally

Hi, I'm Dorothy. I'm the WiR at the Metropolitan New York Library Council, which is a consortium of ~150 libraries and archives in New York City. My position was initially 2 months, but has recently been extended to 5 months. My position is not funded by WMF, but I want to make sure that I am reporting the progress of my work to the appropriate people and generating information as I go so that others can learn from my experience and so that I can just generally feel more connected to the activities of other WiRs. My work as WiR has been focused on engaging with libraries- prominently museum libraries in New York, and consulting with them about possible Wikipedia projects and events, the GLAM-Wiki movement, discussing how they can incorporate Wikipedia into their daily workflow. I have created a GLAM page for us, where I have been posting updates about our activities. See it here: Wikipedia:GLAM/Metropolitan New York Library Council. I have also created an account with notes from all of the meetings I've had. I have looked at the Edit-a-Thon logic model, and I will use that in the future. But I'm mainly interested in finding out if there are any particular platforms or WMF people that I should be engaging with as a WiR to put information about the trainings/ events that I have had. Wikipedia:User:OR drohowa 17:06, 5 September 2013 (UTC)[reply]

Hi Dorothy, welcome! That's a great project, congratulations on your new role, and it's great it has been extended. It's great that you want to report about your program, even if it's not funded by WMF. It's really important - as it can serve as a teaching tool for other Wikipedians in Residence, the staff that manages them, and can also engage and inspire other organizations to consider a similar role at their own institution. I'm glad the edit-a-thon Logic Model was useful. We're in the process of developing further resources on how to evaluate them, which will be here in the future. There really aren't any WMF people that would be appropriate to ping about your work, unless you're seeking funding or analytics input, however, the best place is generally the GLAMWiki.org community, or the US GLAM-Wiki community. The mailing lists they have are generally the best places to share experiences, and the main GLAMWiki page has a place for case studies and models. And we're always happy here to do our best to help you with evaluation of activities. WMF basically placed GLAM into the communities hands last year, so it's more community focused, and no staff time is being devoted to supporting it, hence why I suggested those two community groups. You can also post your efforts and updates in the This Month in GLAM newsletter. Oh, and if you are interested in writing a blog for the Wikimedia Foundation blog about your work, we totally support that. Hope this helps some! SarahStierch (talk) 17:58, 10 September 2013 (UTC)[reply]

How and when are programs evaluated against strategy?

How should programs - while in design, and while in operation - be measured against strategic goals? Are there good models for continuous comparison or alignment, feedback loops that also include revision of strategies based on experiences, and sharing this evolution as it happens? –SJ talk 02:07, 14 September 2013 (UTC)[reply]

Hi SJ, are you referring to the strategic goals of the Foundation or are you referring to the strategic goals of individual movement entities (chapters, thematic organizations, etc.)? Or both? --Frank Schulenburg (talk) 17:25, 16 September 2013 (UTC)[reply]

This was a general question, not specific to Wikimedia. In our movement, I would be interested particularly in an answer for the WMF's annual program decisions, and strategic goals. –SJ talk 01:50, 7 November 2013 (UTC)[reply]

We have suggested using a logic model as a general mapping of one's program theory (i.e., anticipated cause and effect chain of outcomes to strategic goals and objective targets) and basic road map for guiding program monitoring and evaluation planning (see here) However, the logic model is neither an evaluation nor project management plan, there are several program/project management models for continuous implementation monitoring many of which come from a standard business perspective while others come from more of a social, systems change perspective. There is actually a lengthy article on en.wikipedia as well as you may find some helpful resources on our resources page, however, those links are focused on program evalaution rather than management. Most important is that the project management model fit the ethos of the program and organization and that there is a clear plan, along with frequent and regular review, to prevent things like scope creep or failure to document major changes in implementation. As we tried to emphasize at the Budapest workshop, programming, and its evaluation, is an iterative process and shifts and changes are often a requisite to success in a real world setting. JAnstee (WMF) (talk) 17:57, 16 September 2013 (UTC)[reply]

Thank you JAnstee, this is most helpful. I would like to read your thoughts on how we can apply some of this thinking to Wikimedia programs. For instance, some of our strategic goals do not have clear metrics; nor have we defined them in the past few years. And in some cases there are 3 or more metrics used across the movement for strategic goals such as those for amount of content, proportion of contributors from different demographics, and quality. In other areas, some programs (by various movement entities) have been described after the fact as low-impact. But I have rarely seen this sort of feedback provided with a pointer to specific metrics that should apply; or feedback provided as the program develops, so that it can improve - rather than afterwards, in assessing its worth. Finally, software changes that are rolled out have a detailed philosophy behind them, with implicit metrics that should be able to detect desired changes. However the measurement of those metrics is not always active or prioritized. When community groups give feedback on the effectiveness and impact of a new rollout, they are often observing and reflecting back the impact on some of our goals; though not necessarily the same goals that the designers were targeting. It may help to ahve models for how to welcome the perspective of those directly affected, and how to measure a big project's impact on all related strategic goals (not just those focused on in its design document). –SJ talk 01:50, 7 November 2013 (UTC)[reply]