Grants talk:PEG/WM US-DC/Workshop Facilitator Training

Evaluation by the GAC

GAC members who oppose this request

1. Don't thinks the amount of funds requested is commensurate with the impact from this grant, sorry. NLIGuy (talk) 11:53, 3 August 2014 (UTC)
2. See comments of NLIGuy and myself MADe (talk) 18:27, 6 August 2014 (UTC)

GAC comments

The project seems good, but I am not sure about the metrics: how can you guarantee that the participant will later hold their edit-a-thons (with appropriate number of attendees)?

How will you pick out participants? For instance, should I be a applicants “if funding is available”? Of course, I don’t plane to do it – I could be more useful as a co-trainer :-) --Packa (talk) 17:45, 2 August 2014 (UTC)

We can't guarantee that participants will hold workshops, but part of the curriculum is for each participant to come up with a plan for hosting his or her own workshop program. If these plans do not come to fruition, we will figure out why and adjust the program accordingly. A failure to meet objectives is as much a learning opportunity as a success.

Participants are submitting applications to my colleague Emily. So far we have at least 15 applications for 10 spots, and we are accepting applications until the 15th, so I expect it be a competitive process. I am also inviting DC-area participants based on their experience with hosting edit-a-thons. harej (talk) 22:23, 2 August 2014 (UTC)

Thank you, harej, I think you have told about it and you are prepare to do all for success of this course. So I only support my colleagues

• to reduce the accommodation fee: 4 days are OK, but cost per night not,
• and, if is it possible, to limit it for instance only to the East USA/Canada participants (I know, it could not be easy, it depends on the number of applicants, their structure and so on).

Last question: why have you submit your application so late? We need some time for evaluating (particularly during holidays) and you know it, I hope. --Packa (talk) 17:40, 5 August 2014 (UTC)

Speaking purely for myself at this point but I want to see if we can combine some of the days, bringing the length of the program from three days to two and saving costs that way. I regret that the application was submitted late but we were having some trouble finding a partnering library for the event; our choice of partner determines what accommodations we use, meaning huge differences in the budget. The upside of this is that a lot of the event is already planned; should funding be approved we can act quickly to implement. As far as geography of applicants goes, I'll see if we can focus on eastern candidates though I'm not sure there actually is a huge difference in airfare. If we go over the $4,500 budget allotment for travel there are extra (non-WMF) funds we can use. harej (talk) 06:24, 6 August 2014 (UTC)

Thanks! Three days are after my experience appropriate length – the program sounds good for me. I think you should change the hotel – it will reduce also the room expenses. --Packa (talk) 21:43, 7 August 2014 (UTC)

Comments MADe

6k$ to stay in the Marriot? Seems a bit overkill. My first reaction would be to do some research about time/location of the event. As clarified in the grant request, this could drastically limit the accomodation and travel costs. MADe (talk) 14:23, 3 August 2014 (UTC)

I'm not fond of the choice either, but we have few options. The weekend we selected is the only weekend our trainer is available, and an important component of the training is working with a library with access to a rich collection of source material. The only library that was able to accommodate us was at the University of Maryland, which has limited public transportation options. A cheaper hotel, farther from the venue, would have introduced additional transportation costs plus logistical complications, as people unfamiliar to the area would need to find their way to the seminar. While this option costs more, we believe we get more out of having people in one compact area. harej (talk) 17:26, 4 August 2014 (UTC)

Ilario's comment

I am a little bit unsure to understand this project. It would collect people from whole USA and Canada to come in DC and to have a training paid by WMF? Is not it? The project itself is relevant and it may be important but I don't understand why from whole USA and why the costs are not shared. In my opinion cancelling the costs of the travel and putting them in charge of the attendees may give several advantages. One is to have people "really" interested, the second is to reduce the distances and to create a closer community around DC. People from West USA can come and having this workshop if they are really motivated.

I am not convinced that this course can give good advantage if someone from California, for instance, go there. The links may only be virtual afterwards and he/she not well connected, after several time they will be broken. Basically this event may be good if he would create a community, but I am not convinced that this community can survive after the event. In my opinion the plan has to be reviewed and should be limited only to the East USA/Canada. --Ilario (talk) 08:17, 4 August 2014 (UTC)

Making people pay their own travel expenses would dramatically limit attendance. It doesn't matter how much you want to go if you cannot afford to, and we want to make opportunities available to the community regardless of economic limitations. Allowing the participation of even those with limited means is a hallmark of inclusion and of diversity—something Wikimedia needs more of, not less.

Community building is long-term work, and no matter what we do we are not going to see a thousand new chapters bloom overnight. In the meantime, there is very much an opportunity for the participants to build connections with each other, which we observed from GLAM Boot Camp, an event with national participation. Though it was 16 months ago at this point, we are still seeing the benefits from that workshop, particularly as the people who participated in it still talk to each other and rely on each other for social support. This social support is what breeds long-term volunteering, and it is evidence that you can build a community in this way. harej (talk) 17:50, 4 August 2014 (UTC)

Comments by Polimerek

Regarding success measurement - I would expect not only the numbers of future workshops organized by trained people or number of attendees of these workshops but also measurement of retention of the Wikimedia project's users who will participate in future workshops. If the last one is proved to be high - that could be a real surplus for Wikimedia projects... More details of the effectiveness of the similar Chicago initiative would be beneficial. How the effectiveness of the techniques developed at Loyola University Chicago and applied by Emily Temple-Wood to the training the Wikimedia trainers been proved to be effective? Would love to read more about these techniques as we are thinking about similar training however on smaller, local scale... Polimerek (talk) 19:24, 13 August 2014 (UTC)

More impact needed

Hi, thanks for the proposal - it seems well thought of, and well planned. However, I would expect a 16K$ grant to bear more of an impact for the Movement and having held dozens of edit-a-thons and wiki workshops, and having worked in the National Library of Israel for a long while (where many of these workshops took place), I think that having a three-day plan to have people learn to create better workshops or edit-a-thons is quite an investment of time and money that could easily be transformed into a 1-day event, 2 days max, thus cutting costs by 50%. Have you considered the following:

• Pre-preparing some of the materials as videos (so that not only the 15 participants can use them)?
• Creating guides for using library related resources and materials? I'm sure the people from the Wikipedia Library would love to lend a hand in that.
• Dividing participants according to geography and having them do the 3rd day in their area instead of spending a good ~5000$ on accommodation, meals, venue?

I don't want to sound too reprimanding, but I think that while your plan is well thought of, it seems you neglected to explain why the specific method was chosen over much cheaper methods. The only explanation I can see for that is,

"Wikimedia DC has a strong track record of three-day training events, including GLAM Boot Camp
in 2013 which led to the development of highly active outreach volunteers..."

I find the lack of "real", lasting results for such an amount alarming. What materials will stay for the rest of the community? What long term impact do you expect this to bear? And I don't mean more edit-a-thons? As you stated, you don't have any way of ensuring that this will result in concrete improvement of quantity and quality of participation, so I would expect the budget and plan to be commensurate with this fact. I don't get the impression of this here, sorry. I'm open for discussion, however. Let me know what you think. NLIGuy (talk) 11:53, 3 August 2014 (UTC)

Hello NLIGuy. We are planning on recording the presentations and making them available to the public; however, passive viewing of presentations is no substitute for active participation in a seminar. When you participate in person, you can collaborate, ask questions, and get feedback in real time. What you recommend is more in the style of a system of self-help. It is an option, but not the best option. Self-help does not build connections with likeminded volunteers, it does not allow you to get feedback in real time (an important part of learning), and the lack of an existing social network means that there is no ready source of help available. We are looking to invest in the creation of a network of volunteers that supports itself. This network will exist to design programs around recruiting local communities to participate in the sharing of knowledge through writing Wikipedia articles. These programs will be based on a methodology designed by Emily Temple-Wood, an IEG recipient who has seen more editor retention through her events than any Wikimedia chapter. We want others to learn from her success so that we can be more effective in recruiting editors and improving content. harej (talk) 18:04, 4 August 2014 (UTC)

At this point I suppose harej: video cannot substitute, even partially, real interactive presentation. Moreover nobody will see three hour length lecture. --Packa (talk) 21:51, 7 August 2014 (UTC)

Tony1 comments and questions

James, thank you for this proposal. I must say, two weeks before you start spending money is not long for this critical process. Regrettably, I have to agree with many of the points made by NLIGuy, Ilario, and MADe—this is despite my view that both you and Emily are exceptional Wikimedians.

Please note the need for special justification for travel. It's a pity we seem cornered by circumstances into high costs for accommodation. (If the trainer is doing it gratis, I don't mind a decent hotel room and flight paid.) Nearly $17,000 for 10 participants (at the lower end of attendance expectations) means we need to justify $1700 in value per person. This is not helped by the absence of clear themes, skills that might be acquired, in relation to criteria a and b in the table below.

Meetups are hard to justify in terms of direct benefits to readers, but some attempt should be made. In view of these costs, and a key finding of the July 2014 PEG impact learning series, that "the largest cost driver by far was travel; the value should be carefully considered", I wonder whether participants might be asked to pay for a portion of the travel and accommodation (this is common in the real world). Full scholarships for those unable to pay but still keen could be offered. That report also posed "questions to ask when reviewing and executing a program": "clear outcome", "clear way of engaging the community", "clear on-wiki impact".

Random comments:

• "participants will learn how to work with librarians to get access to specialized sources" on the first day. I'm surprised we need to go to such expense to show skilled editors how to gain accesss to specialised sources, and that this should take a whole day. What interests me is the design of the source material portfolio that facilitators might prepare for their own sessions: getting to the core of the skills and experience down the line that participants in editor training and editathons need requires a ready-made portfolio so that online wiki-skills can be the focus. It's hard enough to get people using sources when they're dished up on a plate, let alone diluting the learning with library-based "how to get it in the first place" skills. This is apparently relegated to Day 3; but perhaps it should be the focus of the whole seminar from the start—reverse-engineering the outcomes facilitators want from their trainees.
• A whole day on "intructions for creating a friendly and welcoming space for workshop participants"—really? Better as tips sprinkled throughout the seminar days.
• "username tracking and tracking event attendance rates" as a metric? I'd expect after such largesse and with these small numbers that there wouldn't be much truancy. Or perhaps I'm confusing those attend this seminar with those who the participants might then go back to their local communities to train. Quality assurance might well be part of the curriculum, including questionnaires and follow-up.

Running scores against PEG criteria and strategic priorities.

This scoring table is work in progress. I'd like to know later whether you think it's useful, and soon if you think any of the scores are unfair at this juncture. The general idea is that both scores and certainties should change (hopefully upwards!) as the application and talkpage develop. Do you have the time to beef up the design? If so, these scores could rise significantly.

Key:

• 1 = very weak or no alignment
• 3 = weak alignment
• 5 = passable alignment; room for significant improvement
• 7 = reasonably good alignment
• 10 = excellent alignment

Criterion	Running score	Certainty level of scoring	Notes
(a) High-quality content: Potential for increase at WMF sites (e.g. Spanish Wikipedia, Commons).	2	Medium	It would be more convincing if there were more details about the training methods and the skills expected to be acquired ("newly-acquired skills" are mentioned ... but what will they be?).
(b) High-level participation: Potential for increase at WMF sites.	4	Medium	Some expression of the current problems in facilitating editing sessions would have been helpful for us. What is done badly at the moment? What will you improve?
(c) Volunteers: Their availability, readiness, and skill-base for implementing the activities. Are the scale and scope of the activities commensurate with this?	6	Medium–high	Looks like the minimum of 10 have enrolled. But there's no mention of what criteria were used in selecting them for this subsidy, and whether they're well placed to have an impact on their own communities (which ones?).
(d) Measures of success: Are they realistic and actually measurable in context?	1	Medium–low	I didn't quite understand the DC benchmark. No mention of specific instruments for follow-up, and the time lines for this. Looks like a longer task than the first report will allow for. "This project will primarily be measured according to the number of participants in the training"—circular. Who will be in charge of the sprawling data?
(e) Budget: Matches scale and scope? Strategic justification for labour and materials? Responsible growth/investment?	3	Medium	Can it be tamed?
(f) Sustainability of impact and increase in reach (new partnerships, public awareness of WMF sites). Would the work keep on giving?	6	Low	This score would rise significantly if we could be more certain about the details. There is potential for indirect impact.
(g) Community engagement/support (could include promotion of diversity in the WM movement).	4	Low	I can't get a grip on it.

Tony (talk) 07:25, 11 August 2014 (UTC)

Hello, Tony. I appreciate the extensive feedback. I will respond to your comments point-by-point.

Justification for travel—First, regarding level of participation, we are expecting full capacity for this event. We have over 30 applications for 10 travel scholarships, with those applications solicited through Wikipedia watchlist notices. I am also expecting considerable attendance from the DC community. It would be nice to let more people participate, but this event is specifically an instructional event, not something like a conference with hundreds of people. Thus we need an attendance that reflects a setup that would be appropriate for a classroom, i.e., no more than 15 students. Regarding the justification for travel, we recognize that there is a broad opportunity to engage with organizations and groups throughout the United States, and further, that outreach has often been done by on-the-ground volunteers. Our ambition is to provide support for these volunteers who want to make the leap from Wikipedia editing to in-person Wikipedia outreach.

We have found a particularly effective way of doing this is through small instructional events, where participants have the opportunity to work together and build close connections. It will require further study to figure out how exactly this works, since studying our programs is kind of a new thing to the Wikimedia movement. That said, we've observed from GLAM Boot Camp in 2013 that the attendees spent abundant amounts of time together during the event—they had the event during the day, dinner at night, and stayed at the same hostel. When we surveyed the participants, we had found that they had accomplished a remarkable number of things. More astonishing, I had discovered independently that the participants were in fact continuing to collaborate over Facebook, long after the event. There was empirical evidence of the formation of a social network of participants from all over the place, consisting of people who would not have otherwise formed relationships with each other, offline or online. In-person instruction was the proper intervention to take these motivated volunteers and have them carry on outreach work for the Wikimedia movement.

We could, in the future, figure out other types of intervention that are more cost effective, but this is a specific one that is known for being highly effective. So yes, we are paying for travel, and I understand that travel funding has a problematic reputation at the moment, but I want to clarify that this is travel funding for the purpose of participating in a highly structured event based on an emerging theory of change. Each participant will have travel funded with a clearly delineated participation goal, namely, the development of skills and the planning of local outreach programs.

Full vs. partial travel funding—I have answered the question of full scholarships vs. partial scholarships above, but I would like to further elaborate. We absolutely want to recruit these ten people as our partners in Wikimedia outreach. To make sure this recruitment is most effective, we need to make sure that their needs as humans are covered. There is a demonstrated link between being able to volunteer and having access to financial resources. We are not in a position to provide for all of their needs, unfortunately, but we could at least spare the significant burden of travel that would cause significant hesitation to even volunteer at all. Further, suggesting that we would not necessarily cover expenses would deter applicants, especially women, who feel that they do not have the chops to get the "top prize" of a full scholarship and are better off not applying. Thus from our point of view, we either fully embrace our participants as participants in the event, or we create unnecessary stress in the name of some ambiguous cost-saving objective.

Clear outcome—As a structured program with participation goals for each participant, the intended outcome is for each participant to plan a program of workshops wherever he or she lives, working in partnership with whatever organizations and communities exist. During the training they will learn how specifically to do this, based on the Systemic Bias Kit. The kit will be taught through direct instruction (lecturing) and one-on-one learning pods that reinforce this instruction and allow participants to teach each other. Following this, each participant will be asked to develop plans to carry out what is learned, and each participant will receive continuing support to see whether this plan is acted upon (and if our social network theory holds, the participants will be helping each other as well). Thus we have an outcome very clearly planned, and we are selecting participants based on the likelihood of being able to act on these outcomes. Of course, as with any intervention, there is no guarantee of this working, but the program is deliberately designed to promote that outcome as much as possible. And if the outcome does not happen, we can look into why it does not, and iterate accordingly.

Clear engagement—We are recruiting from the English Wikipedia community. We have done so creating this page, which was subsequently advertised through watchlist notices (geographically targeted for those in the U.S. and Canada). We subsequently received over 30 applications for just 10 slots, indicating strong interest to attain these skills. I am not sure what particular metrics of engagement you had in mind, but I have reason to believe that the Wikipedia community is sufficiently engaged.

Clear on-wiki impact—The entire objective is to teach people to improve content; this training teaches people how to teach people how to improve content. Emily's workshops in Chicago have not only improved retention, they have also improved content at a higher rate than that which typically occurs at edit-a-thons. The Systemic Bias Kit page lists articles created and expanded at her events. So after all is said and done, there will be an improved encyclopedia.

Curriculum contents—The first day of the program takes place at the library, but it won't take the entire day to learn how to obtain access to specialized sources. What you mention is actually what we have in mind. Presumably experienced Wikipedians don't need to learn how to find sources so much as curating a collection of resources that can be made available for a specific edit-a-thon. Facilitators curate collections for the benefit of their workshop attendees, who can focus on developing their Wikipedia writing skills. (Emily could describe better how this works in practice—she's the one who designed the program.) I am also not sure that "instructions for creating a friendly and welcoming space for workshop participants" will take a full day, but it would require more than just giving out tips; compare it to a sensitivity training or a teaching school where you learn specific skills that are necessary for the construction of a classroom setting.

Username tracking—Username tracking refers to tracking the individual usernames at the follow-up workshops, not at this particular training. I believe Emily will also be teaching about questionnaires, since I know she has done them in Chicago.

I will respond to the points made in the table shortly. harej (talk) 21:55, 13 August 2014 (UTC)

WMF Comments

• As harej mentioned above, we suggest condensing this workshop into 2 days (a full day of library training seems too much).
• How will participants be selected for maximum impact? We understand you already have a number applicants. What process will you go through to select those with the highest chances of actually organizing workshops and edit-a-thons in their communities?
• Please note we would rather have fewer participants what are meaningful (committeed to hosting edit-a-thons) rather than funding 10 that aren't as promising.

Alex Wang (WMF) (talk) 23:21, 13 August 2014 (UTC)

Thank you for your comments, Alex. Emily and I have agreed to condense the workshop into two full days and then a half-day on Sunday, meaning no hotel room costs for Sunday night (we send people home Sunday evening instead of Monday morning). Further, we have found alternative venues for Saturday and Sunday, meaning we will not need a costly hotel conference room for those days. I will furnish an updated budget shortly but the new cost should be thousands of dollars cheaper.

Participants are being selected for maximum impact based on the content of their application. Emily tells me that the applications she has received have included very specific agendas for improving Wikipedia content, as well as specifically identified opportunities to carry out the types of programs that we want to see carried out. Emily could tell you more but she is visiting family and her availability is limited at the moment. In any case, all the applicants accepted are meritorious candidates, and I very much look forward to meeting them. harej (talk) 15:40, 14 August 2014 (UTC)

Thanks harej. We appreciate your efforts to make this event as efficient (and effective!) as possible. Looking forward to the updated proposal. Alex Wang (WMF) (talk) 16:52, 14 August 2014 (UTC)

Alex, I have updated the budget. The sticker price has been reduced to $11,909, a reduction of 29%. harej (talk) 19:29, 14 August 2014 (UTC)

A general note on the curriculum—there was a more fleshed out curriculum, but logistical considerations means we have to look at it again and decide how to adapt it. There should be a fuller agenda posted soon; what I wrote in the proposal is more of a summary. harej (talk) 19:37, 14 August 2014 (UTC)

Request Approved

Thank you and the GAC for your participation in this discussion. We understand the GAC's concerns regarding cost, participants, and measures of impact. We appreciate that WMUS-DC has reduced the cost significantly. We are also encouraged by the selection process for participants and their commitment to organizing events after the workshop.

Given Emily's success in organizing edit-a-thon series we are happy to support this capacity building project. Her Systemic Bias Workshop Kit as a model for effective edit-a-thons and in-person trainings should go a long way in building skills within the movement.

In addition to a final report due 60 days after completion of the grant, we will also expect an update on the measures of success one year after completion to better understand the effectiveness of the edit-a-thons organized by participants. Alex Wang (WMF) (talk) 18:24, 15 August 2014 (UTC)

At the request of the GAC, we are providing additional rationale for why we have decided to fund this proposal.

We see great value in this proposal because:

1. The trainer is one of the best outreach practitioners we have identified so far.
2. She has systematized the learnings from her successful outreach activities and created a resource for others to use (the systemic bias kit -- some early evidence the larger efficacy of the methods is documented here, in Emily's IEG midpoint report).
3. The workshop is _capacity building_ on a large national scale -- i.e. the relatively high expenditure is an attribute of the geography of the United States and of the fact this training serves volunteers from the entire country.
4. We confidently expect this workshop to be a useful sharing of specific and tried-and-true skills, far more effective than the average general outreach session. We also expect the structure of the training and materials developed will be shared across the movement.
5. We have made the reporting requirements address the concerns about follow-up: we have asked James and Emily to require their workshop attendees to track participants in their local workshops over the period of a year.
6. Most concerns of the GAC were, in our judgment, adequately addressed.

Alex Wang (WMF) (talk) 21:12, 22 August 2014 (UTC)