Grants talk:IEG/Co-Location Impact Study

Prior research

Hi! I'm curious if you've incorporated prior learning and program evaluation research into your work, and in what ways your research would be novel.

• Editing workshops: https://meta.wikimedia.org/wiki/Programs:Evaluation_portal/Library/Editing_workshops
• Meetups: https://meta.wikimedia.org/wiki/Programs:Evaluations/WWHM
• Editathons: https://meta.wikimedia.org/wiki/Programs:Evaluation_portal/Library/Edit-a-thons

Cheers, Ocaasi (talk) 21:48, 2 April 2014 (UTC)

Hey Ocaasi! This work will certainly be looking at incorporating the prior learning and some of the questions that have been asked by the research you linked to, as well as other research. These past studies have looked a lot at new editors, people who attend workshops, meetups and edit-a-thons for the first time, and how they have been retained, or not been retained as is often the case. This study is looking more at existing editors, and groups, where the editors started out online, formed bonds and relationships with other editors online and then taken part in an co-located events where they interact with other editors in person, some of whom they would have interacted with previously online. What I'm looking at investigating is how this in-person interaction effects participation of existing editors, more especially those that edit as part of a group, like Wikiprojects, and if there is an effect on participation.

There has been much research in other communities, as well as Wikipedia, on the effects of in-person meet ups, and online meetups, but research on the crossover between groups that form online and then meet offline is very limited. And this is the gap I hope this research can fill, as well as answering question related to in-person wikipedia meetups prior research has asked.

Please let me know if you have any other questions about this proposal, and thanks for the feedback!

OSUBrit (talk) 04:28, 6 April 2014 (UTC)

Outcomes of this research for Wikipedia

Hi OSUBrit,

I'm curious to know how you'd plan to release your recommendations from this study, to ensure that they're most likely to be of actionable use to Wikimedians. Although I understand how useful it will be for the academic community to have your findings published in an academic paper, I've also seen a great many recommendations from researchers in published papers go un-implemented by Wikipedians. Any thoughts how you might help ensure your findings reverberate beyond the ivory tower? And remember, for projects to be eligible for funding, materials must be published and released as free and open-source.

Best wishes, Siko (WMF) (talk) 21:48, 7 April 2014 (UTC)

Hi Siko (WMF)

I, too, have observed various studies of Wikipedia where recommendations end up forgotten and thrown aside, and with this research I would make a real push for this not to happen. Firstly this research will likely be encompassed into my dissertation research, which would mean I would continue to work on the project following the end of the 6 month window - so I won't simply drop and run when it's all over. Secondly I would be constantly interacting with the community, both through the events I would be attending, and also through Wikipedia, allowing me to not only get more feedback for the project, but to start getting the actionable results out in to the community as soon as possible.

Additionally, and I think most beneficially, I would like to conclude this project with the creation of a workshop or workshop series, a sort of workshops on workshops really. I've read a few Wikimedia studies where there is some talk of how workshop organizers don't really pass on the knowledge of how to organize a workshop to other community members. In this workshop I would use the experience and lessons I have gained from the study, and my attendance at the various, diverse, meetups to try and impart this material to the community. At first this would likely be a virtual workshop, but in the future there may be opportunity for workshops at in-person events also.

So for this research I like to view the academic paper to be a consolidating end-goal for the 6 month period, not only does it help get the details and results of this study out to the academic community but it helps to serve as a jumping off point for putting those results to real, actionable use. I would be doing that through continued interaction with the community both online and in-person and the creation of a workshop where the results can be imparted to the community quickly, noticeably and effectively.

Thanks for your question, please let me know if you have any follow ups! OSUBrit (talk) 00:23, 8 April 2014 (UTC)

Thanks for the speedy answer! I'd suggest that since IEG is focused on improving Wikimedia projects first and foremost (hint: we're interested in funding direct online impact, and we like to see this even from research proposals), you might consider reversing the order to focus on getting the word out to Wikimedians as part of your 6-month project, and working on academic publication as a longer term goal. Cheers, Siko (WMF) (talk) 00:34, 8 April 2014 (UTC)

Thanks for the suggestion OSUBrit (talk) 01:06, 8 April 2014 (UTC)

Eligibility confirmed, round 1 2014

This Individual Engagement Grant proposal is under review!

We've confirmed your proposal is eligible for round 1 2014 review. Please feel free to ask questions here on the talk page and make changes to your proposal as discussions continue during this community comments period.

The committee's formal review for round 1 2014 begins on 21 April 2014, and grants will be announced in May. See the schedule for more details.

Questions? Contact us.

--Siko (WMF) (talk) 21:48, 7 April 2014 (UTC)

Co-location and Wikimania

The focus on "co-location" that you put in the title makes me think you want to find out something about replicability and so on. I have no idea how Wikimania fits in this; also, Wikimania is an over-observed (over-exposed) event, I really see no gain in including it in this research. Please focus on local events which receive less attention and that cost less. --Nemo 07:06, 15 April 2014 (UTC)

Hi Nemo, thanks for your input. The reference to "co-location" doesn't necessarily related to replicability to environments beyond Wikipedia, if that is what you are referring to. The title refers specifically to the physical, in-person meetings that individuals who have previous only interacted online are having at certain Wiki events, such as Wikimania. However, I do take your point about Wikimania, it is well researched but I don't think that necessarily means it should be discounted altogether. I have chosen here to look at a number of different events of varying sizes, the largest being Wikimania, in order to get a good perspective across the complete spectrum of meetup events that occur in the community. This will allow for a more complete comparison in the analysis of my results. As to the cost, I am, in fact, a British Citizen, which will allow me to mitigate many of the costs associated with research at this particular event due to my contacts, knowledge and ability to enter, research and work in the UK and EU freely. OSUBrit (talk) 23:04, 16 April 2014 (UTC)

Hi OSUBrit, I think Nemo has a good point. I'm not familiar with the research done on Wikimania, but in terms of replicability, robustness and external validity of the results, it might be best to focus on the "average" editing event, which according to the Editathon report, is about 16 participants (I'd actually target the 25% to 75% quartile range of participation; see the report's appendix to calculate). Typically its best to narrow the focus of research so that results can emerge. If you have too much variation in the type of event and the size of events, especially given your small sample size, it will be difficult figure out what it is that the average editor gains from attending offline editing events. Additionally, Wikimania attendees might have inherent qualities that make them different from other events; there is also a huge difference in motivation levels that one gets from being at such a large event that is not typical at all. Hope this helps! EGalvez (WMF) (talk) 18:01, 18 April 2014 (UTC)

Hi EGalvez (WMF), following on from Nemo and other users' feedback I've decided to re-focus the study on smaller social wikipedia meetup events (so those where the primary purpose is social interaction and not editing). These will fall in about the 16 participant area that the Editathon reports and thus should be quite manageable from the perspective of trying to interview as many participants as possible. Narrowing the scope should enable the study to meet the replicability and robustness standards that would expected of such research as well. OSUBrit (talk) 05:17, 19 April 2014 (UTC)

Transcription

This sounds like a waste of money for Wikimedia and an insult to all volunteers who transcribed interviews for Wikinews and other Wikimedia projects. If you care about this project, do it yourself. Nemo 07:06, 15 April 2014 (UTC)

I think the tone of Nemo's response was rude and inappropriate to this forum. However, I agree with the substance of their critique: transcription services are the largest single line-item in your budget, and I question whether they are a necessary expense. In addition to Nemo's point about volunteer transcription within our movement, I will offer that there is also a long and storied history within academia of PhD candidates transcribing their own interview data ;) Jtmorgan (talk) 18:57, 15 April 2014 (UTC)

There is an interesting question here. Is the point of the work simply to collect interviews and transcribe the result? Or, is the point of the proposed work to generate original research and propose solutions based on original results? If the main point were simply to collect data - then maybe requiring transcription as a part of the work would make sense. However, if the goal is to spend time to analyze and generate results - then spending the majority of the time on transcription is probably not the best use of the research time. I will acknowledge that Nemo's text highlights activities by Wikipedians that are focused on collections where the collection and transcription are the primary point. Activities like Wikinews, mentioned by Nemo, are not about generating original research. Dwmc (talk) 03:46, 16 April 2014 (UTC)

Adding transcription costs to this proposal was a difficult decision. I too was struck by how it has turned out to be the biggest line item, so yes Nemo I do find your overall point valid, especially coupled with what Jtmorgan has said. I feel that it is expected of me to do a good deal of my own work here, especially given how this research will be a chunk of my dissertation research, and I do feel the burden of that, and the responsibility to let the community get the most out of the money they would invest in this research. My general concern here however is time, and the time it will take to transcribe around 50 interviews, that could each be well over an hour long, is vast; especially with trying to get good solid results out to the community within 6 months. As Dwmc has said, the point is to generate original research and spending a good deal of time on transcription will have an impact on what research can be produced. A balance has to be struck, however given this feedback I have to say I think my initial inclination for what that balance should be has been wrong. I think it would be much better to invest a little of the requested funds into better aiding my own ability to transcribe (dictation software, for example) and thus removing the majority of the cost of this part of the study, although at the cost of the number of interviews that I can use in the 6-month study (It would not effect the number that would be collected in total). OSUBrit (talk) 00:32, 17 April 2014 (UTC)

Other research projects find it very useful to have professional transcription (OII is a good place to look for examples). This allows sharing transcripts among researchers, even those who were not the interviewees. It jumpstarts the analysis process and professional transcription makes it harder to lose parts of the interview. I suggest keeping the transcription costs! Jodi.a.schneider (talk) 19:27, 19 April 2014 (UTC)

Beyond EnWiki?

Understanding how co-location helps or hinders collaboration is a very compelling research question, but in terms of impact on the Wikimedia Movement I think it could be more valuable to focus on collaborations beyond English Wikipedia, or in geographic locations where the local editing community is still small and fragmentary. Those are the scenarios in which it would be most useful to know how in-person meetups influence collaboration. As you know, there's good reason to suspect that other wikimedia projects are not just tiny Enwiki clones, and that contributors from other parts of the earth have different needs than North Americans/Western Europeans. Have you given any thought to how you might include projects beyond Enwiki, or geographies beyond North America and Western Europe, in your sampling? Cheers Jtmorgan (talk) 19:06, 15 April 2014 (UTC)

Hey Jtmorgan, I've spent some time over the last few days thinking about this, and I also spent time thinking about it before I wrote the proposal for the very reason you suggested, because EnWiki isn't the be all and end all. The main reason I chose the locations I did was because I wanted an international slant on the research, but as I am not fluent in any language other than English and with the limited time available for the study, trying to engage in research in languages other than English would be difficult. However you have a good point about looking at geographies that are not as active, and I think looking at smaller projects which are based in areas that predominately speak English (Welsh wikipedia, for example) would also be a great way to expand the study to areas that are under-studied while getting around that problem. There is also the possibility at looking at English speaking areas outside of North America and Western Europe, such as South Africa (which could also cover Afrikaans Wikipedia) or Australia/New Zealand to try and get a more complete picture. Additionally keeping Wikimania in the study in order to interview individuals from other non-english projects attending the event may also be a good opportunity to increase the diversity of the study. Let me know what you think. OSUBrit (talk) 04:05, 19 April 2014 (UTC)

I would suggest carefully documenting your methods. Perhaps you could find a collaborator in another region, fluent in another language, who might replicate your study elsewhere. Then you could draw larger conclusions together. I think Wikimania is different than local meetups, so I'd be careful about how you use it in your study. Jodi.a.schneider (talk) 19:28, 19 April 2014 (UTC)

Jodi.a.schneider I agree about taking great care with Wikimania, a direct contrast to the much smaller social events this study will be looking at should not be attempted and I wouldn't attempt to do so, other than perhaps some of the smaller socially based satellite events that may crop up around Wikimania due to the large number of editors that will be present. The only reason I am including it in the study at all is in an attempt to question Wikieditors from other languages than EnWiki. At this point I may decide to remove it all together for reasons of clarity. OSUBrit (talk) 02:22, 20 April 2014 (UTC)

Defining productivity of contributors

Defining productivity of contributors has been discussed extensively over the years, and there is a lot of activity that takes place outside of Wikipedia that later impacts what is done on Wikipedia. How would you use your data to come to a definition of productivity? --Pine^✉ 04:04, 16 April 2014 (UTC)

Hi --Pine^✉, this is a great question, especially in reference to the activity that takes place outside of Wikipedia. So this research would be using a Grounded Theory approach, where they definition would come to be informed from the data I would collect. That said, I am very aware of the peripheral activity that goes on beyond Wikimedia platforms where editors interact with each other, so I would attempt to tease information on these activities out of the individuals I will be interviewing. There is some speculation that a documented decrease in participation in previous studies of individuals who meet first online and then interact in-person occurs because individuals move to other platforms for their primary interactions, and so that is definitely something I want to investigate and will lend itself well to defining productivity. It could well be that this research results in multiple definitions of productivity, that include a purely wikimedia platformed based definition and then additional definitions that include all interactions between editors, both on and off these platforms (whether that be in-person, online or both). OSUBrit (talk) 04:26, 19 April 2014 (UTC)

Ah, oops, I didn't see this before I asked my question. I'd suggest that you add this information into the proposal, with a brief methodology section. And I guess looking to past lit would be more appropriate *after* the fact, given grounded theory! Jodi.a.schneider (talk) 19:30, 19 April 2014 (UTC)

Scope of research

I like the idea of increasing the online impact of offline events, but the scope of your study is extremely broad. There are a very large number of variables with online and offline work. What variables will you be considering in your studies where you hope to find useful data for designing offline events? --Pine^✉ 04:08, 16 April 2014 (UTC)

Hi again --Pine^✉, so given some of the other feedback I've recieved I'm going to reign in the scope of the study by looking at one specific type of meetup event - social meetups. So I'm looking at events that could involve some type of hack-a-thon editing, but primarily are there as a social interaction to get to know other wikipedians, discuss community matters and other topics. I'm primarily interested in the social interaction aspects and how they effect participation and given that these types of events are social in nature they seem like the best place to investigate this. From the data I collect at these events hopefully the research will reveal key aspects that can be added to more work centric events, like hack and edit-a-thons to enable increase participation and user retention. OSUBrit (talk) 02:23, 20 April 2014 (UTC)

Variability of events and logistics

An interesting idea. I'm curious what your thoughts are about the variability of events (hackathons, editathons, workshops etc.) and how you might find ways to compare/contrast among these? Do you think it could be easier to focus on one of these event types so it is more feasible to obtain results? Also, what would be the logistics of conducting the interviews (i.e. time length, before/after events, etc.) ? Thanks! EGalvez (WMF) (talk) 02:19, 17 April 2014 (UTC) Revised April 18, 2014

Hi EGalvez (WMF), so based on the feedback i've received I've decided to limit the scope of the study and concentrate on one type of meetup event, the social meetups, where users are interacting primarily to get to know each other and discuss the community and other topics in a social setting, rather than the work setting of a hack-a-thon. I'm primarily interested in the social interaction aspects and how they effect participation and given that these types of events are social in nature they seem like the best place to investigate this. Since these events mostly don't feature the more work centric editing environments of a hack-a-thon it should be easier to engage individuals at these events and interview them for the project. As for the logistics I would attend meetup events primarily to make contacts with whom I could then follow up with to interview at a later time (either online or in person, depending on their preference), of course it may also turn out that I would be interviewing some individuals at the events as well. I hope this answers you questions, please let me know if you have any other questions! OSUBrit (talk) 04:45, 19 April 2014 (UTC)

Evaluation reports and scaling

I think Grants talk:IEG/Women and Wikipedia#Evaluation reports is relevant to this proposal. How would your study differ from what has been done previously?

Also, given the small size of most social meetups, what information do you anticipate from your work that could lead to significant increases in the number or size of social events?

Thanks, --Pine^✉ 05:54, 19 April 2014 (UTC)

Hey --Pine^✉ the core of this study is qualitative research, although I have made sure to include a mix-methods proposal by gathering quantitative data on participant usage, in order to bolster my findings from interviews. I am trying to tease out the benefits of these meetups, and the social interaction and relationship building that occurs in them. Previous studies, are largely looking at numbers such as edit counts, budgets, numbers of featured articles, quality images etc as their metrics of success, but this research is far more centred on the interpersonal relationships that form the core of editors interactions at in-person events and, for many, online friendships and collaborations. That's something that I feel makes this research different from what has largely been done previously.

Additionally, the size of the social events shouldn't really be an issue, as i'm focusing on the interactions between editors, the information I will be collecting will be largely generalisable across meetings of all types, by studying how the social interactions are, or are not, beneficial to editors in these events can enable organizers of other events to better plan their meetings. For example the study may find that by putting in more non wikipedia task-based activities in to editathon planning organizers might find a way to increase retention by enabling editors to build relationships and common ground beyond just editing activities. Thus by finding out what works and what doesn't work in these events, and translating that across to other types of meetup events we can increase user retention and also increase the size of future events by limiting attrition OSUBrit (talk) 01:33, 20 April 2014 (UTC)

Hi OSUBrit. As someone who has attended a number of small meetups I am skeptical that what you're describing here will scale well, and asking for about $10,000 to find out is something that I'm not sure I can support. I'm curious to know what meetups you have attended to date and what you think you have learned from those meetups that is valuable and easily scalable. In other words, before I support putting $10,000 toward this proposal, I would like to know if there is evidence that your approach has worked informally or on a smaller scale. Thanks, --Pine^✉ 07:14, 21 April 2014 (UTC)

--Pine^✉, I have previously attended hack-a-thons and small social gatherings before - although admittedly not a great many - in the Wikipedia community and so I can definitely see an important contrast between the different types of events. In the case of replicability between the social gatherings and task-based meetups I'm very specifically looking at what elements of the social can be applied to the task-based in order to improve retention and productivity, from my attendance at both these types of event, I can definitely say I think that can be done. The hack-a-thons had a very different feel to the social gatherings and I think there is definitely something about the social setting that is valuable, positive and transferable to other event types. One specific research hack-a-thon i attended was much more informal and social in nature than others I had been to, the interaction I had both at the event, and previously (as I had happened to have met some of the attendees socially before hand) made the experience not only more pleasurable but more productive as there seemed to be less barriers to interactions, less reluctance to speak up and ask questions and generally a more positive atmosphere. What I will add to this is that I have a great deal more experience with meetups of varying sizes within the Youtube community and although that community is more socially oriented than task oriented, when compared to wikipedia, I think the core of what I'm looking to investigate in this study is similar. In the cases of the larger meetups you will see generally little difference in interaction to the smaller ones, because in a large environment people tend to splinter off into smaller sub-groups, thus replicating the environment of much smaller social meetings (like the ones I am looking to investigate), this lends itself to suggest that despite the vast difference in numbers of people, the interactions are readily scaleable between small and large events. OSUBrit (talk) 21:02, 21 April 2014 (UTC)

Non-English

"In order to get a diverse range of participants locations have been chosen across a number of countries in cities of varying size and cultural diversity, including smaller non-English Wikimedia projects." But "The Proposed locations are: Seattle, WA; New York, NY; Vancouver, Canada; Toronto, Canada; London, United Kingdom & Birmingham, United Kingdom." -- maybe something has changed? Jodi.a.schneider (talk) 19:24, 19 April 2014 (UTC)

Hi Jodi.a.schneider I'm actually reviewing the locations now to include places where I can look at non-English Wikiprojects located in areas where the contributors would also be fluent in English (for example Welsh or Afrikaans). I added some text to reflect this on the grant page, I'm looking to have those locations, and the cost reflections updated in a few hours, thanks! OSUBrit (talk) 00:29, 20 April 2014 (UTC)

Cool. I'd recommend looking at the Irish (GA) Wikipedia as well in that case. Jodi.a.schneider (talk) 08:07, 20 April 2014 (UTC)

Productivity

This is vague: "Create a definition of productivity in Wikipedia using context from the interviews and based in the collected edit history data" What kind of definition, for what purpose? What methods will you use for teasing this out of "context" and edit history? Anything else you could say about methods at the moment would be interesting. Will you also check the literature, e.g. Geiger, R. Stuart, and Aaron Halfaker. "Using edit sessions to measure participation in Wikipedia." CSCW 2013. ? Jodi.a.schneider (talk) 19:24, 19 April 2014 (UTC)

Saw you above comment Jodi.a.schneider I'll add a methods section to make it more clear, but yes I'll be forming a definition using a Grounded Theory approach, the CSCW literature on participation will definitely be playing a part of that! OSUBrit (talk) 01:40, 20 April 2014 (UTC)

Aggregated feedback from the committee for Co-Location Impact Study

Scoring criteria (see the rubric for background)	Score 1=weak alignment 10=strong alignment
(A) Impact potential Does it fit with Wikimedia's strategic priorities? Does it have potential for online impact? Can it be sustained, scaled, or adapted elsewhere after the grant ends?	5.8
(B) Innovation and learning Does it take an Innovative approach to solving a key problem? Is the potential impact greater than the risks? Can we measure success?	6
(C) Ability to execute Can the scope be accomplished in 6 months? How realistic/efficient is the budget? Do the participants have the necessary skills/experience?	6.8
(D) Community engagement Does it have a specific target community and plan to engage it often? Does it have community support? Does it support diversity?	6.8
Comments from the committee: Scope and proposer’s experience indicate that they have the ability to execute. The research question is unanswered and worth trying. It could be interesting to learn more about how (and what types of) offline meetups cause productivity in smaller communities or sub-communities, as it will bring insights on growing communities. A focus on WikProjects makes some sense. The study’s potential impact is limited, as it doesn't consider cultural aspects. We question the return on investment - it's expensive and the value of the outcome to the community is not clear. We expect that the outcomes of the meetups and hackathons with or without social functions could be self-reported and analyzed by WMF at no additional cost. Surveys, email and Skype could likely accomplish as much as in-person interviews, without a travel budget.

Thank you for submitting this proposal. The committee is now deliberating based on these scoring results, and WMF is proceeding with it's due-diligence. You are welcome to continue making updates to your proposal pages during this period. Funding decisions will be announced by the end of May. — ΛΧΣ²¹ 00:35, 13 May 2014 (UTC)

Round 1 2014 Decision

This project has not been selected for an Individual Engagement Grant at this time.

We love that you took the chance to creatively improve the Wikimedia movement. The committee has reviewed this proposal and not recommended it for funding, but we hope you'll continue to engage in the program. Please drop by the IdeaLab to share and refine future ideas!

Comments regarding this decision:
We’re not yet seeing enough potential impact for the community to justify the costs of a research project along these lines. However, we really appreciate all you’ve done to engage in the proposal process, and we’d be happy to see you back at IEG in a future round with any other ideas you have. Thanks for participating!

Next steps:

1. Review the feedback provided on your proposal and to ask for any clarifications you need using this talk page.
2. Visit the IdeaLab to continue developing this idea and share any new ideas you may have.
3. To reapply with this project in the future, please make updates based on the feedback provided in this round before resubmitting it for review in a new round.
4. Check the schedule for the next open call to submit proposals - we look forward to helping you apply for a grant in a future round.

Questions? Contact us.