Grants:IEG/Full Circle Gap Protocol: Addressing the Unknown Unknowns/Midpoint

This project is funded by an Individual Engagement Grant

Welcome to this project's midpoint report! This report shares progress and learnings from the Individual Engagement Grantee's first 3 months.

Summary

This project has successfully achieved the stated goals of recruiting an undergraduate course and experts to participate in the gap finding pilot project. Other successes are described in detail below. This midway report also chronicles challenges that have arisen in the course of the pilot, including 1) rethinking techniques for working with academic experts in a brainstorming session or other gap finding exercise in a meaningful way and, 2) matching courses with a gap finding session. This report describes the strategies that are being used now, and that might be used in the future, to attend to these challenges. I've also created a learning pattern that builds on successes so far.

Methods and activities

This project is focused on developing a feminist, pilot protocol for doing a feminism-focused gap finding project. Achievements so far:

Recruited five academic experts from the University of Washington to participate in the pilot
Signed Letters of Engagement for each participants (details below)
Organized and hosted a five hour brainstorming session to discuss, and do, gap finding
Contacted a community partner, Ada's Technical Bookstore, where event was hosted
Created Wordpress blog to chronicle the project
Presented in four undergraduate classes and consulted with four professors on doing Wikipedia editing around feminism content gaps with students
Recruited of one course to do Wikipedia editing and use the gap finding list generated by the project
Presented on successes and lessons so far at WikiConference USA (Talk titleː "Working with Academic Experts")
Participated in UW Research Commons editing events, invited to a local feminist to present a reading of Wikipedia as related to her leadership in an association for women in mechanics prior to a public editing event
Currently conducting a close reading gap finding activity with experts to augment brainstorming session

Midpoint outcomes

Participant and Course Recruitment

Five participant experts have been recruited for this pilot, each from the University of Washington in Seattle WA. Three are doctoral candidates, two are assistant professors. The experts are from Gender, Women's and Sexuality Studies (2), the Information School (2), and the Department of Communication (1). Four of the five participants are instructors of record teaching courses this fall and when recruited were interested in teaching with Wikipedia in their courses. Three of their four courses this fall are related specifically to feminism and technology, the focus of the project. The third course is on critical media literacy. Three of the four course instructors opted to include a class or unit on Wikipedia, gender gaps, and feminist interventions in their courses. The course on Critical Media Literacy specifically assigned Wikipedia editing to the students as a final project, and will offer the gap finding list as a resource for the students.

Course Presentations

I made 45-90 minute presentations to three of the courses taught by participants in the pilot. For each presentation, I coordinated with the instructor beforehand, assigned relevant readings, and described the gap finding pilot project to undergraduates. I'll describe each class presentationː

Classː Small upper div. seminar (less than 20 students) on Gender and Online Interaction. Presentationː I sat with the students in their intimate seminar class and heard from each of them how their analysis of the Gamergate talk page was going. Then I made a presentation about "Free labor and the gift" and ended with a discussion of feminist interventions in online organizations, focusing on Wikipedia as one case study.

Classː medium undergraduate course ( ̃40 students) on Critical Media Literacy. Presentationː I made a 45 minute presentation about the relationship between production of Wikipedia and the content available on the site. I focused on a few examples of debates and described how editing works and parts of a Wikipedia page. This foreshadowed work students would do later in editing Wikipedia for their final projects, and I introduced the gap finding list that they were invited to use.

Classː large undergraduate ( ̃100 student) lecture course on Gender and Information Technology. Presentationː I made a 45 minute presentation to students about Wikipedia and systemic bias. I described intervention projects, including the Gap Finding project. Then I conducted a brief, in-class "gap finding" exercise that enlisted students evaluate the Wikipedia page "Women in Computing." Students were invited to summarize the page and provide references on a shared in-class live document that could be added to improve the page. The page directly related to their course theme and materials. (This exercise was generative as an in-class activity but unsuccessful in terms of making a portable/transferable list for other editors to use in part due to time constraints -- 20 minutes was not enough time -- and equipment limitations -- not all students had access to computers or tablets).

Gap Finding Brainstorming Session Outcomes

The goal of the pilot is to develop a protocol process for doing distributed editing. The pilot has focused so far on working with academic experts at University of Washington, and partnering with WikiED to do editing in one class.

The brainstorming session that was conducted with the experts was informal and sought to engage the participants as partners in the development of this protocol. In the session we successfully:

Worked with WikiEd beforehand to generate lists of pages that are highly trafficked and "underdeveloped" (based on criteria for Featured Article) related to feminism and technology

Built a wiki repository to compile links and pages to edit, this list was the focus of the last hour and a half of our brainstorming session

Identified a "bestiary" of gaps on and around Wikipedia in addition to particular, 'unknown unknown' content gaps listed above. The consensus at our session was that gaps, and the ways that systemic bias influence and shape the project, are not limited to content alone. Social and socio-technical gaps affect nature of volunteering, the structure of editing, the wiki community culture, harassment policies, the design of the site, Wiki partnerships, and the future of the project. We created an annotated list of this "bestiary" of gaps on a white board. Only after we identified these was it possible to move into critiques of the pages themselves.

As a result, our conversation focus kept to critique at a meta level, rather than devoting the whole session to evaluating pages

Close reading - A new form of gap finding

After our session, I elected to spearhead a modified approach to gap finding that would augment our accomplishments. The aim of this focus is to provide a more detailed, complete list of gaps to be touchstones for the undergraduates who would be doing editing with the list.

Why augment what we did? The work we did in the session was collaborative and emerged through conversation, thus a challenge in putting the lists to work is that most of the entries maintained an "expert" level familiarity with the content. For instance, at it's most vague, our list includes the following entryː

technofeminism -- Relates to feminist technoscience but could constitute its own post.

This is useful for another expert to take note of, but does not offer material for a novice in feminist theories of technology to use. That said, other entries were slightly more developed, such as this comment on ENIACː

ENIAC -- Add reference Janet Abbate's book Recoding Gender: Women's Changing Participation in Computing both to reference list/citations and to expand on section on programming, female programmers, and perhaps elsewhere throughout. https://mitpress.mit.edu/books/recoding-gender

However, without already knowing that ENIAC was an early computer programming language developed by women, this entry would again be esoteric to new editors and novices with feminist theories of technology.

To enable undergraduates or other new editors to do a "deep dive" in to criticizing content, I decided our list needed further explanation. Indeed, one of the many challenges of working with academic experts together is conversing using a shared vocabulary. Because we were speaking to each other, rather than to undergraduates or non-experts, we used familiar foundational concepts and terms without explaining them.

Thus, by inviting participants to conduct a second brainstorming session in a different format -- and in return opt out of attending the presentations -- I hoped to overcome this shortcoming.

I provide more details about the close reading activity in this learning pattern, and will further comment on how this goes in the final report.

To implement the close reading activity, I reached out individually to each participant by email or phone. My aim was to engage each person in a private conversation about their own interests and reflections on the activity, and invite them personally to do a second round of brainstorming. I hoped by asking them to consider ways we might deepen our list with a reading of one page, they would be personally compelled to participate in this new format. I also invited them to choose the presentation format that would be most personally meaningful to them. Below are the suggested formats:

How to present your close reading:
Decide how to share the close reading. I’ve listed three suggestions (with brief justifications) below and invite you to choose one or propose a different format. Please help the project by briefly explaining why the close reading format you’ve chosen is meaningful for you as a participant in this pilot.
1. A questionnaire/question-answer format (e.g. a Google Forms). This format is the most buttoned in way to conduct a close reading. You analyze a Wikipedia page by responding to prompts. The benefit is such a formulaic format is that it is easy to follow. There’s also the benefit that all close readings are presented in a relatively similar format.
2. A written narrative. This format invites you to decide how to craft your close reading. Do you wish to devote your close reading to analyzing one particular section and not cover in depth all the dimensions of the article? Write up the narrative in an article or blog format, or another writerly genre. The benefit of an open written narrative is that you decide where to devote your critical attention. For future editors, variation in format motions to existing variation in articles – not all are the same nor need the same sorts of editing attention.
3. An oral presentation (e.g. you give a lecture/presentation). This presentation proposal format invites you to shine as an academic presenter and make a close reading to a crowd – either before an editing event, at a conference, or at a panel, or public event. For this pilot project, an oral presentation would be filmed and with your permission shared with future editors (rather than staged at a conference, not all that feasible given the quarterly time constraints). This format foregrounds you as a content area expert doing a public service, critiquing the encyclopedia’s coverage of a specific area. As with the written narrative, the presentation styles and concentrations might vary widely, foregrounding your reading of the page and your speaking abilities.
4. Something elseǃ

Outcomes of Close Reading Activity

Three of the five participants accepted the invitation. One was unsure how her contributions would be useful at this stage and opted not to participate in the close reading. The instructor of the course using the list did not provide a close reading, but did augment her list of suggested readings for her students. Of the three who choose to do an additional close reading, each chose a different presentation format. I will describe and analyze these formats hereː

1. Oral close reading - interviewː I conducted a close reading "interview" with one participant. For about thirty minutes, she analyzed one Wikipedia page that was underdeveloped and directly connected to her area of expertise. She summarized what was presented. She commented on what literature was used to make claims. She described what was missing, she analyzed the presentation style, framing, and links. She read through the talk page and commented on controversies. She recommended five additional references to be added that would assist in deepening the page. She also discussed the politics of visibility and also explained why it would be important to attend to these oversights. I recorded and transcribed her interview, including it as an entry on our Gap Finding repository list. The oral close reading interview took about forty minutes total -- and about an hour or so for me to transcribe. In my view, this method was by far the most successful and would be one I recommend to future projects interested in engaging academic experts. The oral commentary required little preparation by the expert. The dialogue unfolded in an interative, interesting fashion. It did require synthesis workː I reorganized the conversation rahter than simply reproducing the transcript and organized it into three sectionsː "summary of page" "what's missing" and "references - annotated" with her critical reflections woven in as direct quotations. (updateː this was one of the entries that was chosen by students to edit).

2. Written close readingː One participant opted to do a close readings in written form. She briefly described what is on a pag that is relevant to her research interests, what's missing, and listed two references to improve it. She said she ran out of time in doing this activity.

3. Engaging a course in a close readingː The third participant suggested doing her close reading as a class exercise with her students. I worked with her and spearheaded this activity when I gave a guest lecture and have already described earlier in this report. Her course focuses on "Gender and Information Technology" so the students were invited do a close reading of the page "Women and Technology" (which was changed at some point prior to the class to "Gender and Technology"). I came to her course and made a presentation that synthesized readings on Wikipedia, described forms of intervention, and we did a live chat where students read the page and provided citations from their course materials that would deepen or further enrich the material. This went very well. The students had much to contribute. The major constraint was time, we did this activity for only 20 minutes. However, building an activity around having a "hive" of students who have recently read material on a particular optic make critiques of a Wikipedia page can be very successful. The instructor was interested including this activity in a future iteration of the course over multiple class periods, rather than in one one. The activity does, however, require all students have access to the internet and devices. Not all of the students in her course had this, though some effectively used their smart phones, but hosting such an activity in a lab would be a better idea. The students in this course had completed one lab exercise where half the class completed a WikiEd tutorial on learning to edit, or the Wikipedia Adventure, however none of the students who tried the Wikipedia Adventure were able to successfully follow the Adventure -- and so all students did the WikiEd tutorial.

If one limitation of the brainstorming session was the ease of speaking in shared academic discourse, a shortcoming of my change of course correspondence was reaching out to participants individually rather than as a group (in a group email). Reaching out individually not only required more emailing and connecting on my end, but also kept the participants distanced from each other and unaware of what the others were contributing.

Bonus close readingː

On October 26, I invited a local activist to do a close reading of pages related to women and aircraft/aviation mechanics prior to a Wikipedia Lab editing session at the University of Washington. (I am among the organizers of Cascadia Wikimedia User Groups weekly Wikipedia Lab sessions at the UW Research Commons). Trained as an aircraft mechanic, the speaker was the leader of a local chapter of women in aviation mechanics organization. In her presentation, she commented on entries related to women in aviation mechanics and made editing suggestions, most of which were specifically related to her work with the organization and highlighted local press. Honored to be a guest speaker, she invited ten of her colleagues to join, none of whom had ever edited Wikipedia before. Inviting a local expert to present on existing gaps on Wikipedia prior to an editing session sheds light on gaps, brings new voices to Wikipedia, and promotes partnerships with local organizations. This was a successful ad hoc gap finding session partnered with an editing event.

Reflections on the letter of engagement and doing collaborative work in a grant format

As I detail in a blog post, I crafted a Letter of Engagement to clearly lay out expectations with the participants of the pilot project from the beginning. I wanted to make sure that all of the requirements were transparent and that participants would not feel taken advantage of, but would rather understand themselves to be partners in a collaborative feminist project that sought to critically engage with Wikipedia's processes and content.

However, when I reflected on the outcomes of the brainstorming session, and sought to augment the list we created with an additional close reading activity, I was modifying the very terms I was laying out. Moreover, by reaching out to the participants individually and inviting them to a phone call or email conversation, as mentioned, I may have contributed to a sense of breakdown in the collaborative energy that we had generated in the brainstorming session.

A limit of adding in the close reading activity was that participants also voiced frustration at the amount of time necessary to do a "good" close reading of a page, and one simply did not understand what she was supposed to do and opted to not participate at all. Was this a limitation of the letter? Participants also voiced a sense of distance between what they could contribute and what the aims of the pilot were -- was it to give the instructor's students reasonable tasks to work on or to share their own expertise? If the former, then many felt unprepared to do so, given that the course content was not related to their areas of expertise. If the latter, the experts felt this would be too time consuming.

Based on the way that this close reading pilot unfolded with respect to the letter of engagement, I recommend that future pilot projects seeking to do collaborative work maintain the collaborative focus of the pilot and keep the Letter of Engagement open to changes. For projects seeking to work with academic experts, I would still recommend using Letters of Engagement (particularly if experts are compensated for their time and enlisted as co-collaborators) but foreground the close reading into the project from the beginning of the activity, as I describe in this learning pattern on working with academic experts. In particular, the oral close reading was very successful (as described) and could be a useful model for future close readings.

Finally, as I presented at WikiConference USA, it's important to remember that academic experts are often tasked with a large number of "community service" activities. Many, if not all of these, are not relevant to their degree progress or tenure. It's paramount to respect their time and levels of interest.

Finances

Proposed Budget

$5,500 - Project manager at 10hr/week, $25/hr for ~220 hours
$1,000 - A $200 honorarium for five subject matter advisers
$500 - Refreshments (for two events), Space rental costs, discretionary funds for copying and/or childcare costs

TOTAL $7,000

Distributed Funds

$3,500.00 (5/18)

Funds Allocation at Midpoint

$3,250.00 - Project manager at 10hr/week, $25/hr for ~130 hours (July-November)
$205 - Space rental, refreshments

TOTAL $3,455

Funds Requested for Remaining Grant Period

$1,625.00 - Project manager at 10hr/week, $25/hr for ~65 hours (November-January)
$1,000 - A $200 honorarium for five subject matter advisers

TOTAL $2,625

Unused Funds, Do Not Distribute

̩* $875 ( $250 + $625)

TOTAL $875

Learning

The best thing about trying something new is that you learn from it. We want to follow in your footsteps and learn along with you, and we want to know that you are taking enough risks to learn something really interesting! Please use the below sections to describe what is working and what you plan to change for the second half of your project.

What are the challenges

As described in the report above, one of the challenges of this pilot project has been thinking about how to engage non-Wikipedian experts in a meaningful way to do content critique, while also attending to the many ways in which Wikipedia processes and procedures support forms of systemic bias. The way the initial brainstorming session was organized led to many fruitful conversations, but also invited further reflection on how to engage academic experts to do close readings of content in ways that would be meaningful for them. There is also the challenge of how to meaningfully rethink and redesign the editing process in a way that is attentive to feminist values -- that honor and respect the labors that go into editing. This is difficult and does not have an easy answer.

What is working well

Recrafting brainstorming to invite experts to do close readings overall has worked well. The close reading experiment has been a rich and fruitful addition to the project, as have the different methods of doing it. They are a great supplement to the brainstorming file we initially wrote together. As the course that will be drawing on these resources is only now beginning their final projects, how the close readings will be taken up is yet to be seen. That said, I've created an initial Learning Pattern that outlines the possibilities of a close reading exercise.

Next steps and opportunities

Transcribing the remaining close readings

Working with the courses doing Wikipedia editing to assist them in using the lists, providing assistance with editing as needed

Sharing the list and readings with others in the Wikipedia community

Participating as an observer along with other interested experts in student presentations that critically reflect on their assignment

Convening a final meeting with the participants in the pilot in early January to discuss the pilot and craft a final report on doing a feminism-focused and feminist distributed editing project

Grantee reflection

What an honor to receive this grant and develop this project. The successes and challenges have been instructive to me and the Wikipedia community. I received positive feedback and had invigorating conversations with Wikipedians at WikiConference USA. I've become more active in the local Wikimedia Cascadia User Group meetings and been inspired by my own critiques and made more edits myself. The collaboration with WikiEd has revealed instructive ways of thinking through editing on Wikipedia and the constraints that enable/prevent academics from working with Wikipedia, and what students experience when they encounter the encyclopedia project as a classroom activity.

I wish to thank the participants in this pilot project, all the Wikipedians who have supported the project and found it valuable, and the students who have given feedback, asked questions, and gone into edit as a result of my engagement. Many encounters, activities, presentations, and reflections were made possible by this grant that would not have otherwise been possible. In addition, many "unknown unknowns" have been identified -- and feasibly, will be acted upon, as a result.