Research talk:Voice and exit in a voluntary work environment

Interesting

Very interesting proposal. I might be interested. Note I am male, white, straight and cis, though... OTOH, I find discrimination both ignorant and abhorrent. And baffling... The gender gap should not exist, yet it does. Any attempt to correct this can only be a good thing Iadmc (talk) 01:04, 26 January 2017 (UTC)

Thanks for putting the time to read it and leaving a comment here, Iadmc. :) I would not take into account the demographics of the people who are up for helping at this point (and thanks for bringing that up, btw). I can see how it can be interesting to learn, for example, if there is a difference in response level based on the different demographic profiles of teammates. If we have enough data we can look into it, but at this point, that complicates/constraints the research too much. I'll wait for a week for others to leave feedback here as well and then will get in touch with you and other interested people to figure out how to move forward with this. :) --LZia (WMF) (talk) 01:13, 26 January 2017 (UTC)

OK. Thanks Iadmc (talk) 01:14, 26 January 2017 (UTC)

bad actors

It's a great idea and I look forward to seeing the detailed experimental design. It would be great to see an combinatoric design which was implicitly intersectional. I recommend that you consider what effect, if any, trolls and those gaming the system are likely to have on the experimental design. Feel free to ping me when you're signing up support teams. [I came here from there. ] Stuartyeates (talk) 08:30, 26 January 2017 (UTC)

Thanks for your interest and offer for help, Stuartyeates. We'll keep you posted. In the mean time, can you expand on what kind of effect you have in mind when you talk about trolls and those who gamify? I understand it's hard to tell exactly without us specifying the experiment, but I want to understand more what you have in mind and take it into account in designing the experiment. --LZia (WMF) (talk) 18:49, 26 January 2017 (UTC)

There are established accounts of those who present themselves differently online, women whose online persona is male or neutral to avoid the pickup attempts, trolls etc. Maybe we could ask contributors their demographics more than once, once at the gateway into the community and then again several weeks / months later once we have established their trust. If they differ significantly, flag them for follow-up. I doubt this will catch the trolls, but it may catch others. https://genderize.io/ is also an interesting service, which might also detect whether they username is gendered differently to their reported identity; many usernames are non-gendered, of course. We'd have to design the information capture in such a way as to not exclude the identity-fluid and those on voyages of self-discovery, of course. Stuartyeates (talk) 04:12, 27 January 2017 (UTC)

I see what you mean. Thanks for expanding. I'll make a note of possible follow up surveys. --LZia (WMF) (talk) 00:26, 31 January 2017 (UTC)

Count me in!

OMG what took you so long? We really need a study on this stuff, and after 10+ years onwiki, I am certain that all the other WP editor retention gaps can be inferred from this one. This is just the biggest one. Though everybody has a mother and no one is out to disrespect her, the end result is that everyone does. This is also true for systemic bias in western academia, and Wikipedia has no choice but to amplify that bias through lack of reliable sources. Jane023 (talk) 09:47, 26 January 2017 (UTC)

Oh and I think it would also be incredibly valuable to find out how many academic studies have been inspired by the gendergap; I mean people who tried to remedy the situation with some edits to Wikipedia only to discover that there are no reliable sources, leading them to create reliable sources elsewhere. :) Jane023 (talk) 09:50, 26 January 2017 (UTC)

Jane023, thanks for your support. We will keep you posted. Regarding your second point: that is a good question, but it is outside of the scope of this specific proposal. I've made a note of it in case I run into relevant studies to pass them your way. :) --LZia (WMF) (talk) 19:13, 26 January 2017 (UTC)

Please note them here, for now. I will add links to some on-wiki lists tomorrow. Rich Farmbrough 00:15 28 January 2017 (GMT).

Interesting; interested

I think this is an interesting idea and I'm interested in learning more about it. --Rosiestep (talk) 17:00, 26 January 2017 (UTC)

Hi Rosiestep. Thanks for the note and interest. I'll ping you offlist to have a chat with you about some of your recent projects and see if we can reuse some of your learnings. If you have specific pointers you can share before we chat for me to educate myself more, please share them here. Thanks! --LZia (WMF) (talk) 21:09, 26 January 2017 (UTC)

Thanks for the call yesterday, Rosiestep. :) Here are a few highlights of our takeaways:

• We discussed onboarding tasks and sequences: typo fixing, type-editing, adding a section to an already existing article, and translations.
• What you have observed recently is that women may contribute more if they are in an environment that they know it's pretty much just them. This was based on an observation in one of your recent editathons where a large number of tech women (~150) were in a room with very few men (~3). You recommended that we keep an eye on this observation and see if we can use/assess it as part of the experiments. For example, this may mean that if the two participants know the gender of the other side and they are both women, we expect more engagement from them (and engagement should be defined, of course).
• We talked about teaming up two new-comers vs a new-comer and a more experienced editor (the latter needs to be defined) and that you expect if two new-comers are teamed up, they engage less (engagement needs to be defined).
• We talked about the upcoming editathons in March and the possibility of considering those as an opportunity for such experiments. (The way the current experiment is proposed, we will focus on those who organically open an account, and not those that attend an editathon in person necessarily.)

@SalimJah: and/or Rosie, if I have missed something, please add them here. Thank you! --LZia (WMF) (talk) 21:50, 1 February 2017 (UTC)

A note about editathons: for the editathons I have organised, most users created their accounts in ways that couldn't be tied to the editathon at the time of creation. They were either created in advance in anticipation of the editathon or were created via 3G/4G (associated with personal telecom provider not venue). Stuartyeates (talk) 08:17, 2 February 2017 (UTC)

Stuartyeates - This has been my experience, too. Account creation data doesn't sync up with event attendance. So if using the in-person editathon model for this research, the newcomer pairing would have to be configured differently --Rosiestep (talk) 17:37, 2 February 2017 (UTC)

Addressing under-representation along other lines: race, culture, etc.

I think this is a great project and am excited to see progress made to understand why the gender gap exists in contributions to Wikipedia, as well as to devise ways to begin to balance out contribution rates.

Seeing that the term "minority" is also being added to the research topic, I'd be curious to know how neatly the researchers believe gender and other identifiers (race, culture, sexual orientation, language, economic/educational status) can be grouped, examined, and addressed?

I would like to see more evidence that the 'minority gap' is driven by the same factors that influence the gender gap - not entirely clear to me, yet.

I consider this some of the most important work in which WMF / the community can engage, so thank you! SPatton_(WMF) 18:05, 26 January 2017 (UTC)

Good point, SPatton_(WMF), and thanks for your comment and encouragement. To be completely clear: we don't know if the same drivers of gender gap are the drivers of other kinds of, what I loosely call, misrepresentation in contributor demographics. We put the minority gap phrase there for now, as a reminder to ourselves and everyone reading the proposal that we should keep the broader problem in mind, though for the start, we will need to start from one specific population (and btw, we need to be clear what we mean by minority: women are not minority in general as we all know, but seem to be minority in the contributor population based on the references.) --LZia (WMF) (talk) 21:31, 26 January 2017 (UTC)

Well actually we do have a pretty good first approximation idea about drivers of some of these gaps. For example the primary driver of the contributor gender gap appears to be confidence related. Rich Farmbrough 00:19 28 January 2017 (GMT).

It might be because of personal failings but then again it could be the rape threats and the death threats. —Neotarf (talk) 02:52, 10 February 2017 (UTC)

Translation

Hey LZia (WMF). Great to see this research initiative. Can I mark this page up for translation so non-English speakers can weigh in on the discussion and help participate? Thanks, I JethroBT (WMF) (talk) 22:32, 26 January 2017 (UTC)

I JethroBT (WMF), thanks for bringing this up. I'm not sure having the research and discussion pages flagged for translation at this point is a good use of volunteers' time though (and I'm happy to be corrected by you or others who know more). Once we know which communities may be interested to be more involved, and if the decision is to go with a non-English speaking community, I would feel comfortable asking people to translate the page to that language. Again, I may be wrong, but I don't know how to differentiate this project from all the other research projects we do, and I don't think we want all of those to be translated (in an ideal world I would have loved to, btw.:). --LZia (WMF) (talk) 00:40, 28 January 2017 (UTC)

Great!

Please do not use the debunked 10% figure though.

I foresee one large issue here. The percentage of registrants that go on to make an edit is small, those that make more than one smaller still. Thus the danger is that we team a keen newbie with someone who will never log in again.

Rich Farmbrough 00:13 28 January 2017 (GMT).

Hi Rich. When you say "debunked 10% figure", what are you referring to?

Thanks for flagging the second point. I made a note of it for us to keep track of it. One thing that may happen is that if you team up people, people that normally would not contribute may end up contributing (which btw, would be an interesting finding on its own). If this doesn't happen and many do not do even the first few edits, we should think about alternative design approaches. --LZia (WMF) (talk) 00:35, 31 January 2017 (UTC)

A few comments

Hello, thank you for doing this. I think it might be more challenging than you foresee, but good luck with it. A couple of comments/questions. 1) How will you onboard the new people? 2) It says somewhere in here - can't find it now so this is a paraphrase - that the reason that women don't edit is a lack of confidence. This is repeated several times throughout the literature, and I tried to track it down and finally gave up, it seems like one big echo chamber where everyone is just repeating it. There was one mention of some kind of confidence survey where women were less likely to make something up if they didn't know the answer to something, and it strikes me that this might be a positive quality for a project that depends on verifiability and reliable sources. It may very well be that women don't edit because of personal failings and a lack of cognitive ability to evaluate their own circumstances, instead of taking on board what men explain to them, but IMO if you are going to say something negative about a group that is so underrepresented, that it needs more context, so it doesn't just sound like repeating a negative stereotype. 3) There are some successful groups that have a place to organize on social media outside of WP, this is particularly useful for people who have met each other IRL. —Neotarf (talk) 03:11, 10 February 2017 (UTC)

Thanks for sharing those thoughts, Neotarf! A few comments on your points 1 and 2:

1) There are two distinct steps to this project as it stands. The first step is to develop an edit recommendation system for new users according to their revealed interests. This will allow us to i) ask new users to do something useful that (ii) is relevant to their interests. Both elements should contribute to an increase in participation and retention. The second step is to use this system to identify pairs of users who share similar interests and evaluate the effectiveness of a "matched teams" intervention with regards to the gender gap.

2) The existing literature looks at gender differences in competitiveness and self-confidence while holding gender differences in ability constant. Those differences are notably well established in the controlled environment of the lab, where people are asked to perform tasks in which men and women perform equally well. This is not to say anything negative about women per se. I can actually think of many situations in which cooperativeness and cautiousness dominate self-assertiveness and overconfidence as a mode of operating. Regarding underconfidence, I actually think that we're saying the same thing: it's a systematic downward bias in women's evaluation of their own ability to perform some task, so that they often don't voice when in fact they should. This is the issue that we're trying to solve by tweaking the environment in which those women operate. Unlike previous interventions (see for instance this study in which a simple self-affirmation intervention aimed at boosting confidence reduces the gender achievement gap in college science), however, we try to implement this in a sustainable/ scalable way. SalimJah (talk) 19:42, 18 February 2017 (UTC)

@SalimJah Interesting study, I'm not really current with the field, but it looks like 2) "confidence" is an over-simplified description of what they are measuring, which has more to do with resisting "self fulfilling prophecy" or maybe "stereotype threat" or intentional or unconscious bias, not so much a lack of self esteem, but a coping mechanism for external pressures that underrepresented groups automatically face. There are some other assumptions here, that women don't edit more because they think it's too hard, which it isn't since most women already know how to use a keyboard, or that reading and writing are men's skills, which I think most people would argue the other way around, that women are considered to have more verbal skill than men, also the above exercise was for STEM, where women peak in about the 3rd or 4th grade, but men pull ahead in high school, exactly when attraction starts to happen, again trying to remember something I read a looong time ago. My concern I think was about creating a new barrier or prejudice that people think is a proven scientific principle.

As to the first step 1) I think if you can develop a successful model of onboarding newbies there will be a huge interest in it. But how to ask newbies to do some meaningful task, that is difficult given the current culture. There used to be a feature you could subscribe to that delivered automated editing suggestions to your talk page. Unfortunately a good many articles are guarded by users who believe they own the article, and anyone who makes an edit to the article will quickly find themselves in a dispute. I have heard this story over and over on other social media, and have come to believe it is the rule rather than the exception. Also the arbitration committee often breaks up groups of editors they believe might become "too powerful", and various staffers believe "women need to dial it back". So your best bet is to work with something already established like these people who are experienced in finding meaningful editing tasks; I am delighted to see the contact has already been made. Also just offhand I would say groups of two or three are small; how many academic "group grade" experiences can you think of that did not end in a horror story? You might try four or five, depending on experience, might that end up being one of your variables?

And 4) - I've lost count - please please please think really hard about primum non nocere and the ethics of what you are trying to do, encouraging unsuspecting women to step into what many insiders believe to be a "buzzsaw culture". This is probably much more common than is generally known, and I would not wish it on anyone, an unmarked pdf is here. Also this discussion needs to happen. —Neotarf (talk) 23:56, 19 February 2017 (UTC)

Self-fulfilling prophecies are a real risk, so this is a worthwhile discussion. Nemo 17:09, 20 March 2017 (UTC)

Thanks. I've tried to clarify the newbie onboarding strategy on the page. The self-fulfilling prophecy concern is an important one: while we explicitly list our research hypotheses on this page, the new users targeted by the intervention won't be exposed to them. The intervention will not explicitly encourage women to step-in. Based on the existing scientific evidence, it merely implements a slight modification of the environment in which they evolve (i.e., the creation of matched "teams" upon registration) to evaluate its impact on their intrinsic willingness to contribute. Nothing more. SalimJah (talk) 15:44, 20 April 2017 (UTC)

State of the funnel

@EGalvez (WMF): do you know of any surveys that have run recently (past six months) and can tell us something about the split in Wikimedia/Wikipedia readership by gender (or other demographics)? I'm ready to bet there is at least one survey that has collected this but I can't tell from Community_Engagement/Calendar. Thanks for your help in advance. --LZia (WMF) (talk) 16:10, 20 April 2017 (UTC)

Hi LZia (WMF) -About readers specifically, the only one I can think of off the top of my head is a Welsh readership survey done by WMUK. I'm not sure if they have results yet. You haven't been asking about gender in your quicksurveys? I also believe that for the strategy, we are working on understanding awareness of Wikipedia in GN as well as a few more GS countries. I am wondering if use (I read wikipedia) can be added to those surveys?--EGalvez (WMF) (talk) 19:17, 28 April 2017 (UTC)

Thanks EGalvez (WMF). No gender question in reader research so far (we're trying to keep those surveys very tight and short). How about editors? You recently conducted a large scale survey. Anything you can share from there? Thanks. --LZia (WMF) (talk) 19:20, 28 April 2017 (UTC)

Hi LZia (WMF) We are still working on getting the reporting completed and published. Probably a few more weeks until we can get something out. Do you need this more urgently? --EGalvez (WMF) (talk) 18:52, 1 May 2017 (UTC)

EGalvez (WMF), it would be great if we can get this one number out earlier. Can you help us with it? --LZia (WMF) (talk) 20:06, 1 May 2017 (UTC)