Grants talk:IEG/Reimagining Mentorship on Wikipedia (old)

How will this be better?

Hey guys. I'm glad to see this idea getting fleshed out. Two key question about this project that I hope to be stated in plain terms are,

• "What's currently lacking in Wikipedia's current mentoring spaces?" and
• "How will our proposed effort improve the current state?"

--EpochFail (talk) 18:17, 24 September 2013 (UTC)

Thanks for the questions!

Problem: The exiting mentoring program differs from person to person, has no outreach program (see below), is fragmented, lacks any sort of remotely standardized curriculum, and lacks reinforcers for program completion. The existing program also has an extensive time toll on mentors.

Solution: Using our proposed outreach program, we will identify more people who would benefit from this program. Using standardized curriculum in modules, we can isolate what a user needs to know in shorter segments that they are more likely to complete while retaining individualized attention. We will also implement a modulated reward system to encourage program completion and, through standardization and semi-automation, reduce workload on mentors without sacrificing quality. --Jackson Peebles (talk) 20:16, 24 September 2013 (UTC)

+1 to the question and answers here. Jackson, if your concise definition of the problem and solution could be added to the main page (I'm missing it, if it's there, perhaps just lost in lots of other text?), it would probably help most readers of your proposal get a better handle on the overall intention. Feel free to add a new section towards the top called "the problem" or something if that helps. I feel like the 2 biggest pieces of the problem with existing adoptions that you may be experimenting with solutions for in a project like this is 1) proactive outreach to new editors who might not know about mentorship as an option, rather than waiting for them to be bold enough to ask for a mentor themselves, and 2) breaking 1:1 learning into smaller tasks that people need help with and can easily teach, so that mentors and mentees don't wander off partway through (as I understand they often do in the current adopt-a-user setup). I wish there was some good baseline data beyond anecdotal evidence to demonstrate these problems, in terms of number of new editors who seek adoption per month and number who complete it...I wonder if part of this project would involve collecting that, or if anyone has these numbers from the current adoption program? Siko (WMF) (talk) 21:20, 24 September 2013 (UTC)

I definitely agree that having some baseline data to get a good idea of where we stand now would definitely give us an idea on where we can improve, and that this should be the first thing done in the grant (or leading up to it starting. I do think a decent survey tool would come in handy here, getting feedback from both current and past mentees and mentors would be invaluable. I know that WMF has access to a survey tool, not sure if that's available to us but if not, would request to add a small amount so access to a suitable survey tool is possible. I agree that the focus should be on starting things on a small scale, focusing more on teaching core skills (that the mentee wants to learn) rather than the focus being on formal pairing ups (but that can be done if the two want to do that). We could then measure the effect this method has against the current, and see if it has had a positive effect. I think this is the best way to start this project. Steven Zhang (talk) 12:35, 25 September 2013 (UTC)

Proactive newcomer support

One of the most interesting aspects of wiki-mentorship for me is building the software support necessary to enable mentors to proactively find newcomers who need support. My primary concern about the current state of mentorship in Wikipedia is based on research that shows/argues that the primary failure of mentorship in Wikipedia is that it doesn't come soon enough[1][2], while reverts[3], warnings[4] and deletions[5] come almost immediately. I think that we can have a big impact by building a practice around getting a live, helpful person to good newcomers sooner.

It doesn't look like this proactive newcomer discovery work is part of the current proposal. It seems perfectly reasonable to me that such efforts would be outside of the scope of work that you guys have in mind. If not, I'd be happy to capture my ideas within the proposal. --EpochFail (talk) 18:17, 24 September 2013 (UTC)

EpochFail, I respectfully disagree. Please see Scope and Activities, Bullet 3 and Project Map Steps 2, 3, and 4. Reaching out to proactively find newcomers is actually one of our highest priorities. We want to reach the largest possible audience of interested individuals who fit criteria for inclusion (see Meet Your Match and Matchmaking sections). We repeatedly reference Snuggle (although this would not be exclusive) as a means to identify Good Faith users who need extra training. --Jackson Peebles (talk) 20:16, 24 September 2013 (UTC)

I see. It could be that I'm missing something, but it looks like, starting at step 1, newcomers would be expected navigate to a specific page and read some documentation. Then with step 2, newcomers would fill out a profile. Only at step 3 could mentors interact with mentees by matching profiles and offering to start a mentoring relationship. Surely, we should expect that many newcomers would not care to go through such a protracted process even if they received an invitation. I don't mean to lessen the importance of such a mentoring program. It's just that, once a newcomer decides to go through the whole process of requesting mentorship, they've probably already survived Wikipedia's impersonal wall of warning templates and decided to invest their time and energy into an entirely separate endeavor than actually editing the wiki[6].

I realize that what I've said may read like an attack on formal mentorship. Please know that I don't intend it as such. I suspect that formal mentor-mentee relationships are the way that many prolific Wikipedians begin their editing careers. It's just that I see the proactive, informal mentorship that I've designed Snuggle to support as a complementary activity that occurs for all good-faith newcomers who run into trouble, whether they would like to enter a formal mentor-mentee relationship or not. In fact, I see a clear path by which early support for newcomers could guide many promising newcomers towards the formal mentorship practice described in this proposal. --EpochFail (talk) 21:04, 24 September 2013 (UTC)

Forgot to ping Jackson Peebles --EpochFail (talk) 22:59, 24 September 2013 (UTC)

EpochFail, I don't think any of us took that as an attack on formal mentorship. Your concerns are very valid. I see how the wording could have been interpreted that way and how I could have overlooked that phrasing since I have my "mental idea" on how this would work. I assure you that our intent is to reach out to others. In fact, I envision a banner, at some point. The spirit is to seek out those who would benefit, not require them to seek us out! --Jackson Peebles (talk) 16:30, 25 September 2013 (UTC)

Jackson Peebles. Awesome. It looks like we're on the same page. Maybe we could split the components of this proposal into two phases of mentorship: informal mentorship/detection and formal mentorship/training. Informal mentorship would commonly occur before "step 1" and involve both the detection of good-faith newcomers and support without starting a formal relationship. For those newcomers who accept and invitation or find the formal mentoring spaces themselves, formal mentorship and training would begin with step 1.

I imagine that we'd target the masses with phase 1 (informal mentorship) and target a smaller set of interested newcomers with phase 2 (formal mentorship). Does that sound right to you? Behold my ASCII art:

                .-----------------------------------------.
                |                                         V
[New registrations] ---> [Snuggle & other tools] --> [Formal mentorship]
                |              ^     |                    ^
                |              |     V                    |
                '-------> [Informal mentorship]-----------'

So, most good newcomers would proceed directly through Snuggle and receive Informal mentorship and/or an invitation to apply for formal mentorship. A smaller proportion would obtain informal mentorship without being caught by Snuggle users and a very small proportion would proceed directly to formal mentorship. --EpochFail (talk) 20:54, 25 September 2013 (UTC)

you might want to consider welcome / invitations, to newbies to mentee, as was done at teahouse. Slowking4 (talk) 21:33, 30 September 2013 (UTC)

Community notification

I'd suggest posting community notification in both EN:WP's Teahouse and Adopt-a-user program, as this project would probably have some impact on those folks and you'd want to make sure they're aware, endorsing or giving feedback, etc. If you've already done that, can you please post the links in the notification section on the main page? I'm also generally guessing this would be piloted as a new space on EN:WP, but if the experiment proved successful, there might be a future aim to join forces or propose changes based on what was learned and piloted here into the existing Adopt-a-user program, or sharing outward to other language Wikipedias with other 1:1 adoption programs? Cheers! Siko (WMF) (talk) 21:41, 24 September 2013 (UTC)

PS - I think there is a typo in the budget table - 3,000 or 5,000 for programming?

Fixed I've read (thoroughly) all of the other ideas/suggestions/questions, and I'll address them just as soon as I can! Thank you, everyone, for the thorough feedback. Steven Zhang, can I have some assistance on these? I'm pretty swamped. --Jackson Peebles (talk) 05:09, 25 September 2013 (UTC)

Yup, let me check what you've done and I will post in some other places. I think we should think of a name for the pilot space (and something better than "New Adoption", maybe, er, Mentorship 2.0? Steven Zhang (talk) 11:53, 25 September 2013 (UTC)

Eligibility confirmed, round 2 2013

This Individual Engagement Grant proposal is under review!

We've confirmed your proposal is eligible for round 2 review. Please feel free to ask questions here on the talk page and make changes to your proposal as discussions continue during this community comments period.

The committee's formal review for round 2 begins on 23 October 2013, and grants will be announced in December. See the schedule for more details.

Questions? Contact us.

Community Notifications

Hi Steven and Jackson,
As you know, the IEG committee will begin their review of round 2 proposals on 23 October 2013. I'm looking over the community notifications section of each eligible application to expedite review. Thanks for the links to your Teahouse and Adopt-a-User posts. Please also provide links to your Wikimedia-l and Wikimedie-e posts. This will help the committee follow up more easily. Thanks!
Best of luck! :)
Anna Koval (WMF) (talk) 19:23, 23 October 2013 (UTC)

Hi Anna. Took some searching, but have added them now. I also posted to WikiEN-L today. Steven Zhang (talk) 01:51, 24 October 2013 (UTC)

Thanks, Steven. :) Good luck! :) --Anna (WMF) 15:50, 25 October 2013 (UTC)

My thoughts

I haven't participated in this project's formation as much as I would have liked, but Steven pinged me on enwp to comment, and I would say that, after reading the proposal as it stands, this sounds to me like potentially the best thing we have done for newcomers since the Teahouse, and the two can "collaborate" to provide newcomers with a dynamic support system. The idea of a "lighter weight" adoption system would take pressure off both newcomers and those running the program, as speaking from experience, running a personal adopt-a-user program can be very time consuming. Overall, I applaud Jackson and Steven for their leadership of this endeavor, and look forward to participating once it gets approved for a grant :-). Note: Ping me on my en.wp talk page if needed, thanks! Go Phightins! (talk) 02:23, 24 October 2013 (UTC)

Automatically matching mentors/mentees

Can you tell me more about how you envision an algorithm for matching mentors and mentees might work? Are potential mentees expected to fill out a profile about themselves before receiving mentorship? Is this not a higher barrier to entry than the current system for requesting mentorship? --EpochFail (talk) 15:16, 24 October 2013 (UTC)

I'm tossing up between a few ideas. It could perhaps be automatically requested as part of a trial modification to the new account creation process (they answer a few brief questions). I'm also considering the idea of an automated process monitoring the edits of new users, and then recommending a lesson for them based on those (i.e - a bot (like ClueBot) watches the recent changes feed, an editor with less than ~50 edits or so inserts a reference, but it's format is broken or is a bare URL, so they're sent a message recommending a lesson they can take.). I'm tossing up between these two options at present. Steven Zhang (talk) 21:04, 24 October 2013 (UTC)

Measures of success

The first measure of success, "Editors that go through the new mentorship program go on to edit longer, more often, and more diverse contributions [...]," is difficult to identify without controlled experimentation. The problem lies in the filter. Newcomers who are willing to request mentorship are categorically more active/productive editors(http://dx.doi.org/10.1145/2038558.2038586). I recommend adding to the success measures that "more newcomers will seek and attain mentorship" independent of their outcomes post mentorship. Note that if you are highly successful in getting more newcomers to participate (by opening the filter/lowering the barrier of entry), I'd expect to see the overall outcomes fall because you'll end up measuring the activities of less dedicated newcomers along-side the highly dedicated that made it over the higher barriers that are currently in place. --EpochFail (talk) 15:30, 24 October 2013 (UTC)

That's a good measure to add - I'm going to add it. I think that measurement of success that you have mentioned can be identified. Part of this project will involve analysing the current state of mentorship, so I would look at 10 or so editors that go through the existing mentorship methods, and measure things like number, frequency and diversity of edits over a period of time. We would then implement the changes we come up with as part of this project, and run it in parallel for a trial, and measure the results (20 use the old method, 20 use the new method). I think this is quite feasible, and definitely can be measured. Steven Zhang (talk) 21:37, 24 October 2013 (UTC)

Some questions from Deskana

In order to evaluate this, some additional information would be nice, namely:

1. The estimated duration of the project.
2. The usernames of the people who will be receiving the relevant funds. In the case that these are not presently known, please state that.
3. The estimated time commitment per week for each person involved in the project (e.g. 37.5 hours per week), and the duration of their involvement (e.g. 1 month).

--Deskana (talk) 16:23, 24 October 2013 (UTC)

1. Hi Dan. I estimate the project will take approximately 6 months, the first six weeks will be about clearly establishing a baseline to improve on, so looking at users that go through existing mentorship processes, and gathering from that how often, how much and how diverse their edits are. I also think a survey would be effective here, of existing adopters and editors that have gone through mentorship, to get their feedback and ideas on ways forward. The next six weeks would be used to develop the tools (the matching software) and write the skill lessons, with the following two months an initial trial of the new mentorship process, evaluated half way through the trial. We would then measure the results in the following month, ask those who participated would be asked to give their feedback on the process, and write up our final results.
2. Jackson and myself would receive 3,000 and 7,000 for our involvement, respectively. At this time, we do not have programmers or designers in mind, but will be seeking them out shortly.
3. I plan to dedicate at least 15 hours a week to this project, for the duration of the grant. Jackson will contribute around 10 hours. We have discussed the distribution of the stipend, but I'll let Jackson comment on that further. Steven Zhang (talk) 12:40, 25 October 2013 (UTC)

Comments

First, I'll say that the need for a continual supply of new volunteers is the most important thing for the sustainability of our projects. Many of our larger projects are suffering from a decline on volunteers and if this mentorship project works, it might be able help that situation. I'll just add that I have a lot of faith in Jackson and Steven because of their commitment and dedicated work at enwp. Also, having Aaron's support and help is a big plus because he's done the research and knows the problems we face and many/most of the solutions. Therefore, I think this is an important project and fully support it.

Having said that, I'll echo Deskana's comment in that I'd like to see a more detailed breakdown of the expenses. Also, I believe Jackson had an earlier IEG at en:Wikipedia talk:Video and Interactive Tutorials. I'd just like a quick comment if that project is completed and what he thought of it. Just noting that I made some suggestions on the talk page for that project. Lastly, I'm certain Aaron will make sure we get good data and analysis, but I'd like to make sure there's a plan in place to get before and after data that's statistically valid. That way we can measure the effectiveness of the project. The 10-20 editors in the current plan seems a little low from a statistics standpoint, but I'm not an expert. 64.40.54.157 04:45, 25 October 2013 (UTC)

Hi there. I've given a bit more info regarding the breakdown of funds. 10-20 is a rough figure, but when I consider the level of detail we will need to go into for each editor, it may be a reasonable sample (we are going to be looking at more than just edit count, for example). I would welcome suggestions on this if anyone feels a sample of 20 to establish a baseline is too statistically insignificant, of course :) Steven Zhang (talk) 12:42, 25 October 2013 (UTC)

Thanks very much for the response, Steven. I didn't realize the depth of detail you were getting in to with each editor. 10-20 could very well be an appropriate number. As I mentioned, I'm not an expert and would defer to others, including Aaron who has a great deal of experience in this area. Thanks. 64.40.54.76 05:27, 26 October 2013 (UTC)

Just noting that I've found Grants:Jackson Peebles/Video and Interactive Tutorials/Report, so my questions to Jackson are now answered. 64.40.54.76 06:10, 26 October 2013 (UTC)

What about social norms?

I really like the approach of this project. I do wonder if, and I could be perceiving this incorrectly, the emphasis on skills is overlooking some of the importance of mentoring newcomers about the social norms of participating in Wikipedia. For example, in the education program I always encourage professors and new editors to introduce themselves to Wikiproject members or prominent editors of an article so that they do not take them by surprise by the sudden influx of edits. Another example might be mentoring newcomers around how to interact with editors that are reverting their work or how to appeal to other editors for support in such cases. With the technical barrier being lowered by such tools as visual editor, I wonder if part of this initiative should also place some emphasis on "social" skills rather than focusing only on technical skills. Gabrielm199 (talk) 14:25, 1 November 2013 (UTC)

I've got to agree. I see social norms as the larger barrier to entry into Wikipedia than a lack of wiki/encyclopedia editing skills. If one knows how to behave, then resolving technical issues seems like a more minor concern. --EpochFail (talk) 14:35, 1 November 2013 (UTC)

It might be worth thinking about designing the interview protocol for the first phase of this project to include questions that address normative behavior and newcomers. Happy to help with this if others are interested. Gabrielm199 (talk) 13:49, 4 November 2013 (UTC)

Aggregated feedback from the committee for Reimagining Mentorship on Wikipedia

Scoring criteria (see the rubric for background)	Score 1=weakest 5=strongest
Potential for impact
(A) The project fits with the Wikimedia movement's strategic priorities	4
(B) The project has the potential to lead to significant online impact.	4
(C) The impact of the project can be sustained after the grant ends.	4
(D) The project has potential to be scaled or adapted for other languages or projects.	4
Ability to execute
(E) The project has demonstrated interest from a community it aims to serve.	4
(F) The project can be completed as scoped within 6 months with the requested funds.	4
(G) The budget is reasonable and an efficient use of funds.	4
(H) The individual(s) proposing the project have the required skills and experience needed to complete it.	4.5
Fostering innovation and learning
(I) The project has innovative potential to add new strategies and knowledge for solving important issues in the movement.	4
(J) The risk involved in the project's size and approach is appropriately balanced with its potential gain in terms of impact.	3.5
(K) The proposed measures of success are useful for evaluating whether or not the project was successful.	4
(L) The project supports or grows the diversity of the Wikimedia movement.	4
Comments from the committee: A very interesting concept that could be suitable for most of the bigger wikipedias and other wikimedia projects. May become a “keystone piece” in the new editor onboarding process. Taking a 'Teahouse approach' in building sustained motivation and preventing editor dropouts is a wonderful opportunity to develop a true mentor-mentee support system that would increase the activity of new contributors. Proposers are highly qualified and driven mentors with a useful background in teaching new editors and understanding the learning process. Improved mentorship could also reduce the workload of experienced editors as less housekeeping will have to be done. With some concerns about game elements like badges and scores, we’d recommend that the proposal would be trialed and tested on a small scale initially. Concerns about lack of focus and funding for a plan to bring in “good” mentors, which may be as critical to success as software and design changes. Unclear how the project can be adapted and used in other languages or projects. Measurable goals could be more clear and include quantitative targets.

Thank you for submitting this proposal. The committee is now deliberating based on these scoring results.

Funding decisions will be announced by December 16. — ΛΧΣ²¹ 00:25, 14 November 2013 (UTC)

Status update

This project has not been selected for an Individual Engagement Grant at this time.

We love that you took the chance to creatively improve the Wikimedia movement. The committee has reviewed this proposal and not recommended it for funding, but we hope you'll continue to engage in the program. Please drop by the IdeaLab to share and refine future ideas!

Comments regarding this decision:
We enjoyed seeing this idea, and mourn the passing of Jackson Peebles, who was clearly a great asset to this project as well as to Wikipedia as a whole. We strongly encourage the remaining project members to return with an updated team for consideration in a future round of IEG. We’d also enjoy hearing more about the team’s plans to use badges and identify mentors in a future proposal.

Next steps:

1. Review the feedback provided on your proposal and to ask for any clarifications you need using this talk page.
2. Visit the IdeaLab to continue developing this idea and share any new ideas you may have.
3. To reapply with this project in the future, please make updates based on the feedback provided in this round before resubmitting it for review in a new round.
4. Check the schedule for the next open call to submit proposals - we look forward to helping you apply for a grant in a future round.

Questions? Contact us.

Alright folks (Steven Zhang, Matty.007, Gabrielm199, Go Phightins! & TheOriginalSoni), what's next? I propose that we get on a voip call or IRC to reorganize this week. My schedule is pretty open, so I'll let someone who's more constrained propose a time. In the meantime, I'll autojoin #wikipedia-en-help^connect as "halfak". Feel free to ping me. --EpochFail (talk) 23:51, 16 December 2013 (UTC)

Hmm. I'm not on WikiMedia much to a TB on wikipedia will probably get me better. I am usually free at weekends on IRC, though the times vary. Thanks, Matty.007 (talk) 20:47, 17 January 2014 (UTC)

questions about allocation of funds - javascript

was there an intention or assumption that the javascript work (a few thousand dollars) would go to technical13?

If not, presumably the work would go out to tender? How would this work? Demiurge1000 (talk) 21:00, 2 April 2014 (UTC)

Hey Demiurge1000, This is the old proposal that wasn't funded because Jackson passed. The new proposal is here: Grants:IEG/Reimagining_Wikipedia_Mentorship. Cheers, Ocaasi (talk) 21:11, 2 April 2014 (UTC)