Grants talk:IdeaLab/Project Accuracy

I think it would be a mistake to fund this project before the community has had a say in whether it will accept and collaborate with the project. Generally the en.wiki community is highly allergic to any thing that smells the slightest of top-down control and/or paid editing. This has aspects of both and while I agree in ideal with the aim and method I think in practice it is a major departure from our community ideals. This is a major risk that I think the current description does not identify. Also another risk is that it could discourage editor recruitment and retention to suddenly have a layer of authority that is not open to the normal processes of consensus building and participation. It would for example not take many times for some of my writing to be "overruled" by a "Review Board" (unless I were in agreement with their corrects) before I would consider whether to continue contributing. I have added some other thoughts to user:Atsme's talkpage--Maunus (talk) 00:45, 26 March 2016 (UTC)

Paid editing? What exactly do you mean, Maunus? Are you referring to the Grant? Atsme^📞📧 20:02, 26 March 2016 (UTC)

Further response to the concerns: posted at the project's TP: ....the RAAFA status would be a level above in that FAs would be reviewed by a qualified Editorial Review Board (ERB) comprising members that have met certain qualifications; the latter of which will be determined by the various project teams. Currently, we have 4731 FAs which will keep the ERB quite busy for a while. The intention is not to undermine or lessen the impact of FAs or the current review process of quality assessment - quite the opposite - rather it is a program designed to further validate and enhance those promoted articles with a level of review that can be "marketed" (recognized) and further promoted outside WP as reliable because of the way Project Accuracy and the ERB will be structured; i.e., a well-balanced, qualified team of editors following an established set of criteria to help ensure accuracy by corroborating information in the article with the cited sources, etc. One could liken it to a WP image enhancement campaign in that it will be further boosted by publicity designed to improve public perception of WP as a reliable source. Of course, the finer details still need to be worked out and there will be some learning curves after launch.

I have no idea of any basis for most of the expressed concerns. We're talking about article quality and promotion of quality articles. How it became a speculative view of paid editing, overruling, layer of authority, and so on is beyond me. Atsme^📞📧 20:07, 26 March 2016 (UTC)

Yes, I think the grant would risk coming across that way unless there is total transparency and community wide involvement and agreement with the goals and organization of the project. (Basically I dont even see what the requested 90,000 dollar grant would be spent of if as you claim the project will work entirely within the established structure of volunteer wikipedia editing). I think it is worrying that you not only did not foresee these types of complaints, but that you also seem not to understand them, since understanding them will be necessary in order to answer them satisfactorily when they inevitably arise.--Maunus (talk) 16:50, 27 March 2016 (UTC)

Maunus, to begin, it wasn't until after I started filling in some of the blanks here that I was encouraged to expand my proposal into a grant application. I wasn't even considering a grant in the beginning but what I'm finding is that it's difficult to find highly qualified editors who can or perhaps are even willing to volunteer that much of their time because they have to work for a living. What happened to AGF? Also, you may not have noticed but my original idea - after all, this is an "IdeaLab" - included a proposal to create a Dallas/Ft. Worth chapter of WP which would have required renting a conference room, paid advertising, promotion, etc. Nothing is set in stone yet - I haven't even completed the application - so I'm wondering what role you have assumed in this process, other than trying to shoot it down before it's even off the ground? You've also done a good job of misrepresenting the Project's mission when you expressed your conspiratorial ideas of potential risk, and I find that rather disconcerting. I am well aware that my presentation is incomplete which is why I sought guidance, requested input at the Project's TP, and haven't completed the application. While I can appreciate constructive criticism, your input of late has been rather negative and extremely critical based on misinterpretations. Sorry, but I don't consider that being helpful. Atsme^📞📧 20:45, 27 March 2016 (UTC)

@Atsme:. That sounds very much like you were considering using the grant money for compensation for qualified fact writers. That of course is a form of paid editing, and while I personally am not opposed to that a priori, I know many editors are. Personally I would be happy to receive payment for the valuable content work I believe I do on wikipedia, and would also approve of oithers doing so if they were transparent about it. You clearly should have thought that through before filing the grant proposal (and before creating the project on Wiki) and as you thought about the risks. I am not actually espousing any of these conspiracies, and I am happy to assume good faith, but I think you clearly have not sufficiently thought out the strategy for how to get community acceptance and participation in this project. That s why I strongly recommend against funding the project until the issue of community acceptance has been settled (or at least addressed). Similar projects in the past have turned into absolute disasters both for wikipedias editing climate and from a publicity perspective - and starting up top-down projects that are then rejected by the comunity is a major source of frustration and a waste of money for the wikimedia foundation. That is why the risk of an adverse community reaction cannot be taken lightly. I think you severely underestimate the magnitude of the project you are proposing and of the risks that its implementation faces. I actually think that pointing out to you that you are underestimating the effort it probably the best help for you at this early stage in the process where you can still stop up and consider those obstacles and prepare for how to move around them. I think you should start by trying to get broad community input at en.wiki through the village pump, so that a formal proposal to create the project can be started with full community wide support. If you think I am being harsh, then you will be in for a surprise when you are confronted with people who are actually completely opposed to the idea. First of all you will need to be able to formulate clear and convincing answers to the five questions that I made at the project talkpage, cause those will be the major hurdles the project will face. --Maunus (talk) 22:32, 27 March 2016 (UTC)

• I made some comments at Wikimedia Foundation Board noticeboard#Build or rebuild which may be relevant here. There is a lot of anxiety felt when innovation is proposed which appears to change the establised way of doing things, especially at such long-established projects as the English-language Wikipedia. However, some of the innovations proposed are better seen as layered on top of, or a new way of accessing, existing projects. If the proposal here is to create a new way of presenting knowledge, building on the existing Wikipedia(s) as a foundation but adding a layer of expert assessment, peer-review or whatever, then that is a complement to, not a threat to, the current knowledge bases. Of course, looking at it that way implies things about the presentation technically and culturally. It is a fair challenge to say that this is a new thing and needs new and additional modes of curation and support, and to ask what resources would be needed to develop and support it. But the implicit suggestion that it would be additional work for, or a change to the work of, existing volunteers is not, it seems, a logical barrier. If the existing volunteers do not wish to do this work, then there should be no pressure on them to do so: but that work will have to come from somewhere. Rogol Domedonforsr talk:Rogol Domedonfors (talk) 21:05, 27 March 2016 (UTC)

Yes Rogol Domedonfors (talk · contribs), there is a lot of unwarranted anxiety around innovation. But if we can learn anything from the obstacles projects of major innovation have faced in the past it is that community anxiety must not be underestimated and that it needs to be addressed head on, with full transparency from the outset. Any innovation that comes as a surprise and without community support will be rejected, no matter how beneficial it can potentially be. That is why all the potential objections I list must be addressed before even making the suggestion to the wider community.--Maunus (talk) 22:51, 27 March 2016 (UTC)

Maunus, should I be concerned you may be seeing my grant proposal as competition [1], since they are similar? I was not aware that you applied for a grant after I created Project Accuracy on February 3, 2016 and had been requesting input from that point forward? Perhaps Rogol Domedonfors or I JethroBT can help advise if our two grants could be considered conflicting. Atsme^📞📧 14:46, 28 March 2016 (UTC)

My idea does not involve a grant request.--Maunus (talk) 18:30, 29 March 2016 (UTC)

Funding

With respect to formal review, IMO this should be done in collaboration with already established journals. Maybe the Wikimedia Foundation could cover part of the "open access" costs at PLOS Medicine for those who wish to bring an article through this process? Doc James (talk · contribs · email) 15:44, 28 March 2016 (UTC)

By the way if you look at the right upper hand corner of the Dengue fever article you will notice a little journal icon. If you click on it it takes you to pubmed commons to the formally published version. And then interestingly people are referencing the published version of the Wikipedia article.[2] Doc James (talk · contribs · email) 22:33, 30 March 2016 (UTC)

Peer review

By the way we currently have the Wikiversity Journal of Medicine that provides peer review.

And we just had a second journal open on Wikiverity called the Second Journal of Science

There are no publication fees. Doc James (talk · contribs · email) 22:27, 30 March 2016 (UTC)

What are the funds for?

Can you provide a breakdown of what the funds are for? Doc James (talk · contribs · email) 22:33, 30 March 2016 (UTC)

Thank you, Doc James. I had an encouraging discussion with WMF staff this morning, and have since revised the grant proposal. As discussion continues here and on the TP of WP:Project Accuracy, our goals and ways to achieve those goals are continuously developing. (continuing below)

Okay. Will wait for further details. Best Doc James (talk · contribs · email) 23:21, 30 March 2016 (UTC)

┌─────────────────────────────────┘
Our posts resulted in an edit conflict, so I'll start over. Thank you, Doc James. I had an encouraging discussion with WMF staff this morning, and have since revised the grant proposal. As discussion continues here and on the TP of WP:Project Accuracy, our goals and ways to achieve those goals are continuously developing. Following are some of the highlights for the efforts of our Primary Project Coordinators (PPC) and why funds would be needed:

1. outreach efforts to recruit qualified WP editors as well as off-wiki academics/experts/professionals to serve on the editorial review board (ERB),
2. collaborate with various project teams to assemble FAs for review, and help coordinate an ERB best suited for reviewing the articles - drawing from the 100+/- reviewers who have agreed to participate to form the pool;
3. initially help with the project's site design, or help line-up experienced design people we can hire;

End result - a balanced mix of reviewers will comprise the ERB - possibly 50 to 100 we can draw from - including academics, writers/experts with RW experience in public dissemination who can translate highly specified jargon for the general public, experts in the areas relevant to the articles they'll be reviewing, and so forth. The PPC will coordinate the reviews and collaborate with the project teams (or individuals) who submit the FAs for promotion. I also have plans for "promotion drives" that will offer prizes (either monetary or product or both) which will create incentive and encourage quality. Atsme^📞📧 23:28, 30 March 2016 (UTC)

Okay so money and prices for who exactly? The reviewers?

Google spent a huge among of money on reviewers for medical content per here and it was not successful. Doc James (talk · contribs · email) 23:33, 30 March 2016 (UTC)

Wow, Doc - I wasn't aware of that project. The approach I had in mind was quite different but now I'm thinking I need to test the waters first because it appears the medical articles are going to be the biggest hurdle to clear but not impossible. Give me a little time to think about it, put feelers out and get input from some of my close friends who are specialists (heart, gastro, etc.) and we'll visit this issue again. Atsme^📞📧 03:06, 31 March 2016 (UTC)

Hi Atsme, who are the "100+/- reviewers who have agreed to participate to form the pool" mentioned above? Are there reviewers actually lined up, or do you mean that you hope 100 editors (or someone else?) will volunteer, or that you want to employ 100? SarahSV ^talk 20:58, 3 April 2016 (UTC)

Hi, SlimVirgin - Proj Med already has willing reviewers but I don't have all the details to know exactly how everything will tie-in. I'm diligently working on it and have 3 appointments lined up next week. Just the other day, I began a dialogue with an academic who is also a practitioner. Better yet, he has an "in" to a world-wide network. He didn't quite understand what I was proposing in my email - it's a bit harder to pitch when they don't understand how WP works - but he will be in Dallas on the 17th so I'll be able to visit with him in person. That will give me time to put together a legible proposal he can take back to the Dean and share with others in his network. As soon as I get input from him with regards to the best approach, I will begin presenting it to various other universities, practitioners and experts, that way we will have a mix of academics and experienced private sector experts to draw from but with an actual commitment. I also have a dialogue started with WMF staff regarding the potential for an IEG but can't really move forward on that until I know the particulars, and why/if we would need funding. My plan is to prepare a presentation showing the universities (and private sector experts) how the ERB will prove mutually beneficial. Doc James is helping me when he can. I am amazed at Doc's energy and ability to do so many different things. He rarely ever skips a beat. I am also in awe of Project Med, their website layout and how much that team has accomplished. Hopefully Project Accuracy will grow with help from various other projects so we can focus our energy and efforts on achieving our much needed goals. It has been a decade or so since I put together a presentation but I'm pretty sure I can still do it with a little help. I will need more eyes reviewing what I get done which is why I was hoping to recruit at least 5 primary project coordinators. This project is probably going to be a build it and they will come effort. Atsme^📞📧 21:45, 3 April 2016 (UTC)

Thanks, Atsme. One problem is that you haven't said what will motivate reviewers. If you pay them enough, that will do it, but then you have to say how you will raise the funds. If you don't pay them, you'll have to offer something else. You could arrange for academic reviewers to be given academic credit, but that would involve a lot of work and persuasion. For non-academic reviewers, what might motivate them? Reviews are a great deal of work, with nothing to show for it (unlike writing your own articles). People often offer FAC reviews because they want someone to review their own FACs, so there is an indirect quid pro quo. But there is a constant shortage of reviewers nevertheless. You're proposing even more stringent and time-consuming reviews, and you want to involve academics who won't need editors here to review their work, so why would they do it? That's the first problem to solve, I would say. SarahSV ^talk 22:55, 3 April 2016 (UTC)

SlimVirgin - funding will be part of the discussion for the meeting on Monday, specifically the possibility of WMF providing small grants to participating universities. Another thought was to provide participating universities (along with the grant?) in-kind support to help set-up Wiki projects and as you mentioned, give academic credit for reviews (the latter being the part that my initial email contact - the academic/practitioner - didn't quite understand). For fauna & flora (biological sciences internships and the like), I plan to approach USF&WS and various State game & fish resource agencies (note: as a public outreach effort, the respective agencies can set aside their own sludge funds to cover it). I'm thinking Army Corps will be another resource for engineering, and NOAA for another. Once we get the medical, science, and biology FAs promoted by a team of qualified reviewers, the WMF will be more open to funding other things we need, such as website design (we may need 3 pages to list all members of our ERB with headshot and accreditation), media promotion for the new editorial review board project that will help improve WP's image and snowball in a most positive way. The 5 primary project coordinators are the ones who will initially help get things off the ground, and that's where I was thinking a grant would be quite useful, but first I want to get all my ducks in a row and get phase one launched. Does that make sense? Atsme^📞📧 00:05, 4 April 2016 (UTC)

Break

Atsme, thanks again for the reply. I'm still not sure how you see this, so a few questions:

• Would the reviewers be Wikipedians or external specialists?
• How will you motivate them? (Reviewing is time-consuming and unpleasant work, particularly fact-checking.)
• If they are going to be paid, how will you raise the money? And how will you persuade the FA writers that they should write for free so that others can review their work for pay?
• Would the reviewers post a review somewhere or edit the FAs? (The latter would not go down well.)
• FAs have to be kept up to date. That's especially true with science. What happens to the accuracy seal when the FA writers update the article?

SarahSV ^talk 05:14, 5 April 2016 (UTC)

Thank you for your questions, SlimVirgin. My answers are in green text interspersed with a recap of your questions. Atsme^📞📧 16:48, 5 April 2016 (UTC)

• Would the reviewers be Wikipedians or external specialists?

• There may be times when it will be a combination - contingent upon the topic of the submission(s) - and while anonymity is allowed, validation of credentials will be required to become a member of the ERB.

• How will you motivate them? (Reviewing is time-consuming and unpleasant work, particularly fact-checking.)

• Academics will be motivated by their university curriculum. There are also mutual advantages associated with internships.
• Another motivation will be the media publicity surrounding WP's announcement of an ERB, along with the prestige of being included on the project's ERB page(s), complete with headshot and academic accreditation/expert credentials.
• The plan also involves WMF awarding a small annual grant to the participating universities which will start with a few and hopefully grow in numbers. Establishing participating universities and setting up the program will require exerted effort by our primary project coordinators (PPC); the exerted effort part contributing to why I'm having such a hard time finding volunteers. 😊
• We could tap into our own academic resources for help and direction in getting to the right people with our proposal. Doc James said he already has willing reviewers but needs the quality articles. Perhaps he will interject. Community colleges, trade schools and smaller state-funded colleges are an obvious choice for non-medical articles and topics respective to their areas of expertise.
• In summary, the entire review process is contingent upon the submissions and topics to be reviewed, and will be dependent on our PPC's ability to get submissions to the relevant ERB. We can use categories as the basis for establishing ERBs.

• If they are going to be paid, how will you raise the money? And how will you persuade the FA writers that they should write for free so that others can review their work for pay?

• At this time, direct payment to individual reviewers is not on the table.
• Grants to universities are a consideration and I've been trying to schedule a time with WMF staff to discuss our options (which I mistakenly thought would happen yesterday).
• Our PPCs will need funds to get certain things done once the project is up and operating, such as website design, attending meetings involved with our recruiting efforts - all of which I see as a one-time initial cost that the WMF can consider renewing annually depending on the results.
• Re: FA writers - contests with prizes and awards, recognition and peace of mind knowing their work will finally have a level of protection it didn't have previously. The monetary awards for the competitions will be included in our grant request.

• Would the reviewers post a review somewhere or edit the FAs? (The latter would not go down well.)

Reviews will involve a 3 Tier process as follows:

• Tier 1 - the process begins with a list of FAs to be reviewed by the relative ERB. The list includes FAs submitted by project teams, individual authors and/or Project Accuracy's primary project coordinators (PPC). Submissions will be organized according to category and presented to the proper ERB for review. (The submission process will be similar to that of FAC)
  
• Tier 2 - each submission will undergo review by the ERB (5 to 7 reviewers). Credentialed academics, practitioners, and experts will comprise the ERB (some may be WP editors), and will alternate depending on topic. For example, if a medical article is submitted for review, the PPC will organize a team of qualified medical reviewers and possibly an expert(s) in public dissemination to help maintain a general public rating for readability. Hopefully, we will have a pool of at least 20+/- reviewers to choose from in the beginning. We will have a page for our ERB with a headshot of each participant along with their credentials, something like what this Journal has done. The ERB's review and assessment will focus on (fact-checking) corroborating information with the cited sources to determine (1) accuracy, (2) reliability of the cited sources, and (3) quality of the overall presentation. The ERB will not actually be editing anything. Once the members of the ERB have each completed their reviews, they will discuss their findings (on the ERB's review page) and reach a collaborative assessment that will be presented as a critique to the authors (editors) and/or project team if there is one. The submission may pass the initial review and receive the gold seal or it may need revision. The critiques will be used by the authors to make the necessary revisions during a specified time frame - they may be simple revisions that can be done in a day such as an upgrade to higher quality source, or it may require more research and take longer which will require resubmission. Reasons will vary.
  
• Tier 3 - if an author requires further discussion about the assessment, the ERB may assign one or more of their members (ideally experts on the subject) to answer questions or help the author(s) with the necessary revisions. When that process is complete and the revisions have been made, the article can be submitted again beginning with the Tier 1 review.
  

• FAs have to be kept up to date. That's especially true with science. What happens to the accuracy seal when the FA writers update the article?

• To begin, in order for any accuracy/fact-checking/reliability status to work, the reviewed and approved articles must have a level of protection to prevent vandalism and incompetence from destroying their reliability. Updates can certainly be made but with a slight detour that involves a review process not too unlike what we currently have in place, such as pending changes review but a beefed-up version of it. We also have "stewardship" at the project team level and watchlists that are maintained by bots and individual article creators/authors. Perhaps code can be written that specifically targets the gold sealed articles and affords them the type of protection we need while maintaining the anyone can edit policy.
• If a preliminary review determines the proposed changes pass the acid test (for lack of a better term), a review of the proposed change(s) by an ERB will occur. The majority of those articles will not require the same time and effort as the original review so I don't see it creating a problem. Also, keep in mind that previously reviewed and approved articles are a level above our customary FA reviews, a substantial number of which did not receive the equivalency of a fact-checking review as will be performed by verified experts comprising our ERB. Granted, the entire process will be a collaborative effort dependent in part on the combined efforts of individual project teams and the PPC. One could liken it to an expanded version of Project Med but with an extra layer of reassurances.

Keeping in mind that the project is still in its initial stages of development, it remains open to ideas for improvement. What I see as the hardest obstacle to overcome is moving beyond status quo which appears to have stagnated despite WMFs concerns over the substantial drop in readership and editor retention. Atsme^📞📧 16:48, 5 April 2016 (UTC)

┌─────────────────────────────────┘
Hi Atsme, none of these replies make sense to me, I'm afraid. That academics would want internships or be motivated by their curricula, etc., I can't see it. The link you provided below is about students editing articles, not specialists reviewing them.

When I've persuaded specialists to review articles, I've had a particular reason: it's a widely read article, or it's going to be on the main page. And I've sought out people who devote their lives to the issue (not only to the general area), so they're highly motivated to make sure everything is accurate.

Re: updating FAs: "If a preliminary review determines the proposed changes pass the acid test ..." etc, that isn't how things work. When UNICEF updates its prevalence figures for FGM, we have to respond fairly quickly and change all the figures in the article, the map, the tables, include additional countries if they have, and change any text that makes comparisons that rely on the old figures. It's fiddly, dull work. No one is going to waste time talking about it rather than doing it. SarahSV ^talk 01:26, 6 April 2016 (UTC)

Added note

SlimVirgin, please also consider the following: "By 2014, more than 98% of undergraduates said they used Wikipedia. Crucially, the action taken most often with Wikipedia’s content among those undergraduates was verification — checking to see if what Wikipedia said accurately reflected what they’d read in class. That tells us that as Wikipedia has permeated the culture, students are becoming more aware of how to read Wikipedia through a critical lens." [3]. Atsme^📞📧 18:01, 5 April 2016 (UTC)

• "Academics will be motivated by their university curriculum." I don't understand what this means. Currently, in the academic world that I know, everything in University curricula motivates against participating in non-prestigeous un-remunerated review activities. I don't know any academics who would care about the "mutuality of internships". There is a degree of peer pressure and respect around contributing peer reviews to academic journals - but this is not currently the case for contributing to wikipedia in any way.--Maunus (talk) 21:43, 5 April 2016 (UTC)

• I gleaned something totally different from this but I may have misinterpreted the intended direction and involvement of academia as it relates to Wikipedia. Atsme^📞📧 21:53, 5 April 2016 (UTC)

• That link is about student editing, not specialist reviews. SarahSV ^talk 01:28, 6 April 2016 (UTC)

┌─────────────────────────────────┘
What I saw at the first link wasn't just about student editing. What I saw were key points to growing this project:

• "Today, the Wiki Education Foundation has worked with 478 instructors at 282 universities..." <--- major participation
• "But it started with just a handful of pioneering instructors." <---- it grew
• "That project became a siren call for universities to open up access to their own materials." <---sharing knowledge = contributing quality articles and editors to WP
• "Crucially, the action taken most often with Wikipedia’s content among those undergraduates was verification — checking to see if what Wikipedia said accurately reflected what they’d read in class. That tells us that as Wikipedia has permeated the culture, students are becoming more aware of how to read Wikipedia through a critical lens." <---- they are already fact-checking; therefore, with the proper presentation and a little nurturing, I see mutual benefits resulting from the creation of an ERB under the supervision of scholars.

What I saw at the second link was further assurance that the programs academia already has in place can be easily expanded to provide qualified reviewers for the ERB:

• "These students learn communication skills essential for any practicing health professional, and they improve the public’s access to reliable, accurate knowledge about health and disease." <---key words = reliable, accurate
• "The results of the first three cycles of Dr. Azzam’s Wikipedia-editing medical school course will be published later this year in a top-tier medical education journal." <---a medical education journal = peer reviewed + editorial review board.
• "Dr. Azzam sets students to work on fact-checking an article’s content, and adding sections that are missing. This term, all seven of his students collaborated on a single article: hepatitis." <---fact-checking article content. It can also serve as a model for other topics.

4th year medical students fact-checking under the supervision and guidance of a lead academic. Atsme^📞📧 03:03, 6 April 2016 (UTC)