Research talk:How LLMs impact knowledge production processes

Requested clarifications

Can you please make a number of clarifications.

1. How long for and where will collected data be stored? What privacy protections have been incorporated? What procedures are being used to ensure anonymity?
2. What is "IRB" at the University of Minnesota?
3. Is the research part of any larger project? How is the research being funded?
4. Who is responsible for any (potential) ethics or privacy related complaints? How can they be contacted? They should be independent of the researchers conducting the project.
5. A question on method - how do you control for non-users of LLMs in the research? Many editors on English Wikipedia have raised numerous ongoing concerns about the use of LLMs.

Regards, --Goldsztajn (talk) 03:57, 14 January 2025 (UTC)

Hi @Goldsztajn, thank you for your interest! Hopefully the following answers would help clarify:

1. The collected data (the recording for audio) will be stored safely in University of Minnesota BOX (cloud storage) until when we finished the data analysis. Currently we estimated to finish the study by end of April. Even if the study takes longer than we expected, it will be deleted within 6 months after the interview.

2. IRB stands for Institutional Review Board, which is a group that reviews research involving human subjects to make sure that the research is ethical and protects the rights of participants.

3. This is a standalone study as of now, and there is no funding. We hope that the results from this study could inspire future work, and establish the foundation for a larger project that may have funding. Sadly, this also means that we do not have monetary compensation for participants.

4. copied from our IRB information sheet: "This research has been reviewed and determined exempt by the Institutional Review Board (IRB), which is part of the Human Research Protections Program (HRPP). If there is an issue you would like to discuss with someone who is not on the research team you are encouraged to call the HRPP Research Participants’ Advocate Line at 612-625-1650 (Toll Free: 1-888-224-8636) or go to z.umn.edu/participants."

5. This is a good question - in this study, we do not consider the non-users of LLMs. We hope to understand how LLMs users interact with LLMs in Wikipedia, thus get a realistic pictures of how LLMs are used. To make sure that our participants have actually used LLMs, participants need to fill out a survey first (which has a link on the project page), and we will determine their qualification from their answers.

Again, thank you so much for your questions! Please let me know if you have more questions!

Best, Phoebezz22 (talk) 16:34, 14 January 2025 (UTC)

Hi @Phoebezz22 thanks for the reply. A couple of responses.

1. The response indicates audio interviews will be retained for no more than six months - do your instructions for setting up the interviews include methods to ensure cameras are not turned on? Does the message indicate that informants' interviews will be recorded?
2. Qualtrics is being used for the initial survey - Qualtrics tracks IPs - are participants informed this is happening? - or that IP tracking has been turned off?
3. It is good to be transparent regarding the "IRB" - this information indicates that the research is exempt from ... what? Registration with human research ethics compliance? What is the basis of the exemption? This is potentially an international research project - do you only expect your participants to come from the USA? That phone number is not convenient for anyone outside North America. Can I suggest you make the link to the actual website where there is a feedback form prominent and visible on the project page.
4. I'm not sure your method will address the problem of the disruption caused by LLMs. In my opinion it's too selective a data set - you appear to be entirely focused on "good" actors.

Regards, Goldsztajn (talk) 01:13, 15 January 2025 (UTC)

@Goldsztajn Hi thanks for your reply and suggestions.

1. Yes we inform our interview participants.

2. We acknowledge that IP addresses would be automatically tracked, and we turned off IP tracking, since it is not information that directly relates to the research.

3. I will put the link you suggested on the project page. As for information, here is the reason from IRB for exemption: "This study met the following category(ies) for exemption: (2) Research that only includes interactions involving educational tests (cognitive, diagnostic, aptitude, achievement), survey procedures, interview procedures, or observation of public behavior (including visual or auditory recording) if at least one of the following criteria is met: (ii) Any disclosure of the human subjects’ responses outside the research would not reasonably place the subjects at risk of criminal or civil liability or be damaging to the subjects’ financial standing, employability, educational advancement, or reputation"

4. I will think more about this. Phoebezz22 (talk) 14:02, 15 January 2025 (UTC)