Research:Community-centered Evaluation of AI Models on Wikipedia/Study Recruitment

Greetings!

Are you someone who enjoys reviewing edits on the English Wikipedia? We could really use your assistance (with a maximum compensation of $150 USD)!

About Us:

We are a research team from Carnegie Mellon University, working together with the creator of ORES. Our research aims to help Wikipedians evaluate the quality of AI models on Wikipedia and identify areas for improvement. These models, such as ORES and Liftwing, power Huggle, SWViewer, recent change filters, etc., and aid patrollers in reviewing edits. We believe Wikipedians' voices are essential in evaluating AI models from their individual and collective perspectives, especially given the increasing use of AI models in Wikipedia nowadays.

Based on a formative study in our research, we developed Wikibench, a system that facilitates the AI evaluation process while centering Wikipedians' voices. We would be thrilled if you could participate in our pilot testing for this system!

The Studies

Our pilot testing involves two studies:

1. Onboarding session: The first study consists of a 1-hour onboarding session to help you become familiar with our system. We will show you how the system work and gather your initial reaction and feedback on the system.
2. Exit interview: After the first study, we kindly request that you continue using the system for one week while performing your patrolling work (with lightweight daily requirements that take less than 20 minutes per day). The second study takes place after one week, when we will learn about your experience with the system through a brief 30-minute exit interview.

Both studies will be conducted via Zoom (audio and screen-share only). We may also explore other options (e.g., text-based) if that works better for you. Please feel free to specify your preference in the screening survey below.

Compensation

Upon successfully completing both studies, you will be eligible to receive a maximum compensation of $150 USD.

Specifically, we will provide $30 as compensation for the one-hour onboarding session, and $20 for the 30-minute interview. Additionally, if you meet the lightweight daily participation requirements for using our system during your one-week patrolling period, you will receive $20 per day, for up to 5 days.

How to participate

This study has been completed, and we are not currently recruiting participants. To receive updates about Wikibench and future study opportunities, please sign up here.

Contact us

If you have any questions, please don't hesitate to contact us. We greatly appreciate your time and consideration!