Personalized Feedback Versus Money: The Effect on Reliability of Subjective Data in Online Experimental Platforms
Ye, Teng; Reinecke, Katharina; Robert, Lionel
2017-02-25
Citation
Ye, T., Reinecke, K. and Robert, L. P. (2017). Personalized Feedback Versus Money: The Effect on Reliability of Subjective Data in Online Experimental Platforms, Proceedings of the 20th ACM Conference on Computer Supported Cooperative Work and Social Computing Companion, February 25 - March 01, 2017, Portland, OR, USA
Abstract
We compared the data reliability on a subjective task from two platforms: Amazon's Mechanical Turk (MTurk) and LabintheWild. MTurk incentivizes participants with financial compensation while LabintheWild provides participants with personalized feedback. LabintheWild was found to produce higher data reliability than MTurk. Our findings suggest that online experiment platforms providing feedback in exchange for study participation can produce more reliable data in subjective preference tasks than those offering financial compensation.Publisher
ACM
Other DOIs
Subjects
Crowdsourcing Online Experimentation Mechanical Turk Compensation Crowdworker Compensation Crowdsourcing compensation incentives crowdsourcing incentives motivation crowdsourcing motivation online study data quality online experimentation data reliability online studies data reliability LabinWild
Types
Article
Metadata
Show full item recordShowing items related by title, author, creator and subject.
-
Innis, Jeffrey W.; Goodman, Frances R.; Bacchelli, Chiara; Williams, Thomas M.; Mortlock, Douglas P.; Sateesh, Praveen; Scambler, Peter J.; McKinnon, Wendy; Guttmacher, Alan E. (Wiley Subscription Services, Inc., A Wiley Company, 2002-05)
-
Guevara-Fujita, Maria; Fahrner, Stacey; Buraczynska, Kinga; Cook, Jason; Wheaton, Dianna; Cortes, Fanny; Vicencio, Cesar; Pena, Marcela; Fishman, Gerald A.; Mintz-Hittner, Helen; Birch, David; Hoffman, Dennis; Mears, Alan J.; Fujita, Ricardo; Swaroop, Anand (John Wiley & Sons, Inc., 2001-02)
-
Hoefele, Julia; Sudbrak, Ralf; Reinhardt, Richard; Lehrack, Silvia; Hennig, Steffen; Imm, Anita; Muerb, Ulla; Utsch, Boris; Attanasio, Massimo; O'Toole, John F.; Otto, Edgar A.; Hildebrandt, Friedhelm (Wiley Subscription Services, Inc., A Wiley Company, 2005-04)
Remediation of Harmful Language
The University of Michigan Library aims to describe library materials in a way that respects the people and communities who create, use, and are represented in our collections. Report harmful or offensive language in catalog records, finding aids, or elsewhere in our collections anonymously through our metadata feedback form. More information at Remediation of Harmful Language.
Accessibility
If you are unable to use this file in its current format, please select the Contact Us link and we can modify it to make it more accessible to you.