Eliciting and Leveraging Input Diversity in Crowd-Powered Intelligent Systems
dc.contributor.author | Song Kwon, Jean Young | |
dc.date.accessioned | 2020-01-27T16:25:07Z | |
dc.date.available | NO_RESTRICTION | |
dc.date.available | 2020-01-27T16:25:07Z | |
dc.date.issued | 2019 | |
dc.date.submitted | 2019 | |
dc.identifier.uri | https://hdl.handle.net/2027.42/153428 | |
dc.description.abstract | Collecting high quality annotations plays a crucial role in supporting machine learning algorithms, and thus, the creation of intelligent systems. Over the past decade, crowdsourcing has become a widely adopted means of manually creating annotations for various intelligent tasks, spanning from object boundary detection in images to sentiment understanding in text. This thesis presents new crowdsourcing workflows and answer aggregation algorithms that can effectively and efficiently improve collective annotation quality from crowd workers. While conventional microtask crowdsourcing approaches generally focus on improving annotation quality by promoting consensus among workers, this thesis proposes a novel concept of a diversity-driven approach. We show that leveraging diversity in workers' responses is effective in improving the accuracy of aggregate annotations because it compensates for biases or uncertainty caused by the system, tool, or the data. We then present techniques that elicit the diversity in workers' responses. These techniques are orthogonal to other quality control methods, such as filtering, training or incentives, which means they can be used in combination with existing methods. The crowd-powered intelligent systems presented in this thesis are evaluated through visual perception tasks in order to demonstrate the effectiveness of our proposed approach. The advantage of our approach is an improvement in collective quality even in settings where worker skill may vary widely, potentially lowering barriers to entry for novice workers and making it easier for requesters to find workers who can make productive contributions. This thesis demonstrates that crowd workers' input diversity can be a useful property that yields better aggregate performance than any homogeneous set of input. | |
dc.language.iso | en_US | |
dc.subject | Input Diversity | |
dc.subject | Human-Computer Interaction | |
dc.subject | Crowdsourcing | |
dc.subject | Human Computation | |
dc.subject | Artificial Intelligence | |
dc.subject | Machine Learning | |
dc.title | Eliciting and Leveraging Input Diversity in Crowd-Powered Intelligent Systems | |
dc.type | Thesis | |
dc.description.thesisdegreename | PhD | en_US |
dc.description.thesisdegreediscipline | Electrical Engineering: Systems | |
dc.description.thesisdegreegrantor | University of Michigan, Horace H. Rackham School of Graduate Studies | |
dc.contributor.committeemember | Lasecki, Walter | |
dc.contributor.committeemember | Pierce, Casey Brittany Spruill | |
dc.contributor.committeemember | Corso, Jason | |
dc.contributor.committeemember | Griffin, Brent | |
dc.contributor.committeemember | Kim, Juho | |
dc.subject.hlbsecondlevel | Computer Science | |
dc.subject.hlbsecondlevel | Electrical Engineering | |
dc.subject.hlbtoplevel | Engineering | |
dc.subject.hlbtoplevel | Science | |
dc.description.bitstreamurl | https://deepblue.lib.umich.edu/bitstream/2027.42/153428/1/jyskwon_1.pdf | |
dc.identifier.orcid | 0000-0003-4379-3971 | |
dc.identifier.name-orcid | Song, Jean Young; 0000-0003-4379-3971 | en_US |
dc.owningcollname | Dissertations and Theses (Ph.D. and Master's) |
Files in this item
Remediation of Harmful Language
The University of Michigan Library aims to describe library materials in a way that respects the people and communities who create, use, and are represented in our collections. Report harmful or offensive language in catalog records, finding aids, or elsewhere in our collections anonymously through our metadata feedback form. More information at Remediation of Harmful Language.
Accessibility
If you are unable to use this file in its current format, please select the Contact Us link and we can modify it to make it more accessible to you.