The Effect of Listener Experience and Social Expectation on Illusory Percepts

Abstract

Listeners’ expectations and predictions about their interlocutors during the course of speech perception have had a central role in the development of sociophonetic models of speech perception. These models emphasize that listeners use their expectations about a speaker’s social identity and their knowledge of the structured phonetic variation associated with those identities to guide perception as listeners perceptually adapt to their interlocutors. In a similar vein, multisensory integration has also been shown to modulate linguistic percepts when listeners are presented with incongruent auditory and visual cues. Under these conditions visual cues eclipse auditory information generating illusory percepts where listeners report hearing what they see. This dissertation explores the intersection of these two literatures and probes whether dialect specific visual signals facilitate socially indexed perceptual adaptation without acoustic reinforcement. Two experiments were conducted using illusory stimuli to assess how Indian English and American English participants shifted their categorization and speech shadowing strategies when listening to model talkers from each dialect. In both experiments, participants interacted with illusory stimuli that were comprised of an acoustic voiced bilabial stop paired with visual articulations constructed to induce coronal percepts (alveolar or retroflex stops or interdental fricatives) or labial percepts (labiovelar approximants or labiodental fricatives or approximants). Crucially, these stimuli required participants to confront phonological substitutions, mergers, or splits from the other dialect, which they were exposed to through a video that provided experience with the real-world speech patterns of the model talkers. For example, in labial conditions, American English participants were predicted to learn a merger between their labiodental fricative and labiovelar approximant categories that would reflect the labiodental approximant in Indian English, while Indian English participants were predicted to learn to split their labiodental approximant category into labiodental fricatives and labiovelar approximants to reflect the American English contrasts. In the categorization experiment, participants’ categorization of illusory stimuli before and after exposure suggest that, as predicted, American English participants learned to shift (in a subset of conditions) their categorization strategies, reflecting the merged patterns of the Indian English model talker even in light of multisensory incongruity. In comparison, Indian English participants’ responses only showed within dialect categorization shifts, but in a manner that was rooted in both model talker specific expectations and experience. In the shadowing experiment, conducted to assess intra-category sensitivity to illusory stimuli, participants produced baseline productions of target words and, after receiving experience with the model talkers, shadowed, or imitated, illusory (audiovisual incongruent) and veridical (audiovisual congruent) stimuli. F2-F3 values were extracted to measure post-exposure shifts in production of coronal stimuli and normalized F2 values were extracted for labial stimuli. Measures were compared across the baseline, veridical, and illusory conditions. Against predictions, results showed that model talker specific shadowing was confined to veridical conditions, where both participant groups imitated congruent audiovisual stimuli across dialect boundaries, a finding that suggests that imitation may require acoustic reinforcement. This dissertation makes multiple contributions to research on linguistic expectation and perceptual adaptation. Results of the categorization experiment suggest that sociophonetic perception persists in light of multisensory incongruity given the right linguistic experience and broad inter-category measures. Results from shadowing suggest that imitation may need acoustic reinforcement when targeting socially indexed intra-category variation. Taken together, illusions provide a novel path forward for researching the structure of sociophonetic perception and whether these percepts depend on acoustics.