Show simple item record

Visual speech differentially modulates beta, theta, and high gamma bands in auditory cortex

dc.contributor.authorKarthik, G.
dc.contributor.authorPlass, John
dc.contributor.authorBeltz, Adriene M.
dc.contributor.authorLiu, Zhongming
dc.contributor.authorGrabowecky, Marcia
dc.contributor.authorSuzuki, Satoru
dc.contributor.authorStacey, William C.
dc.contributor.authorWasade, Vibhangini S.
dc.contributor.authorTowle, Vernon L.
dc.contributor.authorTao, James X.
dc.contributor.authorWu, Shasha
dc.contributor.authorIssa, Naoum P.
dc.contributor.authorBrang, David
dc.date.accessioned2021-12-02T02:27:50Z
dc.date.available2022-12-01 21:27:46en
dc.date.available2021-12-02T02:27:50Z
dc.date.issued2021-11
dc.identifier.citationKarthik, G.; Plass, John; Beltz, Adriene M.; Liu, Zhongming; Grabowecky, Marcia; Suzuki, Satoru; Stacey, William C.; Wasade, Vibhangini S.; Towle, Vernon L.; Tao, James X.; Wu, Shasha; Issa, Naoum P.; Brang, David (2021). "Visual speech differentially modulates beta, theta, and high gamma bands in auditory cortex." European Journal of Neuroscience 54(9): 7301-7317.
dc.identifier.issn0953-816X
dc.identifier.issn1460-9568
dc.identifier.urihttps://hdl.handle.net/2027.42/170934
dc.description.abstractSpeech perception is a central component of social communication. Although principally an auditory process, accurate speech perception in everyday settings is supported by meaningful information extracted from visual cues. Visual speech modulates activity in cortical areas subserving auditory speech perception including the superior temporal gyrus (STG). However, it is unknown whether visual modulation of auditory processing is a unitary phenomenon or, rather, consists of multiple functionally distinct processes. To explore this question, we examined neural responses to audiovisual speech measured from intracranially implanted electrodes in 21 patients with epilepsy. We found that visual speech modulated auditory processes in the STG in multiple ways, eliciting temporally and spatially distinct patterns of activity that differed across frequency bands. In the theta band, visual speech suppressed the auditory response from before auditory speech onset to after auditory speech onset (−93 to 500 ms) most strongly in the posterior STG. In the beta band, suppression was seen in the anterior STG from −311 to −195 ms before auditory speech onset and in the middle STG from −195 to 235 ms after speech onset. In high gamma, visual speech enhanced the auditory response from −45 to 24 ms only in the posterior STG. We interpret the visual‐induced changes prior to speech onset as reflecting crossmodal prediction of speech signals. In contrast, modulations after sound onset may reflect a decrease in sustained feedforward auditory activity. These results are consistent with models that posit multiple distinct mechanisms supporting audiovisual speech perception.Intracranial EEG data from a large cohort (n = 21) of patients shows that audiovisual speech elicits multiple distinct patterns of neural activity within the STG and adjacent cortex, occurring across separate frequencies and spatial/temporal distributions. Our results suggest that visual modulation of auditory speech perception utilizes multiple mechanisms, potentially reflecting independent sources of information.
dc.publisherAcademic Press
dc.publisherWiley Periodicals, Inc.
dc.subject.otherspeech
dc.subject.otheraudiovisual
dc.subject.otherECoG
dc.subject.otheriEEG
dc.subject.otherintracranial
dc.subject.othermultisensory
dc.subject.othersEEG
dc.titleVisual speech differentially modulates beta, theta, and high gamma bands in auditory cortex
dc.typeArticle
dc.rights.robotsIndexNoFollow
dc.subject.hlbsecondlevelNeurosciences
dc.subject.hlbtoplevelHealth Sciences
dc.description.peerreviewedPeer Reviewed
dc.description.bitstreamurlhttp://deepblue.lib.umich.edu/bitstream/2027.42/170934/1/ejn15482-sup-0001-Supplemental-Materials.pdf
dc.description.bitstreamurlhttp://deepblue.lib.umich.edu/bitstream/2027.42/170934/2/ejn15482.pdf
dc.description.bitstreamurlhttp://deepblue.lib.umich.edu/bitstream/2027.42/170934/3/ejn15482_am.pdf
dc.identifier.doi10.1111/ejn.15482
dc.identifier.sourceEuropean Journal of Neuroscience
dc.identifier.citedreferenceOkada, K., Venezia, J. H., Matchin, W., Saberi, K., & Hickok, G. ( 2013 ). An fMRI study of audiovisual speech perception reveals multisensory interactions in auditory cortex. PLoS ONE, 8 ( 6 ), e68959. https://doi.org/10.1371/journal.pone.0068959
dc.identifier.citedreferenceKaiser, J., Hertrich, I., Ackermann, H., & Lutzenberger, W. ( 2006 ). Gamma‐band activity over early sensory areas predicts detection of changes in audiovisual speech stimuli. NeuroImage, 30 ( 4 ), 1376 – 1382. https://doi.org/10.1016/j.neuroimage.2005.10.042
dc.identifier.citedreferenceKaiser, J., Hertrich, I., Ackermann, H., Mathiak, K., & Lutzenberger, W. ( 2005 ). Hearing lips: Gamma‐band activity during audiovisual speech perception. Cerebral Cortex, 15 ( 5 ), 646 – 653. https://doi.org/10.1093/cercor/bhh166
dc.identifier.citedreferenceKaras, P. J., Magnotti, J. F., Metzger, B. A., Zhu, L. L., Smith, K. B., Yoshor, D., & Beauchamp, M. S. ( 2019 ). The visual speech head start improves perception and reduces superior temporal cortex responses to auditory speech. eLife, 8, e48116. https://doi.org/10.7554/eLife.48116
dc.identifier.citedreferenceKayser, C., & Logothetis, N. K. ( 2009 ). Directed interactions between auditory and superior temporal cortices and their role in sensory integration. Frontiers in Integrative Neuroscience, 3, 7. https://doi.org/10.3389/neuro.07.007.2009
dc.identifier.citedreferenceKleiner, M., Brainard, D., & Pelli, D. ( 2007 ). “What’s new in Psychtoolbox‐3?” Perception 36 ECVP Abstract Supplement
dc.identifier.citedreferenceKumar, G. V., Halder, T., Jaiswal, A. K., Mukherjee, A., Roy, D., & Banerjee, A. ( 2016 ). Large scale functional brain networks underlying temporal integration of audio‐visual speech perception: An EEG study. Frontiers in Psychology, 7, 1558. https://doi.org/10.3389/fpsyg.2016.01558
dc.identifier.citedreferenceLazic, S. E. ( 2010 ). The problem of pseudoreplication in neuroscientific studies: Is it affecting your analysis? BMC Neuroscience, 11 ( 1 ), 1 – 17. https://doi.org/10.1186/1471-2202-11-5
dc.identifier.citedreferenceLazic, S. E., Clarke‐Williams, C. J., & Munafò, M. R. ( 2018 ). What exactly is ‘ N ’ in cell culture and animal experiments? PLoS Biology, 16 ( 4 ), e2005282. https://doi.org/10.1371/journal.pbio.2005282
dc.identifier.citedreferenceLega, B., Germi, J., & Rugg, M. D. ( 2017 ). Modulation of oscillatory power and connectivity in the human posterior cingulate cortex supports the encoding and retrieval of episodic memories. Journal of Cognitive Neuroscience, 29 ( 8 ), 1415 – 1432. https://doi.org/10.1162/jocn_a_01133
dc.identifier.citedreferenceLewis, A. G., & Bastiaansen, M. ( 2015 ). A predictive coding framework for rapid neural dynamics during sentence‐level language comprehension. Cortex, 68, 155 – 168. https://doi.org/10.1016/j.cortex.2015.02.014
dc.identifier.citedreferenceLuke, S. G. ( 2017 ). Evaluating significance in linear mixed‐effects models in R. Behavior Research Methods, 49 ( 4 ), 1494 – 1502. https://doi.org/10.3758/s13428-016-0809-y
dc.identifier.citedreferenceMcGrath, M., & Summerfield, Q. ( 1985 ). Intermodal timing relations and audio‐visual speech recognition by normal‐hearing adults. The Journal of the Acoustical Society of America, 77 ( 2 ), 678 – 685. https://doi.org/10.1121/1.392336
dc.identifier.citedreferenceMesgarani, N., Cheung, C., Johnson, K., & Chang, E. F. ( 2014 ). Phonetic feature encoding in human superior temporal gyrus. Science, 343 ( 6174 ), 1006 – 1010. https://doi.org/10.1126/science.1245994
dc.identifier.citedreferenceMicheli, C., Schepers, I. M., Ozker, M., Yoshor, D., Beauchamp, M. S., & Rieger, J. W. ( 2020 ). Electrocorticography reveals continuous auditory and visual speech tracking in temporal and occipital cortex. European Journal of Neuroscience, 51 ( 5 ), 1364 – 1376. https://doi.org/10.1111/ejn.13992
dc.identifier.citedreferenceOlasagasti, I., Bouton, S., & Giraud, A. L. ( 2015 ). Prediction across sensory modalities: A neurocomputational model of the McGurk effect. Cortex, 68, 61 – 75. https://doi.org/10.1016/j.cortex.2015.04.008
dc.identifier.citedreferenceOzker, M., Schepers, I. M., Magnotti, J. F., Yoshor, D., & Beauchamp, M. S. ( 2017 ). A double dissociation between anterior and posterior superior temporal gyrus for processing audiovisual speech demonstrated by electrocorticography. Journal of Cognitive Neuroscience, 29 ( 6 ), 1044 – 1060. https://doi.org/10.1162/jocn_a_01110
dc.identifier.citedreferenceOzker, M., Yoshor, D., & Beauchamp, M. S. ( 2018 ). Converging evidence from electrocorticography and BOLD fMRI for a sharp functional boundary in superior temporal gyrus related to multisensory speech processing. Frontiers in Human Neuroscience, 12, 141. https://doi.org/10.3389/fnhum.2018.00141
dc.identifier.citedreferencePeelle, J. E., & Sommers, M. S. ( 2015 ). Prediction and constraint in audiovisual speech perception. Cortex, 68, 169 – 181. https://doi.org/10.1016/j.cortex.2015.03.006
dc.identifier.citedreferencePlass, J., Brang, D., Suzuki, S., & Grabowecky, M. ( 2020 ). Vision perceptually restores auditory spectral dynamics in speech. Proceedings of the National Academy of Sciences, 117 ( 29 ), 16920 – 16927. https://doi.org/10.1073/pnas.2002887117
dc.identifier.citedreferencePlass, J., Guzman‐Martinez, E., Ortega, L., Grabowecky, M., & Suzuki, S. ( 2014 ). Lip reading without awareness. Psychological Science, 25 ( 9 ), 1835 – 1837. https://doi.org/10.1177/0956797614542132
dc.identifier.citedreferenceRay, S., Crone, N. E., Niebur, E., Franaszczuk, P. J., & Hsiao, S. S. ( 2008 ). Neural correlates of high‐gamma oscillations (60–200 Hz) in macaque local field potentials and their potential implications in electrocorticography. Journal of Neuroscience, 28 ( 45 ), 11526 – 11536.
dc.identifier.citedreferenceReale, R. A., Calvert, G. A., Thesen, T., Jenison, R. L., Kawasaki, H., Oya, H., Howard, M. A., & Brugge, J. F. ( 2007 ). Auditory‐visual processing represented in the human superior temporal gyrus. Neuroscience, 145 ( 1 ), 162 – 184. https://doi.org/10.1016/j.neuroscience.2006.11.036
dc.identifier.citedreferenceReuter, M., Rosas, H. D., & Fischl, B. ( 2010 ). Highly accurate inverse consistent registration: A robust approach. NeuroImage, 53 ( 4 ), 1181 – 1196. https://doi.org/10.1016/j.neuroimage.2010.07.020
dc.identifier.citedreferenceRiha, C., Güntensperger, D., Kleinjung, T., & Meyer, M. ( 2020 ). Accounting for heterogeneity: Mixed‐effects models in resting‐state EEG data in a sample of tinnitus sufferers. Brain Topography, 33 ( 4 ), 413 – 424. https://doi.org/10.1007/s10548-020-00772-7
dc.identifier.citedreferenceSakowitz, O. W., Quiroga, R. Q., Schürmann, M., & Başar, E. ( 2005 ). Spatio‐temporal frequency characteristics of intersensory components in audiovisually evoked potentials. Cognitive Brain Research, 23 ( 2‐3 ), 316 – 326.
dc.identifier.citedreferenceSchroeder, C. E., & Lakatos, P. ( 2009 ). Low‐frequency neuronal oscillations as instruments of sensory selection. Trends in Neurosciences, 32 ( 1 ), 9 – 18. https://doi.org/10.1016/j.tins.2008.09.012
dc.identifier.citedreferenceSchroeder, C. E., Lakatos, P., Kajikawa, Y., Partan, S., & Puce, A. ( 2008 ). Neuronal oscillations and visual amplification of speech. Trends in Cognitive Sciences, 12 ( 3 ), 106 – 113. https://doi.org/10.1016/j.tics.2008.01.002
dc.identifier.citedreferenceSmith, E., Duede, S., Hanrahan, S., Davis, T., House, P., & Greger, B. ( 2013 ). Seeing is believing: Neural representations of visual stimuli in human auditory cortex correlate with illusory auditory perceptions. PLoS ONE, 8 ( 9 ), e73148. https://doi.org/10.1371/journal.pone.0073148
dc.identifier.citedreferenceStouffer, S. A., Suchman, E. A., DeVinney, L. C., Star, S. A., & Williams, R. M. Jr. ( 1949 ). The American soldier: Adjustment during army life (Vol. 1 ). Princeton University Press.
dc.identifier.citedreferenceVan Wassenhove, V., Grant, K. W., & Poeppel, D. ( 2005 ). Visual speech speeds up the neural processing of auditory speech. Proceedings of the National Academy of Sciences, 102 ( 4 ), 1181 – 1186. https://doi.org/10.1073/pnas.0408949102
dc.identifier.citedreferenceVatakis, A., & Spence, C. ( 2007 ). Crossmodal binding: Evaluating the “unity assumption” using audiovisual speech stimuli. Perception & Psychophysics, 69 ( 5 ), 744 – 756. https://doi.org/10.3758/BF03193776
dc.identifier.citedreferenceViola, P., & Wells III, W. M. ( 1997 ). Alignment by maximization of mutual information. International Journal of Computer Vision, 24 ( 2 ), 137 – 154.
dc.identifier.citedreferenceVroomen, J., & Stekelenburg, J. J. ( 2010 ). Visual anticipatory information modulates multisensory interactions of artificial audiovisual stimuli. Journal of Cognitive Neuroscience, 22 ( 7 ), 1583 – 1596. https://doi.org/10.1162/jocn.2009.21308
dc.identifier.citedreferenceWang, L., Wang, W., Yan, T., Song, J., Yang, W., Wang, B., Go, R., Huang, Q., & Wu, J. ( 2017 ). Beta‐band functional connectivity influences audiovisual integration in older age: An EEG study. Frontiers in Aging Neuroscience, 9, 239. https://doi.org/10.3389/fnagi.2017.00239
dc.identifier.citedreferenceWang, X. J. ( 2010 ). Neurophysiological and computational principles of cortical rhythms in cognition. Physiological Reviews, 90 ( 3 ), 1195 – 1268. https://doi.org/10.1152/physrev.00035.2008
dc.identifier.citedreferenceAarts, E., Verhage, M., Veenvliet, J. V., Dolan, C. V., & Van Der Sluis, S. ( 2014 ). A solution to dependency: Using multilevel analysis to accommodate nested data. Nature Neuroscience, 17 ( 4 ), 491 – 496. https://doi.org/10.1038/nn.3648
dc.identifier.citedreferenceArnal, L. H., Morillon, B., Kell, C. A., & Giraud, A. L. ( 2009 ). Dual neural routing of visual facilitation in speech processing. Journal of Neuroscience, 29 ( 43 ), 13445 – 13453. https://doi.org/10.1523/JNEUROSCI.3194-09.2009
dc.identifier.citedreferenceArnal, L. H., Wyart, V., & Giraud, A. L. ( 2011 ). Transitions in neural oscillations reflect prediction errors generated in audiovisual speech. Nature Neuroscience, 14 ( 6 ), 797 – 801. https://doi.org/10.1038/nn.2810
dc.identifier.citedreferenceBarr, D. J., Levy, R., Scheepers, C., & Tily, H. J. ( 2013 ). Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of Memory and Language, 68 ( 3 ), 255 – 278. https://doi.org/10.1016/j.jml.2012.11.001
dc.identifier.citedreferenceBastos, A. M., Usrey, W. M., Adams, R. A., Mangun, G. R., Fries, P., & Friston, K. J. ( 2012 ). Canonical microcircuits for predictive coding. Neuron, 76 ( 4 ), 695 – 711. https://doi.org/10.1016/j.neuron.2012.10.038
dc.identifier.citedreferenceBeauchamp, M. S. ( 2016 ). Audiovisual speech integration: Neural substrates and behavior. In Neurobiology of language (pp. 515 – 526 ). Academic Press.
dc.identifier.citedreferenceBeauchamp, M. S., Argall, B. D., Bodurka, J., Duyn, J. H., & Martin, A. ( 2004 ). Unraveling multisensory integration: Patchy organization within human STS multisensory cortex. Nature Neuroscience, 7 ( 11 ), 1190 – 1192. https://doi.org/10.1038/nn1333
dc.identifier.citedreferenceBeauchamp, M. S., Nath, A. R., & Pasalar, S. ( 2010 ). fMRI‐guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect. Journal of Neuroscience, 30 ( 7 ), 2414 – 2417. https://doi.org/10.1523/JNEUROSCI.4865-09.2010
dc.identifier.citedreferenceBernstein, L. E., & Liebenthal, E. ( 2014 ). Neural pathways for visual speech perception. Frontiers in Neuroscience, 8, 386. https://doi.org/10.3389/fnins.2014.00386
dc.identifier.citedreferenceBesle, J., Fischer, C., Bidet‐Caulet, A., Lecaignard, F., Bertrand, O., & Giard, M. H. ( 2008 ). Visual activation and audiovisual interactions in the auditory cortex during speech perception: Intracranial recordings in humans. Journal of Neuroscience, 28 ( 52 ), 14301 – 14310. https://doi.org/10.1523/JNEUROSCI.2875-08.2008
dc.identifier.citedreferenceBesle, J., Fort, A., Delpuech, C., & Giard, M. H. ( 2004 ). Bimodal speech: Early suppressive visual effects in human auditory cortex. European Journal of Neuroscience, 20 ( 8 ), 2225 – 2234. https://doi.org/10.1111/j.1460-9568.2004.03670.x
dc.identifier.citedreferenceBourguignon, M., Baart, M., Kapnoula, E. C., & Molinaro, N. ( 2020 ). Lip‐reading enables the brain to synthesize auditory features of unknown silent speech. Journal of Neuroscience, 40 ( 5 ), 1053 – 1065. https://doi.org/10.1523/JNEUROSCI.1101-19.2019
dc.identifier.citedreferenceBrang, D. ( 2019 ). The stolen voice illusion. Perception, 48 ( 8 ), 649 – 667. https://doi.org/10.1177/0301006619858076
dc.identifier.citedreferenceBrang, D., Dai, Z., Zheng, W., & Towle, V. L. ( 2016 ). Registering imaged ECoG electrodes to human cortex: A geometry‐based technique. Journal of Neuroscience Methods, 273, 64 – 73. https://doi.org/10.1016/j.jneumeth.2016.08.007
dc.identifier.citedreferenceBrang, D., Plass, J., Kakaizada, S., & Hervey‐Jumper, S. L. ( 2020 ). Auditory‐Visual Speech Behaviors are Resilient to Left pSTS Damage. bioRxiv.
dc.identifier.citedreferenceChandrasekaran, C., Trubanova, A., Stillittano, S., Caplier, A., & Ghazanfar, A. A. ( 2009 ). The natural statistics of audiovisual speech. PLoS Computational Biology, 5 ( 7 ), e1000436. https://doi.org/10.1371/journal.pcbi.1000436
dc.identifier.citedreferenceChang, E. F., Rieger, J. W., Johnson, K., Berger, M. S., Barbaro, N. M., & Knight, R. T. ( 2010 ). Categorical speech representation in human superior temporal gyrus. Nature Neuroscience, 13 ( 11 ), 1428 – 1432. https://doi.org/10.1038/nn.2641
dc.identifier.citedreferenceChen, T., & Rao, R. R. ( 1998 ). Audio‐visual integration in multimodal communication. Proceedings of the IEEE, 86 ( 5 ), 837 – 852. https://doi.org/10.1109/5.664274
dc.identifier.citedreferenceDale, A. M., Fischl, B., & Sereno, M. I. ( 1999 ). Cortical surface‐based analysis: I. Segmentation and surface reconstruction. NeuroImage, 9 ( 2 ), 179 – 194. https://doi.org/10.1006/nimg.1998.0395
dc.identifier.citedreferenceElliott, T. M., & Theunissen, F. E. ( 2009 ). The modulation transfer function for speech intelligibility. PLoS Computational Biology, 5 ( 3 ), e1000302. https://doi.org/10.1371/journal.pcbi.1000302
dc.identifier.citedreferenceEngel, A. K., & Fries, P. ( 2010 ). Beta‐band oscillations—Signalling the status quo? Current Opinion in Neurobiology, 20 ( 2 ), 156 – 165. https://doi.org/10.1016/j.conb.2010.02.015
dc.identifier.citedreferenceErber, N. P. ( 1975 ). Auditory‐visual perception of speech. Journal of Speech and Hearing Disorders, 40 ( 4 ), 481 – 492. https://doi.org/10.1044/jshd.4004.481
dc.identifier.citedreferenceEskelund, K., Tuomainen, J., & Andersen, T. S. ( 2011 ). Multistage audiovisual integration of speech: Dissociating identification and detection. Experimental Brain Research, 208 ( 3 ), 447 – 457. https://doi.org/10.1007/s00221-010-2495-9
dc.identifier.citedreferenceFischl, B., Sereno, M. I., & Dale, A. M. ( 1999 ). Cortical surface‐based analysis: II: Inflation, flattening, and a surface‐based coordinate system. NeuroImage, 9 ( 2 ), 195 – 207. https://doi.org/10.1006/nimg.1998.0396
dc.identifier.citedreferenceGroppe, D. M., Urbach, T. P., & Kutas, M. ( 2011 ). Mass univariate analysis of event‐related brain potentials/fields II: Simulation studies. Psychophysiology, 48 ( 12 ), 1726 – 1737. https://doi.org/10.1111/j.1469-8986.2011.01272.x
dc.identifier.citedreferenceHickok, G., Rogalsky, C., Matchin, W., Basilakos, A., Cai, J., Pillay, S., & Binder, J. ( 2018 ). Neural networks supporting audiovisual integration for speech: A large‐scale lesion study. Cortex, 103, 360 – 371. https://doi.org/10.1016/j.cortex.2018.03.030
dc.identifier.citedreferenceHullett, P. W., Hamilton, L. S., Mesgarani, N., Schreiner, C. E., & Chang, E. F. ( 2016 ). Human superior temporal gyrus organization of spectrotemporal modulation tuning derived from speech stimuli. Journal of Neuroscience, 36 ( 6 ), 2014 – 2026. https://doi.org/10.1523/JNEUROSCI.1779-15.2016
dc.identifier.citedreferenceKadipasaoglu, C. M., Baboyan, V. G., Conner, C. R., Chen, G., Saad, Z. S., & Tandon, N. ( 2014 ). Surface‐based mixed effects multilevel analysis of grouped human electrocorticography. NeuroImage, 101, 215 – 224. https://doi.org/10.1016/j.neuroimage.2014.07.006
dc.identifier.citedreferenceKadipasaoglu, C. M., Forseth, K., Whaley, M., Conner, C. R., Rollo, M. J., Baboyan, V. G., & Tandon, N. ( 2015 ). Development of grouped icEEG for the study of cognitive processing. Frontiers in Psychology, 6, 1008. https://doi.org/10.3389/fpsyg.2015.01008
dc.working.doiNOen
dc.owningcollnameInterdisciplinary and Peer-Reviewed


Files in this item

Show simple item record

Remediation of Harmful Language

The University of Michigan Library aims to describe library materials in a way that respects the people and communities who create, use, and are represented in our collections. Report harmful or offensive language in catalog records, finding aids, or elsewhere in our collections anonymously through our metadata feedback form. More information at Remediation of Harmful Language.

Accessibility

If you are unable to use this file in its current format, please select the Contact Us link and we can modify it to make it more accessible to you.