Brainstem and Cortical Encoding of Speech-in-Noise Under Visual Speech Cues
DOI:
https://doi.org/10.66238/jaeee19Keywords:
Multisensory Integration, Auditory Brainstem, Cortical Tracking, Speech Perception, ElectroencephalographyAbstract
The human ability to comprehend spoken language in acoustically challenging environments is a complex phenomenon that relies heavily on the integration of auditory and visual sensory streams. While the traditional view posits that audiovisual integration occurs primarily at higher-order cortical levels, emerging evidence suggests that visual speech cues may exert profound top-down modulatory effects on subcortical structures, including the auditory brainstem. This paper presents a comprehensive examination of how visual speech cues influence the neural encoding of continuous speech embedded in varying levels of background noise across both brainstem and cortical processing hierarchies. Through an extensive theoretical review and a simulated experimental framework, we investigate the dynamic interplay between the brainstem frequency-following response and cortical tracking of the speech envelope. The proposed paradigm utilizes high-density electroencephalography to simultaneously capture these distinct neurophysiological markers under auditory-only and audiovisual conditions. Our synthesis of the data reveals that visual information, such as articulatory lip movements, not only enhances the temporal precision of low-frequency cortical entrainment but also significantly shores up the phase-locked representation of fundamental frequencies in the auditory brainstem via corticofugal feedback loops. These findings challenge the purely bottom-up perspective of subcortical sensory processing and underscore the deeply interactive nature of the human nervous system. The implications of this research extend across clinical domains, offering potential avenues for therapeutic interventions in populations experiencing age-related hearing decline or central auditory processing deficits, thereby providing a more holistic understanding of multisensory speech perception mechanisms.
References
1. Lin, Liang, et al. "OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment." arXiv preprint arXiv:2509.24610 (2025).
2. Shan, T., Lalor, E. C., & Maddox, R. K. (2026). Chimeric music reveals an interaction of pitch and time in electrophysiological signatures of music encoding. Journal of Neuroscience, 46(4).
3. Xu, X. (2024). Improving monaural speech enhancement by mapping to fixed simulation space with knowledge distillation. IEEE Signal Processing Letters, 31, 386-390.
4. Shan, T., & Maddox, R. K. (2025). Comparing methods for deriving the auditory brainstem response to continuous speech in human listeners. Imaging Neuroscience, 3, IMAG-a.
5. Huang, Jinbang, et al. "One Demo Is All It Takes: Planning Domain Derivation with LLMs from A Single Demonstration." arXiv preprint arXiv:2505.18382 (2025).
6. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., et al. (2016). WaveNet: A generative model for raw audio. arXiv preprint arXiv:1609.03499.
7. Huang, T., Cui, Z., Du, C., & Chiang, C. E. (2025, June). CL-ISR: A Contrastive Learning and Implicit Stance Reasoning Framework for Misleading Text Detection on Social Media. In 2025 6th International Conference on Electronic Communication and Artificial Intelligence (ICECAI) (pp. 610-616). IEEE.
8. Chen, C., Wang, W., Mou, X., Tang, T., Chen, X., Wang, Q., ... & Yang, Z. (2026). Inner Speech in Motor Cortex: Laying the Foundation for Privacy‐Preserving Speech Neuroprostheses. Medicine Bulletin, 2(2), 244-248.
9. Li, A., Liu, W., Luo, X., Yu, G., Zheng, C., & Li, X. (2021). A simultaneous denoising and dereverberation framework with target decoupling. arXiv preprint arXiv:2106.12743.
10. Xie, Z., Ren, X., Zheng, T., Bai, J., Fan, W., Xu, B., ... & Song, Y. A Survey on AI Agent Harness.
11. Xu, X., Tu, W., & Yang, Y. (2023). CASE-Net: Integrating local and non-local attention operations for speech enhancement. Speech Communication, 148, 31-39.
12. Wu, Y., He, Y., Liu, X., Wang, Y., & Dannenberg, R. B. (2023, June). Transplayer: Timbre style transfer with flexible timbre control. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 1-5). IEEE.
13. Xu, X., Li, J., Zhang, Y., Tu, W., & Yang, Y. (2025). Spatial information aided speech and noise feature discrimination for Monaural speech enhancement. Expert Systems with Applications, 269, 126349.
14. Ren, Y., Wu, Y., Fan, J. W., Khurana, A., Fu, S., Wu, D., ... & Huang, M. (2024). Automatic uncovering of patient primary concerns in portal messages using a fusion framework of pretrained language models. Journal of the American Medical Informatics Association, 31(8), 1714-1724.
15. Zhou, J., Shuang, K., Wang, Q., Qian, B., & Guo, J. (2025). Bi-directional feature learning-based approach for zero-shot event argument extraction. Information Processing & Management, 62(5), 104199.
16. Li, A., Liu, W., Zheng, C., Fan, C., & Li, X. (2021). Two heads are better than one: A two-stage complex spectral mapping approach for monaural speech enhancement. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 1829-1843.
17. Xu, Xinmeng, et al. "Delayed Commitment for Representation Readiness in Stage-wise Audio-Visual Learning." arXiv preprint arXiv:2605.01673 (2026).
18. Shan, T., Wenner, C. E., Xu, C., Duan, Z., & Maddox, R. K. (2022). Speech-in-noise comprehension is improved when viewing a deep-neural-network-generated talking face. Trends in Hearing, 26, 23312165221136934.
19. Li, A., Lei, T., Sun, Z., Chen, R., Yin, E., Li, X., & Zheng, C. (2025). Learning neural vocoder from range-null space decomposition. arXiv preprint arXiv:2507.20731.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Soobin Shin, Ichika Fujiwara
This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles published in the Journal of Advanced Electrical and Electronic Engineering (JAEEE) are distributed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Authors retain copyright of their work while granting the journal the right of first publication.
Under this license, users are permitted to read, download, copy, distribute, print, share, adapt, and reuse published material for both commercial and non-commercial purposes, provided appropriate credit is given to the original author(s), the source of publication is acknowledged, a link to the license is provided, and any changes made are clearly indicated.
The license does not permit users to imply endorsement by the original authors or the journal. No additional restrictions may be imposed that prevent others from exercising the rights granted under CC BY 4.0.
By submitting to the journal, authors confirm that their work is original, does not infringe upon third-party rights, and complies with ethical publishing standards.
