Synchronous Voice Action Dialogue Policies Coordinate Service Robot Turn Taking Control and Robot Motion Timing
DOI:
https://doi.org/10.66238/jaeee28Keywords:
Service Robots, Turn-Taking Control, Dialogue Policy, Multimodal InteractionAbstract
The integration of physical actions with verbal communication remains a critical challenge in the development of socially competent service robots. Human conversation is inherently multimodal, relying on a complex synchronization of speech, gaze, gestures, and utilitarian actions to regulate turn-taking smoothly. However, traditional dialogue systems typically process speech and robotic actions sequentially or asynchronously, leading to unnatural delays, interruptions, and a degraded user experience. This paper proposes a comprehensive framework for synchronous voice-action dialogue policies aimed at optimizing service robot turn-taking control. By treating physical actions and verbal utterances as a unified output space, the proposed model enables robots to anticipate turn transitions and execute synchronized multimodal responses. The research introduces a continuous state-space representation that captures both linguistic context and spatial-temporal features of the interaction environment. Through extensive simulated and real-world human-robot interaction trials, this study evaluates the efficacy of synchronous policies against conventional modular approaches. The findings demonstrate significant improvements in interaction fluency, characterized by a reduction in overlapping speech, lower response latencies, and higher subjective ratings of robot sociability. This research provides a foundational architecture for next-generation service robots, emphasizing the necessity of embodied cognition in spoken dialogue systems and offering new perspectives on multimodal reinforcement learning in domestic and public service environments.
References
1. Xu, X., Tu, W., & Yang, Y. (2025). Efficient audio–visual information fusion using encoding pace synchronization for Audio–Visual Speech Separation. Information Fusion, 115, 102749.
2. Chen, C., Zhang, J., Lakshmikanth, S. K., Fang, Y., Shao, R., Wetzstein, G., ... & Adeli, E. (2025). The language of motion: Unifying verbal and non-verbal language of 3d human motion. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 6200-6211).
3. Xu, X., Tu, W., & Yang, Y. (2023). CASE-Net: Integrating local and non-local attention operations for speech enhancement. Speech Communication, 148, 31-39.
4. Wu, D., Zhang, H., Chen, J., Liu, H., Chng, E. S., Tian, F., ... & Yu, G. (2025). Mind-Paced Speaking: A Dual-Brain Approach to Real-Time Reasoning in Spoken Language Models. arXiv preprint arXiv:2510.09592.
5. Zhang, H., Liu, H., Zhang, X., Zhang, Q., Hu, Y., Zhao, J., ... & Chng, E. S. (2025). Impact of Frame Rates on Speech Tokenizer: A Case Study on Mandarin and English. arXiv preprint arXiv:2505.17076.
6. Dai, L., Li, A., Chi, C., Liang, Y., Li, X., & Zheng, C. (2026). GOMPSNR: Reflourish the Signal-to-Noise Ratio Metric for Audio Generation Tasks. arXiv preprint arXiv:2601.13758.
7. Xu, X., Wang, Y., Xu, D., Peng, Y., Zhang, C., Jia, J., & Chen, B. (2022, May). Vsegan: Visual speech enhancement generative adversarial network. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 7308-7311). IEEE.
8. Xu, X., Tu, W., Yang, Y., Li, J., & Zhang, Y. (2025). Interactive Target Positive and Negative Features Modeling for Monaural Speech Enhancement. IEEE Transactions on Audio, Speech and Language Processing, 33, 4856-4869.
9. Li, Y., Li, K., Yin, X., Yang, Z., Dong, Z., Yao, Z., ... & Lu, Y. (2026, March). Sepprune: Structured pruning for efficient deep speech separation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 38, pp. 31861-31869).
10. Xu, X. (2024). Improving monaural speech enhancement by mapping to fixed simulation space with knowledge distillation. IEEE Signal Processing Letters, 31, 386-390.
11. Zhang, J., Chen, C., Chen, X., Yu, H., Xiang, T., Khan, A. S., ... & Adeli, E. (2026). Vibes: A conversational agent with behaviorally-intelligent 3d virtual body. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 39994-40008).
12. Lv, Z., Wang, Q., Lyu, T., & Dong, J. (2026). A Deep Learning-Based Model for User Emotion Recognition and Behavior Guidance Design: EMoGuideNet-An Emotion-Aware Multimodal Framework for Enhancing Intelligent User Interaction. Journal of Organizational and End User Computing (JOEUC), 38(1), 1-30.
13. Li, A., Zheng, C., Zhang, L., & Li, X. (2022). Glance and gaze: A collaborative learning framework for single-channel speech enhancement. Applied Acoustics, 187, 108499.
14. Xu, Xinmeng, Weiping Tu, and Yuhong Yang. "Pcnn: A lightweight parallel conformer neural network for efficient monaural speech enhancement." arXiv preprint arXiv:2307.15251 (2023).
15. Zhang, H., Chen, J., Wu, D., Li, Y., Zhang, Y., Zhang, X. T., ... & Tian, F. (2026). Duplexsla: A full-duplex spoken language model with synchronized speech, language, and action. arXiv preprint arXiv:2605.20755.
16. Xu, X., Han, C., Zhang, Y., Tu, W., & Yang, Y. (2024, April). Curricular Contrastive Regularization for Speech Enhancement with Self-Supervised Representations. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 10486-10490). IEEE.
17. Li, A., Lei, T., Sun, Z., Chen, R., Yin, E., Li, X., & Zheng, C. (2025). Learning neural vocoder from range-null space decomposition. arXiv preprint arXiv:2507.20731.
18. Yu, H., Zhang, J., Chen, C., Xiang, T., Fang, Y., Niebles, J. C., & Adeli, E. (2026, March). Socialgen: Modeling multi-human social interaction with language models. In 2026 International Conference on 3D Vision (3DV) (pp. 1-17). IEEE.
19. Xu, X., Tu, W., & Yang, Y. (2023, June). Selector-enhancer: learning dynamic selection of local and non-local attention operation for speech enhancement. In Proceedings of the AAAI conference on artificial intelligence (Vol. 37, No. 11, pp. 13853-13860).
20. Li, A., Liu, W., Zheng, C., Fan, C., & Li, X. (2021). Two heads are better than one: A two-stage complex spectral mapping approach for monaural speech enhancement. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29, 1829-1843.
21. Baevski, A., Zhou, H., Mohamed, A., & Auli, M. (2020). wav2vec 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Avery Bailey, Markus Neumann
This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles published in the Journal of Advanced Electrical and Electronic Engineering (JAEEE) are distributed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Authors retain copyright of their work while granting the journal the right of first publication.
Under this license, users are permitted to read, download, copy, distribute, print, share, adapt, and reuse published material for both commercial and non-commercial purposes, provided appropriate credit is given to the original author(s), the source of publication is acknowledged, a link to the license is provided, and any changes made are clearly indicated.
The license does not permit users to imply endorsement by the original authors or the journal. No additional restrictions may be imposed that prevent others from exercising the rights granted under CC BY 4.0.
By submitting to the journal, authors confirm that their work is original, does not infringe upon third-party rights, and complies with ethical publishing standards.
