Open-Vocabulary 3D Scene Understanding with Gaussian Splatting and Vision-Language Alignment
DOI:
https://doi.org/10.66238/jaeee17Keywords:
Scene Understanding, Gaussian Splatting, Vision-Language Models, Semantic Alignment, Neural RenderingAbstract
The integration of open-vocabulary comprehension into three-dimensional scene representations represents a critical frontier in computer vision, robotics, and spatial computing. Traditional methodologies have heavily relied on closed-set categorical labels and computationally expensive volumetric rendering techniques, which fundamentally limit real-time applicability and generalization capabilities. This paper introduces a comprehensive framework for open-vocabulary three-dimensional scene understanding by synergizing three-dimensional Gaussian splatting with vision-language alignment models. By embedding high-dimensional semantic features directly into the attributes of three-dimensional Gaussians, the proposed system transcends the limitations of predefined object categories, enabling arbitrary text-based spatial queries. The research meticulously details the feature distillation process, whereby spatial knowledge from pre-trained two-dimensional vision-language models is transferred into the three-dimensional domain via differentiable rasterization. Extensive methodological analysis and subsequent experimental validations demonstrate that this approach achieves unprecedented rendering speeds while maintaining high fidelity in both geometric reconstruction and semantic segmentation. Furthermore, the integration of contrastive language-image pre-training embeddings ensures that the resultant scene representations can intelligently respond to complex, long-tail semantic queries. Through rigorous quantitative and qualitative evaluations, this paper establishes a novel benchmark in the intersection of neural rendering and multimodal semantic alignment, offering significant implications for autonomous navigation and interactive spatial environments.
References
1. Zhang, P., Ge, Z., Song, L., & Lam, E. Y. (2023). Neuromorphic imaging with density-based spatiotemporal denoising. IEEE Transactions on Computational Imaging, 9, 530-541.
2. Wang, W., Mi, L., Cen, S., Dai, H., Li, Y., Fu, X., & Liu, Y. (2025). Region-based Content Enhancement for {Efficient} Video Analytics at the Edge. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25) (pp. 613-633).
3. Wang, C., Song, R., Muller, R., Monteuuis, J. P., Celik, Z. B., Petit, J., ... & Li, M. (2026, March). Cp-freezer: Latency attacks against vehicular cooperative perception. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 2, pp. 1114-1122).
4. Peng, Q., Planche, B., Gao, Z., Zheng, M., Choudhuri, A., Chen, T., ... & Wu, Z. (2024). 3d vision-language gaussian splatting. arXiv preprint arXiv:2410.07577.
5. Dai, S., Wu, Y., Chen, S., Huang, R., & Dannenberg, R. B. (2023, November). SingStyle111: A Multilingual Singing Dataset With Style Transfer. In ISMIR (pp. 765-773).
6. Zhao, H., Lu, T., Gu, J., Zhang, X., Zheng, Q., Wu, Z., ... & Jiang, Y. G. (2024, September). Magdiff: Multi-alignment diffusion for high-fidelity video generation and editing. In European Conference on Computer Vision (pp. 205-221). Cham: Springer Nature Switzerland.
7. Yang, D., Gao, Y., Wang, X., Yue, Y., Yang, Y., & Fu, M. (2025, May). Opengs-slam: Open-set dense semantic slam with 3d gaussian splatting for object-level scene understanding. In 2025 IEEE International Conference on Robotics and Automation (ICRA) (pp. 8486-8492). IEEE.
8. Liu, Y., Liu, M., Huang, S., & Lv, J. (2025, April). Asymmetric visual semantic embedding framework for efficient vision-language alignment. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 39, No. 6, pp. 5676-5684).
9. Dong, J., Liu, J., Qu, X., & Ong, Y. S. (2025). Confound from All Sides, Distill with Resilience: Multi-Objective Adversarial Paths to Zero-Shot Robustness. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 624-634).
10. Peng, Q., Zheng, C., & Chen, C. (2024). A dual-augmentor framework for domain generalization in 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 2240-2249).
11. Tu, P., Huang, Y., Zheng, F., He, Z., Cao, L., & Shao, L. (2022, June). Guidedmix-net: Semi-supervised semantic segmentation by using labeled images as reference. In Proceedings of the AAAI conference on artificial intelligence (Vol. 36, No. 2, pp. 2379-2387).
12. Qi, Z., Wang, S., Zhang, W., & Huang, Q. (2024). Uncertainty-boosted robust video activity anticipation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12), 7775-7792.
13. Yang, D., Wang, X., Gao, Y., Liu, S., Ren, B., Yue, Y., & Yang, Y. (2025, October). Opengs-fusion: Open-vocabulary dense mapping with hybrid 3D Gaussian splatting for refined object-level understanding. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 21135-21142). IEEE.
14. Xie, C., Zhu, D., Wang, Z., Zhang, H., & Wei, Z. (2026). Compliance-Aware Discharge Agent for Auditable ICU Discharge Planning: A Pilot Feasibility Study Using Structured eICU Records. Available at SSRN 6429758.
15. Qu, W., Shao, Y., Meng, L., Huang, X., & Xiao, L. (2024). A conditional denoising diffusion probabilistic model for point cloud upsampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20786-20795).
16. Peng, Q., Bai, C., Zhang, G., Xu, B., Liu, X., Zheng, X., ... & Lu, C. (2025, October). NavigScene: Bridging local perception and global navigation for beyond-visual-range autonomous driving. In Proceedings of the 33rd ACM International Conference on Multimedia (pp. 4193-4202).
17. Tan, Y., & Wu, L. (2023). Neuro‐adaptive practical prescribed‐time control for pure‐feedback nonlinear systems without accurate initial errors. International Journal of Adaptive Control and Signal Processing, 37(4), 915-933.
18. Wang, C., Man, Y., Muller, R., Li, M., Celik, Z. B., Gerdes, R., & Petit, J. (2024, December). Physical ID-Transfer Attacks against Multi-Object Tracking via Adversarial Trajectory. In 2024 Annual Computer Security Applications Conference (ACSAC) (pp. 957-973). IEEE.
19. Wang, S., Yu, Y., Feldt, R., & Parthasarathy, D. (2025). Automating a complete software test process using llms: An automotive case study. arXiv preprint arXiv:2502.04008.
20. Tang, Y., Zhang, G., Liu, J. K., & Qin, R. (2025). Weakly supervised land-cover classification of high-resolution images with low-resolution labels through optimized label refinement. International Journal of Remote Sensing, 46(5), 1913-1937.
21. Zhang, P., Zhu, S., Wang, C., Zhao, Y., & Lam, E. Y. (2024). Neuromorphic imaging with super-resolution. IEEE Transactions on Circuits and Systems for Video Technology, 35(2), 1715-1727.
22. Qu, W., Wang, J., Gong, Y., Huang, X., & Xiao, L. (2025). An end-to-end robust point cloud semantic segmentation network with single-step conditional diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 27325-27335).
23. Yang, Huan, et al. "Kvshare: An llm service system with efficient and effective multi-tenant kv cache reuse." arXiv preprint arXiv:2503.16525 (2025).
24. Dong, J., Koniusz, P., Feng, L., Zhang, Y., Zhu, H., Liu, W., ... & Ong, Y. S. (2025). Robustifying zero-shot vision language models by subspaces alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 21037-21047).
25. Liu, Y., Feng, W., Liu, Z., Huang, S., & Lv, J. (2025). Aligning information capacity between vision and language via dense-to-sparse feature distillation for image-text matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 21679-21688).
26. Lv, Q., Deng, X., Chen, G., Wang, M. Y., & Nie, L. (2024). Decision mamba: A multi-grained state space model with self-evolution regularization for offline rl. Advances in neural information processing systems, 37, 22827-22849.
27. Zhang, W., Zhang, C., Gu, C., Kou, J., Yuan, H., Fang, X., ... & Fang, Y. (2024, October). Hallucination in Large Language Models: From Mechanistic Understanding to Novel Control Frameworks. In 2024 7th International Conference on Universal Village (UV) (pp. 1-36). IEEE.
28. Qi, Z., Yuan, Y., Ruan, X., Wang, S., Zhang, W., & Huang, Q. (2024, March). Bias-conflict sample synthesis and adversarial removal debias strategy for temporal sentence grounding in video. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 5, pp. 4533-4541).
29. Song, S., Tang, Y., & Qin, R. (2025). Synthetic Data Matters: Re-training with Geo-typical Synthetic Labels for Building Detection. IEEE Transactions on Geoscience and Remote Sensing.
30. Zhao, H., Wang, Q., Zhan, G., Min, W., Zou, Y., & Cui, S. (2022). Need only one more point (NOOMP): Perspective adaptation crowd counting in complex scenes. IEEE Transactions on Multimedia, 25, 1414-1426.
31. Dong, J., Koniusz, P., Qu, X., & Ong, Y. S. (2025, July). Stabilizing modality gap & lowering gradient norms improve zero-shot adversarial robustness of vlms. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1 (pp. 236-247).
32. Guo, Hanzhong, et al. "Leveraging verifier-based reinforcement learning in image editing." arXiv preprint arXiv:2604.27505 (2026).
33. Zhao, H., Gu, J., Wang, S., Lu, T., Zhang, X., Wu, Z., ... & Jiang, Y. G. (2026). LSTD: Long Short-Term Temporal Diffusion for Video Generation. IEEE Transactions on Multimedia.
34. Wang, C., Muller, R., Song, R., Monteuuis, J. P., Petit, J., Man, Y., ... & Li, M. (2025). From Threat to Trust: Exploiting Attention Mechanisms for Attacks and Defenses in Cooperative Perception. In 34th USENIX Security Symposium (USENIX Security 25) (pp. 7387-7406).
35. Huang, J. (2025, September). SCIAI: RELIABLE LARGE LANGUAGE MODEL REASONING FOR SCIENTIFIC LITERATURE VERIFICATION AND HYPOTHESIS VALIDATION. In The 5th International scientific and practical conference “Trends in the development of science by young scientists and students”(September 30-October 03, 2025) Warsaw, Poland. International Science Group. 2025. 122 p. (p. 16).
36. Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., & Ng, R. (2020). NeRF: Representing scenes as neural radiance fields for view synthesis. In Proceedings of ECCV.
37. Lv, Qi, et al. "Spatial-temporal graph diffusion policy with kinematic modeling for bimanual robotic manipulation." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.
38. Zhang, W. (2026). A 5-6 GHz PVT Robust Current Mode Passive Mixer for Direct Down-Conversion Receiver.
39. Redmon, J., Divvala, S., Girshick, R., & Farhadi, A. (2016). You only look once: Unified, real-time object detection. In Proceedings of CVPR.
40. Lv, Qi, et al. "F1: A vision-language-action model bridging understanding and generation to actions." arXiv preprint arXiv:2509.06951 (2025).
41. Guo, Hanzhong, et al. "Real-time identity defenses against malicious personalization of diffusion models." arXiv preprint arXiv:2412.09844 (2024).
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Betty Debby, Bill Debra, Kyuri Min
This work is licensed under a Creative Commons Attribution 4.0 International License.
All articles published in the Journal of Advanced Electrical and Electronic Engineering (JAEEE) are distributed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Authors retain copyright of their work while granting the journal the right of first publication.
Under this license, users are permitted to read, download, copy, distribute, print, share, adapt, and reuse published material for both commercial and non-commercial purposes, provided appropriate credit is given to the original author(s), the source of publication is acknowledged, a link to the license is provided, and any changes made are clearly indicated.
The license does not permit users to imply endorsement by the original authors or the journal. No additional restrictions may be imposed that prevent others from exercising the rights granted under CC BY 4.0.
By submitting to the journal, authors confirm that their work is original, does not infringe upon third-party rights, and complies with ethical publishing standards.
