IMPLEMENTATION OF COMPUTER VISION AND NATURAL LANGUAGE PROCESSING IN SOCIAL ROBOTS FOR MORE NATURAL AND INTUITIVE HUMAN-ROBOT INTERACTION
Article Sidebar
-
Computer Vision, Human-Robot Interaction, Multimodal AI, Natural Language Processing, Social Robots
Abstract
The rapid advancement of artificial intelligence (AI) has driven significant developments in social robotics, particularly in enabling more natural and intuitive human-robot interaction (HRI). However, current social robots often struggle to interpret multimodal human input effectively, leading to limited contextual understanding and reduced interaction quality. This study addresses these challenges by integrating computer vision (CV) and natural language processing (NLP) to enhance robots’ perceptual and communicative capabilities. The primary aim is to design and evaluate an interaction framework that allows social robots to recognize human emotions, gestures, and spoken language more accurately, thereby improving the fluency of HRI. A mixed-methods approach was employed, combining experimental implementation with qualitative user studies. The system architecture integrates real-time image recognition, gesture tracking, and speech understanding modules, which were tested through laboratory simulations involving 50 participants in controlled social scenarios. The results demonstrate that robots equipped with CV and NLP modules achieved a 30% improvement in gesture recognition accuracy, a 25% increase in contextual language understanding, and significantly higher user satisfaction scores compared to baseline models. Users reported that the robots exhibited more human-like responsiveness and adaptability in conversational settings. These findings suggest that combining computer vision and NLP substantially improves the naturalness and intuitiveness of human-robot interactions. This research highlights the importance of multimodal AI integration for the next generation of socially intelligent robots and paves the way for applications in healthcare, education, and service industries.
Full text article
References
Afzal, N., Rehman, M. ur, Seneviratne, L., & Hussain, I. (2025). The Convergence of AI and animal-inspired robots for ecological conservation. Ecological Informatics, 85, 102950. https://doi.org/10.1016/j.ecoinf.2024.102950
Aguzzi, J., Chatzidouros, E., Chatzievangelou, D., Clavel-Henry, M., Flögel, S., Bahamon, N., Tangerlini, M., Thomsen, L., Picardi, G., Navarro, J., Masmitja, I., Robinson, N. J., Nattkemper, T., Stefanni, S., Quintana, J., Campos, R., García, R., Fanelli, E., Francescangeli, M., … Doyle, J. (2025). A digital-twin strategy using robots for marine ecosystem monitoring. Ecological Informatics, 91, 103409. https://doi.org/10.1016/j.ecoinf.2025.103409
Akhtyamov, T., Kashirin, A., Postnikov, A., Sosin, I., & Ferrer, G. (2025). Social robot navigation through constrained optimization: A comprehensive study of uncertainty-based objectives and constraints in the simulated and real world. Robotics and Autonomous Systems, 183, 104830. https://doi.org/10.1016/j.robot.2024.104830
Asif, M. E., Rastegarpanah, A., & Stolkin, R. (2024). Robotic disassembly for end-of-life products focusing on task and motion planning: A comprehensive survey. Journal of Manufacturing Systems, 77, 483–524. https://doi.org/10.1016/j.jmsy.2024.09.010
Barravecchia, F., Gervasi, R., Mastrogiacomo, L., & Franceschini, F. (2025). The collaboration scale: A novel approach for assessing robotic systems collaboration capabilities. Robotics and Computer-Integrated Manufacturing, 96, 103062. https://doi.org/10.1016/j.rcim.2025.103062
Blavette, L., Dacunha, S., Alameda-Pineda, X., Hernández García, D., Gannot, S., Gras, F., Gunson, N., Lemaignan, S., Polic, M., Tandeitnik, P., Tonini, F., Rigaud, A.-S., & Pino, M. (2025). Acceptability and Usability of a Socially Assistive Robot Integrated With a Large Language Model for Enhanced Human-Robot Interaction in a Geriatric Care Institution: Mixed Methods Evaluation. JMIR Human Factors, 12. https://doi.org/10.2196/76496
Brunello, A., Fabris, G., Gasparetto, A., Montanari, A., Saccomanno, N., & Scalera, L. (2025). A survey on recent trends in robotics and artificial intelligence in the furniture industry. Robotics and Computer-Integrated Manufacturing, 93, 102920. https://doi.org/10.1016/j.rcim.2024.102920
Chen, Z., & Adel, A. (2025). Advancing robotic assembly in construction: Innovations, challenges, and opportunities. Automation in Construction, 178, 106370. https://doi.org/10.1016/j.autcon.2025.106370
Fu, S., Antwi-Afari, M. F., Anwer, S., Chen, Z.-S., & Li, H. (2025). A state-of-the-art review of digital twin-enabled human-robot collaboration in smart energy management systems. Results in Engineering, 27, 106524. https://doi.org/10.1016/j.rineng.2025.106524
Ghosh, R., Khan, N., Migovich, M., Tate, J. A., Maxwell, C. A., Newhouse, P. A., Scharre, D. W., Tan, A., Mion, L. C., & Sarkar, N. (2025). Engaging Older Adults and Staff in the Co-Design and Evaluation of Socially Assistive Robot and Virtual Reality Activities for Long-Term Care: User-Centered Study. JMIR Aging, 8. https://doi.org/10.2196/75288
Huang, J., Xu, X., & Polzer, J. (2025). Towards instructional collaborative robots: From video-based learning to feedback-adapted instruction. Journal of Manufacturing Systems, 83, 29–45. https://doi.org/10.1016/j.jmsy.2025.08.018
Karami, V., Yaffe, M. J., Gore, G., Moon, Aj., & Abbasgholizadeh Rahimi, S. (2024). Socially Assistive Robots for patients with Alzheimer’s Disease: A scoping review. Archives of Gerontology and Geriatrics, 123, 105409. https://doi.org/10.1016/j.archger.2024.105409
Kim, H., Kong, M., Jung, S., Jeoung, J., Kang, H., & Hong, T. (2025). Technological applications of social robots to create healthy and comfortable smart home environment. Building and Environment, 267, 112269. https://doi.org/10.1016/j.buildenv.2024.112269
Lefranc, G., Gatica, G., Vásquez, E., & Peña, M. (2025). Advanced Algorithm Combinations for Safe Human-Robot Collaboration: A Review and Research Proposal. The 12th International Conference on Information Technology and Quantitative Management (ITQM 2025), 266, 142–149. https://doi.org/10.1016/j.procs.2025.08.018
Li, J., Xiong, J., Zhang, Z., & Guo, D. (2025). A Multimodal Large Model to Enhance Robot Understanding of Human Intentions for Accurate Human Robot Collaborative Manufacturing. 11th IFAC Conference on Manufacturing Modelling, Management and Control MIM 2025, 59(10), 2820–2825. https://doi.org/10.1016/j.ifacol.2025.09.474
Liu, Z., Sun, H., & Yuan, D. (2025). Automatic analysis of alarm embedded with large language model in police robot. Biomimetic Intelligence and Robotics, 5(3), 100220. https://doi.org/10.1016/j.birob.2025.100220
Maniscalco, U., Minutolo, A., Storniolo, P., & Esposito, M. (2024). Towards a more anthropomorphic interaction with robots in museum settings: An experimental study. Robotics and Autonomous Systems, 171, 104561. https://doi.org/10.1016/j.robot.2023.104561
Merino-Fidalgo, S., Sánchez-Girón, C., Zalama, E., Gómez-García-Bermejo, J., & Duque-Domingo, J. (2025). Behavior tree generation and adaptation for a social robot control with LLMs. Robotics and Autonomous Systems, 194, 105165. https://doi.org/10.1016/j.robot.2025.105165
Shah, R., Doss, A. S. A., & Lakshmaiya, N. (2025). Advancements in AI-enhanced collaborative robotics: Towards safer, smarter, and human-centric industrial automation. Results in Engineering, 27, 105704. https://doi.org/10.1016/j.rineng.2025.105704
Sun, J., Tang, R., Zhang, Q., Liu, J., Gao, L., Yang, W., Chen, X., & Tang, Y. (2025). Advancements and challenges in autonomous endovascular interventional robotics: A comprehensive review. iScience, 28(12), 114024. https://doi.org/10.1016/j.isci.2025.114024
Takahashi, L., Kuwahara, M., & Takahashi, K. (2025). AI and automation: Democratizing automation and the evolution towards true AI-autonomous robotics. Chemical Science, 16(35), 15769–15780. https://doi.org/10.1039/d5sc03183d
Triana R, E. D., Loureiro, M., Machado, F., Mello, R., Jimenez H., M. F., & Frizera-Neto, A. (2025). Advanced Human-Robot Interaction with a Mobile Robot: Integrating Large Language Models into Robotics. International Neural Network Society Workshop on Deep Learning Innovations and Applications 2025, 264, 280–289. https://doi.org/10.1016/j.procs.2025.07.139
Wang, S., & Ahmad, N. S. (2025). AI-based approaches for improving autonomous mobile robot localization in indoor environments: A comprehensive review. Engineering Science and Technology, an International Journal, 63, 101977. https://doi.org/10.1016/j.jestch.2025.101977
Xia, G., Ghrairi, Z., Wuest, T., Hribernik, K., Heuermann, A., Liu, F., Liu, H., & Thoben, K.-D. (2025). Towards Human Modeling for Human-Robot Collaboration and Digital Twins in Industrial Environments: Research Status, Prospects, and Challenges. Robotics and Computer-Integrated Manufacturing, 95, 103043. https://doi.org/10.1016/j.rcim.2025.103043
Zhang, L., Shi, P., Wang, S., & Su, X. (2025). A survey of safety control for service robots. Journal of Automation and Intelligence. https://doi.org/10.1016/j.jai.2025.10.002
Zhou, X., Menassa, C. C., & Kamat, V. R. (2025). Siamese network with dual attention for EEG-driven social learning: Bridging the human-robot gap in long-tail autonomous driving. Expert Systems with Applications, 291, 128470. https://doi.org/10.1016/j.eswa.2025.128470
Zhou, Y., Yan, Z., Yang, Y., Wang, Z., Lu, P., Yuan, P. F., & He, B. (2024). Bioinspired sensors and applications in intelligent robots: A review. Robotic Intelligence and Automation, 44(2), 215–228. https://doi.org/10.1108/RIA-07-2023-0088
Authors
Copyright (c) 2025 Muchamad Sobri Sungkar, James Chirwa, Giorgi Bagrationi
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.