Naturalistic and Engaging Human-Robot Interactions Through Affective Computing with Application to Apathy for Older Adults with Mild Cognitive Impairment

Abstract

This dissertation presents the design of a socially assistive robotic system within a virtual reality environment (SAR-VR) called Non-Immersive Robot and Virtual Reality Activities in Aging (NIRVANA), along with an enhanced feedback framework to address apathy in older adults with mild cognitive impairment (MCI). NIRVANA features co-operative virtual activities for older adult pairs, integrating physical, social, and cognitive elements. A humanoid robot (Nao) serves as a coach, delivering individualized verbal feedback to promote engagement. Apathy is highly prevalent in individuals with MCI—affecting up to 82% of those with dementia—and is linked to decreased quality of life and increased caregiver burden. Existing interventions are often resource-intensive and hard to personalize. To address this, four virtual activities were co-designed with older adults (n=14) and an interdisciplinary team of engineers, nurses, and physicians. A hierarchical feedback control system was developed to allow the robot to deliver targeted, naturalistic feedback in multi-user interactions using multimodal inputs and fuzzy logic. Three fuzzy inference systems (FIS) were developed to support context-aware robot feedback grounded in emotional intelligence and human-machine communication: one for affect-informed feedback, one for performance-based feedback, and a third that integrates the previous two for multi-participant interaction. Fuzzy logic was chosen for its interpretability and ability to incorporate expert knowledge. Each FIS used multimodal data—such as facial expressions, stress index, and task performance—collected during the NIRVANA clinical trial and was aligned with therapeutic strategies for promoting engagement. Key FIS design elements, such as membership function tuning and rule-base reduction, were optimized using real-world data. This work contributes a novel SAR-VR system that advances naturalistic, multi-user human-robot interaction for addressing apathy in older adults with MCI. It lays a foundation for future feedback frameworks that dynamically integrate affective states and therapeutic goals to support more personalized, impactful technologies.