Designing Autonomous Agents using Neurosymbolic Reinforcement Learning with Behavior Trees

Abstract

Autonomous agents operating in complex, dynamic, and adversarial environments require both the adaptability of reinforcement learning (RL) and the interpretability and structure of symbolic reasoning. Neurosymbolic reinforcement learning (NeSy RL) combines these complementary capabilities. However, designing NeSy RL agents for complex long-term tasks remains a significant challenge. This dissertation presents an end-to-end framework for designing NeSy RL agents using behavior trees (BTs), a reactive and modular symbolic representation for decomposing tasks into subtasks and coordinating their execution. The framework organizes agent design into three stages: (1) construction of a high-level symbolic control structure, (2) optimization of RL components within that structure, and (3) safe and robust deployment in previously unseen environments. To address these stages, novel methods are developed for automated BT construction, RL optimization, safety assurance, and adaptation. For autonomous UAV navigation, BTs are automatically generated through genetic programming and integrated with hierarchical RL components optimized under safety constraints, while a runtime verification method based on real-time reachability analysis enables safe deployment in unseen environments. For autonomous cyber defense, NeSy RL agents using BTs are developed to coordinate defensive actions against adaptive attackers, supported by adversarial RL methods for adapting defender policies to emerging attack strategies. Furthermore, this dissertation introduces BTs as structured reward and action masking functions, which improve sample efficiency and can be automatically generated and formally verified. Experimental results across UAV navigation and cyber-defense domains demonstrate improvements in safety, robustness, sample efficiency, and explainability compared to state-of-the-art approaches. Collectively, this work establishes a systematic methodology for constructing, optimizing, verifying, and deploying NeSy RL agents using BTs for complex long-term tasks.