| dc.description.abstract | Among the fundamental AI techniques, deep neural networks (DNN) have been widely applied in many industrial systems including safety-critical systems because of their versatility. DNNs are based on simple mathematical operations combining neurons containing non-linear activation functions in layers, whose functionality, however, is difficult to be interpreted by humans. This lack of transparency leads to one major barrier in realizing trustful autonomy. Therefore, in this dissertation, I propose a framework based on computational geometry to conduct reachability analysis of DNNs including feedforward neural networks and convolutional neural networks, such that (1) the safety of DNNs, particularly neural network controllers in autonomous systems, can be formally verified efficiently, (2) input spaces that lead to unsafe behaviors of DNNs can be fully computed, and (3) the adversarial robustness of image-classification can be fairly evaluated. At the core, our reachable-set-based methods utilize a facet-vertex incidence matrix and a face lattice structure to encode the combinatorial structure of convex polytopes. Our methods include exact reachability analysis and fast reachability analysis. The former computes the exact output reachable domain of a DNN for an input domain while the latter under approximates the domain by considering selected sensitive neurons rather than all neurons in layers. Besides, our approach is also capable of backtracking to the input domain given an output reachable set. Therefore, the entire input spaces that caused the safety violation in a DNN can be identified. Furthermore, based on these methods, I propose another framework to repair unsafe DNNs and produce provably safe models on multiple safety properties with negligible performance degradation, even in the absence of training data. Our repair method uses reachability analysis to calculate the unsafe reachable domain of a DNN, and then uses a novel loss function to construct the distance to the safe domain during the retraining process. Since subtle changes of the DNN parameters can cause unexpected performance degradation, we also present a minimal repair approach where the DNN deviation is minimized. We also explore applications of our method to repair DNN agents in deep reinforcement learning with seamless integration with learning algorithms. | |
