Machine Learning for Multimodal Medical Data

Abstract

The rapid advancement of diagnostic technologies in healthcare has heightened the demand for physicians to integrate heterogeneous yet complementary data generated during routine practice, including radiology images, pathology slides, genomic data, and clinical features. Recent progress in multi-modal learning offers new opportunities to address these challenges by facilitating the effective fusion of diverse data modalities. This dissertation focuses on leveraging machine learning, particularly deep learning methods, to tackle challenges in multi-modal learning. It begins with an exploration of unimodal approaches, addressing segmentation and anomaly detection tasks associated with various modalities in medical domains. The core of the dissertation shifts to multi-modal learning, examining its applications in diagnosis, prognosis, and segmentation tasks that integrate multiple medical modalities. The first half proposes pipelines that combine image and non-image data for diagnostic and prognostic tasks related to gliomas and soft tissue tumors, with a focus on addressing the challenge of missing modalities during training and inference. The latter half investigates the application of multi-modal learning in medical image segmentation, highlighting the fine-tuning potential of segment-everything models using weak annotations. Additionally, language is integrated to guide pre-trained vision models, facilitating a more flexible multitask segmentation pipeline applicable to kidney pathology. In the end, the dissertation concludes with a summary of contributions and an outline of future research directions.