Multimodal and robust prediction of radiotherapy toxicities: foundation model fusion and domain adaptation for low-data-rate scenarios

Radiotherapy is an essential modality in cancer treatment, enabling the selective destruction of tumour cells and control of disease progression. Although effective in many cancers, it can also damage surrounding healthy tissue, leading to adverse effects that can compromise patients' quality of life [1]. It is therefore essential to predict the toxicity of treatments in order to anticipate complications and adapt treatment planning to each patient.

Several studies have explored the use of AI to predict radiotherapy-induced toxicities [2]. Most of these studies are based on conventional techniques that mainly explore radiomic characteristics. In a recent study, Elhaminia et al. [3] showed that multimodal CNN models, combining CT scan images with clinical and dosimetric data from patients, can improve prediction performance. However, the models proposed in the literature are trained on small databases, often consisting of a small number of patients and focused on a single type of cancer, which severely limits their generalisation capacity. Furthermore, none of these models yet exploit foundation models, even though these offer promising potential, as we have shown in one of our recent studies on predicting response to lymphoma treatment [4].

The main objective of this thesis is to develop predictive models of radiotherapy-induced toxicities that are robust, accurate and transferable to different clinical contexts. More specifically, the work aims to exploit foundation models in order to extract high-level representations capable of capturing complex and discriminating features from images and dosimetric data. The next step will be to design inter-modal and inter-model fusion strategies that integrate representations from different latent spaces by modality (CT, PET, clinical and dosimetric data) and by foundational model in order to take advantage of their complementarity. Finally, domain adaptation techniques will be explored to transfer these enriched representations to a specific type of cancer, enabling the training of high-performance models even with limited data sets.

The subject of this thesis, led by Halim Benhabiles, stems from his EULiST research mobility project at the LAIMBIO (Medical Image Analysis and Biometrics Laboratory) at URJC (Rey Juan Carlos University) in Madrid. One of the objectives of this mobility is to finalise the scientific proposal for the thesis, co-supervised by Professor Norberto Malpica. The collaboration will also involve other teacher-researchers with complementary expertise in medical data analysis, notably Vincent Jaouen from IMT Atlantique/LATIM. On the IMT Nord Europe side, the future young teacher-researcher who will join the HIDE research group and is scheduled to be recruited in December 2025 will be involved in the project. The project will also benefit from the support of the Madrid radiotherapy centre (Centro de Protonterapia Quironsalud) and the Oscar Lambret cancer centre in Lille, providing access to a variety of medical data for experimentation and validation of the approaches developed, as well as clinical expertise for interpreting the results and assessing their medical relevance. Funding for this thesis will strengthen links between the various institutions, pave the way for mobility between them, particularly between France and Spain, and create a solid foundation for jointly preparing European calls for projects.