FROM TUMOR DYNAMICS TO TREATMENT OPTIMIZATION: INTEGRATIVE MATHEMATICAL MODELING AND MACHINE LEARNING IN ONCOLOGY

Abstract

This thesis presents an interdisciplinary approach that combines mathematical modeling, optimal control theory, and deep learning to investigate key problems in cancer research. Focusing on glioblastoma multiforme (GBM), we develop reaction–diffusion models that incorporate phenotypic switching and necrosis, capturing critical aspects of tumor heterogeneity and invasion dynamics. To support therapeutic design, Stepanova’s model of tumor–immune interactions is extended with optimal control formulations, as well as adaptive methods based on fuzzy systems and reinforcement learning. In parallel, the thesis addresses challenges in lymphoma diagnostics through the application of deep learning techniques, with a particular focus on the StarDist architecture for accurate nuclear segmentation in histopathological images. Together, these contributions advance current modeling and computational strategies for understanding tumor progression, evaluating treatment responses, and improving diagnostic accuracy.