SarcomaNet: A Privacy-Preserving Multi-Task Deep Learning Framework for Soft Tissue Sarcoma Analysis from Multi-Modal Imaging

Abstract

Soft tissue sarcomas (STSs) are a rare and biologically heterogeneous group of malignancies comprising more than 100 histological subtypes, with an annual incidence of fewer than 5 per 100,000 persons. Their rarity makes large, well-annotated imaging datasets difficult to assemble, limiting the use of deep learning for automated tumour segmentation, histological grade classification, and survival risk estimation—three clinically relevant tasks that are often performed separately and with limited quantitative imaging support. This thesis presents SarcomaNet, a unified, privacy preserving tri-head deep learning framework that addresses all three tasks jointly from multi-modal imaging using the publicly available TCIA Soft Tissue Sarcoma dataset (𝑁 = 51). SarcomaNet is built on a shared 3D Residual U-Net (ResU-Net) encoder–decoder backbone and introduces four complementary innovations to address the constraints of rare-cancer imaging. (1) Cross-Modal Masked Autoencoding (CrMAE) is a self-supervised pre-training strategy that leverages the complementary biophysics of multi-modal imaging by masking a randomly selected modality channel (T2FS MRI, FDG-PET, or CT) at 50% patch density and training the encoder to reconstruct the masked modality from the two visible channels. Unlike spatial masked autoencoders, CrMAE encourages the encoder to learn physics-grounded cross-modal correspondences (oedema ↔ metabolic activity ↔ tissue ensity) hat are directly informative of tumour biology. Over 40 pre-training epochs on all 51 unlabelled patient volumes, CrMAE reduces reconstruction MSE by 87.7% without requiring external data...