Interpretable Skin Lesion Analysis System with Hair-Robust Segmentation

Abstract

The final report gives a comprehensive overview of the end-to-end interactive skin lesion analysis system development that was built within a two-semester long senior project work. In the system, a user uploads a dermoscopic image and receives an automated fully detailed report about the lesion that includes not only textual information of classification with confidence score and interpretable rule-based finding, but also a visual representation of segmentation mask with lesion boundaries. The core uniqueness of the system lies on our FSSM-ResNet model, a hair-aware joint segmentation-and-classification architecture we developed and got accepted at a conference. Together with an object detection, a deterministic classical-CV assessment layer, a tree-based confidence classifier, and a separate LLM reporting layer at the backend level, we wrapped them all up under a user-friendly front-end for dermatologists’ use. Throughout this journey, we did extensive research on different technical directions before setting on the final approach. Initially, we explored self-supervised learning literature for feature extraction, then experimented with interpretable prototype based networks such as ProtoPNet, as well as examining superpixel-based segmentation methods. Each had limitations that pushed us to look for better solutions. Eventually, we arrived at a fuzzy logic based approach for handling hair occlusion, the most persistent problem in dermoscopy. This resulted in FSSM-ResNet, which embeds a Fuzzy Structural Suppression Module directly inside the segmentation network and shares its encoder with a binary classification head.