Incremental learning approach for semantic segmentation of skin histology images

Abstract

This study presents an incremental learning framework to enhance the generalization and robustness of transformer-based deep learning models for segmenting skin cancer and related tissue structures. While deep learning models often perform well on data distributions similar to their training sets, their accuracy typically degrades when exposed to novel scenarios–limiting their clinical utility in skin cancer diagnosis. To address this, we propose a biologically inspired incremental learning strategy tailored for skin cancer classification and segmentation, allowing the model to incorporate new data progressively while reducing catastrophic forgetting. Our approach integrates multiple loss functions to preserve existing knowledge while adapting to additional magnification levels. Experimental results on the indistribution test set demonstrate consistent performance improvements: achieving 89.05% accuracy with 10× magnification, 92.68% with 10× and 5× combined, and 95.53% when incorporating 10×, 5×, and 2× magnifications. These findings highlight the potential of our method to improve the adaptability and reliability of deep learning systems for empirical generalization in skin cancer classification tasks.