Breast Cancer Classification using ResNeXt50_32x4d with Hybrid Loss

Authors: A. Hashemi, K. Azizi, M.K. Titkanlou, R. Moucek
Status: Submitted to Biomedical Signal Processing and Control, under review.

Abstract & Overview

This work studies binary breast cancer classification from histopathological images using a deployment-aware medical AI workflow. The goal is to support reliable benign-versus-malignant screening while keeping the experimental protocol rigorous enough for clinical research settings.

Using the BreaKHis dataset (7,909 images at 40x, 100x, 200x, and 400x magnifications), the model is fine-tuned with a patient-wise split to reduce leakage risk. A diverse augmentation pipeline and a hybrid loss function (weighted BCE plus hard triplet loss) improve generalization and embedding separability. The model achieves 99.37% test accuracy and an AUC-ROC of 0.9976, demonstrating stable performance across magnification levels.

Keywords & Key Concepts

Breast Cancer Classification, Histopathological Images, ResNeXt50_32x4d, Hybrid Loss Function, Data Augmentation, Multi-Resolution Analysis, Deep Learning, Transfer Learning

Dataset & Preprocessing

Dataset: BreaKHis contains 7,909 images from 82 patients. Classes include benign subtypes (adenosis, fibroadenoma, phyllodes tumor, tubular adenoma) and malignant subtypes (ductal carcinoma, lobular carcinoma, mucinous carcinoma, papillary carcinoma). Images are RGB histopathology samples across 40x, 100x, 200x, and 400x magnifications.

Preprocessing: The study uses a patient-wise 70/20/10 train-validation-test split. Class balance is handled through sampling strategies, and augmentation includes resizing, cropping, flips, affine transforms, color jitter, Gaussian blur, random erasing, MixUp, and CutMix.

Proposed Method & Architecture

CNN & Transfer Learning: Modified ResNeXt50_32x4d architecture with custom classifier head for binary output. Fine-tuning applied for histopathology images.

Model Architecture: ResNeXt50_32x4d backbone with ResNeXt+CBAM (Convolutional Block Attention Modules after layer3 & layer4). Hybrid loss: weighted BCE + hard triplet loss. Optimizer: AdamW with ReduceLROnPlateau, early stopping, and mixed precision training.

Evaluation & Results

Overall Accuracy: 99.37% with strong magnification-level performance: 40x: 100%, 100x: 99.54%, 200x: 99.49%, 400x: 98.33%.

Other Models: ResNeXt+CBAM: 99.24% overall, improved AUC-ROC at some levels. ViT-B/16: 98.86% accuracy, excels at high magnifications. DenseNet201: 98.23%, lower at higher resolutions.

Training curves show fast convergence (~54 epochs) with minimal overfitting. Confusion matrices and AUC-ROC confirm robustness. Proposed framework achieves state-of-the-art performance in binary breast cancer classification.

Hybrid loss + advanced augmentation ensures high accuracy, generalization, and robust embedding separation. Multi-resolution evaluation proves applicability across clinical histopathology images. Future work includes transformer-CNN hybrids, attention modules, and multi-class classification.