Synthesis: Multimodal fusion (omics + imaging + clinical)

Evidence grades (GRADE-adapted): A high — multiple well-conducted RCTs or systematic reviews converge. B moderate — single pivotal RCT or consistent observational evidence. C limited — single observational study, mechanistic, or expert consensus. D preclinical / hypothesis-generating.

Why multimodal fusion

Each TNBC data modality captures partial information about tumor biology and likely clinical outcomes:

• Genomics / transcriptomics captures molecular drivers but doesn't capture spatial organization or stromal composition
• Histopathology captures morphology and immune infiltrate spatial patterns but doesn't capture sub-resolution molecular features
• Radiology captures macroscopic tumor extent, multifocality, and treatment response but doesn't resolve cellular-level features
• Clinical features capture patient context, prior treatment history, comorbidities, and performance status that none of the biological modalities measure
• Blood-based markers / ctDNA capture systemic disease status and emerging resistance signals

A unified predictor that integrates all available modalities should, in principle, capture more of the relevant biology than any single-modality approach. Whether and how to achieve this integration is the multimodal fusion research question.

Fusion architectures

ML approaches to multimodal fusion fall into several categories:

Early fusion (feature concatenation)

Features from each modality are extracted independently, concatenated into a single feature vector, and processed by a standard classifier (random forest, SVM, neural network). Simple but doesn't model cross-modality interactions explicitly. Often a baseline approach.

Late fusion (decision-level aggregation)

Each modality has its own classifier that produces a probability score; scores are aggregated (averaged, weighted, or combined via a meta-classifier) to produce the final prediction. Modular and robust to missing modalities but doesn't model cross-modality interactions.

Intermediate fusion (representation-level)

Each modality has its own encoder that produces a learned representation; representations are combined in a shared latent space, with a unified classifier on top. Allows cross-modality interactions while respecting modality-specific structure.

Attention-based fusion

Cross-attention mechanisms (from the transformer architecture) explicitly model interactions between modalities, allowing the model to learn which modality contributes most to each prediction. Multi-head attention can capture multiple interaction patterns simultaneously.

Graph-based fusion

Multi-modal data is represented as a graph where nodes are features or modalities and edges capture relationships. Graph neural networks process these structured representations.

TNBC-specific multimodal applications

Outcome prediction

Multimodal models for predicting pCR, EFS, OS, or other clinical outcomes have been developed integrating various data combinations:

• H&E + clinical — the simplest multimodal combination; consistently outperforms either alone
• H&E + RNA-seq + clinical — adds molecular information; typically improves AUC by 0.03–0.08 over unimodal
• H&E + RNA-seq + radiology + clinical — comprehensive multimodal; gains additional modest improvement
• H&E + ctDNA + clinical — emerging combination leveraging non-invasive ctDNA monitoring

Published TNBC multimodal models for pCR prediction report AUC values in the 0.80–0.88 range — higher than the 0.65–0.80 of single-modality approaches. Validation across independent cohorts remains limited.

Subtype refinement

Multimodal subtype classification combining bulk transcriptomics, methylation, mutation data, and IHC features (the Burstein-framework methodology, modernized with ML) has been applied to produce refined subtype calls. The FUSCC Jiang 2019 framework is the most prominent example^[1]B.

Treatment selection

Reinforcement learning approaches modeling sequential treatment decisions in metastatic TNBC have been proposed but are at very early research stage. The data requirements (large numbers of treatment-sequenced patients with detailed multi-modal profiling) limit current applicability.

Foundation models and pretraining

Large pre-trained foundation models for biomedical data are an active research direction with TNBC applications:

• Pathology foundation models (UNI, CONCH, Path Foundation, Phikon) pretrained on millions of pathology slides, fine-tuned for TNBC-specific tasks
• Transcriptomic foundation models (scGPT, Geneformer) pretrained on large-scale single-cell or bulk transcriptomic data
• Multi-modal biomedical foundation models attempting unified pretraining across modalities

Whether foundation-model-based multimodal fusion outperforms task-specific training in TNBC clinical prediction tasks is being demonstrated.

Data and validation challenges

Multimodal TNBC research faces substantial data challenges:

• Patient cohorts with all modalities are rare. Few public datasets include matched H&E, transcriptomic, radiologic, and clinical data for the same patients. TCGA-BRCA has many of these but predates modern IO therapy era; other datasets are smaller or have specific modality limitations.
• Modality-specific quality variation. H&E slides from different institutions vary in preparation; RNA-seq quality varies; radiology imaging varies in modality and resolution. Multimodal models must handle this heterogeneity.
• Missing modalities. Real-world patients often lack one or more modalities (no available tissue for RNA-seq, no MRI for some patients). Robust models must handle missing inputs gracefully.
• Validation across centers. A model trained on one institution's multi-modal data may not generalize to others. Multi-institutional validation is essential but logistically challenging.

Regulatory and operational considerations

Multimodal clinical-decision-support tools face regulatory complexity. A multimodal pCR predictor combining H&E features, RNA-seq, and clinical features would require:

• Standardization of each input modality (H&E preparation, RNA-seq protocol, clinical data definitions)
• Validation that the model performs as specified across the variation in each modality
• Clear specification of which modalities are required vs optional and how the model behaves under missing data
• Integration into clinical workflows that don't currently produce all modalities for every patient

The most likely first-deployment multimodal tools will combine routinely-collected data (H&E and clinical features at minimum; possibly imaging) rather than requiring specialized molecular profiling. Tools requiring routine RNA-seq are unlikely to reach broad clinical adoption until RNA-seq becomes routine in TNBC workup (which is not currently the case).

Evidence table

Open questions and active investigation

• What's the optimal fusion architecture for TNBC? No single architecture has clearly won across published TNBC multimodal studies. Attention-based and graph-based methods show promise but require larger validation datasets.
• Foundation models vs task-specific training. Whether pretraining on large biomedical datasets and fine-tuning on TNBC outperforms TNBC-specific training is being tested across multiple tasks.
• How to handle missing modalities. Practical clinical deployment requires graceful handling of missing inputs. Architectures with explicit missing-data handling (modality dropout during training, learned modality-imputation) are being developed.
• Reasoning over time-series multimodal data. Patients accumulate multi-modal data over their disease course. Models that integrate longitudinal information are more clinically useful than snapshots but harder to train and validate.
• Interpretability of multimodal predictions. Clinicians want to know which modalities and which features drove a specific prediction. Attention-based architectures provide some interpretability; how to make this clinically actionable is an active question.
• Real-world prospective validation. Most multimodal TNBC ML work is retrospective on archival cohorts. Prospective deployment with measurement of clinical decision impact is needed for regulatory adoption and routine use.

For the component-modality ML directions, see the ML subtype classification synthesis, the ML pathology pCR synthesis, and the ai pathology synthesis. For the biological subtype frameworks that multimodal models often try to refine, see the Burstein synthesis and the Lehmann/Pietenpol synthesis.

References

Each citation links to the original publication via DOI. The same records are searchable in the evidence library by title or DOI.

1. Jiang YZ, Ma D, Suo C, et al. Genomic and transcriptomic landscape of triple-negative breast cancers: subtypes and treatment strategies. Cancer Cell. 2019;35(3):428–440.e5. doi:10.1016/j.ccell.2019.02.001. ↩
2. Lipkova J, Chen RJ, Chen B, et al. Artificial intelligence for multimodal data integration in oncology. Cancer Cell. 2022;40(10):1095–1110. doi:10.1016/j.ccell.2022.09.012. ↩
3. Chen RJ, Lu MY, Williamson DFK, et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell. 2022;40(8):865–878. doi:10.1016/j.ccell.2022.07.004. ↩

Last reviewed: 2026-06-04. Researcher-layer synthesis page. Evidence grades follow the GRADE-adapted rubric defined at the top of this page.