Multimodal AI frameworks for cancer phenotype discovery.

MOSCARD addresses a core challenge in multimodal medical imaging: bias introduced when one modality systematically confounds another. By integrating causal reasoning directly into the fusion architecture, MOSCARD learns de-confounded representations that are more robust across patient populations and imaging conditions.

The model uses chest X-ray (CXR) as the primary modality and ECG as a complementary guiding modality. A co-attention mechanism learns which regions of the CXR are most relevant given the ECG signal. Encoders based on Vision Transformer (ViT) and MedCLIP provide modality-specific representations.

A structural causal model (SCM) is integrated into the training pipeline to explicitly model and remove confounding relationships between modalities. The framework supports four training modes: Baseline, Causal, Conf, and CaConf, enabling controlled ablation studies of each causal component.

A knowledge-grounded adaptation strategy for vision-language models (VLMs) applied to screening mammography. The framework constructs unique case-sets designed for resident training and few-shot adaptation, addressing the challenge of limited annotated mammography data.

Mini-batch selective sampling is used to build case-sets that maximize representational diversity within each adaptation batch. Two VLMs are evaluated: MedCLIP (in-domain, trained on medical imaging) and ALBEF (out-of-domain, trained on general vision-language pairs).

The approach is validated across zero-shot, few-shot, and supervised settings on UW Madison datasets, with external validation on Mayo Clinic. Authors include Aisha Urooj Khan, John Garrett, Tyler Bradshaw, Lonie Salkowski, Jiwoong Jeong, Amara Tariq, and Imon Banerjee. Code and model checkpoints are publicly available.

A multimodal spatiotemporal graph neural network for predicting 30-day all-cause hospital readmission. MM-STGNN fuses two distinct data streams: longitudinal chest radiographs (capturing imaging trajectory over time) and electronic health records (capturing structured clinical measurements and events).

The architecture combines GraphSAGE for learning patient similarity graphs from the EHR modality with a Gated Recurrent Unit (GRU) for modeling temporal dynamics in both imaging and clinical sequences. A cross-modal attention mechanism aligns representations from the two modalities before final prediction.

Evaluated on the MIMIC-IV dataset, MM-STGNN achieved AUROC 0.79 on both evaluation splits, demonstrating that fusing longitudinal imaging with EHR data provides a meaningful improvement over single-modality baselines. Authors include Siyi Tang, Amara Tariq, Jared A. Dunnmon, and Imon Banerjee, among others.