A Multimodal Foundation Model for Longitudinal Patient Representation and Scalable Insight Generation in Oncology

The oFM is a longitudinal multimodal oncology foundation model trained by Tempus AI on a 1.67M-patient real-world cohort. It converts daily clinical and molecular events—including diagnoses, treatments, DNA, RNA, and pathology findings—into time-stamped episodes, encodes them with GatorTron-base-2k and Fourier Number Embedding, incorporates H&E whole-slide representations from Virchow2 and PRISM2, and integrates the trajectory with a two-layer temporal Transformer into a 1024-dimensional patient-state embedding. A three-stage curriculum combines TSDAE reconstruction, masked multimodal reconstruction, JEPA-style future latent prediction, VICReg regularization, and Cox survival supervision. Against curated clinical and molecular features, frozen oFM embeddings raised overall-survival AUC to 0.774 from 0.563, with corresponding gains for progression-free survival and treatment response. Across 11 comparative-treatment cohorts, treatment-benefit ranking reached tAUTOC/SD 4.61 versus 1.38 for baseline features, and the oFM ranked benefit better in 9 of 11 cohorts. For interpretation, a sparse autoencoder, latent-space steering, temporal precedence graphs, and retrieval-augmented reasoning with MedGemma 4B connect predictions to clinical and biological mechanisms. The oFM has 421M trained parameters, while its pathology encoders remain frozen. The authors emphasize that these retrospective results support representation quality and hypothesis generation, not validated treatment biomarkers or causal mechanisms.