Advances in digital pathology and artificial intelligence have presented the potential to build models for objective diagnosis, prognosis and therapeutic-response and resistance prediction. In this talk we will discuss our work on: (1) Data-efficient methods for weakly-supervised whole slide classification with examples in cancer diagnosis and subtyping (Nature BME, 2021), identifying origins for cancers of unknown primary (Nature, 2021) and allograft rejection (Nature Medicine, 2022) (2) Discovering integrative histology-genomic prognostic markers via interpretable multimodal deep learning (Cancer Cell, 2022; IEEE TMI, 2020; ICCV, 2021; CVPR, 2024; ICML, 2024). (3) Building unimodal and multimodal foundation models for pathology, contrasting with language and genomics (Nature Medicine, 2024a, Nature Medicine 2024b, CVPR 2024). (4) Developing a universal multimodal generative co-pilot and chatbot for pathology (Nature, 2024). (5) 3D Computational Pathology (Cell, 2024) (6) Bias and fairness in computational pathology algorithms (Nature Medicine, 2024; Nature BME 2023).

Biomedical image analysis is a core part of clinical decision-making and scientific discovery. However, most clinicians and clinical researchers lack the computational resources or machine learning expertise required to train models for their specific imaging tasks. Existing solutions are often narrowly tailored to predefined tasks, and even foundation model-based approaches typically require fine-tuning, limiting accessibility and generalizability.

I will present frameworks for a new paradigm for biomedical image analysis -- where a single multimodal model is designed to solve a broad spectrum of imaging tasks without the need for task-specific training, fine-tuning, or ML expertise. We design new adaptable systems that tackle diverse and unseen tasks out-of-the-box. By removing the barriers of computational cost and ML specialization, these approaches promise to empower clinicians and researchers to tackle workflows that were not previously possible.

Machine learning is the dominant approach to solving biomedical imaging tasks and our group at Cornell has been at the forefront of much of the progress in this field over the last decade. In this talk, I will give an overview of some of the most recent projects that we have been working on, much of which comes from real world clinical needs. In each project, I will describe our motivation, present our machine learning formulation, and show experimental results that go beyond simple accuracy metrics. We will cover longitudinal image analysis in tracking disease progression and cancer screening, extracting quantitative measurements in a clinically feasible manner that transforms care delivery, and present a simple method to handle missing data. 

With the rapid advancements in AI models, there has been growing concern in recent literature about the risk of unintended biases affecting individuals unfairly based on factors like race, gender, and other clinical characteristics. One of the key challenges in analyzing AI bias is that the factors contributing to unfairness are often interconnected and can amplify one another. It is crucial to understand how individual, institutional, and societal biases may be either mitigated or exacerbated when adopting AI systems. Common approaches to eliminating bias, such as developing demographic-specific models, often face the challenge of insufficient demographic representation. In this talk, we present an alternative computational approach to creating 'fair' models that aim to reduce bias without relying on demographically balanced datasets. Our observations indicate that techniques aimed at decoupling demographic information from task predictions typically fail to match the performance of baseline models. While preserving the model performance, we will explore a novel ‘fair’ AI modeling approach that incorporates adversarial, causal, and contrastive learning techniques, with a focus on applications in digital healthcare.

There is undeniable innovation occurring in imaging brought forth by advances in artificial intelligence. Despite all of this innovation being touted over the past two decades, and certainly since the advent of AlexNet in 2012, we've seen relatively little adoption in clinical workflows. Here I'm discussing some of the incentives in venture capital and novel startups as they pertain to imaging as well as opportunities and groundwork that needs to be laid to actually make sure that this innovation comes forth into clinical and broadly life sciences workflows.

Medical foundation models can identify prognostic signals in patient data at scale, yet their black-box nature prevents us from translating these learned signals into testable scientific hypotheses. In this talk, we will demonstrate how to transform these models into engines for scientific discovery. Our pipeline uses mechanistic interpretability, specifically Sparse Autoencoders (SAEs) combined with large language models (LLMs), to decompose abstract features from 3D CT scans and lab values into hundreds of understandable clinical concepts. We used it to generate over 1,200 hypotheses aimed at explaining progression to major disease outcomes, including cancer, dementia, and cardiovascular, kidney, and metabolic diseases. One example hypothesis indicated that an imaging feature scoring the level of "incidental, benign, or indeterminate lesions in solid organs (liver, kidneys, adrenal glands, pancreas, or spleen)," was more strongly associated with onset of primary cancer (OR of 3.46 (95% CI 2.52–4.77)) than clinical risk factors like age (OR 1.29 (95% CI 1.00–1.67)) and smoking (OR 1.46 (95% CI 1.03–2.07)). Overall, this work demonstrates a scalable framework for turning any medical AI from a simple prediction tool into an engine for generating new clinical insights.

Human radiologists develop comprehensive understanding of medical imaging through training across diverse clinical contexts, enabling them to interpret CT, MRI, and PET scans across body regions and seamlessly transition between clinical tasks. Current AI models fundamentally lack this capability, remaining confined to narrow, task-specific domains due to disease-specific datasets, inability to adapt without retraining, and prohibitive annotation costs. In this talk, I present three progressive works that systematically address these barriers toward universal 3D medical imaging understanding. Hermes enables a single model to learn from diverse, heterogeneous medical data across multiple modalities and body regions. Iris introduces efficient in-context learning where the model segments novel anatomical structures absent from its training data using only one reference example without any retraining, matching supervised performance while being orders of magnitude faster than existing adaptive methods. MASS reduces annotation dependence through mask-guided self-supervised learning, enabling training on vast unlabeled imaging data while preserving adaptation capability. Together, these works establish a framework for universal medical imaging AI that is scalable, generalizable across modalities and body regions with reduced dependence on expert annotation, advancing toward AI systems with medical imaging understanding as broadly as human radiologists.

The field of biomedical AI is undergoing a paradigm shift, evolving from static prediction tools to generalist agents capable of reasoning, tool use, and scientific discovery. This talk explores this transition from passive models to active agents and presents Google DeepMind’s latest advancements in the space, including TxGemma, MedGemma, and Cell2Sentence (C2S).