README.md

---
license: mit
widget:
- src: >-
    https://prod-images-static.radiopaedia.org/images/566180/d527ff6fc1482161c9225345c4ab42_big_gallery.jpg
  candidate_labels: enlarged heart, pleural effusion
  example_title: X-ray of cardiomegaly
library_name: open_clip
pipeline_tag: zero-shot-image-classification
---

# Model Card for WhyXrayCLIP 🩻

#  Table of Contents

1. [Model Details](#model-details)
2. [Get Started](#how-to-get-started-with-the-model)
3. [Uses](#uses)
4. [Training Details](#training-details)
5. [Evaluation](#evaluation)
6. [Citation](#citation)

## Model Details

WhyXrayCLIP can align X-ray images with text descriptions. It is fine-tuned from [OpenCLIP (ViT-L/14)](https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K) on [MIMIC-CXR](https://physionet.org/content/mimic-cxr/2.0.0/) with clinical reports processed by GPT-4. WhyXrayCLIP significantly outperforms PubMedCLIP, BioMedCLIP, etc. in zero-shot and linear probing on various chest X-ray datasets. (See results in [Evaluation](#evaluation)) While our CLIP models excel with careful data curation, training converges quickly, suggesting the current contrastive objective might not fully exploit the information from the data, potentially taking shortcuts, such as comparing images from different patients instead of focusing on diseases. Future research should explore more suitable objectives and larger-scale data collections to develop more robust medical foundation models.

- **Paper:** https://arxiv.org/pdf/2405.14839
- **Website:** https://yueyang1996.github.io/knobo/
- **Repository:** https://github.com/YueYANG1996/KnoBo


## How to Get Started with the Model

Use the code below to get started with the model.

```bash
pip install open_clip_torch
```

```python
import torch
from PIL import Image
import open_clip

model, _, preprocess = open_clip.create_model_and_transforms("hf-hub:yyupenn/whyxrayclip")
model.eval()
tokenizer = open_clip.get_tokenizer("ViT-L-14")

image = preprocess(Image.open("test_xray.jpg")).unsqueeze(0)
text = tokenizer(["enlarged heart", "pleural effusion"])

with torch.no_grad(), torch.cuda.amp.autocast():
 image_features = model.encode_image(image)
 text_features = model.encode_text(text)
 image_features /= image_features.norm(dim=-1, keepdim=True)
 text_features /= text_features.norm(dim=-1, keepdim=True)

 text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)
```


## Uses

As per the original [OpenAI CLIP model card](https://github.com/openai/CLIP/blob/d50d76daa670286dd6cacf3bcd80b5e4823fc8e1/model-card.md), this model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero-shot medical image (X-ray) classification. We also hope it can be used for interdisciplinary studies of the potential impact of such models.

### Direct Use

WhyXrayCLIP can be used for zero-shot X-ray classification. You can use it to compute the similarity between an X-ray image and a text description.

### Downstream Use

WhyXrayCLIP can be used as a feature extractor for downstream tasks. You can use it to extract features from X-ray images and text descriptions for other downstream tasks.

### Out-of-Scope Use

WhyXrayCLIP should not be used for clinical diagnosis or treatment. It is not intended to be used for any clinical decision-making. Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.


## Training Details

### Training Data
We utilize the [MIMIC-CXR](https://physionet.org/content/mimic-cxr/2.0.0/) dataset, specifically selecting only the PA and AP X-rays, which results in 243,334 images, each accompanied by a clinical report written by doctors. We preprocess these reports by extracting medically relevant findings, each described in a short and concise term. In total, we assemble 953K image-text pairs for training WhyXrayCLIP.

### Training Details

We utilize the training script from [OpenCLIP](https://github.com/mlfoundations/open_clip) and select [ViT-L/14](https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K) as the backbone. Training is performed on 4 RTX A6000 GPUs for 10 epochs with a batch size of 128 and a learning rate of 1e−5. We choose checkpoints based on the lowest contrastive loss on validation sets.

## Evaluation

### Testing Data

We evaluate on 5 X-ray classification datasets: [Pneumonia](https://pubmed.ncbi.nlm.nih.gov/29474911/), [COVID-QU](https://arxiv.org/pdf/2003.13145), [NIH-CXR](https://www.kaggle.com/datasets/nih-chest-xrays/data), [Open-i](https://www.kaggle.com/datasets/raddar/chest-xrays-indiana-university), and [VinDr-CXR](https://vindr.ai/datasets/cxr). We report the zero-shot and linear probing accuracy on the above 5 datasets.

### Baselines
We compare various CLIP models, including [OpenAI-CLIP](https://huggingface.co/openai/clip-vit-large-patch14), [OpenCLIP](https://huggingface.co/laion/CLIP-ViT-L-14-laion2B-s32B-b82K), [PubMedCLIP](https://huggingface.co/flaviagiammarino/pubmed-clip-vit-base-patch32), [BioMedCLIP](https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224), [PMC-CLIP](https://huggingface.co/ryanyip7777/pmc_vit_l_14) and [MedCLIP](https://github.com/RyanWangZf/MedCLIP). We evaluate these models in both zero-shot and linear probe scenarios. In zero-shot, GPT-4 generates prompts for each class, and we use the ensemble of cosine similarities between the image and prompts as the score for each class. In linear probing, we use the CLIP models as image encoders to extract features for logistic regression. Additionally, we include [DenseNet-121](https://github.com/mlmed/torchxrayvision) (fine-tuned on the pretraining datasets with cross-entropy loss) as a baseline for linear probing.

### Results
The figure below shows the averaged Zero-shot and Linear Probe performance of different models on five chest X-ray datasets.

![Results](X-ray-results.png)

## Citation

Please cite our paper if you use this model in your work:

```
@article{yang2024textbook,
 title={A Textbook Remedy for Domain Shifts: Knowledge Priors for Medical Image Analysis}, 
 author={Yue Yang and Mona Gandhi and Yufei Wang and Yifan Wu and Michael S. Yao and Chris Callison-Burch and James C. Gee and Mark Yatskar},
 journal={arXiv preprint arXiv:2405.14839},
 year={2024}
}
```