The official code for the paper "Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Data"
In this project, we collect a large-scale medical multi-modal dataset, MedMD, with 16M 2D or 3D images. We train a new medical multi-modal generative model RadFM on it, enabling both 2D and 3D scans, multi-image input and visual-language interleaving cases.
All Datasets are released! We have updated the links in our dataset table. You can find all our text part data in https://huggingface.co/datasets/chaoyi-wu/RadFM_data_csv.
For decompressing the splited compression files in most cases, please check the following code in linux:
cat zip.z* > myzip.zip
unzip myzip.zip
For quick start, you can check the Quick_demo path.
We demonstrate a simple diagnosis case here to show how to inference with our model.
Feel free to modify it as you want.
-
S1. Download Model checkpoint or form baiduyun (No need for decompressing).
-
S2. Decompress the original zip file, you can get a
pytorch_model.bin. -
S3. put
pytorch_model.binunder pathQuick_demo/. -
S4. python
test.pyand you can get a conversation as:Input:
Can you identify any visible signs of Cardiomegaly in the image?
Output: yes
By the way, never try to perform this in cpu and gpu is all you need :）.
For re-training a model on our dataset or large-scale testing our pre-train model, you can check src.
Simply, train.py for training and test.py for testing.
- Check the data_csv to get how different datasets are processed and download them into
src/Dataset/data_csv - Modify the path as you disire, and check
src/train.pyto pre-train orsrc/train.pyto test.
Some cases produced by our final model:
Datasets downloading URL:
The split of each dataset can be found in https://huggingface.co/datasets/chaoyi-wu/RadFM_data_csv you just need to download the image part from each datasets.
Check the following table to see how to process each dataset and how each file in https://huggingface.co/datasets/chaoyi-wu/RadFM_data_csv is linked to each dataset:
| Dataset Name | Process Dataset Code | Related Filename |
|---|---|---|
| Rad3D-series | jpg2nii Process Code, nii2npy Process Code, Final Datset to Read npy and Related Texts | radiology_article_npy_train/test.json |
| MPx-series | MedPix Dataset | MedPix_muli_train/test.csv, MedPix_single_train/test.csv |
| PMC-Inline | Paper-inline Dataset | paper_train.csv (This dataset is not used for evaluation) |
| PMC-CaseReport | Case-report Dataset | filtered_case_report_train/test.csv |
| VinDr-Mammo | Diagnosis Open Format Dataset, Diagnosis Close (yes/no) Format Dataset | mammo_balance_train/test.csv |
| VinDr-SpineXR | Diagnosis Open Format Dataset, Diagnosis Close (yes/no) Format Dataset | spinexr_balance_train/test.csv |
| VinDr-PCXR | Diagnosis Open Format Dataset, Diagnosis Close (yes/no) Format Dataset | pcxr_balance_train/test.csv |
| PMC-OA | Pmcoa Dataset | pmcoa_image_caption_train/test.csv |
| PMC-VQA | vqa Dataset | pmcvaq_train/test.csv |
| VQA-RAD | vqa Dataset | vqarad_train/test.csv |
| SLAKE | vqa Dataset | slakevqa_train/test.csv |
| MIMIC-CXR | CXR Open Captioning Dataset | mimic_caption_train/test.csv |
| VinDr-CXR | Diagnosis Open Format Dataset, Diagnosis Close (yes/no) Format Dataset | chestxray_balance_train_new.csv, chestxray_balance_test.csv |
| NIH ChestXray14 | Diagnosis Open Format Dataset, Diagnosis Close (yes/no) Format Dataset | chestxray_balance_train_new.csv, chestxray_balance_test.csv |
| CheXpert | Diagnosis Open Format Dataset, Diagnosis Close (yes/no) Format Dataset | chestxray_balance_train_new.csv, chestxray_balance_test.csv |
| Covid-CXR2 | Diagnosis Open Format Dataset, Diagnosis Close (yes/no) Format Dataset | chestxray_balance_train_new.csv, chestxray_balance_test.csv |
| NLM-TB | Diagnosis Open Format Dataset, Diagnosis Close (yes/no) Format Dataset | chestxray_balance_train_new.csv, chestxray_balance_test.csv |
| Object-CXR | Diagnosis Open Format Dataset, Diagnosis Close (yes/no) Format Dataset | chestxray_balance_train_new.csv, chestxray_balance_test.csv |
| OpenI | Diagnosis Open Format Dataset, Diagnosis Close (yes/no) Format Dataset | chestxray_balance_train_new.csv, chestxray_balance_test.csv |
| RSNA | Diagnosis Open Format Dataset, Diagnosis Close (yes/no) Format Dataset | chestxray_balance_train_new.csv, chestxray_balance_test.csv |
| SIIM-ACR | Diagnosis Open Format Dataset, Diagnosis Close (yes/no) Format Dataset | chestxray_balance_train_new.csv, chestxray_balance_test.csv |
We sincerely thank all the contributors who uploaded the relevant data in our dataset online. We appreciate their willingness to make these valuable cases publicly available.
If you have any questions, please feel free to contact wtzxxxwcy02@sjtu.edu.cn.
