Multimodal Structured Retrieval Augumented Radiology Report Generator

I am building a clinically robust, retrieval-augmented framework that produces structured radiology reports from chest-x-ray images and associated text. Accuracy and clinical relevance drive every design choice, so I want the system to learn equally from both the IU X-ray and MIMIC-CXR datasets.

The pipeline I envision looks like this:

• Visual encoding with ViT-B16 to obtain global image embeddings.
• Retrieval of the top-k similar studies from the training corpus to steer generation toward clinically plausible language and findings.
• Text generation with Clinical T5, producing both the “Findings” and “Impression” sections.
• Relation-aware validation using RadGraph, with a specific focus on analyzing relationships between clinical entities rather than mere keyword spotting.
• Evaluation scripts that compute BLEU, ROUGE, and CheXbert F1 so we can quantify linguistic quality and clinical correctness side by side.

Deliverables I expect:

1. A reproducible codebase (Python, PyTorch, Hugging Face ecosystem) that trains, fine-tunes, and serves the model end-to-end.
2. A modular retrieval component whose top-k parameter is easily adjustable.
3. Inference scripts that accept a new chest-x-ray, run retrieval, generate the report, and output RadGraph-structured relations.
4. An evaluation notebook or CLI that outputs BLEU, ROUGE, CheXbert F1, and RadGraph consistency scores on a held-out test set.
5. Clear README detailing environment setup, dataset preprocessing for both corpora, and expected runtimes on a single GPU.

Experience needed is medical imaging, Vision Transformers ViTB16, Clinical T5 .