VLM AI Model Development
Budget: ₹12,500 – ₹37,500 INR
Project Title: Fine-tune a Vision-Language Model to read vital signs from patient monitor photos
OVERVIEW
We are a healthcare technology startup building a remote patient-monitoring product for hospitals. We need an AI model that looks at a photo of any patient monitor (NICU / ICU bedside monitor) and extracts the vital signs as structured data.
The model must work across DIFFERENT monitor brands and layouts (Philips, GE, Drager, Mindray, Nihon Kohden, SLE, etc.) — not just one fixed brand. It must understand that different labels mean the same vital (e.g. "HR", "PR", "Pulse", "Heart Rate" all mean heart rate).
This is a semantic understanding problem, NOT a simple OCR or bounding-box detection task. Please do not propose Roboflow / YOLO / Tesseract-only solutions — they cannot generalize across unseen layouts.
WHAT THE MODEL MUST DO
- Input: one image of a patient monitor (often in a cluttered real-world hospital scene, with staff/equipment in frame)
- Output: clean JSON, e.g.
{ "hr": 142, "spo2": 98, "rr": 45, "bp_sys": 70, "bp_dia": 40, "temp": 36.8 }
- Use null for any vital not visible
- Must be robust to different fonts, colors, screen layouts, glare and angles
SCOPE OF WORK
1. Fine-tune an open-source VLM. Our preferred base model is Qwen2.5-VL-7B (Apache 2.0 license). You may propose an alternative (e.g. InternVL2) only with strong justification, but Qwen2.5-VL is our default choice.
2. Build a data pipeline: we will provide raw monitor images; help us set up an auto-labeling step (using an external vision API to generate first-pass labels) plus a human-verification workflow.
3. Run the fine-tune using LoRA/QLoRA (free GPU environments like Kaggle/Colab are acceptable — keep training cost near zero).
4. Export the final model to GGUF format so it can run locally via Ollama.
5. Deliver a deployment guide so the model runs offline on BOTH macOS (Apple Silicon) AND Windows machines — no internet and no cloud cost.
6. Provide an accuracy report on a held-out test set.
CONSTRAINTS (IMPORTANT)
- Final model must run OFFLINE and CROSS-PLATFORM — it must work on both macOS (Apple Silicon) and Windows PCs (with or without an NVIDIA GPU). Patient data cannot leave the hospital — privacy requirement.
- Target hardware is a mid-range machine with 16GB RAM. Inference speed of a few seconds per image is acceptable.
- License must allow commercial use (Apache 2.0 / MIT preferred). No models with research-only licenses. (This is one reason we prefer Qwen2.5-VL-7B, which is Apache 2.0.)
- Inference cost after deployment must be zero (self-hosted only).
DELIVERABLES
- Fine-tuned model weights + exported GGUF file
- Training/fine-tuning code (documented, reproducible)
- Auto-labeling + data-prep scripts
- Deployment instructions for BOTH macOS and Windows (via Ollama)
- Accuracy evaluation report
IDEAL SKILLS AND EXPERIENCE
- Proven experience fine-tuning Vision-Language Models (Qwen-VL strongly preferred; also InternVL, LLaVA, PaliGemma, etc.)
- Strong Python; PyTorch; Hugging Face Transformers; PEFT/LoRA; Unsloth a plus
- Experience exporting models to GGUF and running with Ollama / llama.cpp on both Mac and Windows
- Comfortable with QLoRA on limited/free GPU resources
- Bonus: prior medical / OCR / document-AI projects
TO APPLY
Please share:
1. Your proposed approach in 3-4 lines
2. Rough timeline and milestone breakdown
We have a starter dataset of real monitor photos ready to share with shortlisted candidates.
OVERVIEW
We are a healthcare technology startup building a remote patient-monitoring product for hospitals. We need an AI model that looks at a photo of any patient monitor (NICU / ICU bedside monitor) and extracts the vital signs as structured data.
The model must work across DIFFERENT monitor brands and layouts (Philips, GE, Drager, Mindray, Nihon Kohden, SLE, etc.) — not just one fixed brand. It must understand that different labels mean the same vital (e.g. "HR", "PR", "Pulse", "Heart Rate" all mean heart rate).
This is a semantic understanding problem, NOT a simple OCR or bounding-box detection task. Please do not propose Roboflow / YOLO / Tesseract-only solutions — they cannot generalize across unseen layouts.
WHAT THE MODEL MUST DO
- Input: one image of a patient monitor (often in a cluttered real-world hospital scene, with staff/equipment in frame)
- Output: clean JSON, e.g.
{ "hr": 142, "spo2": 98, "rr": 45, "bp_sys": 70, "bp_dia": 40, "temp": 36.8 }
- Use null for any vital not visible
- Must be robust to different fonts, colors, screen layouts, glare and angles
SCOPE OF WORK
1. Fine-tune an open-source VLM. Our preferred base model is Qwen2.5-VL-7B (Apache 2.0 license). You may propose an alternative (e.g. InternVL2) only with strong justification, but Qwen2.5-VL is our default choice.
2. Build a data pipeline: we will provide raw monitor images; help us set up an auto-labeling step (using an external vision API to generate first-pass labels) plus a human-verification workflow.
3. Run the fine-tune using LoRA/QLoRA (free GPU environments like Kaggle/Colab are acceptable — keep training cost near zero).
4. Export the final model to GGUF format so it can run locally via Ollama.
5. Deliver a deployment guide so the model runs offline on BOTH macOS (Apple Silicon) AND Windows machines — no internet and no cloud cost.
6. Provide an accuracy report on a held-out test set.
CONSTRAINTS (IMPORTANT)
- Final model must run OFFLINE and CROSS-PLATFORM — it must work on both macOS (Apple Silicon) and Windows PCs (with or without an NVIDIA GPU). Patient data cannot leave the hospital — privacy requirement.
- Target hardware is a mid-range machine with 16GB RAM. Inference speed of a few seconds per image is acceptable.
- License must allow commercial use (Apache 2.0 / MIT preferred). No models with research-only licenses. (This is one reason we prefer Qwen2.5-VL-7B, which is Apache 2.0.)
- Inference cost after deployment must be zero (self-hosted only).
DELIVERABLES
- Fine-tuned model weights + exported GGUF file
- Training/fine-tuning code (documented, reproducible)
- Auto-labeling + data-prep scripts
- Deployment instructions for BOTH macOS and Windows (via Ollama)
- Accuracy evaluation report
IDEAL SKILLS AND EXPERIENCE
- Proven experience fine-tuning Vision-Language Models (Qwen-VL strongly preferred; also InternVL, LLaVA, PaliGemma, etc.)
- Strong Python; PyTorch; Hugging Face Transformers; PEFT/LoRA; Unsloth a plus
- Experience exporting models to GGUF and running with Ollama / llama.cpp on both Mac and Windows
- Comfortable with QLoRA on limited/free GPU resources
- Bonus: prior medical / OCR / document-AI projects
TO APPLY
Please share:
1. Your proposed approach in 3-4 lines
2. Rough timeline and milestone breakdown
We have a starter dataset of real monitor photos ready to share with shortlisted candidates.