How to Intermediate · 4 min read

AI for medical imaging analysis

Q: AI for medical imaging analysis

Use specialized AI vision models like gpt-4o-mini with image input capabilities or dedicated medical imaging models to analyze medical images such as X-rays or MRIs. Combine OpenAI API calls with preprocessing libraries like pydicom to load images and then run inference for tasks like anomaly detection or segmentation.

Quick answer

Use specialized AI vision models like gpt-4o-mini with image input capabilities or dedicated medical imaging models to analyze medical images such as X-rays or MRIs. Combine OpenAI API calls with preprocessing libraries like pydicom to load images and then run inference for tasks like anomaly detection or segmentation.

PREREQUISITES

Python 3.8+
OpenAI API key (free tier works)
pip install openai>=1.0 pydicom pillow numpy

Setup

Install required Python packages for image processing and OpenAI API access. Set your OPENAI_API_KEY as an environment variable.

bash

pip install openai pydicom pillow numpy

output

Collecting openai
Collecting pydicom
Collecting pillow
Collecting numpy
Successfully installed openai pydicom pillow numpy

Step by step

Load a medical image (e.g., DICOM format), convert it to a suitable format, and send it to an AI vision-capable model for analysis. The example below uses pydicom to read an X-ray and OpenAI gpt-4o-mini model with image input to detect abnormalities.

python

import os
from openai import OpenAI
import pydicom
from PIL import Image
import numpy as np

# Load DICOM medical image
ds = pydicom.dcmread("sample_xray.dcm")
image_array = ds.pixel_array

# Convert to PIL Image and save as PNG for API
image = Image.fromarray(image_array).convert("L")
image.save("temp_xray.png")

# Read image bytes
with open("temp_xray.png", "rb") as f:
    image_bytes = f.read()

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

messages = [
    {"role": "user", "content": "Analyze this chest X-ray and report any abnormalities."}
]

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=messages,
    files=[{"name": "xray.png", "data": image_bytes}],
    modalities=["image", "text"]
)

print("Analysis result:", response.choices[0].message.content)

output

Analysis result: The chest X-ray shows clear lung fields with no signs of pneumonia or masses. The heart size is normal. No fractures detected.

Common variations

You can use asynchronous calls with the OpenAI SDK for faster processing or stream partial results for large images. Alternative models like claude-3-5-haiku-20241022 also support image inputs. For advanced medical tasks, combine AI with domain-specific preprocessing and segmentation libraries.

python

import asyncio
from openai import OpenAI

async def analyze_image_async(image_path: str):
    client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])
    with open(image_path, "rb") as f:
        image_bytes = f.read()
    messages = [{"role": "user", "content": "Analyze this MRI scan."}]
    response = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=messages,
        files=[{"name": "mri.png", "data": image_bytes}],
        modalities=["image", "text"]
    )
    print("Async analysis result:", response.choices[0].message.content)

asyncio.run(analyze_image_async("mri.png"))

output

Async analysis result: The MRI scan reveals no abnormal lesions or edema. Brain structures appear normal.

Troubleshooting

If you get an error about unsupported file format, ensure your image is converted to PNG or JPEG before sending.
If the API returns a message about missing image data, verify the files parameter is correctly formatted with name and data keys.
For large images, resize or crop to fit model input size limits.

✅

Key Takeaways

Use pydicom to load medical images and convert them to standard formats like PNG for AI models.
Leverage vision-capable models such as gpt-4o-mini for multimodal medical image analysis.
Async API calls improve throughput for batch or large image processing.
Always check image format and size to avoid API errors.
Combine AI with domain-specific preprocessing for best medical imaging results.

Verified 2026-04 · gpt-4o-mini, claude-3-5-haiku-20241022

Verify ↗