Visual-Pill-ID: Building an AI Pharmacist with GPT-4o and SAM 💊

Ever stared at a handful of loose pills and wondered, “Wait, was the blue one for my allergies or my blood pressure?” 😅 You’re not alone. Medication errors are a massive global health challenge.

In this tutorial, we are building Visual-Pill-ID, a cutting-edge computer vision pipeline that solves the “multi-pill confusion” problem. By combining the geometric precision of the Segment Anything Model (SAM) with the multimodal reasoning of GPT-4o, we can transform a messy photo of mixed medication into a structured, verified prescription list.

We’ll be diving deep into Computer Vision, Instance Segmentation, and Multimodal LLMs to create a production-ready OCR and identification system.

The Architecture: Precision Meets Intelligence 🧠

The biggest challenge in pill identification isn’t just “seeing” the pill; it’s isolating it from a crowded background and understanding its specific markings. Our pipeline follows a “Segment-then-Analyze” pattern.

graph TD
    A[Raw Image of Multiple Pills] --> B[SAM: Segment Anything Model]
    B --> C{Instance Masks}
    C --> D[OpenCV: Crop & Preprocess]
    D --> E[GPT-4o Vision: Multi-modal Analysis]
    E --> F[OCR & Pill Identification]
    F --> G[Prescription Validation & Safety Logic]
    G --> H[Final Structured JSON Output]

Enter fullscreen mode Exit fullscreen mode

Prerequisites 🛠️

To follow along, you’ll need:

• PyTorch: For running the SAM weights.
• Segment Anything Model (SAM): The vit_h or vit_b checkpoints.
• OpenCV: For image manipulation.
• GPT-4o API Key: For the heavy lifting in multimodal reasoning.
• Python 3.9+

Step 1: Isolating Pills with Segment Anything (SAM)

Traditional bounding boxes often overlap when pills are touching. We need Instance Segmentation to get the exact pixels of each pill.

import numpy as np
import torch
import cv2
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator

# Load the SAM model
device = "cuda" if torch.cuda.is_available() else "cpu"
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h_4b8939.pth").to(device)

# Generate masks automatically
mask_generator = SamAutomaticMaskGenerator(sam)

def get_pill_masks(image_path):
    image = cv2.imread(image_path)
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    masks = mask_generator.generate(image)

    # Filter by area to remove tiny artifacts
    filtered_masks = [m for m in masks if m['area'] > 500]
    return filtered_masks, image

print(f"🚀 Detected {len(get_pill_masks('pills.jpg')[0])} potential pills!")

Enter fullscreen mode Exit fullscreen mode

Step 2: Preprocessing and GPT-4o Multimodal Analysis

Once we have the masks, we crop each pill. However, sending 10 separate images to GPT-4o is expensive. Instead, we create a “Collage of Interest” or send them in a structured batch. GPT-4o is incredible at OCR on curved surfaces, which is typical for medication.

import base64
import requests

def encode_image(image_np):
    _, buffer = cv2.imencode('.jpg', image_np)
    return base64.b64encode(buffer).decode('utf-8')

def identify_pills(pill_crops):
    # Constructing the multimodal prompt
    prompt_content = [
        {"type": "text", "text": "Identify each pill in these images. Extract markings, color, and shape. Compare with standard medical databases."}
    ]

    for crop in pill_crops:
        base64_image = encode_image(crop)
        prompt_content.append({
            "type": "image_url",
            "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}
        })

    # Call GPT-4o
    # API implementation details...
    # Return structured JSON

Enter fullscreen mode Exit fullscreen mode

Step 3: The Secret Sauce — Prescription Verification

It’s not enough to know it’s “Ibuprofen 200mg.” We need to know if it matches the user’s prescription. By feeding the OCR text from the medicine bottle (also captured in the pipeline) and the identified pills into GPT-4o, we can perform a cross-check.

💡 The “Official” Way to Scale

While this DIY pipeline is great for prototyping, building a production-grade medical vision system requires handling edge cases like glare on blister packs and HIPAA-compliant data handling.

For more production-ready examples and advanced patterns on integrating LLMs with specialized computer vision models, check out the detailed guides at WellAlly Tech Blog. They cover deep-dives into AI reliability that are crucial for healthcare applications. 🥑

Implementation Code: Putting it all Together

Here is a snippet showing how we combine the SAM mask with an OpenCV crop to feed the vision model:

def process_pipeline(img_path):
    masks, original_img = get_pill_masks(img_path)
    pill_data = []

    for i, mask in enumerate(masks):
        # Create a bounding box from the mask
        x, y, w, h = mask['bbox']
        crop = original_img[y:y+h, x:x+w]

        # In a real app, you'd send this to GPT-4o
        # id_result = call_gpt4o_vision(crop)

        pill_data.append({
            "id": i,
            "position": mask['point_coords'],
            "confidence": mask['stability_score']
        })

    return pill_data

# Example output structure
# [
#   {"id": 1, "label": "Metformin", "color": "white", "shape": "oblong"},
#   {"id": 2, "label": "Lisinopril", "color": "pink", "shape": "round"}
# ]

Enter fullscreen mode Exit fullscreen mode

Conclusion: The Future of Vision-AI 🚀

By combining SAM’s spatial awareness with GPT-4o’s semantic intelligence, we’ve built a pipeline that understands both the where and the what. This multi-stage approach is much more robust than using a single “end-to-end” model which might hallucinate pill counts.

What’s next for Visual-Pill-ID?

1. 3D Reconstruction: Using Gaussian Splatting to see all sides of a pill.
2. Edge Deployment: Running a quantized SAM on mobile devices.

Are you working on Multimodal AI? Drop a comment below or share your thoughts on medical AI safety! 👇

If you enjoyed this technical deep dive, don’t forget to visit wellally.tech/blog for more insights on high-performance AI architectures!