Benchmarking Serverless GPUs: Modal vs RunPod vs Replicate Cold Starts (2026)

Deploying open-source LLMs (like Llama-3) or real-time Whisper transcription in production often forces a difficult architectural trade-off: keep dedicated GPUs running 24/7 (expensive) or rely on serverless scale-to-zero (cold start latency penalty).

To evaluate container spin-up overhead, we benchmarked median cold start latencies and per-second execution costs across the major serverless GPU platforms.

Benchmark Results

Provider GPU Median Cold Start Equiv. Hourly Rate Scale-To-Zero Modal A100 (40GB) 1.8s ~$2.85 / hr Yes RunPod Serverless A100 (80GB) 4.2s ~$2.59 / hr Yes Replicate A100 (80GB) 6.5s ~$4.14 / hr Yes Together AI H100 Cluster Instant (Pooled) Token-based N/A Lambda Labs A100 (80GB) VM Boot (~45s) $1.89 / hr No

Key Observations

1. Snapshot Restoration: Modal’s filesystem and memory snapshotting drastically cut container initialization down to under 2 seconds, making user-facing on-demand LLM calls viable.
2. Cost-Efficiency: For asynchronous batch jobs or high-throughput queues, RunPod Serverless remains the most cost-effective choice per GPU second.
3. Managed Inference: If you do not require custom CUDA extensions or proprietary container code, pooled inference APIs like Together AI provide instantaneous responses without cold-start engineering.

The full benchmark dataset, hardware configurations, and testing scripts are maintained at ServerlessGPUBench.

Raw benchmark metrics are also open-sourced on GitHub: awesome-serverless-gpu-latency.