API 시간 초과: 중단점을 찾아 안전하게 재시도

API Timeout: How to Diagnose Dropouts and Configure Safe Retries for Long LLM Generations

An API Timeout means that one participant in the request path stopped waiting. The client may have failed to connect, a proxy may have closed an idle stream, the application’s total deadline may have expired, or a gateway may not have received an upstream response in time. The message alone does not prove that the model is unavailable.

The safe response is to record the exception class, HTTP status and body when present, elapsed time, and request ID, then change only the limit owned by the failing layer. Increasing every timeout at once hides the cause and may repeat work whose result is still unknown.

If the request used BetterToken, open the Dashboard before retrying and match its time, model, status and Token usage. That immediately separates a request that reached the API from a failure before the gateway and reduces the chance of replaying an unknown result.

Where Drops Occur: HTTP Request Lifecycle Phases

An LLM request crosses several independently controlled timers:

[Client] --- (1. Connect Timeout) ---> [API Gateway]
[Client] --- (2. Write Timeout)   ---> [Prompt Upload]
[Gateway] --- (3. Upstream wait)  ---> [Model provider]
[Client] <--- (4. Read / stream)   --- [Response chunks]
[App]    --- (5. Total deadline)  ---> [Whole operation]

Enter fullscreen mode Exit fullscreen mode

1. Connect timeout belongs to DNS, TCP and TLS connection establishment.
2. Write timeout applies while the client sends a chunk of the request body.
3. Read or stream-idle timeout applies while the client waits for the next response chunk, not necessarily the whole generation.
4. Pool timeout is client-side waiting for an available pooled connection.
5. Application deadline caps the whole business operation; upstream timeout is a separate gateway or provider limit. Changing one does not extend the others.

Timeout Diagnostics Matrix

Symptom / Exception Failure Phase Underlying Cause Engineering Resolution httpx.ConnectTimeout and no HTTP response DNS, TCP or TLS Name resolution, route, certificate chain, proxy or outbound policy Reproduce from the same runtime; compare DNS, CA and proxy settings before changing the connect limit httpx.ReadTimeout before or between chunks Client read/idle timer No chunk arrived during that timer Compare first-byte and inter-chunk timing; inspect intermediate proxy idle limits HTTP status and error body Gateway or upstream The request reached an HTTP server Save status, body and request ID; follow that provider’s error contract instead of treating it as a connect timeout httpx.PoolTimeout Client pool No connection became available Measure concurrency and pool occupancy; change limits only if the pool is confirmed saturated Application cancellation at a fixed elapsed time Total deadline Caller, job runner or reverse proxy stopped the operation Identify the owner of that deadline and compare it with all lower-layer timers

A prioritized diagnostic playbook

1. Capture evidence before retrying. Record start/end time, exception class, HTTP status and body, request ID, whether any response chunk arrived, model, endpoint, and the runtime that sent the request.
2. Test the client and network. Repeat from the same container or host. If another environment succeeds, compare DNS, TLS CA, proxy variables and outbound firewall rules.
3. Inspect every intermediary. Compare the client read timer with reverse-proxy and corporate-proxy idle/read timers. A longer SDK timeout cannot extend a shorter intermediary timeout.
4. Separate gateway from upstream. An HTTP response proves that a server answered. Use its status, body and request ID to distinguish a gateway rejection, rate limit or upstream failure.
5. Change one variable and re-test. Keep the model, prompt, network, endpoint and proxy unchanged. This preserves a causal comparison.

Granular Timeout Configuration in Python (HTTPX)

HTTPX documents separate connect, read, write and pool timeouts. The values below are an example for a controlled test, not universal production recommendations; derive real values from measured request phases and your own end-to-end deadline.

import os
import httpx
from openai import OpenAI

API_KEY = os.environ.get("BETTERTOKEN_API_KEY", "your_api_key_here")

custom_timeout = httpx.Timeout(
    connect=5.0,    # Example: socket and TLS establishment
    read=120.0,     # Example: maximum silence between received chunks
    write=10.0,     # Example: maximum silence while sending a chunk
    pool=10.0       # Example: wait for a pooled connection
)

http_client = httpx.Client(
    timeout=custom_timeout,
    limits=httpx.Limits(max_keepalive_connections=50, max_connections=100)
)

client = OpenAI(
    base_url="https://www.bettertoken.ai/v1",
    api_key=API_KEY,
    http_client=http_client
)

response = client.chat.completions.create(
    model="YOUR_CURRENT_MODEL_ID",
    messages=[
        {"role": "system", "content": "You are a senior systems architect."},
        {"role": "user", "content": "Design a high-throughput distributed message broker."}
    ],
    stream=True
)

for chunk in response:
    delta = chunk.choices[0].delta.content or ""
    print(delta, end="", flush=True)

Enter fullscreen mode Exit fullscreen mode

Safe retries across the full SSE lifecycle

Treat a timeout after submission as an unknown result until evidence shows otherwise:

1. Do not automatically replay a partially received stream. Re-sending may create a second generation and additional usage; preserve the partial result and reconcile the first request.
2. Exponential Backoff with Full Jitter: Retries must be spaced using progressive, randomized delays to prevent thundering herd problems.
3. Idempotency Requires Server Support: A local task ID helps correlate logs but does not prevent duplicate server work. Retry automatically only when documented idempotency-key support and request-status reconciliation prove the outcome; otherwise reconcile an unknown result with usage records first.

import time
import random
import httpx

def stream_with_safe_retry(send_request, outcome_known_not_accepted, max_attempts=3):
    """send_request() returns a context-managed iterable response."""
    base_delay = 1.0
    for attempt in range(max_attempts):
        received_any = False
        try:
            with send_request() as response:
                for chunk in response:
                    received_any = True
                    yield chunk
            return
        except (httpx.ConnectTimeout, httpx.ReadTimeout,
                httpx.RemoteProtocolError) as err:
            # This catches failures raised while iterating the SSE response too.
            if (received_any or attempt == max_attempts - 1
                    or not outcome_known_not_accepted(err)):
                raise
            sleep_time = random.uniform(0, base_delay * (2 ** attempt))
            time.sleep(sleep_time)

Enter fullscreen mode Exit fullscreen mode

outcome_known_not_accepted must rely on documented server behavior or an external status check. It must not infer non-acceptance merely from a client timeout. Bound both the attempt count and the total operation deadline.

Two tests that prove or disprove the fix

1. Short control request: request a small response and record status, time to first byte, total duration, request ID and whether the stream completed. If this fails, investigate connectivity, authentication and endpoint configuration before testing a long generation.
2. Controlled long request: after the short test succeeds, increase only the expected output or restore the original workload. Keep model, endpoint, network and proxy unchanged. If only this test breaks, compare read/idle timers, total deadline and intermediary limits.

Recovery is confirmed only when the short test and one repetition of the original scenario complete with the expected status and result, without an unexplained duplicate. If a request reached BetterToken, use the Dashboard to match its time, model, status and Token usage, then open the API reference and verify the endpoint contract before changing retry behavior.

If the response is 429, use the separate rate-limit and Retry-After guide. If only the event stream breaks, continue with the SSE streaming diagnostics.

Sources

• HTTPX timeout types
• OpenAI API error handling
• Claude API errors and request IDs

Originally published on the BetterToken blog.

BetterToken provides pay-as-you-go access to AI model APIs through
OpenAI-compatible and Anthropic-compatible endpoints — useful if you are wiring
Claude Code, Codex, or your own tooling to a custom base URL.
See the docs to get started.