Multi-provider LLM resilience in Python without provider-specific code

OpenAI, Anthropic, and Google expose different APIs, message formats, tool-calling conventions, error types, and response structures.

That difference is manageable while an application uses only one provider. It becomes more expensive when the application needs retries, circuit breakers, fallback routes, observability, and recovery across several providers.

Without a shared abstraction, resilience logic tends to be implemented repeatedly:

OpenAI integration
├── request conversion
├── retry logic
├── error classification
├── circuit breaker
└── tool-call recovery

Anthropic integration
├── request conversion
├── retry logic
├── error classification
├── circuit breaker
└── tool-call recovery

Google integration
├── request conversion
├── retry logic
├── error classification
├── circuit breaker
└── tool-call recovery

Enter fullscreen mode Exit fullscreen mode

I did not want to build resilience three times.

I wanted to define the recovery policy once and apply it across every provider supported by the application.

That became llm-api-resilience, a Python library for retries, ordered failover, circuit breakers, and checkpoint recovery across multiple LLM providers.

Quick start

pip install llm-api-resilience

Enter fullscreen mode Exit fullscreen mode

Once the provider adapters are configured, an ordered fallback plan takes only a few lines:

from llm_api_resilience import RecoveryPlan, ResilientLLM, Route

llm = ResilientLLM(
    RecoveryPlan(
        [
            Route("openai-primary", openai_adapter),
            Route("anthropic-backup", anthropic_adapter),
            Route("google-last-resort", google_adapter),
        ]
    )
)

response = llm.chat(
    [{"role": "user", "content": "Explain circuit breakers briefly."}]
)

print(response.selected_route)
print(response.content)

Enter fullscreen mode Exit fullscreen mode

• GitHub: llm-api-resilience
• Provider adapter: llm-api-adapter

The library is built on top of llm-api-adapter, which provides one interface for OpenAI, Anthropic, and Google.

Because provider-specific differences are handled by the adapter, the resilience layer can operate on a shared contract instead of understanding three separate APIs.

The result is a Python library with:

• ordered failover across models and providers;
• configurable retry policies;
• circuit breakers;
• attempt metadata;
• checkpoint recovery for tool-calling sessions.

One interface across three providers

Before adding resilience, I had already built llm-api-adapter to normalize the main differences between provider APIs.

The application can create adapters for different providers and call the same chat() method:

import os

from llm_api_adapter.universal_adapter import UniversalLLMAPIAdapter

openai_adapter = UniversalLLMAPIAdapter(
    organization="openai",
    model="gpt-5.6-sol",
    api_key=os.environ["OPENAI_API_KEY"],
)

anthropic_adapter = UniversalLLMAPIAdapter(
    organization="anthropic",
    model="claude-sonnet-5",
    api_key=os.environ["ANTHROPIC_API_KEY"],
)

google_adapter = UniversalLLMAPIAdapter(
    organization="google",
    model="gemini-3.5-flash",
    api_key=os.environ["GOOGLE_API_KEY"],
)

messages = [
    {
        "role": "user",
        "content": "Reply with exactly: OK",
    }
]

adapters = {
    "openai": openai_adapter,
    "anthropic": anthropic_adapter,
    "google": google_adapter,
}

for provider, adapter in adapters.items():
    response = adapter.chat(
        messages=messages,
        max_tokens=64,
    )

    print(
        provider,
        adapter.model,
        "->",
        response.content,
    )

Enter fullscreen mode Exit fullscreen mode

The important part is not only that the method names are the same.

The adapter creates a stable boundary around:

• messages;
• tool definitions and results;
• normalized responses;
• finish reasons;
• structured outputs;
• provider errors.

The resilience layer does not need to know how Anthropic represents a tool result, which OpenAI parameter controls structured output, or how Google formats its request.

It only needs to understand the adapter contract.

Once that contract existed, models from different providers could become routes in the same recovery plan.

Defining recovery once

A recovery plan contains an ordered collection of routes.

Each route contains an adapter, a RoutePolicy for retries, timeouts, and backoff, and an optional CircuitBreaker.

from llm_api_resilience import (
    CircuitBreaker,
    RecoveryPlan,
    ResilientLLM,
    Route,
    RoutePolicy,
)

primary_breaker = CircuitBreaker(
    failure_threshold=3,
    cooldown_s=30,
)

llm = ResilientLLM(
    RecoveryPlan(
        [
            Route(
                "openai-primary",
                openai_adapter,
                policy=RoutePolicy(
                    timeout_s=15,
                    max_attempts=2,
                    backoff_s=0.25,
                ),
                breaker=primary_breaker,
            ),
            Route(
                "anthropic-backup",
                anthropic_adapter,
                policy=RoutePolicy(
                    timeout_s=20,
                    max_attempts=1,
                ),
            ),
            Route(
                "google-last-resort",
                google_adapter,
                policy=RoutePolicy(
                    timeout_s=30,
                    max_attempts=1,
                ),
            ),
        ]
    )
)

response = llm.chat(
    messages,
    max_tokens=128,
)

print("Selected route:", response.selected_route)
print("Response:", response.content)

Enter fullscreen mode Exit fullscreen mode

The application sends one normalized request.

The adapter handles provider-specific integration. The resilience layer decides which route to use and what to do when it fails.

Application
     ↓
llm-api-resilience
     ↓
llm-api-adapter
     ↓
OpenAI / Anthropic / Google

Enter fullscreen mode Exit fullscreen mode

The application does not need separate fallback branches that rebuild messages, tools, and parameters for every provider.

Not every failure should be retried

Adding a retry loop around every exception is easy.

Deciding which failures should actually be retried is harder.

A timeout may succeed on the next attempt. A server error may disappear after a short delay. A rate limit may require backoff or a switch to another route.

An invalid API key will not be fixed by retrying. Neither will an invalid schema or an incorrectly configured tool.

In the current default classifier, only normalized timeout, rate-limit, and server errors are retryable:

from llm_api_adapter.errors import (
    LLMAPIRateLimitError,
    LLMAPIServerError,
    LLMAPITimeoutError,
)

from llm_api_resilience import DefaultFailureClassifier

classifier = DefaultFailureClassifier()

print(
    [
        error_type.__name__
        for error_type in classifier.retryable_errors
    ]
)

# [
#     "LLMAPITimeoutError",
#     "LLMAPIRateLimitError",
#     "LLMAPIServerError",
# ]

Enter fullscreen mode Exit fullscreen mode

Authentication, configuration, invalid tool or schema errors, and ordinary ValueError failures are not retried by default.

This is another benefit of placing resilience above a provider adapter.

Each provider exposes different exceptions, but the recovery layer does not need three unrelated classification systems. Provider-specific failures are mapped into shared error types before the recovery policy evaluates them.

Retry first, then fail over

A route can receive multiple attempts before the recovery plan selects the next route.

primary route
      ↓
first attempt fails
      ↓
retry policy allows another attempt
      ↓
retry fails
      ↓
route is exhausted
      ↓
backup route selected

Enter fullscreen mode Exit fullscreen mode

Failover is ordered rather than random.

This allows the application to express preferences such as:

• use a cheaper model first;
• keep a similar model as the first backup;
• move to another provider after local retries are exhausted;
• reserve the most expensive model for the final route.

But every attempt has a cost.

Retries add latency and token usage. A backup model may have a different price, context window, or behavioral profile.

A resilience policy should therefore not mean “keep trying until something works.” It should define how much latency, cost, and model variation the application is willing to accept.

Circuit breakers avoid predictable failures

Retries help with isolated errors. They are less useful when a route is already degraded.

Without a circuit breaker, every new request may wait for the same route to fail several times before moving to a healthy provider.

A circuit breaker remembers recent route health and increments its failure count only for retryable errors such as timeouts, rate limits, and server errors:

closed
   ↓
retryable failures reach threshold
   ↓
open
   ├─ cooldown not expired → request skipped
   │
   └─ cooldown expired + next request
          ↓
      half-open
       ├─ probe succeeds → closed
       └─ probe fails    → open

Enter fullscreen mode Exit fullscreen mode

The circuit does not move to half-open automatically when the timer expires. That transition happens when the next request calls allow_request() after the cooldown.

While the circuit is open, the recovery plan can skip that route instead of repeatedly sending requests that are likely to fail.

The breaker is attached to a route rather than necessarily disabling an entire provider. One model or endpoint may be degraded while another route from the same provider remains usable.

Automatic fallback should not hide failures

A resilient request may eventually succeed even when several things went wrong first.

For example:

OpenAI attempt 1: timeout
OpenAI attempt 2: timeout
Anthropic attempt 1: success

Enter fullscreen mode Exit fullscreen mode

From the application’s perspective, the request succeeded.

Operationally, however, the primary route failed twice and the backup provider handled the request. That affects reliability, latency, and cost.

llm-api-resilience exposes attempt metadata so the recovery path remains visible:

response = llm.chat(
    messages,
    max_tokens=128,
)

for attempt in response.attempts:
    print(
        attempt.route_name,
        attempt.provider,
        attempt.model,
        attempt.duration_s,
        attempt.success,
        attempt.error_type,
    )

Enter fullscreen mode Exit fullscreen mode

An output could look like this:

openai-primary     openai     gpt-5.6-sol       2.41  False  LLMAPITimeoutError
openai-primary     openai     gpt-5.6-sol       2.39  False  LLMAPITimeoutError
anthropic-backup   anthropic  claude-sonnet-5   1.12  True   None

Enter fullscreen mode Exit fullscreen mode

The attempt records also contain start time and error information. They do not include a separate retry number or retry count; the final selected route is available on response.selected_route.

Because error_message is currently derived from str(error), it may contain sensitive details from an upstream exception. Applications should sanitize or restrict error messages before exporting attempt metadata to logs or external observability systems.

This makes it possible to detect:

• increasing fallback frequency;
• a route becoming unreliable;
• latency caused by repeated attempts;
• an expensive backup handling too much traffic;
• circuit breakers opening frequently.

The goal is not only to eventually receive a response. It is also to understand how that response was produced.

Provider fallback is only the first layer

Request-level failover is relatively straightforward when nothing has happened between model calls.

A tool-calling session introduces a more dangerous failure mode.

Imagine this sequence:

user asks the agent to report an incident
        ↓
model requests create_incident_ticket
        ↓
application creates ticket #12345
        ↓
tool result is returned
        ↓
model continuation fails

Enter fullscreen mode Exit fullscreen mode

The external side effect has already happened.

The ticket exists. The application cannot safely pretend that the turn never started.

Restarting the entire turn may make the model request the same tool again and create ticket #12346 for the same incident.

Switching providers does not automatically solve this. The resilience layer must rebuild the provider-neutral session from the checkpoint captured before the first tool round and replay the result of the tool that already executed.

This is where ordinary request fallback stops being sufficient.

Checkpoint recovery after tool execution

A resilient tool-calling session creates a checkpoint before the external tool is executed.

The application then executes the tool and passes its result back through continue_with().

If the continuation fails with a retryable error, the library can restore the checkpoint, select a backup route, and reuse the stored tool result instead of executing the side effect again.

tool call requested
        ↓
checkpoint created
        ↓
tool executed
        ↓
result stored in journal
        ↓
continuation fails
        ↓
backup route selected
        ↓
checkpoint restored
        ↓
stored result replayed
        ↓
session continues

Enter fullscreen mode Exit fullscreen mode

Here is a simplified incident-ticket example:

import json

from llm_api_adapter.models.tools import ToolSpec
from llm_api_resilience import ToolResult

tools = [
    ToolSpec(
        name="create_incident_ticket",
        description="Create an incident ticket.",
        json_schema={
            "type": "object",
            "properties": {
                "incident_id": {"type": "string"},
                "severity": {"type": "string"},
            },
            "required": ["incident_id", "severity"],
            "additionalProperties": False,
        },
    )
]

session = llm.session(
    [
        {
            "role": "user",
            "content": "Create a high-severity ticket for INC-42.",
        }
    ],
    tools=tools,
    tool_choice="auto",
    max_tokens=256,
)

first_response = session.start()

if first_response.tool_calls:
    # The checkpoint exists before the external side effect.
    assert session.checkpoint is not None

    tool_results = []

    for tool_call in first_response.tool_calls:
        # The application performs the external action.
        ticket_id = f"TICKET-{tool_call.arguments['incident_id']}"

        tool_value = {
            "ticket_id": ticket_id,
            "status": "created",
        }

        tool_results.append(
            ToolResult(
                tool_call_id=tool_call.call_id or tool_call.name,
                content=json.dumps(tool_value),
                idempotency_key=(
                    f"incident-ticket:"
                    f"{tool_call.arguments['incident_id']}"
                ),
                replay_policy="side_effecting",
            )
        )

    recovered_response = session.continue_with(tool_results)

    print("Final route:", recovered_response.selected_route)
    print("Answer:", recovered_response.content)

    assert session.checkpoint is not None
    assert len(session.journal.entries) == 1

Enter fullscreen mode Exit fullscreen mode

There is no public restore_checkpoint() call in the application code.

Recovery happens inside continue_with().

If the primary continuation fails, the session can restore its checkpoint and continue through the next route. When the backup route produces a replay-compatible tool call, the journal returns the stored result and the application does not intentionally execute the side effect again. If the backup model changes the tool or its arguments, the application receives a new tool call that requires its own decision.

This requires more than an HTTP retry wrapper.

The checkpoint itself preserves:

• the original messages from before the first tool round;
• provider-neutral request parameters;
• the route identity;
• the operation ID.

Provider-specific previous_response state is intentionally excluded because it may only be compatible with the same provider and model.

The tool journal separately preserves:

• the completed tool call;
• its arguments;
• the returned result;
• an idempotency identifier;
• the replay policy.

A backup provider should receive enough normalized context to continue the session without requiring provider-specific recovery code in the application.

Checkpoint recovery is not the same as idempotency

Checkpoint recovery has an important boundary.

It prevents the resilience layer from deliberately calling a completed tool again during session recovery. It does not automatically make every external system idempotent.

Consider a different failure:

application sends request to create ticket
        ↓
external service creates the ticket
        ↓
network fails before the response returns

Enter fullscreen mode Exit fullscreen mode

The application does not know whether the operation succeeded.

There is no confirmed tool result to store or replay.

A local checkpoint cannot resolve that uncertainty by itself.

Important side effects should still use mechanisms such as:

• idempotency keys;
• unique operation identifiers;
• database uniqueness constraints;
• status lookup before retry;
• transactional outbox patterns.

Checkpoint recovery and external idempotency solve related but different problems.

Checkpoint recovery protects progress after the tool result is known.

Idempotency protects the external operation when its execution status is uncertain.

Testing recovery without waiting for a real outage

Real provider failures are unpredictable and expensive to reproduce.

A resilience library needs deterministic tests that control exactly when every route succeeds or fails.

For example, a test can define this sequence:

primary start: tool call
        ↓
tool executes once
        ↓
primary continuation: timeout
        ↓
backup start: reconstructed tool call
        ↓
backup continuation: success

Enter fullscreen mode Exit fullscreen mode

SequenceAdapter is a test helper that returns a predefined sequence of responses and exceptions.

The important assertions are not limited to the final answer:

final_response = session.continue_with(result)

assert final_response.selected_route == "backup"
assert final_response.content == "Ticket created successfully."

assert [
    attempt.route_name
    for attempt in session.attempts
] == [
    "primary",
    "primary",
    "backup",
    "backup",
]

assert len(session.journal.entries) == 1
assert session.checkpoint is not None

Enter fullscreen mode Exit fullscreen mode

These assertions verify that the backup route completed the session, all route attempts were recorded, one tool result was journaled, and the checkpoint still exists.

They do not by themselves prove that the external side effect ran only once. That requires an explicit execution counter around the tool implementation:

executions = []

def create_ticket(arguments):
    ticket_id = f"TICKET-{arguments['incident_id']}"
    executions.append(ticket_id)

    return {
        "ticket_id": ticket_id,
        "status": "created",
    }

# After recovery:
assert executions == ["TICKET-INC-42"]

Enter fullscreen mode Exit fullscreen mode

With that assertion, the test proves that recovery reused the stored result without executing the external action again.

Deterministic adapters are also useful for testing retries and circuit breakers. Tests should control error order and route behavior rather than hoping that a real provider fails at the right moment.

Why use a library instead of only a gateway?

Managed LLM gateways are useful.

They can centralize routing, credentials, quotas, observability, billing, and organization-wide policies.

llm-api-resilience targets a different boundary: the Python application itself.

A library-level approach can be useful when the application needs to:

• use existing provider accounts and API keys;
• define recovery routes in Python;
• keep control over retry and failover decisions;
• test recovery locally and deterministically;
• access conversation state and completed tool results;
• recover stateful tool-calling sessions.

A generic gateway can route inference requests, but it may not understand the application’s checkpoint, tool journal, or whether an external side effect has already happened.

This does not mean that a gateway and an application-level resilience library are mutually exclusive.

A gateway can manage infrastructure-level routing while the application manages stateful recovery.

Current limitations

A resilience layer does not make different providers interchangeable in every way.

Backup models are not behaviorally identical

Different models may interpret the same conversation differently.

A backup route may complete the request but produce another style, tool choice, or decision.

Failover improves availability. It does not make models deterministic substitutes.

Recovery depends on state

Process-local checkpoints are not sufficient for every deployment.

Distributed workers, process restarts, and long-running sessions may require persistent checkpoint storage and coordination between instances.

External side effects still require protection

The journal prevents deliberate replay of a known completed tool result. It cannot guarantee how an external service behaves when the execution result is unknown.

Every attempt has a cost

Retries and fallback increase latency and token usage.

Policies need explicit limits rather than unlimited attempts.

Error classification requires maintenance

Providers add new error types and change response behavior. Normalized error mappings need ongoing tests and updates.

These are not reasons to avoid resilience. They define where a general recovery layer ends and application-specific reliability work begins.

Define resilience once

The main benefit of building on top of a provider adapter is not only shorter configuration.

It is avoiding duplicated architecture.

Without a shared provider boundary, retries, circuit breakers, observability, failover, and session recovery must be integrated separately into every provider implementation.

With a unified adapter contract, the application can define one recovery system across OpenAI, Anthropic, and Google.

one request interface
        ↓
one normalized error model
        ↓
one recovery plan
        ↓
multiple providers

Enter fullscreen mode Exit fullscreen mode

Provider fallback protects the next LLM call.

Checkpoint recovery protects the work that already happened before it.

• GitHub: llm-api-resilience
• Provider adapter: llm-api-adapter