Capital de TokensCalling an AI model from code looks simple:
response = client.responses.create(...)But in production that line lives in a less tidy world: the network can take too long, the API can return a temporary error, a provider can become degraded, a credential can be invalid, or a dependency can fail exactly when real traffic arrives.
That is why a resilient AI client is not simply a wrapper around an SDK. It is a small piece of architecture where several classic design patterns and several operational resilience patterns come together.
In this article we will take apart a concrete resilient-client example and see which patterns appear, why they are there, and how they relate to one another.
The reference example is available in this Gist: Resilient AI client: timeout, retries, circuit breaker, fallback, and metrics.
This article brings those ideas down to code architecture. For the broader framework —SRE, availability, SLOs, error budgets, checkpoints, idempotency, recovery, and observability— start with Reliability engineering: how to design systems that fail well.
The overall map
The main patterns are:
| Pattern | Role inside the client | Benefit |
|---|---|---|
| Strategy | AIProvider and its implementations | Switch providers without changing the resilient client |
| Adapter | OpenAIProvider | Translate the real SDK into the internal contract |
| Dependency Injection | Provider arrives through the constructor | Decouple creation from use |
| Circuit Breaker | CircuitBreaker | Temporarily stop hammering an unhealthy dependency |
| State machine | CLOSED, OPEN, HALF_OPEN | Change behavior based on circuit state |
| Simple Factory | build_client() | Centralize construction and configuration |
| Facade | ResilientAIClient | Expose a simple API over several internal policies |
On top of those, we add resilience patterns such as timeout, retry, exponential backoff, jitter, fallback, and observability.
The important idea is this:
Resilience rarely comes from one pattern. It comes from composing several small responsibilities around a critical operation.
1. Strategy: choose the provider without changing the client
The Strategy pattern lets us encapsulate different ways of executing the same operation behind a common contract.
In Python, a Protocol fits very well:
class AIProvider(Protocol):
def generate(self, prompt: str, timeout: float) -> str:
...The application can have different implementations:
class DemoProvider:
def generate(self, prompt: str, timeout: float) -> str:
...
class OpenAIProvider:
def generate(self, prompt: str, timeout: float) -> str:
...The resilient client does not need to know which one it is using:
class ResilientAIClient:
def __init__(self, provider: AIProvider):
self.provider = provider
def generate(self, prompt: str) -> str:
return self.provider.generate(prompt, timeout=10)Conceptually:
AIProvider
▲
┌──────┴───────┐
│ │
DemoProvider OpenAIProvider
▲ ▲
└──────┬───────┘
│
ResilientAIClientThe benefit is enormous: the resilience policy stops depending on OpenAI, Anthropic, a local model, or any other specific provider.
That reduces coupling and makes testing easier.
Strategy also helps testing
If ResilientAIClient received a real SDK directly, testing retries or the circuit breaker would require network calls or much more intrusive mocks.
With Strategy we can inject a controlled provider:
class AlwaysFailProvider:
def generate(self, prompt: str, timeout: float) -> str:
raise RetryableAIError("temporary failure")And test exactly how the resilience layer reacts.
2. Dependency Injection: depend on abstractions
Strategy works especially well because the provider is injected from outside:
client = ResilientAIClient(provider=OpenAIProvider())That is Dependency Injection.
The client does not do this:
class ResilientAIClient:
def __init__(self):
self.provider = OpenAIProvider()If it did, it would become coupled to a specific implementation.
The injected version better respects dependency inversion:
high level
ResilientAIClient
│
▼
abstraction
AIProvider
▲
│
low level
OpenAIProviderThe high-level component depends on the contract, not on SDK details.
3. Adapter: tame the external SDK
OpenAIProvider also plays another role: it acts as an Adapter.
The application wants a simple interface:
provider.generate(prompt, timeout)The external SDK, by contrast, may work like this:
client.responses.create(
model=model,
input=prompt,
)The adapter translates between the two worlds:
ResilientAIClient
│
│ AIProvider.generate()
▼
OpenAIProvider
│
│ adapts
▼
OpenAI SDKThat keeps SDK-specific structures from leaking throughout the application.
It also lets us translate technical exceptions into domain-specific errors:
RetryableAIError
NonRetryableAIErrorThat classification matters because the system needs to know which failures are worth retrying and which are not.
4. Circuit Breaker: know when to stop insisting
One of the most important patterns in distributed systems is the Circuit Breaker.
Imagine an external API is completely down.
Without a circuit breaker:
request → fails
retry → fails
request → fails
retry → fails
request → fails
retry → failsThe client keeps spending connections, time, and resources against a service that is clearly unhealthy.
With a circuit breaker:
several failures
↓
open circuit
↓
reject calls temporarily
↓
wait for recovery
↓
try one probe callThe circuit usually has three states:
class CircuitState(str, Enum):
CLOSED = "CLOSED"
OPEN = "OPEN"
HALF_OPEN = "HALF_OPEN"
CLOSED
Calls pass normally. If consecutive failures exceed a threshold, the circuit opens.
OPEN
Calls are temporarily blocked so the dependency is not continuously hammered.
HALF_OPEN
After a recovery interval, one probe call is allowed.
If it succeeds, the circuit returns to CLOSED.
If it fails, it returns to OPEN.
Is this also the State pattern?
Precision matters here.
We have a state machine, but that does not necessarily mean we are using the GoF State pattern in its full form.
A classic State implementation might use separate objects:
ClosedState
OpenState
HalfOpenStateEach state would encapsulate its own behavior.
If instead we use an Enum and conditionals inside CircuitBreaker, the more accurate description is a simplified state machine.
That distinction is useful because it prevents us from calling every piece of code with a state variable a “State Pattern.”
5. Simple Factory: centralize construction
Suppose the application has a function like this:
def build_client(use_openai: bool) -> ResilientAIClient:
if use_openai:
provider = OpenAIProvider()
else:
provider = DemoProvider()
return ResilientAIClient(provider=provider)This centralizes the decision about which objects to create.
It is a Simple Factory.
There is no need to turn every constructor function into a formal Factory Method hierarchy. A simple function is often enough.
The advantage is that the rest of the application can ask for:
client = build_client(use_openai=True)without knowing all the construction details.
6. Facade: one entry point
From the consumer’s perspective, ResilientAIClient also acts as a Facade.
Calling code sees something simple:
result = client.generate(prompt)But behind it, many things may happen:
request
↓
timeout
↓
retry
↓
backoff + jitter
↓
circuit breaker
↓
provider adapter
↓
fallback
↓
metrics + logsThat is exactly the value of a facade: hide internal complexity behind a small, stable interface.
Resilience patterns are not necessarily GoF patterns
An important distinction appears here.
Strategy, Adapter, and Facade are classic object-oriented design patterns.
By contrast, timeout, retry, and circuit breaker belong more to the world of resilience and distributed-systems patterns.
That does not make them less important. In real services, they are often more decisive than many GoF patterns.
Timeout: put a limit on time
Without a timeout, a call can remain blocked for too long.
The timeout defines a budget:
"if there is no useful response in 10 seconds,
I consider this attempt failed"That lets us regain control and execute the next policy: retry, fallback, or error.
A timeout does not fix the dependency. It prevents that dependency from monopolizing our resources indefinitely.
Retry: repeat only when it makes sense
A temporary error may disappear on the next attempt.
For example:
HTTP 429
HTTP 502
HTTP 503
transient timeoutBut other failures should not be retried:
invalid credential
malformed request
nonexistent model
business-rule violationThat is why classification into retryable and non-retryable errors is a fundamental design decision.
Exponential Backoff
Retrying immediately can make an outage worse.
A typical policy waits progressively longer:
attempt 1
↓ fails
wait 0.5 s
attempt 2
↓ fails
wait 1 s
attempt 3
↓ fails
wait 2 s
attempt 4
↓ fails
wait 4 sThat is exponential backoff.
The system gives the dependency more time to recover.
Jitter: avoid the thundering herd
Now imagine a thousand clients receive the same failure at the same time.
If all of them wait exactly one second, all of them will hit the server again exactly one second later.
That can create a new wave of synchronized traffic.
Jitter adds a small random variation:
client A → 1.08 s
client B → 0.91 s
client C → 1.17 s
client D → 0.96 sLoad becomes better distributed over time.
Fallback: degrade instead of failing completely
When the primary provider cannot respond, we may have an alternative.
Examples:
premium model → cheaper model
remote provider → local model
generative response → cached response
full service → reduced functionalityThis is graceful degradation.
The key is that the fallback should preserve useful functionality without silently hiding important errors.
Metrics and logs: observable resilience
A resilient system that produces no operational signals can be very difficult to maintain.
Useful metrics include:
ai_requests_total
ai_request_failures_total
ai_retries_total
ai_fallbacks_total
ai_circuit_open_total
ai_latency_secondsThey help answer real questions:
- Is the provider degraded?
- Are retries rescuing requests or merely adding latency?
- Is fallback being used too often?
- Does the circuit breaker open frequently?
- What percentage of calls succeed on the first attempt?
Observability turns a resilience mechanism into something that can be operated and improved.
A pattern that would fit very well: Decorator
There is an interesting architectural improvement.
If ResilientAIClient directly contains timeout, retry, circuit breaker, fallback, and metrics, it may eventually become too large a class.
An alternative is to model each responsibility as a Decorator:
MetricsDecorator
↓
FallbackDecorator
↓
CircuitBreakerDecorator
↓
RetryDecorator
↓
TimeoutDecorator
↓
OpenAIProviderEach layer implements the same contract:
class AIProvider(Protocol):
def generate(self, prompt: str, timeout: float) -> str:
...Then policies can be composed:
provider = OpenAIProvider()
provider = TimeoutDecorator(provider)
provider = RetryDecorator(provider)
provider = CircuitBreakerDecorator(provider)
provider = MetricsDecorator(provider)That moves the design toward a highly modular architecture.
But it also introduces more objects and more abstraction.
The right question is not “is Decorator more elegant?”
The question is:
Has the system become complex enough to justify separating each policy into an independent component?
How all this relates to SOLID
This design touches several SOLID principles.
Single Responsibility Principle
The provider talks to the SDK.
The circuit breaker protects against unhealthy dependencies.
The factory constructs objects.
The metrics layer observes.
The clearer those boundaries are, the easier the system is to maintain.
Open/Closed Principle
We can add a new provider without rewriting the resilient client:
class LocalLLMProvider:
...The system is open to extension and relatively closed to modification.
Dependency Inversion Principle
ResilientAIClient depends on AIProvider, not on a concrete SDK.
That keeps infrastructure details from dominating application design.
A complete architectural view
We can summarize the system like this:
application
│
▼
ResilientAIClient
Facade
│
┌──────────┼──────────┐
│ │ │
Retry Circuit Metrics
│ Breaker │
└──────────┼──────────┘
│
AIProvider
Strategy
│
┌───────┴────────┐
│ │
DemoProvider OpenAIProvider
│
Adapter
│
▼
OpenAI SDKAround that structure sit timeout, backoff, jitter, and fallback.
What to say in an interview
If you had to describe this design compactly, a good answer would be:
The client uses Strategy to abstract providers, Adapter to isolate the external SDK, Dependency Injection to decouple construction from use, Simple Factory to create the configuration, Facade to expose a simple interface, and Circuit Breaker as a resilience pattern. Around that it applies timeout, retry with exponential backoff and jitter, fallback, and metrics.
And I would add one precision:
CLOSED,OPEN, andHALF_OPENform a state machine; I would only call it the full GoF State Pattern if each state encapsulated its behavior in separate objects.
The most important lesson
The value of this example is not in collecting pattern names.
It is in seeing how each pattern removes a specific kind of coupling or failure:
Strategy → avoids coupling to one provider
Adapter → prevents SDK details from leaking through the app
Dependency Injection → makes substitution and testing easier
Circuit Breaker → stops hammering a failed dependency
Retry → tolerates transient failures
Backoff + Jitter → avoids amplifying incidents
Fallback → preserves useful functionality
Metrics → makes behavior observable
Facade → keeps the public API simpleThat is practical architecture.
The point is not to introduce patterns because they appear in a book. It is to recognize a concrete force in the system —coupling, latency, failure, saturation, provider variability— and use the right tool to contain it.
When an AI client starts participating in important processes, that difference separates a demo that works from a component that can live in production.