Reliability engineering: how to design systems that fail well

A system can work perfectly during a demo and still be unreliable.

It can respond quickly when everything is healthy, pass all its tests, and deploy without errors. But the important question comes later:

what happens when a dependency stops responding, an external API returns errors, a machine restarts, or a pipeline stage fails after twenty minutes of work?

That is where reliability engineering begins.

The central idea is simple: do not design under the assumption that nothing will fail; design knowing that eventually something will.

What is reliability engineering?

Reliability Engineering is the discipline of making a system consistently deliver the expected behavior throughout the period for which it was designed.

In software, that does not simply mean “having few bugs.”

Reliability also depends on operational questions:

• What percentage of the time is the system available?
• What happens when a dependency fails?
• Can we quickly detect degradation?
• Can we recover without manual intervention?
• Can we retry an operation without duplicating effects?
• Can we deploy a new version without taking down the whole service?
• Do we know exactly what happened after an incident?

A reliable system is not necessarily one that never fails.

It is one where failures are expected, observable, contained, and recoverable.

The difference between “it works” and “it is reliable”

Imagine a pipeline that automatically generates an episode:

Article
   ↓
Script
   ↓
Audio
   ↓
Images
   ↓
Video
   ↓
Publication

A functional validation might say:

The pipeline works because it successfully generated and published a complete episode.

Reliability engineering asks different questions.

What happens if the fourth image fails after the script and audio have already been generated?

A fragile design might restart the whole process.

Image 4 fails
      ↓
start over
      ↓
regenerate script
      ↓
regenerate audio
      ↓
regenerate images

That wastes time, money, and compute. It also introduces new opportunities for failure.

A reliable design preserves valid work and resumes from a safe point.

Article     ✅
Script      ✅
Audio       ✅
Image 1     ✅
Image 2     ✅
Image 3     ✅
Image 4     ❌

     retry
       ↓
Image 4     ✅
Video       ✅
Publication ✅

Three especially important concepts appear here: checkpoints, idempotency, and retries.

Checkpoints: preserve valid progress

A checkpoint records that a stage completed correctly and that its result can be reused.

In a long pipeline, this makes it possible to resume from the last consistent point instead of repeating the entire process.

The idea looks like this:

stage A ✅ → save state
stage B ✅ → save state
stage C ❌

restart
   ↓
read state
   ↓
resume from C

Checkpoints are especially useful when stages are expensive: video generation, training, model inference, processing large files, or calls to external APIs.

Idempotency: repeat without duplicating effects

An operation is idempotent when it can run several times and produce the same observable result as if it had run only once.

This matters enormously in distributed systems.

Suppose an application sends a request to publish a video and then loses the connection before receiving the response.

The application does not know whether the publication happened.

If it simply repeats the operation, it could create two publications.

A reliable strategy uses a stable identifier:

publish(video, operation_id="episode-184-youtube")

The server can recognize that the operation has already been processed and return the existing result instead of repeating it.

Without idempotency, retries can become a source of corruption.

Retries: try again, but with judgment

Automatic retries can greatly improve reliability, but badly designed retries can make an incident worse.

A typical pattern uses exponential backoff:

attempt 1
  ↓ fails
wait 1 s
attempt 2
  ↓ fails
wait 2 s
attempt 3
  ↓ fails
wait 4 s

A small random component —jitter— is usually added as well, to prevent thousands of clients from retrying at exactly the same time.

But not every error should be retried.

A temporary timeout may justify a retry.

An invalid credential probably does not.

Reliability is not about putting retry() around everything. It is about classifying failures and defining an explicit policy for each one.

Availability, reliability, and resilience are not the same thing

These terms are often used interchangeably, but they describe different properties.

Availability

Availability measures how much of the time a system can be used.

A service with 99.9% availability may be unavailable for tens of minutes in a typical month.

Availability mainly answers:

Is the service accessible when the user needs it?

Reliability

Reliability asks whether the system produces correct and consistent results over the expected period of time.

An endpoint can be available and return HTTP 200 while still returning incorrect results.

That would be availability without true reliability.

Resilience

Resilience describes the system’s ability to withstand problems and recover to a useful state.

A resilient architecture may degrade some features without losing the service entirely.

For example:

main service
      ↓
external API down
      ↓
use cache or fallback
      ↓
continue with reduced functionality

The goal is not always to preserve 100% of features. Sometimes it is to preserve the most important function.

Fault tolerance: keep working despite failure

Fault tolerance goes one step further.

Instead of merely recovering after an error, the system can continue operating while one of its components is broken.

Classic example:

Server A ─┐
Server B ─┼→ load balancer → users
Server C ─┘

If server B stops responding, A and C continue serving traffic.

But redundancy alone does not guarantee reliability.

If all three servers depend on the same non-redundant database, that database remains a single point of failure.

The useful question is always:

Which components can break the entire system by themselves?

Observability: know what is happening

You cannot quickly recover a system if you do not know what is failing.

That is why observability is one of the pillars of modern reliability.

It is usually built around three main sources:

logs
metrics
traces

Logs explain concrete events.

Metrics let us observe trends: latency, errors, CPU usage, number of failed jobs, queue depth.

Traces let us follow one operation across multiple services.

In a distributed system, this completely changes diagnosis.

Without observability:

"the pipeline failed"

With observability:

job: episode-184
step: image_generation
provider: image-api
attempt: 3
error: timeout
latency: 31.4 s
checkpoint: audio_complete

The second version is actionable.

SLI, SLO, and SLA

Reliability engineering needs metrics, not just feelings.

Three fundamental terms appear here.

SLI — Service Level Indicator

A real measurement of service behavior.

For example:

99.95% of requests completed successfully

SLO — Service Level Objective

The internal objective we want to reach.

Our objective is to keep at least 99.9% of requests successful.

SLA — Service Level Agreement

A formal commitment to the customer.

We contractually guarantee 99.9% monthly availability.

It can be summarized as:

SLI = what we are measuring
SLO = what we want to achieve
SLA = what we promise externally

That distinction matters because not every internal target should become a contractual promise.

The error budget

A useful consequence of defining an SLO is that we can also define how much failure we are willing to tolerate.

That is known as an error budget.

If our objective is not 100%, we are acknowledging an important reality: pursuing absolute availability can be extremely expensive.

The error budget helps balance two forces that constantly compete:

change velocity ←→ stability

If the system is operating far above the objective, the team can take more risk and deploy changes faster.

If it is quickly consuming the error budget, it makes sense to reduce changes and focus on stability.

Reliability stops being a subjective discussion and becomes an operational decision based on data.

So what is SRE?

Site Reliability Engineering, or SRE, applies these ideas to running software systems in production.

A useful way to think about it is:

Software engineering
        +
Operations
        +
Reliability engineering
        ↓
       SRE

SRE tries to solve operational problems through software and automation.

An SRE team may work with:

• health checks;
• alerts;
• retries;
• circuit breakers;
• rate limiting;
• autoscaling;
• canary deployments;
• automatic rollback;
• disaster recovery;
• capacity management;
• postmortems;
• SLO definition;
• elimination of repetitive manual work.

The goal is to avoid making system operations depend on a person watching constantly.

Circuit breakers: stop hammering a broken dependency

Suppose your application calls an external service.

If that service is down and every request keeps calling it, the problem can propagate.

A circuit breaker detects a high failure rate and temporarily stops sending traffic to that dependency.

requests
   ↓
external service
   ↓
many failures
   ↓
circuit breaker OPEN
   ↓
fail fast / use fallback

After an interval, the system tries again.

If the dependency has recovered, the circuit closes again.

This limits damage and avoids wasting resources waiting for responses that probably will not arrive.

Complete code: a resilient AI client in Python

To see these patterns working together, I prepared a single-file executable example. It implements timeout, retries with exponential backoff + jitter, a circuit breaker with CLOSED / OPEN / HALF_OPEN states, fallback, metrics, and logs. It also includes a simulated provider that runs without external dependencies and an optional OpenAI adapter.

If you want to understand why that client is structured that way, the natural follow-up is Design patterns behind a resilient AI client: it breaks down Strategy, Adapter, Dependency Injection, Facade, Simple Factory, and the circuit breaker’s state machine using this same example.

View resilient_ai_example.py in the public Gist →

It is also available as a direct file on the site:

Open resilient_ai_example.py

You can run it without installing anything else:

python resilient_ai_example.py

And if you want to test the OpenAI adapter:

pip install openai
export OPENAI_API_KEY="..."
python resilient_ai_example.py --openai "Explain circuit breaker in 3 sentences"

The example applies an important idea from this article: not every failure gets the same treatment. Temporary errors can be retried; permanent errors fail without blind retries; and the circuit breaker prevents continuing to hammer a dependency that has already demonstrated degradation.

Reliability in AI systems

Artificial intelligence systems add a new problem: the infrastructure can work perfectly and the output can still be wrong.

A model call can return HTTP 200 and produce defective content.

That is why AI-system reliability needs additional layers.

For example:

prompt
  ↓
model
  ↓
structural validation
  ↓
factual verification
  ↓
quality evaluation
  ↓
approval / fallback

In agentic applications, actions also need to be controlled.

An agent can correctly execute a tool and still have made the wrong decision.

That is why mechanisms appear such as:

• output validation;
• strict schemas;
• automated evaluations;
• autonomy limits;
• per-tool permissions;
• state checkpoints;
• decision traceability;
• conditional retries;
• human approval for high-impact actions;
• compensating actions to reverse effects.

Future reliability will not be only reliable infrastructure.

It will also be verifiable reasoning and controlled execution.

A good system does not try to hide every failure

There is a common temptation: catch every exception and continue.

That can produce systems that look stable while accumulating silent corruption.

A healthier principle is to distinguish between:

recoverable failure
degradable failure
critical failure

A recoverable failure can be retried.

A degradable failure allows the system to continue with limited functionality.

A critical failure should stop the process, preserve evidence, and alert.

Sometimes failing quickly and clearly is more reliable than continuing incorrectly.

Postmortems are part of engineering too

After an important incident, a mature team does not simply fix the bug.

It tries to understand why the system allowed that bug to become an incident.

A good postmortem asks:

• What happened?
• When did it start?
• How was it detected?
• What impact did it have?
• Which mechanisms worked?
• Which mechanisms failed?
• Why was the problem able to propagate?
• Which changes will reduce future probability or impact?

The goal is not to find someone to blame.

The goal is to improve the system.

The most important idea

Reliability engineering begins from an uncomfortable but powerful premise:

every component can fail.

A network can go down.

An API can become saturated.

A machine can restart.

A dependency can change.

A deployment can contain a bug.

An AI model can return an incorrect answer.

What differentiates a robust system is not the total absence of those events.

It is how the system responds when they happen.

failure
  ↓
detection
  ↓
containment
  ↓
recovery
  ↓
learning

That cycle is, in essence, reliability engineering.

And the more we depend on cloud, microservices, automation, pipelines, and AI agents, the more important it becomes.

Because in real systems, the question is never only:

Does it work?

The question that matters is:

What happens when it stops working?