Fuzzing against AI slop: how to test agent-generated code without reading every line

One of the most useful responses to the AI slop debate is not to ask the model to “write better code.” It is to change the way we decide whether that code deserves to reach production.

In September 2026, Business Insider published an exchange between a developer worried about declining code quality and Boris Cherny, the creator of Claude Code. Cherny drew a reasonable distinction: throw-away, low-impact code can sometimes be treated almost as a black box; production code should face an even higher bar than manually written code.

The interesting part was the set of guardrails he described around production code at Anthropic: extensive linting, tests, Claude-driven end-to-end tests, automated code and security reviews, automated refactoring, and Claude-powered fuzzers running daily.

That last item matters because fuzzing fits agentic programming unusually well. An agent can generate code faster than a human can manually enumerate every combination of state, input, and execution path. The answer cannot simply be “review faster.” We need systems that actively try to break the software for us.

This article explains what fuzzing actually is, how modern coverage-guided fuzzers work, how fuzzing differs from property-based testing, and how to make it part of the harness around a coding agent.

The problem is not that AI writes many lines

The problem begins when we confuse generated volume with evidence of correctness.

An agent may deliver an implementation that compiles, passes the examples we gave it, looks reasonable in review, matches repository style, and still fails on an input combination nobody imagined.

Humans have the same problem. The difference is scale: an agent can modify several modules, introduce new parsers or states, and touch multiple contracts in minutes.

As generation speed increases, one question becomes more important:

What independent mechanism is trying to falsify the behavior the agent just implemented?

That is where fuzzing enters.

Fuzzing is not just “random input”

The simplest model looks like this:

program
  +
weird inputs
  ↓
crash?
hang?
overflow?
unexpected exception?
absurd memory usage?

Modern fuzzers are more systematic.

A coverage-guided fuzzer such as libFuzzer keeps a corpus of interesting inputs. It selects an input, mutates it, executes the target, and observes which parts of the program were reached. If a mutation reaches previously unseen code, the fuzzer keeps that input and uses it as material for later mutations.

The conceptual loop is:

seed corpus
    ↓
select input
    ↓
mutate it
    ↓
execute target
    ↓
measure coverage / signals
    ↓
new path?
 yes → keep input
 no  → discard it
    ↓
repeat thousands or millions of times

This turns fuzzing from a random lottery into a feedback-guided search.

That distinction matters when a parser rejects almost every arbitrary byte string before reaching interesting logic. A coverage-guided fuzzer learns which mutations make progress toward deeper branches.

What is the fuzzer looking for?

The simplest oracle is: the process should not break.

A fuzzer can treat the following as failures:

an unexpected exception;
a segmentation fault;
a failed assertion;
a timeout;
excessive memory consumption;
undefined behavior detected by a sanitizer;
an invalid memory read or write.

For C and C++, libFuzzer is commonly combined with AddressSanitizer or UndefinedBehaviorSanitizer so that silent memory bugs become detectable failures.

But fuzzers can also check semantic invariants:

parse(serialize(x)) == x

decode(encode(x)) == x

final_balance >= 0

a terminal job never returns to pending

At that point, fuzzing and property-based testing begin to overlap.

Unit tests, property tests, fuzzing, and mutation testing

These techniques solve different problems.

Technique	What changes	What it checks
Unit test	hand-written examples	known concrete behavior
Property-based testing	generated inputs from a modeled domain	invariants that should hold across many cases
Fuzzing	generated or mutated inputs, often coverage-guided	crashes, hangs, memory bugs, unexpected paths, broken properties
Mutation testing	production code itself	whether the test suite notices deliberately introduced defects
Lint / static analysis	no runtime input	unsafe patterns, static contracts, style and classes of defects

The line between property testing and fuzzing is deliberately blurry.

Hypothesis generates cases from strategies, remembers previous failures, searches for edge cases, and shrinks a large failure into a minimal reproducer. Its documentation also includes targeted mutation phases and explicitly points heavy users toward coverage-guided fuzzing for very large search budgets.

A classic engine such as libFuzzer starts more directly from a corpus, mutations, and coverage. Both approaches are trying to explore a state space too large to enumerate manually.

A property-based example

Imagine a service that receives URLs and normalizes an identifier.

A normal unit test might be:

def test_extract_id():
    assert extract_id("https://example.com/watch?v=abc123") == "abc123"

That proves one example works.

With Hypothesis we can express a property:

from hypothesis import given, strategies as st

@given(st.text(min_size=1, max_size=64))
def test_roundtrip_video_id(video_id):
    url = f"https://example.com/watch?v={video_id}"
    assert extract_id(url) == video_id

Now the tool can explore Unicode, reserved characters, strange lengths, and combinations we would probably never write by hand.

We can also establish a robustness contract:

@given(st.text())
def test_parser_never_crashes(raw):
    try:
        extract_id(raw)
    except InvalidUrl:
        pass

This does not prove that the parser is correct. It does prove something useful: arbitrary input must either produce a valid result or a controlled error, not an unexpected explosion.

Hypothesis can replay previous failures and shrink generated failures into small, understandable examples.

Coverage-guided fuzzing in Python

Python developers can use Atheris, Google’s coverage-guided Python fuzzing engine built on libFuzzer.

A minimal target looks like this:

import atheris
import sys

with atheris.instrument_imports():
    from app.parser import parse_payload


def TestOneInput(data: bytes):
    parse_payload(data)


atheris.Setup(sys.argv, TestOneInput)
atheris.Fuzz()

Atheris instruments Python bytecode to collect coverage. When a mutated input reaches new code, it can be retained in the corpus.

For structured input, the target can also encode domain invariants:

import json


def TestOneInput(data: bytes):
    try:
        obj = json.loads(data)
    except (UnicodeDecodeError, json.JSONDecodeError):
        return

    result = normalize_request(obj)
    assert result.status in {"accepted", "rejected"}

Now the fuzzer is not only hunting crashes. It is checking a contract.

Fuzzing tends to work especially well on parsers, binary formats, protocols, serializers, validators, compiler front ends, regular-expression engines, API boundaries, and transformation libraries: small input surface, large internal branching structure.

The corpus is operational memory

A useful fuzzer does not necessarily start from nothing. It can receive representative valid and invalid seeds:

corpus/
  valid-small.json
  empty.json
  malformed-header.bin
  unicode-edge-case.txt
  previous-crash-001

When fuzzing discovers an input that expands coverage, that input can be retained. When it finds a crash, the reproducer is saved.

That creates a valuable lifecycle:

bug discovered
    ↓
reproducing input
    ↓
corpus / regression test
    ↓
never silently reintroduce the same bug

For autonomous agents, this matters enormously. The agent that fixes the bug may be temporary. The reproducer should outlive the agent.

“The agent says it is fixed” is not a result

Suppose a nightly fuzzer finds an input that causes an unexpected exception.

A weak workflow is:

fuzzer finds crash
→ agent changes code
→ agent says "fixed"
→ merge

A stronger workflow is:

fuzzer finds crash
→ save reproducer
→ add regression test
→ agent proposes fix
→ run reproducer
→ run unit/property tests
→ short confirmation fuzz run
→ review / CI
→ merge

The agent can participate in almost every step. It should not be the only oracle judging its own work.

Why fuzzing fits AI slop so well

AI slop often has one dangerous property: local plausibility.

A function can look clean. Naming is sensible. The happy path passes. The diff reads professionally. What is missing are the interactions nobody explicitly asked for:

empty input
+ unusual Unicode
+ retry
+ partially persisted state
+ timeout
+ duplicate request
+ race
+ huge payload
+ permissive parser
+ dependency exception

LLMs are very good at producing a coherent path from a specification. Fuzzers are good at attacking the space where the specification was incomplete.

They are complementary tools.

Coverage-guided does not mean correctness-guided

Coverage can also be misleading.

A fuzzer might execute 95% of a module and still fail to notice a wrong commission, a bad timestamp rounding rule, or two swapped valid states.

Coverage answers:

What code did I execute?

It does not answer:

Was the result correct?

The strongest fuzz targets combine exploration with semantic oracles, reference implementations, differential checks, or metamorphic properties.

Examples:

new_implementation(x) == reference_implementation(x)

sort(sort(x)) == sort(x)

decode(encode(x)) == x

result(x) == result(equivalent_transform(x))

The better the oracle, the more classes of defects the fuzzer can expose.

Differential fuzzing for AI-generated refactors

A particularly useful pattern during AI-generated refactors is to keep the old implementation temporarily as an oracle.

def TestOneInput(data):
    old = old_parser(data)
    new = new_parser(data)
    assert normalize(old) == normalize(new)

The agent may have written a much cleaner parser. Differential fuzzing tries to find inputs where old and new disagree.

That does not prove the old version is perfect, but it dramatically reduces the risk that a “clean” refactor accidentally deletes historical behavior that was never fully documented.

It mechanically asks:

Did the agent change the implementation, or did it accidentally change the contract too?

Structure-aware fuzzing

Raw byte mutation works well for simple inputs. For complex JSON, SQL, ASTs, protobufs, or stateful protocols, most arbitrary mutations may die before reaching meaningful business logic.

Atheris and libFuzzer support custom mutators; other systems support grammars and structured generators.

Instead of only producing arbitrary bytes, a structure-aware generator can create valid objects with pathological internal values:

{
  "items": [],
  "retry": -1,
  "timeout_ms": 2147483648,
  "metadata": {"x": "\u0000"}
}

That shifts exploration from the parser shell toward deeper domain behavior.

Sanitizers expand the definition of failure

In unmanaged languages, a visible crash is only one subset of interesting failures.

Fuzzers are often paired with instrumentation such as:

ASan  → invalid memory access
UBSan → undefined behavior
MSan  → use of uninitialized memory

For C or C++ generated or modified by coding agents, fuzzing critical targets with sanitizers should be close to a default practice.

A practical CI design for agent-generated code

You do not need hours of fuzzing on every commit.

A good pattern is to split the budget by layer.

On each pull request

Run fast feedback:

lint
static analysis
unit tests
property tests
30–120 seconds of focused fuzzing on affected targets

The purpose is to catch obvious regressions without destroying iteration speed.

After merge or nightly

Run deeper jobs:

10–60 minutes per critical target
larger historical corpus
sanitizers
more workers
larger inputs
differential fuzzing

Continuously

For suitable projects, OSS-Fuzz demonstrates the model at scale: distributed continuous fuzzing with engines including libFuzzer, AFL++, Honggfuzz, and Centipede, plus sanitizers.

Its CI tooling can also fuzz a pull request for a short time budget; if a reproducible crash is found, the CI job fails and the crashing input is preserved.

The architecture becomes:

PR
 ↓
fast checks
 ↓
merge
 ↓
nightly fuzzing
 ↓
crash artifact
 ↓
issue / regression test
 ↓
agent proposes fix
 ↓
CI tries to falsify it again

Let the agent write fuzzers, but not the truth

Cherny referred to “Claude-powered fuzzers.” That suggests a useful pattern: let the coding agent help generate fuzz targets, strategies, seed corpora, and properties.

An agent can inspect an API and propose:

which inputs to mutate;
which invariants should hold;
which parsers are good fuzz targets;
which numeric boundaries deserve attention;
which exceptions are acceptable;
which seeds should enter the corpus.

But one rule is essential:

The agent should not unilaterally define both the implementation and the criterion that declares that implementation correct.

If the same model writes the feature and a test that merely restates its assumptions, independence has been lost.

The strongest properties come from external contracts:

specifications
protocols
domain invariants
business rules
reference implementations
historical compatibility
database constraints

The agent can encode those rules. Their meaning should come from something more stable than the diff being validated.

Fuzzing + property tests + mutation testing

These techniques reinforce each other.

A useful loop is:

1. property tests encode invariants
2. fuzzing searches for inputs that break them
3. shrinking/minimization produces a small failure
4. that input becomes a regression test
5. mutation testing deliberately changes production code
6. we verify that the suite detects those changes

Fuzzing asks:

Can I find an input that breaks the program?

Mutation testing asks:

If the program were broken, would my tests notice?

Together they attack two sides of the same problem.

The harness matters more as the model improves

It is tempting to assume that a better model reduces the need for aggressive verification.

The opposite may happen.

A more capable model can touch more files, perform larger refactors, work longer without supervision, complete entire issues, and generate code that looks increasingly convincing. Manual review becomes more expensive precisely because the agent is more productive.

The harness becomes the trust boundary:

coding agent
    ↓
implementation
    ↓
formatter / lint
    ↓
static analysis
    ↓
unit + integration tests
    ↓
property tests
    ↓
fuzzing
    ↓
security checks
    ↓
review
    ↓
production + observability

Not every layer applies to every change. The key is that confidence comes from accumulated evidence, not from the model sounding confident.

The useful lesson from the AI slop debate

The choice is not “read every line” versus “vibe code with no controls.”

There is a third and more interesting option:

delegate more generation while raising the quality of the verifiers.

That changes the engineer’s job. Some of the work moves from manually producing every implementation to designing the system that decides when an implementation has earned trust.

Fuzzing fits that transition because it does not try to judge whether code looks good. It repeatedly executes the code under conditions the author — human or agent — probably did not imagine.

The goal is not to prove that AI-generated code is correct.

The goal is to give it thousands or millions of chances to prove itself wrong before production does it for us.