Dream-RSI: when an agent's history becomes a world for learning how to explore

The phrase recursive self-improvement often evokes an extreme picture: a model rewrites its own weights, becomes more capable, and repeats the process.

Dream-RSI proposes something much more concrete—and arguably more useful for agent systems we can build today: keep the coding agent fixed and recursively improve the policy that decides how to explore.

The paper, Dream-RSI: Recursive Self-Improvement through Evolving Worlds, starts from a simple observation. After hundreds or thousands of attempts on a hard problem, an agent leaves behind a rich history: explored branches, failed implementations, promising candidates, evaluator feedback, costs, and decisions about when to continue or abandon a direction.

That history is usually treated as memory or prompt context.

Dream-RSI does something different:

it turns discovery history into a replay simulator where new exploration strategies can be tested without rerunning the expensive underlying agent.

That changes the economics of search. The system can evaluate many exploration policies cheaply and reserve expensive online executions for strategies that look more promising.

The problem is not only generating good solutions

Many LLM-driven discovery systems follow a loop like this:

propose
   ↓
execute
   ↓
evaluate
   ↓
refine
   ↓
repeat

As the horizon grows, another question becomes critical: where should the next attempt be spent?

A system may have several branches open at once:

root
├── approach A
│   ├── A1
│   └── A2
├── approach B
│   ├── B1
│   └── B2
└── approach C
    └── C1

The next agent call could refine A2, repair B1, start a new branch, continue C1, launch several choices in parallel, or stop.

That is not the same problem as generating code. It is a search orchestration problem.

Dream-RSI makes that layer explicit and programmable.

What actually self-improves

One of the most important parts of the paper is what does not change.

During the recursive loop, the following remain fixed:

the coding agent;
the evaluator;
the execution interfaces;
the mechanism that runs and scores attempts.

What changes is the executable exploration policy.

Conceptually:

def exploration_policy(tree):
    return nodes_to_continue

The policy observes the currently revealed discovery tree and decides which branches to continue, whether to open new roots, how many attempts to batch in parallel, and when to stop.

In other words, recursive improvement happens in the harness, not in the model weights.

From history to a “world”

Suppose an online run produced this tree:

root
├── A → score 0.61
│   ├── A1 → 0.74
│   │   └── A2 → 0.78
│   └── A3 → compile error
├── B → 0.55
│   └── B1 → 0.49
└── C → 0.69
    └── C1 → 0.82

Each node can preserve more than a score: workspace state, generated artifacts, execution errors, diagnostics, and evaluator observations.

After the run finishes, Dream-RSI freezes the tree and treats it as a reusable world.

A different exploration policy can then “play” over that recorded tree:

Policy 1:
root → A → A1 → A2

Policy 2:
root → C → C1

Policy 3:
root → A + C in parallel

Because the outcomes are already stored, replay does not need to call the coding agent or evaluator again. It only reveals historical outcomes corresponding to the selected trajectory.

The paper calls this process dreaming.

The analogy to world models is straightforward: instead of always interacting with the expensive real environment, an agent improves decisions inside a reusable model of previous experience.

Dream-RSI’s replay world is conservative, though. It does not predict arbitrary unseen outcomes; it only replays regions of the search space that were actually observed.

The full loop

Dream-RSI organizes the process into three stages:

1. ONLINE EXPLORATION
   current policy
        ↓
   coding agent + evaluator
        ↓
   discovery tree

2. REPLAY WORLD
   discovery tree
        ↓
   historical simulator

3. DREAMING
   many candidate policies
        ↓
   cheap replay
        ↓
   select a better policy
        ↓
   redeploy online

The improved policy drives the next real run. That run creates another discovery tree, which expands the replay-simulator pool.

The loop is recursive because each online round generates experience that can improve the strategy governing the next round.

What the policy optimizes

The objective is not simply “find the highest score.”

The replay value balances three factors:

value =
    best quality discovered
    - exploration cost
    + useful parallelism bonus

The method penalizes the number of represented generation-evaluation attempts and rewards policies that batch useful work efficiently across available workers.

That prevents the trivial strategy:

“Explore everything.”

A strong policy must find high-quality solutions with fewer calls and make good use of parallel execution.

The clearest result: Lasso

In an algorithm-engineering task, the authors use agents to discover efficient implementations of the Lasso regularization path.

With Gemini 3.1 Pro:

Method	Agent calls	Average runtime
Recursive Fixed Exploration	550	3587.1 ms
Dream-RSI	317	2931.0 ms

Dream-RSI uses roughly 1.7× fewer calls than the fixed policy while producing a faster solver.

With Gemini 3.7 Flash:

Method	Agent calls	Average runtime
Recursive Fixed Exploration	3200	2516.7 ms
Dream-RSI	1879	2350.6 ms

The contrast with SimpleTES is even larger: SimpleTES reports 51,200 generations, while Dream-RSI reaches competitive results with orders of magnitude fewer agent calls.

It also works outside conventional coding tasks

The authors evaluate the same mechanism on eight tasks across three domains.

Mathematical optimization

On Sum–Difference, Dream-RSI reaches 1.145427, above the other methods in the reported comparison table.

On Circle Packing, it reaches 2.635983, matching the strongest reported result among the compared systems.

On Autocorrelation, it remains competitive but does not achieve the best score. That distinction matters: the paper does not establish universal superiority.

GPU kernel engineering

On KernelBench tasks:

VGG16 reaches comparable performance with 2.43× fewer generations;
LayerNorm uses 1.79× fewer generations;
ConvDiv reaches up to 2.09× higher performance at a comparable budget;
ConvMax improves performance by 1.44× at a similar budget.

Again, the model itself did not become smarter. The system simply learned to spend the exploration budget more effectively.

A counterintuitive result: putting memory in the prompt can hurt search

A natural alternative is to summarize previous experience and tell the next agent:

“These directions worked before. Focus here.”

The authors test a version of that idea by injecting historical directional guidance into the prompt.

Under equivalent discovery budgets, the explicit guidance variant performs worse than using history as an interactive replay simulator.

Their interpretation is important: in long-horizon search with many parallel threads, strong semantic guidance can create premature inductive bias and reduce exploratory diversity.

The difference can be summarized like this:

MEMORY AS PROMPT

history
   ↓
"this worked before"
   ↓
agent
   ↓
new generation


HISTORY AS SIMULATOR

history
   ↓
replay world
   ↓
many hypothetical policies
   ↓
select strategy
   ↓
agent

This is not just better remembering. It is learning how to allocate search budget.

An implementation failure does not mean the idea is bad

The policy-development prompt included in the paper contains a rule that is especially useful for coding agents: a local implementation failure does not, by itself, prove that the parent direction is poor.

The policy must reconstruct the branch trajectory before closing it.

For example:

promising idea
   ↓
implementation 1
   ↓
shape mismatch
   ↓
abandon?

Dream-RSI tries to distinguish:

bad algorithmic direction
        vs.
repairable implementation

The paper treats output/correctness mismatches, shared-memory or resource limits, variable/code problems, mask errors, layout errors, and shape errors as normally repairable categories rather than immediate proof that the branch is conceptually wrong.

A naive orchestrator can easily kill a good direction simply because its first implementation failed to compile.

What this could look like in a coding-agent system

Imagine an autonomous agent working on a real issue.

A simple harness might use static rules:

if tests_failed:
    retry()

if ci_green:
    open_pr()

A Dream-RSI-inspired layer would preserve the full trajectory:

Issue #431
├── branch A
│   ├── tests fail: import
│   ├── fix import
│   └── 98% tests pass
├── branch B
│   ├── architectural rewrite
│   └── performance regression
└── branch C
    ├── minimal patch
    └── all tests pass

The system could then evaluate offline policies for questions such as:

when should a branch be repaired?
when should a new branch be opened?
how many workers should be allocated?
how long should a direction be refined before abandoning it?
when should the system exploit a strong candidate?
when should it return to broad exploration?

It would not need to rerun every historical pull request to test those decision rules. Much of the observed consequence structure is already present in the history.

Logs, tests, and execution trees therefore become more than memory: they become training data for the orchestrator.

Important limitations

Dream-RSI is promising, but its boundaries matter.

1. Replay only knows the observed world

The simulator can reveal outcomes already present in a discovery tree. It cannot faithfully answer what would have happened for a completely new idea that was never explored.

It is not a fully generative simulator of the search space.

2. Better replay score does not guarantee better future behavior

The paper selects policies according to average replay performance over historical worlds.

That means the chosen policy is no worse on that replay set, under the defined objective. It does not guarantee perfect generalization to a new task or an unseen region of the search space.

3. Quality depends on historical coverage

If early rounds explore poorly and create weak trees, the replay simulator will also have weak coverage.

The loop must keep returning online to expand its experience.

4. The evaluation domains are structured

The experiments cover algorithm engineering, mathematical optimization, and GPU kernels, all of which provide relatively clear automated evaluators.

Applying the same approach to open-ended architecture work, product decisions, ambiguous debugging, or human-centered tasks requires much more careful evaluation design.

Why this matters for agent engineering

Many current agent systems focus almost entirely on the LLM and treat the harness as secondary infrastructure.

Dream-RSI flips that intuition.

A system can improve substantially without changing the base model if it gets better at deciding:

where to explore;
when to persist;
when to repair;
when to abandon;
when to parallelize;
when to stop.

That suggests a practical architecture:

fixed model
   +
observable harness
   +
structured history
   +
reproducible evaluators
   +
evolving search policy
   =
an agent that learns to spend compute better

The most useful idea in Dream-RSI may not be the phrase recursive self-improvement itself.

It is this:

your agents’ history does not have to be only memory; it can become the environment in which they learn to orchestrate themselves better.

For teams already running coding agents, CI, benchmarks, and multiple workers, that idea is immediately actionable.