Build your own LLM benchmark: why I would start with competitive programming

Every new model arrives with a collection of numbers: reasoning, math, coding, multimodality, tool use, long context, and agents.

The problem is that those numbers mostly answer the questions chosen by the people who designed the benchmark.

If you need to decide which model works best for your own systems, it can be more useful to build an evaluation layer that you can control, version, and repeat.

The goal is not to invent a single “super benchmark.” It is to build a benchmark harness: infrastructure where different task suites can run under the same rules, against different models, with auditable results.

And there is one domain that is especially attractive as a starting point:

competitive programming.

Not because it represents all of software engineering, but because it has three excellent properties for model evaluation: clearly specified problems, executable answers, and an objective judge.

A public benchmark may not measure what you need

A leaderboard can be useful for discovering trends, but it has important limitations.

The first is contamination. If a problem, its solution, or a close variant appeared in training data, a high score can reflect prior exposure in addition to genuine generalization.

The second is benchmark quality itself. In July 2026, OpenAI published an audit of SWE-Bench Pro in which its automated pipeline flagged 27.4% of tasks as problematic and a human annotation campaign identified 34.1%. The failures included overly strict tests, incomplete prompts, low-coverage tests, and misleading task descriptions.

That matters because a broken benchmark can end up measuring a model’s ability to guess quirks in the test suite instead of its ability to solve the intended problem correctly.

The third limitation is more practical: the benchmark may simply not resemble your workload.

A team building coding agents may care about very different things from a lab evaluating mathematical reasoning. It may want to know, for example:

how much it costs to complete a task;
how many attempts the agent needs;
whether it can repair a program after an error;
how long the task takes;
whether it uses the shell correctly;
whether it introduces regressions;
or whether a new model version becomes worse at a task the previous version solved.

That is where a private evaluation system starts to make sense.

Benchmark #1: competitive-programming problems

Competitive programming gives us a remarkably clean evaluation unit.

The model receives a problem statement and must produce a program. The system then compiles or executes that program against tests and produces one of the classic online-judge outcomes:

AC  Accepted
WA  Wrong Answer
TLE Time Limit Exceeded
MLE Memory Limit Exceeded
RE  Runtime Error
CE  Compilation Error

You do not need to ask another LLM whether the answer “looks good.”

The code either works or does not work under a specific test suite.

A problem could be represented like this:

id: cp-000314
source: codeforces
source_id: 2045-C
published_at: 2026-08-17
difficulty: hard
topics:
  - dynamic-programming
  - strings

languages:
  - python
  - cpp

time_limit_ms: 2000
memory_limit_mb: 256

prompt: problem.md
tests: private

The runner then performs a deterministic sequence:

model
  ↓
generated code
  ↓
compile / execute
  ↓
private tests
  ↓
AC / WA / TLE / MLE / RE / CE

The design is simple, but the signal is extremely useful.

LiveCodeBench already showed why fresh problems matter

There is an important precedent for this idea: LiveCodeBench.

The project continuously collects contest problems from LeetCode, AtCoder, and Codeforces and retains their publication dates. That makes it possible to evaluate a model over specific time windows.

If you approximately know a model’s training cutoff, you can select only problems published after that date.

Conceptually:

model training cutoff
          │
          ▼
──────────┼──────────────────────── time
          │   fresh problems
          │   used for evaluation

LiveCodeBench also demonstrates why “coding” should not be reduced to a single task. Beyond code generation, it evaluates capabilities such as self-repair, code execution, and test-output prediction.

The key lesson is not to copy LiveCodeBench exactly.

It is to adopt its principle of freshness.

Your benchmark could maintain views such as:

ALL
POST-CUTOFF
FRESH-365D
FRESH-90D
FRESH-30D
PRIVATE

A score then gains much more context:

Model X

All problems             81%
Post-cutoff              68%
Last 90 days             64%
Last 30 days             61%
Private problems         59%

That 61% or 59% may tell you more than an impressive score on a dataset that has circulated on the public web for years.

The online judge should be a source, not your evaluation infrastructure

There is an obvious temptation:

LLM
 ↓
submit to Codeforces
 ↓
wait for verdict

But depending on an external judge for every evaluation introduces too many variables:

latency;
availability;
accounts and authentication;
usage limits;
platform changes;
difficulty reproducing an identical run;
and terms that may limit automation or redistribution of content.

A more robust design separates two roles.

Online judges can be a source of problems and historical reference. Your own platform should try to execute evaluations in its own sandbox whenever the license, available tests, and source terms allow it.

Online judges / private tasks
             ↓
       problem ingestion
             ↓
      benchmark registry
             ↓
        local sandbox
             ↓
    deterministic evaluator

Now the infrastructure is under your control.

That means you can freeze a benchmark version and repeat it six months later without an external service changing underneath you.

Do not mix “model” and “agent” into one ranking

This distinction is critical.

There are two different evaluations that are often blended together.

Model-only

The model receives the problem and produces one response.

problem → model → solution

No shell. No test execution. No browsing. No feedback.

Here you are trying to measure primarily the model.

Agent

The system may write code, execute it, observe failures, modify the code, and try again.

problem
  ↓
agent writes solution
  ↓
runs tests
  ↓
observes failure
  ↓
repairs
  ↓
runs again

Now you are measuring something else:

model + harness + tools + iteration strategy.

Both rankings are useful, but they do not mean the same thing.

A model that loses in model-only mode could become the better agent if it uses feedback more effectively, manages context better, or repairs its mistakes more efficiently.

pass@1 is only the beginning

The most intuitive metric is pass@1: what percentage of problems are solved correctly on the first attempt.

But your own benchmark can record much more.

Metric	What it reveals
`pass@1`	ability to solve on the first attempt
`pass@3`	ability with multiple attempts
compile rate	how often generated code is executable
test pass ratio	percentage of tests passed
repair success	ability to recover after feedback
attempts-to-AC	attempts required to reach Accepted
latency	response speed
tokens	context and generation efficiency
cost	monetary cost per run
runtime	efficiency of generated programs
memory	memory consumption of solutions

A comparison stops looking like this:

Model A: 74
Model B: 71

And becomes something much more useful:

                   Model A   Model B
pass@1                74%        71%
repair success        42%        68%
avg attempts          1.8        1.4
median latency        9.2s       15.7s
avg cost/problem     $0.06      $0.03

Now you can make an engineering decision.

The right economic metric may be “dollars per solved task”

Model providers usually publish pricing per million tokens.

For an agentic system, that unit does not always tell the whole story.

Imagine two models:

Model A
$0.03 per attempt
40% success

Model B
$0.07 per attempt
85% success

Model A looks cheaper per call.

But the more important number may be:

cost per correctly solved problem

Once retries, tool executions, and compute time are included, the economic ranking can change completely.

A private benchmark lets you measure that unit directly.

Benchmark #2: real software engineering

Competitive programming is an excellent first laboratory, but it does not represent maintaining a real repository.

The natural second benchmark would look more like SWE-bench:

repository snapshot
+
issue
+
hidden tests
        ↓
agent
        ↓
git diff
        ↓
tests
        ↓
PASS / FAIL

SWE-bench runs repositories in Docker and evaluates whether the generated patch resolves a real issue. That reproducibility is an idea worth keeping.

But the recent SWE-bench experience also leaves another lesson: the benchmark’s prompts and tests need their own QA process.

A task that appears valid can be underspecified, enforce behavior through hidden tests that the prompt never requested, or accidentally expose shortcuts.

For a private benchmark, it can therefore be especially valuable to author new tasks deliberately for evaluation.

That reduces contamination and lets you design tests around the intended behavior instead of reconstructing the intent of an old issue after the fact.

OpenAI’s recent guidance on trustworthy third-party evaluations makes a similar recommendation: prefer private or newly constructed tasks when possible, and treat broken tasks as a standard validity risk.

Store the complete trace for every run

A score without context ages badly.

Every evaluation should produce a reproducible artifact with information such as:

{
  "benchmark": "competitive-programming",
  "benchmark_version": "2026.09.1",
  "problem": "cp-314",
  "model": "provider/model-x",
  "mode": "agent",
  "temperature": 0,
  "prompt_hash": "...",
  "tokens_in": 1820,
  "tokens_out": 2921,
  "latency_ms": 12421,
  "cost_usd": 0.031,
  "attempts": 2,
  "result": "AC",
  "tests_passed": 47,
  "tests_total": 47
}

In agent mode, it is also useful to retain the operational trace:

what it saw
→ what it decided
→ which tool it called
→ what the tool returned
→ which file it changed
→ which test failed
→ how it reacted

That turns the benchmark into a diagnostic tool rather than just a leaderboard.

If a new model falls from 72% to 68%, you can investigate why.

The architecture: build a harness, not a pile of scripts

The system can be separated into five pieces.

Benchmark Registry
       │
       ▼
Benchmark Runner
       │
       ├──────── Model adapters
       │          ├ OpenAI
       │          ├ Anthropic
       │          ├ Google
       │          └ local models
       │
       ▼
Sandbox / executor
       │
       ▼
Evaluator
       │
       ▼
Results DB + traces + dashboard

Each benchmark defines its contract.

Each provider implements an adapter.

The runner executes tasks under consistent rules.

And the evaluator decides what success means.

That makes a simple CLI conceivable:

llmbench run cp \
  --model provider/model-x \
  --suite fresh-30d

llmbench run swe \
  --model provider/model-y \
  --suite private-bugs-v1

llmbench compare \
  provider/model-x \
  provider/model-y

The command itself is not the valuable part.

The valuable part is being able to repeat the exact same evaluation when a new model appears.

Your benchmark does not need to be universal

This may be the most important idea.

A private benchmark does not need to convince the entire industry.

It needs to help you answer concrete questions:

which model should I choose for this agent?
did the new version actually improve?
how much does it cost to solve a real task?
which model repairs its own errors best?
which harness extracts more value from the same model?
what regressions did an update introduce?
how far does performance fall on problems released after the training cutoff?

Competitive programming is an excellent starting point because it provides an objective signal that is relatively cheap to automate.

From there you can add debugging, repository engineering, terminal use, tool use, long context, web research, or even private benchmarks derived from your own workflows.

The end result would not simply be another leaderboard.

It would be something more useful:

your own laboratory for deciding when an LLM is actually better for the work you want it to do.