Pass rates belong on the datasheet.

Every coding dataset in the market claims to be "hard" and "high quality." Almost none of them tell you the one number that would let you check: what fraction of attempts by a current frontier model actually solve each task.

When a lab buys coding tasks, it is really buying gradient. A task a model already solves 100% of the time teaches it very little; a task it solves 0% of the time with no partial signal is nearly as useless. The value is concentrated in the band in between — and where that band sits depends entirely on which model you are training. "Hard" is not a property of a task. It is a relationship between a task and a model.

That is why we treat difficulty as something to be measured, not asserted. Before a batch leaves the building, we run frontier models against every task on our own rollout infrastructure — the same containerized environments, the same hidden test suites the customer receives. Each task's datasheet carries its empirical pass rate, so a customer can slice a delivery by where their model's frontier actually is.

What the rollout catches that review doesn't

Human review checks that a task is well-posed. Rollouts check things review structurally cannot:

• Broken environments. A dependency that resolves differently in a fresh container, a flaky test, a network assumption — these surface immediately when hundreds of agent attempts hit the environment.
• Dishonest gates. If the hidden tests can be passed without solving the stated problem — or the reference solution itself fails them — the task is quietly worthless. The rollout finds both directions.
• Saturation. Tasks that looked hard when authored but that current models now clear at 95%+. These still have uses, but selling them as frontier training data would be dishonest.
• Contamination signals. When a model solves a supposedly novel task suspiciously fast, along a suspiciously direct path, that is a flag worth investigating before the customer finds it.

The number changes, and that's the point

A pass rate measured in August is stale by December — models improve, and the band of useful difficulty moves. We think that is a feature of the methodology, not a bug: it forces the catalog to be re-measured against current frontiers, and it gives customers an honest basis for refresh cycles instead of buying the same "hard" set twice.

If a data vendor can't tell you the pass rate of the tasks they're selling you, they haven't run them. And if they haven't run them, neither the difficulty claim nor the environment has been tested end to end.

This is the standard we hold every coding task to — from repository-scale software engineering through terminal and tool-use work to multi-turn sessions. It is more expensive than shipping untested tasks. It is also the difference between selling data and selling claims.

Want a sample datasheet with measured pass rates? Start an inquiry or write to hello@tidel.ai.