Multimodality · September 2026 · Tidel team

Looks right is not a pass.

Agents now ship things people look at: reports, slide decks, 3D scenes, browser games. Grade them by reading the code and you miss what renders. Grade them by looking at a screenshot and you miss what is broken underneath. Multimodal data needs both.

A growing share of agent work ends in an artifact rather than an answer. GDPval's tasks hand the model reference files — spreadsheets, PDFs, images — and ask for the deliverable a professional would sign off on: a memo, a model, a deck. Code generation has moved the same way. The program is a Blender script, a Three.js scene or a playable game, and the only honest way to judge it is to run it and look.

Rendering is the first verifier

The cheapest improvement available to a multimodal agent is to look at its own output. In the GDPval paper, OpenAI prompted GPT-5 to render its files to images and inspect them before submitting. Self-inspection went from 15% of tasks to 97%. Black-square artifacts, which had affected more than half of its generated PDFs, disappeared. Egregious slide-formatting errors fell from 86% to 64%, and expert preference win rates rose five points. GameDevBench reports the same pattern for games: giving an agent screenshots and video lifted GPT-5.4 from 41.1% to 52.0%.

That loop — build, render, look, fix — is what we want models to learn. So it is also what the data has to contain: not just the final file, but the renders the agent checked and the edits it made because of them.

Pixels are not a verdict either

The obvious next step is to let a vision model judge the render. For ranking models on average, that works well: ArtifactsBench's multimodal judge agrees with WebDev Arena's human rankings 94.4% of the time. For deciding whether one particular output is correct — which is what a reward has to do — it breaks down.

WorldCoder-Bench checks Three.js scenes against their live scene state. Judging from the page's DOM correlated with that state at τ = −0.021, which is statistically random. An eight-step inspection agent, at roughly 400 times the cost, still let through 45.6% of badly broken outputs. GameGen-Verifier found the same gap in games: putting the game into set states and checking the result agreed with human judges 92.2% of the time, while an agent asked to play and judge managed 58.8%.

We see the same thing when we red-team graders before release. One Three.js environment we tested gave full marks to a 389-byte page holding an empty 16×16 canvas — no script, no WebGL — once we replaced the judge program it called with a 40-line stand-in. The grader never checked what it was calling. In another, a fifteen-line page that opened a WebGL2 context and cleared the screen once earned full credit on the check meant to prove a real 3D scene was running. Neither grader looked wrong on good submissions. They fail only on the submissions an optimizer will eventually find.

Two verdicts, not one

The way out is to stop asking one grader to do two jobs.

State decides correctness. Reopen the saved Blender project and measure its geometry, modifiers and materials. Read the scene graph of the running web app. Replay recorded input into the game and check where things end up. These checks are deterministic, and they don't care how good the render looks.
The render decides quality. Experts score what a person would see, criterion by criterion, each tied to evidence in a frame or on a page. Vision models can help as a second opinion on looks — not as the source of truth for behavior.
Hide the scoring views. If the agent is shown four reference renders, score it from cameras it has never seen, so it has to build the object rather than paint four angles.
Attack every grader before it ships. A blank canvas, an empty scene, a boilerplate document, a tampered judge: if any of them scores, the gate gets rebuilt. Deterministic checks are necessary conditions. They should never add up to a pass on their own.

What this means for the data

For a multimodal team, the useful data follows from those two verdicts. Trajectories that show the look-and-fix loop. Environments that ship with their own renderer and a verifier that has already been attacked. And critique data that includes the hard negatives — outputs that look right and are broken underneath, labeled with what is wrong. That last category is what a visual judge needs to be calibrated against, and it is the one almost nobody collects.

If a grader has never been shown a blank canvas, you don't know what it does with one.

We build multimodal tasks — long-document vision, Blender assets, interactive web 3D and browser games — with both verdicts built in. See the datasets or request a sample.