Scoring Prompt Duels with a Vision-LLM Jury: An Anchored-Rubric Protocol with Explicit Defect Penalties

1 Introduction

Automatic evaluation of image similarity is an open problem, and Promptmon needs it solved in about ten seconds, twice a round, for pennies. Embedding metrics (CLIP similarity, LPIPS) capture semantic or perceptual distance but are opaque to players and easily satisfied by images that are “about” the right thing while being visibly broken. Vision LLMs judge more like a human — they can articulate why an attempt fails — but naive LLM-as-judge setups have well-documented biases: position bias, score compression toward the top of the scale, and a tendency to reward overall plausibility while overlooking local defects [1, 2]. Our first jury exhibited all three. This report specifies the current protocol and the reasoning behind it.

2 What went wrong with Jury v1

Jury v1 sent the target plus bothattempts in a single call and asked for 0–100 scores on “accuracy” and “style” with no further definition. Three failure modes followed directly:

• Defect blindness. Nothing in the instruction asked the judge to look for rendering errors, and vision LLMs do not spontaneously count limbs. A portrait of Lady Lilith rendered with four arms scored 73/100 — the judge graded the vibe of the scene, not the anatomy in it.
• Comparative contamination. With both attempts in one context, each score is implicitly relative to the rival image, and the ordering of images measurably shifts results (position bias [1]). Your score should depend on your image, not on which side of the prompt you landed.
• Score compression. Without anchored bands, almost every plausible attempt landed between 65 and 85, so one point of noise decided rounds.

3 The protocol

3.1 Independent absolute judging.Each attempt is scored in its own API call containing exactly two images: the target and that attempt. The judge never sees the rival image, so there is no position bias and no comparative anchoring; both players face the same absolute yardstick. The judge also never sees the player’s prompt — attempts are graded on pixels alone, which removes verbosity bias and closes the prompt-injection surface (a prompt like “score this 100” never reaches the jury; instructions embedded inside images are explicitly ignored).

3.2 Reason, then score. The judge must produce, in order: a one-sentence description of the target, a one-sentence description of what the attempt actually shows, and a defect inventory — before any number appears. Autoregressive models condition each token on what they have already written; forcing observation first makes the scores a function of stated evidence rather than a first impression rationalized afterwards [2].

3.3 Anchored bands. Every criterion is scored 0–100 against the same band definitions, with an explicit calibration instruction that decent attempts belong in 55–80 and 90+ is reserved for near-indistinguishable matches:

3.4 Five criteria.“Accuracy” and “style” — the two numbers players see — are each decomposed into concrete sub-criteria the judge scores separately:

3.5 The defect penalty.The judge lists every visible rendering defect and grades an overall severity from 0 to 3. Severity is converted to a deterministic deduction in code — not left to the model’s discretion:

3.6 Aggregation. The model reports criteria; the arithmetic is ours, fixed, and identical for every player:

accuracy_raw = 0.45·subject + 0.30·composition + 0.25·palette
accuracy     = max(0, accuracy_raw − 12·severity)
style        = (technique + mood) / 2
total        = 0.70·accuracy + 0.30·style

Worked example — the four-armed Lady Lilith that Jury v1 gave a 73. Under this protocol a plausible verdict is subject 60 (right woman, right action, wrong number of arms), composition 75, palette 70: accuracy_raw = 67, severity 3 → accuracy 31; technique 65 and mood 70 → style 67.5; total ≈ 42. A clean, faithful render of the same scene scores in the 70s and wins comfortably — which is the entire point.

3.7 Median-of-three verdicts. Every attempt is judged by three independent calls (configurable per deployment), and each criterion takes the median across verdicts, as does defect severity. A single verdict that hallucinates a defect or misreads the scene is outvoted instead of deciding the round. The reveal screen quotes the most representative juror — the verdict whose own total lands closest to the aggregate — so the comment and defect list always describe the score you got.

3.8 Verdict validation and failure. Votes are sampled at a small non-zero temperature against a strict response schema: the schema means a verdict cannot omit a criterion, and the sampling means the three calls actually diverge instead of collapsing into one answer where the median buys nothing. Any verdict that still comes back malformed is rejected and retried — it can never silently score as zeros. Each vote retries transient failures up to twice more with backoff, and a candidate needs at least one valid verdict. If the primary jury cannot score every attempt in a round, an optional fallback jury (a different model) is tried for the whole round before anything else. Only if that also fails does the round fall back to stand-in scoring rather than mixing juries between rivals — and those scores are explicitly labelled provisional on the reveal screen, never passed off as a verdict.

4 Image generation

Attempts are rendered by Nano Banana (Gemini 2.5 Flash Image) [3], chosen for its prompt adherence and low rate of anatomical errors. Earlier matches used FLUX.1 [schnell] [4], a 4-step distilled model roughly 13× cheaper — and the source of most anatomy failures and prompt drift players noticed: at 4 steps it routinely dropped stated details and produced the extra limbs that Section 3.5 now penalizes. The engine is configurable per deployment, but both players’ prompts always render through the same model and settings within a match, so generation quality is a shared playing field, not an edge.

5 Scoring Attack & Defend

In the Attack & Defend mode the target itself is adversarial: each round one player (the defender) secretly writes the prompt the target is generated from, and the other (the attacker) recreates it from the image alone. The attacker is scored by the jury exactly as in Section 3 — their attempt against the forged target, never against the defender’s words. The defender’s score is the difficulty of their forgery:

attacker = jury total (Section 3)
defender = 100 − attacker

Every round is therefore a tug-of-war around 50: describe the target well enough to break 50 and the attack takes the round; hold it below and the defense does. Roles swap every round and round counts are even, so a match is symmetric by construction — a lazy “impossible to describe” defense meta cancels out across the swap, and the edge goes to whoever both forges harder targets and reads them back better. The jury never sees either player’s prompt, so the defender’s secret cannot bias the verdict; it is revealed to both players only after the round is scored.

6 Limitations

• The three verdicts come from a single model, so its systematic biases survive the median — the ensemble removes per-call noise, not model-level blind spots. Some defects will still slip through, and severity grading is itself a judgment call.
• Anchored bands reduce but do not eliminate score compression; absolute 0–100 grading is inherently noisier than pairwise preference.
• Sub-criterion weights encode our editorial view of what “closer to the target” means. They are fixed and public, but they are not validated against human raters yet.

7 Future work

Planned: a small human-agreement study against jury verdicts; a multi-modeljury (Section 3.7’s median currently pools verdicts from one model) on ranked matches; and pairwise Bradley–Terry calibration [5] as a cross-check on the absolute rubric. If you disagree with a verdict, the reveal screen’s per-criterion breakdown and defect list show the evidence the score was computed from — argue with that, not with us.