Wizhut Research · Experiment
Borrowing a judge's opinion once, then never calling it again
Everyone evaluating prompt quality reaches for a model to do the judging, which makes every score slow, billable, non-repeatable, and a copy of your prompt on someone else's server. Promptivo tests the alternative: consult the judge once, learn what it rewards, freeze that into coefficients, and never call it again.
The judge was consulted once, during calibration, and then frozen into coefficients.
Up from ρ ≈ 0.25 for the heuristic engine alone. MePO hold-out the training never saw.
Gemini vs Sonnet, same prompts, same rubric. That is the ceiling any judge-aligned scorer is measured against.
The optimized prompt scores above its raw twin. The one thing the engine must not get wrong.
The model
Four merits come from the MePO framework (Zhu et al., arXiv:2505.09930); three were added for the properties IFEval tests and MePO does not. Each runs as its own scorer over deterministic linguistic signals, starts from a baseline, applies signed adjustments and clamps to 1–5. The overall score is their mean. Nothing here consults a model, a network, or the clock — the same prompt yields the same seven numbers every time.
Worked examples
These are real outputs of the shipped scorer, not illustrations. Scoring is deterministic, so every number below comes back identical on every run — which is what makes a printed score meaningful in the first place. The seven axes are the merits above, each on the engine’s 1–5 scale, drawn against a 0–5 radius so the shape is not flattered.
Write something about marketing for my business. Make it good and make sure it really works.
Nothing here is checkable. "Good" and "really works" name no test a program could apply, so Constraint Verifiability and Structural Compliance sit near the floor, and the shape collapses inward on the right-hand side.
I would really like you to please think very carefully and step by step about this, taking your time and considering all the angles thoroughly before you answer. It is very important that you do a really good job here. Please write something about marketing for my business. Make it good and make sure it really works. Think deeply. Be thorough. Consider everything.
Four times the length, and the request underneath is word-for-word the same. Concise Chain-of-Thought rises (1.6 → 2.8) because reasoning cues really are present — but Precision, Structural Compliance and Informational Integrity all FALL, because the ask is now a smaller fraction of the prompt. Overall: +0.07.
Write 5 email subject lines for a B2B SaaS product that helps engineering teams track deployment frequency. The audience is VPs of Engineering at companies with 50-200 engineers. Each subject line must be under 60 characters and must not use the words "revolutionary", "game-changing" or "unlock". Return them as a numbered list, one per line, with no commentary.
Clarity maxes out and Precision nearly does: the audience, the count, the character limit and three banned words are all stated. Concise Chain-of-Thought is the outlier at 1.9 — the prompt asks for output, not reasoning, and the engine does not pretend otherwise. A single overall number would hide that.
Summarise the attached incident report.
Return the summary as JSON matching this schema:
{"impact": string, "root_cause": string, "duration_minutes": number, "action_items": [string]}
Rules:
- "impact" must be one sentence, under 30 words.
- "duration_minutes" must be an integer.
- "action_items" must contain between 2 and 5 entries.
- Do not include any field not listed in the schema.Every rule here is mechanically checkable, and Constraint Verifiability answers at 4.8 — the highest of the four. Information Completeness stays at 3.5 because the report itself is not in the prompt, which is the honest reading: the engine scores what is written, not what is attached.
Reproduce any of these with the scorer CLI in contrib/prompt_scoring — engine features v2, calibrated heads, no network.
Calibration
A hand-tuned heuristic ranks prompts at about ρ = 0.25 against a judge, which is not good enough to put a number on a screen. The fix was not to add an LLM to the request path but to spend one offline run learning what the judge rewards, and then throw the judge away.
Gemini 2.5 Flash scored the IFEval and MePO prompt sets against the same seven-merit rubric, five independent times at temperature 0. The consensus of those runs is the training signal; a 20% hold-out of MePO pairs is set aside and never trained on.
Every prompt reduces to a 69-dimension deterministic feature vector — counts, ratios, cue detectors, the seven heuristic scores themselves, and a nonlinear expansion over length. Seven closed-form ridge regressions (λ = 256) are fitted over 3,048 prompts, one per merit.
The fitted weights ship as a JSON file embedded in the binary, versioned against the feature layout — a mismatch panics at startup rather than scoring quietly with the wrong head. Runtime is then a dot product. Held-out rank correlation: ρ ≈ 0.53, against ≈ 0.25 for the heuristic alone.
A regression fitted to a judge will happily learn that long prompts score well. The product's promise is that padding does not work, so the calibrated number is not allowed to say whatever the fit wants.
The calibrated number is clamped to the heuristic score plus 0.7. A regression head that decided some feature was worth two points cannot express it, so the heuristic stays the floor of the argument rather than a suggestion.
The failure mode a length-aware model invites is rewarding volume. Prompts that are long without being specific are held to a quarter of the usual lift.
When the prompt reads like keywords arranged around a vague object, the calibration is skipped entirely and the heuristic score stands. Padding cannot buy a point.
The training data is English. Other languages get structural signals and neutral priors, and the calibrated score is not allowed to drift above the heuristic — an honest floor beats a confident guess.
IFEval · 541 prompts
Google Research's Instruction Following Evaluation set: prompts written by domain experts to test whether a model obeys a specific instruction. They average 3.09, a third grade Good or better, and the best single prompt in the set reaches 3.91. Nothing reaches Excellent. The useful result is the shape below, not the mean.
Even careful writers rarely spell out reasoning structure or an explicit output shape. Concise Chain-of-Thought and Structural Compliance sit at the bottom of a set written by people who do this for a living — which is the single most useful thing IFEval says about prompts in general.
Two judges
Agreeing with the judge you were fitted to proves very little. Claude Sonnet 4.6 scored the same 541 prompts against the same rubric and contributed zero training labels — a fully independent opinion on the engine and on Gemini. It is also the measurement that sets the ceiling: the two judges agree with each other at only ρ = 0.61, and land 0.64 points apart on average.
| Dimension | Promptivo | Gemini 2.5 Flash | Sonnet 4.6 |
|---|---|---|---|
| Overall | 3.09 | 3.45 | 2.81 |
| Clarity | 4.22 | 3.85 | 3.65 |
| Precision | 3.50 | 3.07 | 3.10 |
| Concise Chain-of-Thought | 2.27 | 2.68 | 1.81 |
| Information Completeness | 3.54 | 3.10 | 2.99 |
| Constraint Verifiability below both | 2.69 | 3.79 | 3.85 |
| Structural Compliance below both | 2.46 | 3.38 | 3.07 |
| Informational Integrity | 2.97 | 3.38 | 2.43 |
IFEval, 541 prompts, one rubric. Gemini's column is a five-run mean; per-dimension run-to-run σ is at or below 0.009.
Gemini against Sonnet — how well two frontier judges agree with each other on the same prompts.
The frozen engine against the judge it never trained on, on this deliberately narrow dataset.
Exact grade agreement with the training judge — the least flattering framing of the same result, kept for that reason.
Five independent runs, no shared state, temperature 0. LLM-as-a-judge is usually accused of noise; on this set it is not noisy at all. It is simply a standard that charges tokens, latency and your prompt's privacy every time you consult it — and that moves the day the model is replaced.
| Stability bucket | Prompts | Share | Reading |
|---|---|---|---|
| σ < 0.2 — very stable | 475 | 88.1% | Gemini returns the same score across all five runs |
| σ 0.2–0.5 — minor jitter | 64 | 11.9% | Typically a half-point swing in one run of the five |
| σ ≥ 0.5 — high variance | 0 | 0% | No prompt in the set swung further than that |
MePO · 2,000 pairs
MePO pairs a raw prompt with a version rewritten by a research-grade optimizer, which gives a directional test the mean cannot fake: the optimized twin should score higher. It does on 99.9% of pairs, by +0.61 overall. The dimension-level split is the part worth reading — it is exactly what a single holistic score throws away.
The engine and the judge agree on what rewriting buys and what it does not. Gains concentrate in reasoning structure, context and output shape. Constraint Verifiability barely moves (+9.5%) because rewrites do not invent mechanically checkable rules, and Informational Integrity is flat (+1.8%) because polish adds no entities, sources or testable facts. On the same pairs the judge's own overall mean moves 2.46 → 3.56.
What we learned
The headline number is the least interesting thing an experiment produces. These are the findings that changed how we think about the problem — two of them are why there is a next version of this work rather than a finished one.
Gemini and Sonnet scored the same 541 prompts against the same rubric and landed 0.64 points apart on average, with 15% of prompts more than a full point apart and only ρ = 0.61 rank agreement between them. "Agrees with the judge" has a ceiling, and it is that number, not 1.0.
The two dimensions where the engine sits below both judges are Constraint Verifiability (2.69 against 3.79 and 3.85) and Structural Compliance (2.46 against 3.38 and 3.07). A judge credits a constraint a reader would infer; the engine scores only what is written. This is the trade, not a bug to be fixed — closing it needs comprehension at scoring time.
Expert-written prompts cluster in a thin quality band, which is why rank correlation on IFEval is modest (ρ = 0.30 against the unseen judge) while the means sit close. The wide-range MePO set is where a ranking claim can honestly be made.
36 of 541 prompts diverge from the judge consensus by more than 1.5 points; 43% land within half a point, mean gap 0.65. The stubborn cases share a shape: short prompts whose constraint is obvious to a reader and carries no marker an engine can see.
Real scores fall in roughly 2.4–4.1, not across the full 1–5. The engine's own thresholds assume the full range, so the displayed verdict uses cut points derived from the benchmark distributions instead. The grade you see is a presentation decision with a derivation behind it, and it moves if the distribution moves.
Coefficients were fitted on 24 August 2026 against one judge on one date. Nothing re-fits at runtime — that is the point — but it also means the standard is a snapshot. Re-derive it, and every historical score shifts underneath.
Every correlation here is against two frontier models scored in August 2026 on two English academic datasets. Neither dataset resembles the prompts people actually paste into a chat box, and nothing here has been checked against human raters or against outcomes — whether a higher-scoring prompt produces a better answer is a claim this experiment does not yet make, and it is the one we most want to. Read the direction of the results, not the second decimal.
Does a higher score produce a better answer? Every figure on this page measures agreement with a judge's opinion of a prompt. None of it measures the thing anyone actually cares about. Closing that gap means scoring outputs, not prompts, and it is the single most valuable experiment left.
Can inferable constraints be scored without comprehension? Both judges credit a constraint a reader would infer; the engine only sees markers. That gap is worth 0.9–1.1 points on two dimensions and accounts for most of the stubborn disagreements. Some of it is surely reachable with better detectors.
What happens beyond English? Non-English prompts get structural signals and a conservative floor, which is honest and not much else. A calibration with non-English labels behind it is a different experiment, and a bigger one.
Housekeeping for whoever picks this up: the calibration is a snapshot of one judge on one date, and the displayed grade cut points are derived from the resulting distribution. Any engine or feature change invalidates both — bump the feature version, retrain, re-derive the cut points, and regenerate the figures on this page in the same pass.
Live experiment
Promptivo scores a prompt in milliseconds with no model call, so nothing you paste is sent anywhere and the same prompt always scores the same. There is no quota, no tier and nothing to buy — an account just gives your history somewhere to live.