Promptivo

Everyone evaluating prompt quality reaches for a model to do the judging, which makes every score slow, billable, non-repeatable, and a copy of your prompt on someone else's server. Promptivo tests the alternative: consult the judge once, learn what it rewards, freeze that into coefficients, and never call it again.

0 calls

to a model, at scoring time

The judge was consulted once, during calibration, and then frozen into coefficients.

ρ = 0.53

against the judge, on held-out pairs

Up from ρ ≈ 0.25 for the heuristic engine alone. MePO hold-out the training never saw.

ρ = 0.61

is what two judges manage with each other

Gemini vs Sonnet, same prompts, same rubric. That is the ceiling any judge-aligned scorer is measured against.

99.9%

of MePO pairs rank the right way round

The optimized prompt scores above its raw twin. The one thing the engine must not get wrong.

Seven merits, scored independently.

Four merits come from the MePO framework (Zhu et al., arXiv:2505.09930); three were added for the properties IFEval tests and MePO does not. Each runs as its own scorer over deterministic linguistic signals, starts from a baseline, applies signed adjustments and clamps to 1–5. The overall score is their mean. Nothing here consults a model, a network, or the clock — the same prompt yields the same seven numbers every time.

Clarity MePO: Is the ask unambiguous — few vague quantifiers, few dangling pronouns, readable sentence lengths?
Precision MePO: Does it name specific objects, entities and quantities rather than gesturing at a category?
Concise Chain-of-Thought MePO: Does it ask for reasoning structure without padding the prompt with scaffolding?
Information Completeness MePO: Is the context the model would need actually in the prompt — inputs, audience, examples?
Constraint Verifiability IFEval: Could a program check whether the output obeyed the constraint, or only a human?
Structural Compliance IFEval: Is an output shape specified — sections, ordering, delimiters, a length?
Informational Integrity IFEval: Does it carry checkable substance — named entities, sources, testable facts — or only polish?

Four prompts, and what the engine says about them.

These are real outputs of the shipped scorer, not illustrations. Scoring is deterministic, so every number below comes back identical on every run — which is what makes a printed score meaningful in the first place. The seven axes are the merits above, each on the engine’s 1–5 scale, drawn against a 0–5 radius so the shape is not flattered.

A vague ask

2.26 / 5, Needs Improvement

Write something about marketing for my business. Make it good and make sure it really works.

Nothing here is checkable. "Good" and "really works" name no test a program could apply, so Constraint Verifiability and Structural Compliance sit near the floor, and the shape collapses inward on the right-hand side.

The same ask, padded

2.33 / 5, Needs Improvement

I would really like you to please think very carefully and step by step about this, taking your time and considering all the angles thoroughly before you answer. It is very important that you do a really good job here. Please write something about marketing for my business. Make it good and make sure it really works. Think deeply. Be thorough. Consider everything.

Four times the length, and the request underneath is word-for-word the same. Concise Chain-of-Thought rises (1.6 → 2.8) because reasoning cues really are present — but Precision, Structural Compliance and Informational Integrity all FALL, because the ask is now a smaller fraction of the prompt. Overall: +0.07.

Specific, but unstructured

3.76 / 5, Excellent

Write 5 email subject lines for a B2B SaaS product that helps engineering teams track deployment frequency. The audience is VPs of Engineering at companies with 50-200 engineers. Each subject line must be under 60 characters and must not use the words "revolutionary", "game-changing" or "unlock". Return them as a numbered list, one per line, with no commentary.

Clarity maxes out and Precision nearly does: the audience, the count, the character limit and three banned words are all stated. Concise Chain-of-Thought is the outlier at 1.9 — the prompt asks for output, not reasoning, and the engine does not pretend otherwise. A single overall number would hide that.

Constrained by a schema

4.13 / 5, Excellent

Summarise the attached incident report.

Return the summary as JSON matching this schema:
{"impact": string, "root_cause": string, "duration_minutes": number, "action_items": [string]}

Rules:
- "impact" must be one sentence, under 30 words.
- "duration_minutes" must be an integer.
- "action_items" must contain between 2 and 5 entries.
- Do not include any field not listed in the schema.

Every rule here is mechanically checkable, and Constraint Verifiability answers at 4.8 — the highest of the four. Information Completeness stays at 3.5 because the report itself is not in the prompt, which is the honest reading: the engine scores what is written, not what is attached.

Reproduce any of these with the scorer CLI in contrib/prompt_scoring — engine features v2, calibrated heads, no network.

The judge was asked once, in August.

A hand-tuned heuristic ranks prompts at about ρ = 0.25 against a judge, which is not good enough to put a number on a screen. The fix was not to add an LLM to the request path but to spend one offline run learning what the judge rewards, and then throw the judge away.

1A five-run consensus
Gemini 2.5 Flash scored the IFEval and MePO prompt sets against the same seven-merit rubric, five independent times at temperature 0. The consensus of those runs is the training signal; a 20% hold-out of MePO pairs is set aside and never trained on.
2One ridge head per merit
Every prompt reduces to a 67-dimension deterministic feature vector — counts, ratios, cue detectors, the seven heuristic scores themselves, and a nonlinear expansion over length. Seven closed-form ridge regressions (λ = 256) are fitted over 3,048 prompts, one per merit.
3Coefficients, compiled in
The fitted weights ship as a JSON file embedded in the binary, versioned against the feature layout — a mismatch panics at startup rather than scoring quietly with the wrong head. Runtime is then a dot product. Held-out rank correlation: ρ ≈ 0.53, against ≈ 0.25 for the heuristic alone.

Four guards on the fitted score

A regression fitted to a judge will happily learn that long prompts score well. The product's promise is that padding does not work, so the calibrated number is not allowed to say whatever the fit wants.

A head may lift a merit by at most +0.7: The calibrated number is clamped to the heuristic score plus 0.7. A regression head that decided some feature was worth two points cannot express it, so the heuristic stays the floor of the argument rather than a suggestion.
Long, low-density prompts get +0.2 at most: The failure mode a length-aware model invites is rewarding volume. Prompts that are long without being specific are held to a quarter of the usual lift.
Stuffing suspects get no lift at all: When the prompt reads like keywords arranged around a vague object, the calibration is skipped entirely and the heuristic score stands. Padding cannot buy a point.
Non-English is anchored from below: The training data is English. Other languages get structural signals and neutral priors, and calibration can pull each of their merits at most 0.3 below what the heuristic scored — an honest floor beats a confident guess.

What expert prompts actually score.

Google Research's Instruction Following Evaluation set: prompts written by domain experts to test whether a model obeys a specific instruction. They average 3.09 and a third grade Good or better, but only 13 of the 541 reach Excellent, the best single prompt at 3.91. The useful result is the shape below, not the mean.

Clarity

4.22

Information Completeness

3.54

Precision

3.50

Informational Integrity

2.97

Constraint Verifiability

2.69

Structural Compliance

2.46

Concise Chain-of-Thought

2.27

Even careful writers rarely spell out reasoning structure or an explicit output shape. Concise Chain-of-Thought and Structural Compliance sit at the bottom of a set written by people who do this for a living — which is the single most useful thing IFEval says about prompts in general.

Including one that contributed no training labels.

Agreeing with the judge you were fitted to proves very little. Claude Sonnet 4.6 scored the same 541 prompts against the same rubric and contributed zero training labels — a fully independent opinion on the engine and on Gemini. It is also the measurement that sets the ceiling: the two judges agree with each other at only ρ = 0.61, and land 0.64 points apart on average.

Dimension	Promptivo	Gemini 2.5 Flash	Sonnet 4.6
Overall	3.09	3.45	2.81
Clarity	4.22	3.85	3.65
Precision	3.50	3.07	3.10
Concise Chain-of-Thought	2.27	2.68	1.81
Information Completeness	3.54	3.10	2.99
Constraint Verifiability below both	2.69	3.79	3.85
Structural Compliance below both	2.46	3.38	3.07
Informational Integrity	2.97	3.38	2.43

IFEval, 541 prompts, one rubric. Gemini's column is a five-run mean; per-dimension run-to-run σ is at or below 0.009.

ρ = 0.61

Gemini against Sonnet — how well two frontier judges agree with each other on the same prompts.

ρ = 0.30

The frozen engine against the judge it never trained on, on this deliberately narrow dataset.

31.2%

Exact grade agreement with the training judge — the least flattering framing of the same result, kept for that reason.

The judge is steady with itself, which is the part that surprised us

Five independent runs, no shared state, temperature 0. LLM-as-a-judge is usually accused of noise; on this set it is not noisy at all. It is simply a standard that charges tokens, latency and your prompt's privacy every time you consult it — and that moves the day the model is replaced.

Stability bucket	Prompts	Share	Reading
σ < 0.2 — very stable	475	88.1%	Gemini returns the same score across all five runs
σ 0.2–0.5 — minor jitter	64	11.9%	Typically a half-point swing in one run of the five
σ ≥ 0.5 — high variance	0	0%	No prompt in the set swung further than that

The same task, before and after an optimizer.

MePO pairs a raw prompt with a version rewritten by a research-grade optimizer, which gives a directional test the mean cannot fake: the optimized twin should score higher. It does on 99.9% of pairs, by +0.61 overall. The dimension-level split is the part worth reading — it is exactly what a single holistic score throws away.

RawOptimized

Concise Chain-of-Thought1.48 → 2.52, +69.7%

Information Completeness2.57 → 3.58, +39.4%

Structural Compliance1.64 → 2.27, +38.6%

Precision2.59 → 3.47, +34.0%

Clarity3.31 → 3.76, +13.7%

Constraint Verifiability1.89 → 2.07, +9.5%

Informational Integrity2.84 → 2.89, +1.8%

The engine and the judge agree on what rewriting buys and what it does not. Gains concentrate in reasoning structure, context and output shape. Constraint Verifiability barely moves (+9.5%) because rewrites do not invent mechanically checkable rules, and Informational Integrity is flat (+1.8%) because polish adds no entities, sources or testable facts. On the same pairs the judge's own overall mean moves 2.46 → 3.56.

Six results that complicate the answer.

The headline number is the least interesting thing an experiment produces. These are the findings that changed how we think about the problem — two of them are why there is a next version of this work rather than a finished one.

The target itself moves: Gemini and Sonnet scored the same 541 prompts against the same rubric and landed 0.64 points apart on average, with 15% of prompts more than a full point apart and only ρ = 0.61 rank agreement between them. "Agrees with the judge" has a ceiling, and it is that number, not 1.0.
The engine reads only what is written: The two dimensions where the engine sits below both judges are Constraint Verifiability (2.69 against 3.79 and 3.85) and Structural Compliance (2.46 against 3.38 and 3.07). A judge credits a constraint a reader would infer; the engine scores only what is written. This is the trade, not a bug to be fixed — closing it needs comprehension at scoring time.
Expert prompts are too alike to rank: Expert-written prompts cluster in a thin quality band, which is why rank correlation on IFEval is modest (ρ = 0.30 against the unseen judge) while the means sit close. The wide-range MePO set is where a ranking claim can honestly be made.
A stubborn 6.7% resists the method: 36 of 541 prompts diverge from the judge consensus by more than 1.5 points; 43% land within half a point, mean gap 0.65. The stubborn cases share a shape: short prompts whose constraint is obvious to a reader and carries no marker an engine can see.
The usable range is narrower than the scale: Real scores fall in roughly 2.4–4.1, not across the full 1–5. The engine's own thresholds assume the full range, so the displayed verdict uses cut points derived from the benchmark distributions instead. The grade you see is a presentation decision with a derivation behind it, and it moves if the distribution moves.
A frozen standard ages — by design: Coefficients were fitted on 24 August 2026 against one judge on one date. Nothing re-fits at runtime — that is the point — but it also means the standard is a snapshot. Re-derive it, and every historical score shifts underneath.
What this evidence does not cover: Every correlation here is against two frontier models scored in August 2026 on two English academic datasets. Neither dataset resembles the prompts people actually paste into a chat box, and nothing here has been checked against human raters. The one test against outcomes so far found no link between a prompt's score and whether a model followed it — see the first question below. Read the direction of the results, not the second decimal.

Three questions the next version has to answer

Does a higher score produce a better answer?: Not in the first test of it. Claude Sonnet 4.6 and Claude Haiku 4.5 answered every IFEval prompt three times — 3,246 answers, each checked mechanically for whether it followed its instructions — and no prompt score predicted which ones did: not Promptivo's (an odds ratio of 1.02 per standard deviation, 0.82–1.19), and not either LLM judge's. The question stays open, because this set leaves little to predict: 73% of its prompts were followed perfectly, and the two models agreed only loosely (ρ = 0.54) about which prompts were hard.
Can inferable constraints be scored without comprehension?: Both judges credit a constraint a reader would infer; the engine only sees markers. That gap is worth 0.9–1.1 points on two dimensions and accounts for most of the stubborn disagreements. Some of it is surely reachable with better detectors.
What happens beyond English?: Non-English prompts get structural signals and a conservative floor, which is honest and not much else. A calibration with non-English labels behind it is a different experiment, and a bigger one.

Housekeeping for whoever picks this up: the calibration is a snapshot of one judge on one date, and the displayed grade cut points are derived from the resulting distribution. Any engine or feature change invalidates both — bump the feature version, retrain, re-derive the cut points, and regenerate the figures on this page in the same pass.

It is free, and it will stay that way.

Try Promptivo Other experiments