Can Jev judge passage relevance?
An evaluation of Jev’s Choice, Score, and Noul primitives on the LLMJudge benchmark, with human labels and published LLM judges as reference points.
This is not a blind challenge submission. Test labels and benchmark results were public before this experiment. We expect Jev to have seen the public LLMJudge results during training, although its training data is unverified. These scores may reflect benchmark exposure.
What the labels say
Native Choice comes closest to the paper’s best four-grade judge. At the primary answer-bearing boundary, it reaches κ 0.4180 against a published best of 0.4280. Development-tuned Score and Noul thresholds both reduce test agreement.
Four-grade agreement
Cohen’s κ against human grades 0–3
Answer-bearing decision
Cohen’s κ for grades 0–1 versus 2–3
Each paper-best bar is the strongest of the paper’s 42 submitted judges for that metric; the winning submission differs by column. Bars start at zero and share a 0.5 κ scale.
All judgment boundaries
The 0–1 versus 2–3 split is the primary test of whether a passage provides an answer.
| Judge | Four grades | 0 | 1–3 | 0–1 | 2–3 | 0–2 | 3 |
|---|---|---|---|---|
| Jev ChoiceNative | 0.2617 | 0.3798 | 0.4180 | 0.3308 |
| Jev ScoreNative | 0.2249 | 0.3367 | 0.4059 | 0.3099 |
| Jev ScoreCalibrated | 0.2118 | 0.3592 | 0.3656 | 0.2907 |
| Jev NoulNative | 0.2589 | 0.4149 | 0.3981 | 0.3103 |
| Jev NoulCalibrated | 0.2451 | 0.3847 | 0.3402 | 0.3103 |
| Paper best per column | 0.2863 | 0.4228 | 0.4280 | 0.3215 |
Paper winners: willia-umbrela1 (four grades), Olz-gpt4o (0 | 1–3), h2oloo-fewself (0–1 | 2–3), and willia-umbrela3 (0–2 | 3). Jev Choice exceeds the paper’s best only on the 0–2 | 3 boundary: 0.3308 versus 0.3215.
Do the labels preserve system order?
We scored the same 35 passage retrieval runs with human and Jev labels using trec_eval nDCG@10, then compared their rankings.
| Judge | Kendall τ | Spearman ρ |
|---|---|---|
| Jev ChoiceNative | 0.9284 | 0.9891 |
| Jev ScoreNative | 0.9351 | 0.9915 |
| Jev ScoreCalibrated | 0.9024 | 0.9835 |
| Jev NoulNative | 0.9428 | 0.9922 |
| Jev NoulCalibrated | 0.9226 | 0.9874 |
| Paper best per column | 0.9516 | 0.9919 |
Paper winners: prophet-setting2 for τ and TREMA-4prompts for ρ.
How the experiment works
The benchmark provides graded human judgments for 25 development and 25 test queries. The three primitives saw identical query and passage text, with frozen question wording.
Five questions, one request
- ChoiceSelects one of the four benchmark grade definitions.
- ScorePlaces the passage on an ordered four-level scale.
- Noul × 3Tests relatedness, a partial answer, and a dedicated exact answer independently.
Native and calibrated labels
Native Score uses cutoffs 0.5 / 1.5 / 2.5; native Noul uses 0.5 on each answer. Calibration searched 0.05 steps to maximize four-grade κ on development data only. The selected Score cutoffs were 1.4 / 2.1 / 2.25 and Noul cutoffs were 0.8 / 0.9 / 0.5. They were applied once to test.
Four-grade and binary κ, nominal α, confusion matrices, grade distributions, and 95% query-cluster bootstrap intervals.
All 4,423 test pairs have Jev responses. The report also compares 33 released LLM judges and 35 passage runs.
Jev saw shortened head-and-tail text for 44 overlength passages. Test labels are public, and α conventions differ from the paper.
Read the full record
The site summarizes one experiment. The complete numeric report, raw Jev answers, threshold configuration, and source hashes are available for inspection.