JEV 判官

技术笔记Technology Notes / 论文阅读Paper Reading / JEV 当判官JEV as a Judge

这篇论文做了一件很实用的事：把「只输出决策、不输出理由」的判官当成廉价的第一道评测，用 17 个判官配置、1312 个条目测出它在哪里够用、在哪里必须升级，并且证明置信度本身可以当门控信号用。

The paper does something unusually practical: it treats a decision-only judge as a cheap first pass, measures it against 16 other judge configurations over 1,312 items, and reports where it is enough and where escalation is required — then shows that confidence itself can drive that gate.

阅读结论 Verdict 方法论可直接搬 / 级联数字别当承诺 Reusable method, cautious cascade numbers 评测协议与阈值纪律值得照抄；费用与级联收益是离线模拟，落地前要自己复测。 Copy the protocol and the threshold discipline; re-measure the cost and cascade claims before relying on them.

目录 Contents 问题：评测成本，与「什么时候能信」Problem: judging cost and knowing when to trust 评测协议：怎么把这件事测公平Protocol: how it is measured fairly 适用边界：哪里够用，哪里要升级Operating envelope: enough, or escalate 成本与延迟Cost and latency 置信度门控与冻结策略Confidence gating and frozen policies 信号在哪里失效Where the signal fails 批判性评估Critical assessment 论文自己的五条清单The paper's own checklist 对本机两个项目的意义What it means for two local projects

0.36% JEV 的费用相对最强对照判官 GPT-6 的比例，同时差距在 3 个百分点以内 JEV's fee relative to the strongest comparator, GPT-6, while staying within three points

−14.6 / −19.8 JudgeBench（多步推导）与 RM-Bench 硬对（错误的答案写得更漂亮）上的差距，单位是百分点 Gap in points on JudgeBench (multi-step derivation) and on RM-Bench hard pairs (a better-written wrong answer)

99.6% / 47% 单序级联在 τ=0.9 时保留的准确率与所花费用（作为离线模拟的参考值） Accuracy retained and fee spent by the single-order cascade at τ=0.9, as an offline simulation

论文 Paper

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

Yubo Li、Yidi Miao、Ramayya Krishnan、Rema Padman，卡内基梅隆大学。

Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman, Carnegie Mellon University.

来源 Source

arXiv:2609.26550v1

2026-09-22 提交（稿面日期 2026-09-21）。本页按官方 HTML 版精读，数字均回到正文或表号。

Submitted 2026-09-22 (manuscript dated 2026-09-21). This note reads the official HTML version and traces numbers to the text or tables.

被测对象 What is measured

TypeSafe JEV 1.13

一个托管服务：输入自然语言指令与结构化状态，输出类型化概率（Choice / Noul / Score），不返回任何文字。实现闭源。

A hosted service that takes natural-language instructions and structured state and returns typed probabilities (Choice, Noul, Score). It never returns prose, and the implementation is proprietary.

问题：评测成本，与「什么时候能信」

Problem: judging cost and knowing when to trust

用 LLM 当评委已经从「给固定答案打分」扩展到「判断开放式行为」。这带来两股压力：一是推理型判官按 token 计费，模型每改一版、评测集每扩一次，判官的费用就变成研发成本的一部分；二是被问「你有多确定」的模型经常过度自信，把高概率分配给错误答案。

LLM judges have moved from scoring fixed answers to judging open-ended behaviour, which brings two pressures. Reasoning judges bill by tokens, so the judge's fee becomes part of the cost of developing and operating the system. And models asked to report their own confidence are often overconfident, assigning high probabilities to wrong answers.

论文的问题因此被拆成三件事，缺一不可：这个判官准不准、它的置信度可不可用、以及它到底便宜多少。作者的原话是，一个可用的判定、一个正确的决定、一份可靠的置信度，是三件不同的事。

So the question splits into three parts that must be answered together: is the judge accurate, is its confidence usable, and how much cheaper is it really. As the paper puts it, a usable verdict, a correct decision and reliable uncertainty are three different things.

评测协议：怎么把这件事测公平

Protocol: how it is measured fairly

这部分是全文最值得借鉴的地方，也是最容易被忽略的：作者先冻结数据，再开始推理。

This is the most reusable and most easily skipped part: the data was frozen before any inference happened.

协议要点	Protocol point	具体做法	How it is done
先冻结再推理	Freeze before inference	642 条试点在推理前冻结，公开任务按题目来源 40/60 分成「选择集」与「测试集」；670 条扩展集在查看试点准确率之前冻结，从不参与参数拟合。	The 642-item pilot was frozen before any inference and split 40/60 by source question into a selection set and a test set; the 670-item extension was frozen before pilot accuracy was inspected and never entered a fit.
标签不改	Labels kept as supplied	benchmark 标签一律原样使用；人工修正后的准确率只作为敏感性分析单独报告，不进主结果。	Benchmark labels are used as supplied; human-corrected accuracies appear only as a sensitivity analysis, never in the primary results.
无效输出计为错误	Invalid output counts as an error	校验 schema、标签归属、概率有限性与归一化（容差 0.025）、判定与 argmax 是否一致；无效即算错，不重跑、不修补。概率类指标只在有效输出上计算，但分母保留。	Every output is validated for schema, label membership, finite and normalised probabilities (tolerance 0.025) and verdict–argmax consistency. Invalid outcomes count as errors; nothing is repaired or rerun. Probability metrics condition on validity but keep the denominators.
配对差异用簇自助法	Cluster bootstrap for paired differences	2000 次按来源题目聚类的重采样，把同一问题的两条答案、以及一对样本的所有呈现变体绑在一起。	2,000 source-question cluster resamples keep the two answers to one question, and every presentation variant of a pair, together.
分开测量延迟与费用	Latency measured separately	批量质量运行的计时不用于延迟结论（客户端有记账争用）；延迟来自一块冻结的 120 次判定面板，一次只跑一组模型、八并发、最小起始间隔 0.12 秒。	Bulk-run timings are not used for latency claims because client-side accounting caused contention; latency comes from a frozen 120-decision panel, one model group at a time, eight workers, 0.12-second minimum start interval.
两轮采集要标注	Label the collection windows	初始窗口只评 JEV、三个 GPT 基线与 PairRM；扩展窗口在已知这些结果之后重跑，因此多模型比较被明确标为探索性。	The initial window evaluated JEV, three GPT baselines and PairRM; the expansion reran after those results were known, so the multi-family comparison is explicitly exploratory.

一句话概括这套纪律：把「用来调参的数据」和「用来下结论的数据」在推理开始前就分开，并且把每一个模型是在哪个窗口里跑的都写清楚。

In one line: separate the data you tune on from the data you report on, before any inference starts, and record which window every number came from.

适用边界：哪里够用，哪里要升级

Operating envelope: enough, or escalate

同样的 JEV，换一个 workload 结论就完全不同。分界线很清楚：当题目自带参考答案或证据片段时它站得住，当判断需要检查多步推导或抵抗写得更漂亮的错误答案时它撑不住。

The same judge gets opposite verdicts depending on the workload. The line is clear: it holds up when the item ships with a reference answer or an evidence passage, and it does not when the judgment requires checking a multi-step derivation or resisting a better-written wrong answer.

图 1 · 适用边界。绿色是可以直接用，橙色是差距 9–20 个百分点、需要升级，黄色是必须先本地验证，灰色是所有被测判官都接近随机的区域。数据来源：原文 Table 2，按原数值重绘。

Figure 1 - The operating envelope. Green means use it directly, orange means a 9–20 point gap and escalation, amber means validate locally first, grey is where every tested judge is near chance. Source: Table 2, redrawn from the reported values.

任务类型	Workload	JEV	最强对照	Comparator	差值	Δ	建议	Guidance
普通偏好（RewardBench 400）	Ordinary preference (RewardBench 400)	92.2	GPT-6 93.5	−1.3	可以用 JEV	Use JEV
有证据支撑的事实性（HaluEval 240）	Evidence-grounded factuality (HaluEval 240)	87.5	GPT-6 86.7	+0.8	可以用 JEV	Use JEV
最终答案裁决（既有标注 150）	Final-answer adjudication (150)	94.0	GPT-6 96.7	−2.7	可以用 JEV	Use JEV
困难正确性（JudgeBench 350）	Difficult correctness (JudgeBench 350)	78.6	GPT-6 93.1	−14.6	需要升级	Escalate
同风格配对（RM-Bench 普通 480）	Matched-style pairs (RM-Bench normal 480)	84.0	GPT-6 93.3	−9.4	需要升级	Escalate
文风对抗配对（RM-Bench 硬 480）	Style-adversarial pairs (RM-Bench hard 480)	74.8	GPT-6 94.6	−19.8	需要升级	Escalate
四选一选择（RewardBench 2，100）	Four-way selection (RewardBench 2, 100)	73.0	Skywork 79.0	−6.0	先本地验证	Validate first
有出处的摘要（80）	Grounded summaries (80)	71.2	GPT-5.4 72.5	−1.2	先本地验证	Validate first
无参考的自然语言（80）	Reference-free prose (80)	52.5	GPT-5.4 55.0	−2.5	不支持	Not supported

困难正确性的差距还要看分域：推理 68.4 对 95.9（98 条）、编码 76.2 对 97.6（42 条）、知识 84.4 对 90.9（154 条）。差距集中在推理与代码，也就是「你这个推导对不对」这类问题。

The difficult-correctness gap has structure: reasoning 68.4 against 95.9 (n=98), coding 76.2 against 97.6 (n=42), knowledge 84.4 against 90.9 (n=154). It concentrates in reasoning and code, which is exactly the "is this derivation right" kind of question.

还有一个容易被忽略的细节：人工盲审裁决比 benchmark 标签更偏向 GPT-6。在 RewardBench 的分歧项上，裁决偏向 GPT-6 17 次、偏向 JEV 5 次，把差距从 −1.3 拉到 −3.0；26 个双方都判错的 HaluEval 条目里有 24 个标签本身与证据不符，修正后 JEV 95.8、GPT-6 98.3。所以「在 3 个百分点以内」这个结论是跨过两套标签都成立的，而单看 HaluEval 的持平里有一部分是标签噪声。

One easily missed detail: blinded human adjudication favours GPT-6 more than the labels do. On RewardBench disagreements the adjudicator sided with GPT-6 17 times and JEV 5 times, widening the gap from −1.3 to −3.0; and 24 of the 26 HaluEval items both judges missed carry labels the evidence does not support, which corrects to JEV 95.8 and GPT-6 98.3. The "within three points" summary survives both label sets, while part of the HaluEval tie is label noise near the ceiling.

成本与延迟

Cost and latency

费用与延迟来自同一块冻结的 120 次判定面板，一次只跑一组模型。结果显示的是量级差异，而不是几个百分点的差异。

Fee and latency come from one frozen 120-decision panel, one model group at a time. The result is an order-of-magnitude difference rather than a few percent.

图 2 · 每千次判定 0.044 美元、中位延迟 0.152 秒。对照 GPT-4.1 mini 是 0.390 美元 / 0.548 秒，GPT-6 是 12.182 美元 / 1.885 秒。数据来源：原文 Sec.5「Measured latency and fees」与 Fig.2。

Figure 2 - $0.044 per 1,000 judgments at a median 0.152 seconds. GPT-4.1 mini is $0.390 / 0.548 s and GPT-6 is $12.182 / 1.885 s. Source: Sec.5 measured latency and fees, and Fig.2.

费用口径也要看清：论文用的是采集期价格（JEV 1.13.0 每百万输入 token 0.042 美元、输出免费），包含缓存命中折扣与计费的推理 token；用量缺失时按保守预留计费而不是按零计。所有美元数字都是估算，不是账单。

Read the fee basis carefully: collection-time prices (JEV 1.13.0 at $0.042 per million input tokens, output free), including cache discounts and billed reasoning tokens; missing usage is charged as a conservative reservation rather than zero. Every dollar figure is an estimate, not an invoice.

置信度门控与冻结策略

Confidence gating and frozen policies

整个级联之所以成立，是因为JEV 在自己会错的地方不确定。准确率随最大标签概率 q 单调上升：q 低于 0.6 的 65 条只有 47.7% 正确，q 在 0.95–0.99 的 147 条是 93.9%，q 恰好为 1 的 322 条是 99.1%。同一批条目上 GPT-6 只从 78.5% 升到 99.1%——也就是说它的优势集中在 JEV 不确定的地方。

The cascade works because JEV is unsure where it is wrong. Accuracy rises monotonically with the maximum label probability q: 47.7% correct in the 65 items below 0.6, 93.9% in the 147 items between 0.95 and 0.99, and 99.1% of the 322 items at exactly 1. On the same items GPT-6 only moves from 78.5% to 99.1%, so its advantage sits exactly where JEV is unsure.

图 3 · 置信度把错误排在前面。左侧是论文按 q 分箱的准确率；右侧是 τ=0.9 的策略对比，升级 34% 的条目即可保留 GPT-6 准确率的 99.6%，花 47% 的费用。数据来源：原文 Sec.7「Confidence orders the errors」与「Single-order cascades」，右侧为离线模拟。

Figure 3 - Confidence puts the errors first. Left: accuracy by q bin. Right: at τ=0.9, escalating 34% of items keeps 99.6% of GPT-6's accuracy at 47% of the fee. Source: Sec.7 confidence ordering and single-order cascades; the right panel is an offline simulation.

一个关键的对照：如果在同样的预算下随机升级 34% 的条目，只能到 88.1%；如果按标签只升级那些真正会错的条目（oracle 上界），可以到 94.4%。也就是说 q 捕捉到了大约一半的可获得增益——置信度确实携带信息，但远不是全部。

A key control: escalating the same 34% at random reaches only 88.1%, while escalating only the items that are genuinely wrong (the label-aware oracle bound) reaches 94.4%. Confidence captures about half of the attainable gain — real information, but far from all of it.

冻结的双序策略

The frozen two-order policy

因为候选顺序会改变判定（RewardBench 有 3.25% 的决定被倒序翻转，JudgeBench 高达 11.14%），作者把策略定义成先判两个顺序、再把对齐后的概率求平均：

Because candidate order changes verdicts (reversal flips 3.25% of RewardBench decisions and 11.14% of JudgeBench decisions), the policy judges each pair in both orders and averages the aligned probabilities:

p̄(A) = ½ · [ p₁(A, B) + 1 − p₁(B, A) ]

阈值在 96 个试点选择对上选取，规则是「在保持选择集准确率距 fallback 不超过 2 分的前提下最大化覆盖率」，网格与规则在模型扩展前固定，无效的首段输出一律升级。

The threshold was chosen on 96 pilot selection pairs to maximise coverage while keeping selection-set accuracy within two points of the fallback; the grid and rule were fixed before the model expansion, and invalid first-stage outputs always defer.

Fallback	τ	接受比例	Accepted	级联准确率	Cascade acc.	全程强判官	Fallback acc.	Δ	费用比	Fee ratio
GPT-6 Astra	0.90	53.7%	92.5	93.1	−0.59	0.568 / 0.622
GPT-5.4	0.90	53.7%	91.4	91.6	−0.2	0.639 / 1.096
GPT-5.6 Sol	0.70	81.0%	91.0	93.3	−2.4	0.288 / 0.380

注意第三行：GPT-5.6 在 τ=0.7 时接受了 81% 的对，但掉了 2.35 分，超出了自己设定的 2 分容忍度——阈值不是通用的，换一个 fallback 就要重新选。另外 GPT-5.4 那一行的保守费用比是 1.096，大于 1，因为缺失的用量按预留计费。

Note the third row: with GPT-5.6 at τ=0.7 the policy accepts 81% of pairs but loses 2.35 points, beyond its own two-point tolerance — the threshold is not portable, and each fallback needs its own. The GPT-5.4 row also shows a conservative fee ratio of 1.096, above one, because missing usage is charged as a reservation.

信号在哪里失效

Where the signal fails

把置信度当门控有一个前提：它得能区分对错。用 AUROC 衡量这件事之后，边界就出现了——置信度在第一段有能力但不确定的地方好用，在第一段被「自信地误导」的地方完全没用。

Gating on confidence assumes it can separate right from wrong. Measured as AUROC, the boundary appears: confidence routes well where the first stage is competent but uncertain, and badly where it is confidently misled.

图 4 · 置信度的可用性随 workload 崩塌。文风对抗的硬对上 AUROC 从 0.902 降到 0.770，无参考散文只有 0.518——那时所有判官都接近随机，却依然给出 0.90 以上的平均最大概率。数据来源：原文 Sec.6 Table 7 与 Sec.7。

Figure 4 - Usable confidence collapses by workload. On style-adversarial hard pairs the AUROC drops from 0.902 to 0.770, and reference-free prose sits at 0.518 — where every judge is near chance yet still reports mean maximum probabilities above 0.90. Source: Sec.6 Table 7 and Sec.7.

文风对抗：写得更漂亮的错误答案

Style-adversarial: the better-written wrong answer

在 RM-Bench 硬对上，JEV 对 [0.9, 0.95) 区间的对子判错三分之一，对 [0.95, 0.99) 判错 15%；τ=0.9 的级联只能保留 96.5% 的准确率，而普通对上是 99.6%。达到 98.7% 需要把阈值提到 0.95 并升级 58% 的对。

On RM-Bench hard pairs JEV is wrong on a third of the pairs it scores in [0.9, 0.95) and on 15% of those in [0.95, 0.99). A τ=0.9 cascade retains only 96.5% there, against 99.6% on normal pairs; reaching 98.7% needs τ=0.95 and 58% escalation.

无参考答案：所有判官都接近随机

No reference: every judge is near chance

在没有参考答案的自然语言上，JEV 52.5%、GPT-5.4 55.0%，而三者的平均最大概率分别是 0.90、0.95、0.96。Brier 0.815、AUROC 0.518——不是「信号弱」，是「没有信号，却依然自信」。

On reference-free prose, JEV scores 52.5% and GPT-5.4 55.0%, while their mean maximum probabilities are 0.90, 0.95 and 0.96. Brier 0.815 and AUROC 0.518 — not a weak signal but no signal, delivered confidently.

校准与排序是两件事

Calibration and error ranking are separate

GPT-6 在 JudgeBench 上既更准也更校准（Brier 0.095 对 0.297），但在 HaluEval 上 Brier 反而更差（0.245 对 0.176），错误排序却更好（AUROC 0.899 对 0.863）。只看 Brier 会选错门控信号。

GPT-6 is both more accurate and better calibrated on JudgeBench (Brier 0.095 against 0.297), yet has the worse Brier on HaluEval (0.245 against 0.176) while ranking errors better (AUROC 0.899 against 0.863). Reading Brier alone would pick the wrong gating signal.

温度缩放不通用

Temperature scaling does not port

在试点选择集上拟合出的温度分别是 0.65（RewardBench，锐化）、2.15（JudgeBench）、4.45（HaluEval）；同一组温度让 HaluEval 的 NLL 从 0.333 降到 0.284，却让 JudgeBench 与 RewardBench 变差。校准要按 workload 各自验证。

Fitted temperatures came out at 0.65 (RewardBench, sharpening), 2.15 (JudgeBench) and 4.45 (HaluEval). The same fit improved HaluEval NLL from 0.333 to 0.284 while making JudgeBench and RewardBench worse. Calibration needs per-workload validation.

批判性评估

Critical assessment

强证据 Strong evidence 协议在推理前冻结并划分选择集与测试集；benchmark 标签全程不改，人工修正只作敏感性分析；无效输出计为错误而不是剔除；配对差异用 2000 次按来源题目聚类的自助重采样；延迟另用一块冻结面板测量，避免客户端记账争用污染结论；判官分歧的 183 条由人工盲审逐条裁决。 The protocol froze and split the data before any inference. Benchmark labels are never changed and human corrections appear only as sensitivity analysis. Invalid outputs count as errors rather than being excluded. Paired differences use 2,000 source-question cluster resamples. Latency is measured on a separate frozen panel so client-side accounting cannot contaminate it, and 183 judge-disagreement items were adjudicated blind by a human.

中等或弱证据 Weaker evidence 所有级联结果都是离线模拟，论文明确说模拟费用不能说明真实级联延迟；单序级联的阈值是在被评分的同一批条目上读出来的（post hoc），只有冻结的双序策略是预先固定的规则；人工裁决由一位作者完成全部 183 条，且样本按判官分歧挑选而非随机抽样，因此是敏感性分析而不是共识；JEV 是闭源单版本，训练重叠未知，且推理力度、模型规模、服务实现都不同，所以这不是算力对齐的架构比较；扩展窗口在已知初始结果之后进行，多模型比较被作者自己标为探索性。 Every cascade result is an offline simulation, and the paper states outright that simulated fees say nothing about real cascade latency. The single-order thresholds were read off the items they score, so only the frozen two-order policies test a pre-specified rule. One author adjudicated all 183 items, selected by judge disagreement rather than at random, making it a sensitivity analysis rather than consensus. JEV is one proprietary version with unknown training overlap, and reasoning effort, model size and serving all differ, so this is not a compute-matched architectural comparison. The expansion happened after the initial results were known, which the authors label exploratory themselves.

结论是否超出证据 Does the claim exceed the evidence 标题级结论基本站得住：「3 个百分点以内、0.36% 的费用」跨过两套标签都成立，而且人工裁决的方向是让差距变大而不是变小，作者如实报告了。需要打折扣的是级联：99.6% / 47% 这组数字来自 post hoc 阈值的离线模拟，正式冻结策略的数字是 92.5% 对 93.1%、费用 56.8%，论文对此区分得很清楚，引用时应以后者为准。另外「六个判官在饱和控制上满分」这类结果被作者自己承认没有区分度。 The headline claim mostly holds: "within three points at 0.36% of the fee" survives both label sets, and the human adjudication moves the gap against JEV rather than in its favour, which the authors report plainly. The cascade is where to discount: the 99.6% / 47% figures come from an offline simulation with post hoc thresholds, while the pre-specified frozen policy reports 92.5% against 93.1% at 56.8% of the fee. The paper separates these clearly; cite the latter. The authors also concede that the saturated controls discriminate nothing.

论文自己的五条清单

The paper's own checklist

结论部分直接给了一份操作清单，几乎是全文最可搬运的部分：

The conclusion hands over an operational checklist, which is close to the most portable part of the paper:

1. 01

   配对判定跑两个顺序，并对齐概率后求平均

   Judge pairs in both orders and average the aligned probability

   倒序会翻转 3.25%（RewardBench）到 11.14%（JudgeBench）的决定，而首位偏好并不存在（选第一的比例 48.9% / 48.4%），说明这是真实的不一致而不是位置偏差。

   Reversal flips 3.25% (RewardBench) to 11.14% (JudgeBench) of decisions, and there is no first-position preference (48.9% / 48.4%), so this is genuine inconsistency rather than positional bias.

2. 02

   阈值在本地选择集上选，并在留出条目上复核

   Choose the threshold on a local selection set and re-check on held-out items

   论文里就有反例：GPT-5.6 的策略在 τ=0.7 时接受 81% 却掉了 2.35 分。

   The paper contains its own counterexample: the GPT-5.6 policy accepts 81% at τ=0.7 but loses 2.35 points.

3. 03

   比较判官时，把无效输出算作错误

   Count invalid outputs as errors when comparing judges

   调用失败不是推理错误，但它也不是一个判定。概率类指标与准确率的分母不能混用。

   A failed call is not a reasoning error, but it is no judgment either. Probability metrics and accuracy must not share a denominator.

4. 04

   置信度是升级信号，不是合格证书

   Treat confidence as an escalation signal, not a certificate

   要在文风对抗类输入上单独检查它，温度缩放也要按 workload 验证。JudgeBench 上 JEV 的高概率判定里有 9 条是错的。

   Check it specifically on style-adversarial inputs, and validate any temperature per workload. On JudgeBench, nine of JEV's high-probability judgments are wrong.

5. 05

   扩展到新 workload 之前先做小规模本地验证

   Run a small local validation before extending to a new workload

   边界不是按模型划的，是按任务类型划的；同一个判官在证据充足与无参考两种设定下是完全不同的两个工具。

   The envelope is drawn by task type, not by model. The same judge is a different tool with evidence at hand and without a reference.

对本机两个项目的意义

What it means for two local projects

这篇论文和我的两个本地项目正好对得上，一处是「同款引擎的第三方评测」，一处是「同一套门控思路的另一个应用场景」。

This paper lines up with two of my local projects: one is a third-party evaluation of the exact engine in use, the other is the same gating idea applied somewhere else.

Jev 备考：评审的正是它依赖的引擎

Jev Exam Prep: the engine it depends on, evaluated

这个项目用 TypeSafe JEV 逐得分点判定，而论文测的就是同一个服务。好消息是设计选择被独立验证了：每个得分点都绑定材料原文片段，正好落在「有证据支撑的事实性」这一格——JEV 在那里 87.5%，与最强判官持平。论文同时给出了两条必须避开的边界：不要把多步推导当成一个判定问题（推理分域 68.4 对 95.9），也不要在没有材料依据的情况下判分（无参考时所有判官都接近随机）。

That project grades every rubric point with TypeSafe JEV, and this paper measures the same service. The good news is that the design choice is independently validated: binding each point to a verbatim span puts it squarely in evidence-grounded factuality, where JEV scores 87.5% and ties the strongest judge. The paper also names two boundaries to stay inside: never ask a multi-step derivation as a single judgment (reasoning splits 68.4 against 95.9), and never grade without material grounding, where every judge falls to chance.

本地 Agent Harness：把 CANNOT_VERIFY 接到升级路径上

Local Agent Harness: wire CANNOT_VERIFY to a real escalation path

harness 的类型化决策里已经有 CANNOT_VERIFY，但它的下游目前更像是「标出来给人看」。论文给的是一条可量化的替代路径：便宜判定先跑，只在不确定时调强模型，并用「保留了多少准确率 / 花了多少费用」两个数字记账。它的校验器探针也可以直接吸收论文的对抗样本形态——写得漂亮但错误的东西，正是门控最容易自信放行的那一类。

The harness already has CANNOT_VERIFY among its typed decisions, but downstream it mostly means "flag it for a human". The paper offers a quantifiable alternative: run the cheap judgment first, call the strong model only when unsure, and record two numbers — accuracy retained and fee spent. Its validator probes can also absorb the paper's adversarial shape directly, since a well-written wrong artifact is exactly what a gate passes most confidently.

两个项目共用同一条结论：置信度不能当合格证书用，但可以当分诊台用。前提是自己测过它在你的任务上还灵不灵。

Both projects share one conclusion: confidence is not a certificate, but it is a triage desk — provided you measure whether it still works on your own workload.

这篇论文的价值不在于提出新算法，而在于把「便宜的判官够不够用」这件事变成一组可核验的数字，并且诚实地标出哪些数字是离线模拟。

The value here is not a new algorithm. It is turning "is the cheap judge good enough" into a set of verifiable numbers, and being explicit about which of those numbers are offline simulations.

一句话：任务自带参考答案或证据时，便宜判官够用；需要检查推导或抵抗漂亮错误时，必须升级；而置信度能不能当门控，要在你自己的任务上单独测一遍。 In one line: with a reference or evidence at hand the cheap judge is enough, checking a derivation or resisting a well-written wrong answer is not, and whether confidence works as a gate has to be measured on your own workload.

阅读原文（arXiv）Read the paper (arXiv) Jev 备考项目Jev Exam Prep 本地 Agent HarnessLocal Agent Harness