Notes ·
My model had no accuracy number, so I measured it
The ClaimLens NLI model was published with no benchmark behind it. On 3,000 VitaminC test pairs it scores 65.8 percent against 59.4 for a same-size zero-shot model (65.77 against 59.43 unrounded, a 6.3-point gain), and the evaluation found the app had swapped two of its three labels.
ClaimLens checks a paragraph by splitting it into claims, searching the web for evidence, and asking a fine-tuned DeBERTa-v3 model whether each snippet supports the claim, refutes it, or says nothing about it. I published that model on Hugging Face as Zulfhagez/claimlens-deberta-v3-nli. The project README said it beat the zero-shot facebook/bart-large-mnli model, and nothing anywhere backed that up: no benchmark, no model card, no accuracy figure. My own case file said so. In November 2026 I measured it.
Picking a test that matches the job
The model sees one claim and one evidence sentence and returns SUPPORTED, REFUTED or NOT ENOUGH INFO (NEI). VitaminC (Schuster et al., NAACL 2021) has exactly that shape: a claim, one Wikipedia sentence, and those three labels. Many of its pairs are contrastive: the same claim against a revised sentence that flips the answer, which is how web evidence goes wrong in practice.
I took a simple random sample of 3,000 pairs from the 55,197 in the test split, with seed 42 and no stratification, so the class mix is the natural one: 1,556 SUPPORTED, 1,040 REFUTED, 404 NEI. That size fitted a CPU budget; the whole run took 15 minutes. Every interval below is a 95 percent percentile bootstrap over the examples, 1,000 resamples.
The headline
| Model | Accuracy (95% CI) | Macro F1 (95% CI) |
|---|---|---|
| ClaimLens model | 65.8% (64.0 to 67.5) | 0.625 (0.607 to 0.642) |
cross-encoder/nli-deberta-v3-base, zero-shot, same size |
59.4% (57.8 to 61.1) | 0.572 (0.555 to 0.589) |
facebook/bart-large-mnli, zero-shot |
59.1% (57.5 to 60.9) | 0.561 (0.544 to 0.578) |
| Always SUPPORTED | 51.9% (50.1 to 53.6) | 0.228 |
The paired difference against the same-size zero-shot model is 4.5 to 8.0 accuracy points, and against bart-large-mnli 4.7 to 8.4. Neither interval touches zero, so the README's claim holds on this benchmark, by about 6 to 7 points. That is a smaller and more specific statement than "beats", and it is the one I can defend.
The bug the numbers found
The evaluation also found a bug. The model's own config.json on the Hub says index 0 is SUPPORTED, 1 is NEI and 2 is REFUTED. The ClaimLens code, and the README table copied from it, decoded 1 as REFUTED and 2 as NEI. The two maps agree on SUPPORTED and swap the other two.
A label map is a fact you can test, so I decoded the same 3,000 sets of logits both ways:
| Decoding | Accuracy | Macro F1 |
|---|---|---|
| Hub config order (0 SUPPORTED, 1 NEI, 2 REFUTED) | 65.8% | 0.625 |
| The app's old order (0 SUPPORTED, 1 REFUTED, 2 NEI) | 51.1% | 0.405 |
Under the old order, 534 gold REFUTED pairs come out as NEI and 343 gold NEI pairs come out as REFUTED: the signature of a swap, not of a weak model. The app's order scores below always answering SUPPORTED. The config was right and the code was wrong.
In the running app that meant every time the model meant NEI, the vote counted as REFUTED, and every time it meant REFUTED, it counted as NEI. The model says NEI a lot (more on that below), so the app turned many "not enough evidence" cases into refutations. A comment in the verifier said DeBERTa "commonly misclassifies as REFUTED" near-miss numbers. That observation was most likely this swap.
The fix: the verifier now builds its label map from the checkpoint's own id2label when it loads, so a retrain with a different order cannot swap verdicts silently, and a regression test pins the published order. The README table is corrected, and the repository now has a model card.
Where it fails
Per class, the model is precise when it commits: 0.895 precision on SUPPORTED and 0.887 on REFUTED. Its weakness is the third label. It predicted NEI 1,173 times against 404 true NEI, and 830 of its 1,027 errors are SUPPORTED or REFUTED pairs it called NEI. When the evidence names a different airline or conference than the claim, it tends to treat that as missing information rather than a contradiction. In the pipeline this is the cautious direction, and it is what one of the LLM cross-check triggers (NEI with high evidence relevance) exists to catch.
Two slices were striking, with a warning attached. On real Wikipedia revisions the model scores 64.4 percent against 52.4 and 50.4 for the zero-shot models, and on claims containing a digit 66.1 against 52.3 and 50.3. On synthetic contrastive edits it is behind them (68.1 against 71.4 and 73.9), and on claims without digits it is level. So the whole advantage comes from real revisions and numbers. I chose these slices after reading the failures, so they describe this sample; they are not tested hypotheses.
What this number does not say
- The training data is not documented. The Hub repository had no card and the training notebook is not in the repository, so overlap between the training set and VitaminC cannot be ruled out. If the model was trained on VitaminC or FEVER, this is an in-distribution test.
- It measures the model alone on one evidence sentence. ClaimLens votes over up to five web snippets and then applies three gates and an LLM cross-check, so the pipeline's accuracy will differ.
The method, the slices, the ten failures and the commands to reproduce the run are in eval/README.md, and the model card is in MODEL_CARD.md, both on the eval-2026-11 branch.