Looking for one independent annotator: our LLM raters agree with each other at chance level and we can't tell why

🤗Datasets
1

Short version of the ask: I have 100 scenes, a locked set of human reference labels, and five machine raters that disagree with the human and with each other at roughly chance level. I cannot tell whether that is because the task is genuinely interpretive or because my rule definition is bad. One more independent human annotator would settle it, and I would publish the result either way. Details below in case anyone wants to shoot holes in the setup first — which would also be useful.

Disclosures up front: I designed the annotation scheme being tested, so I am not neutral about the outcome. I was also the sole human rater in the first of the three studies, which is why the second study uses someone else. And this post was drafted with help from Claude, which is one of the four LLMs whose scores are reported — read the framing accordingly. The numbers themselves come from a deterministic script over published label files.

The setup

The corpus is a bilingual Turkish–English set of purpose-written narrative scenes, annotated for six craft features. Two of the six are prohibitions that reduce to surface patterns (no named emotion; no simile). Four require judgement. The one that matters here is materialized metaphor: an abstract inner state rendered as a concrete physical detail rather than named.

100 held-out Turkish scenes were labelled by an independent volunteer with no involvement in the methodology. Those labels were locked to an immutable file before any model was run. Five raters were then scored against them: a deterministic rule-based detector and four LLMs, each given identical instruction blocks.

The result

Human reference: 9 positives out of 100.

Rater Positives Cohen’s κ
Machine A 0 0.000
Machine B 1 0.185
Machine C 40 0.019
Rule-based detector 72 0.015
Machine D 78 0.027

Machine B’s 0.185 comes from one positive call that happened to land, so I do not read it as a shared criterion.

Raw agreement across the five was 74.7%–86.3%, which looks respectable and means nothing here: the class distribution is extreme on four of six features, so majority-class guessing scores well. Classic first kappa paradox.

One feature is well conditioned — atmosphere contradiction, human at 44/100. There two raters clear chance, at κ = 0.184 and 0.269. That is the best coefficient anywhere in the three studies, and it is still weak.

Where I am stuck

Two readings fit the data equally well:

(a) The feature is inherently interpretive; there is no single correct label, so disagreement is signal about the task, not failure by the raters. (b) My definition is underspecified — clear to me, underdetermined for anyone else.

I would prefer (a). I am not entitled to prefer it, and with one human rater per study I cannot distinguish them. Two humans agreeing with each other and diverging from the machines points at (b) and is repairable. Two humans failing to agree points at (a), which is a substantive finding about what can be delegated.

The ask

If anyone is willing to label the same 100 scenes against the same definitions, everything needed is public:

  • scenes and locked reference labels: evaluation/ in leventbulut/objective-projection
  • the ten verbatim instruction blocks each rater received
  • a dependency-free scoring script, plus the seeds and the masked ID mapping

You would get named acknowledgement in the resulting record, and the result gets published whichever way it falls — including if it shows my definitions are the problem, which is the outcome I would bet on.

Two secondary questions I would also welcome answers to:

  1. For a feature with 9% prevalence, is there a better agreement statistic than Cohen’s κ that you actually use in practice? I know the literature on prevalence-adjusted variants; I am asking what people run in real annotation projects.
  2. Has anyone here systematically compared the same annotation prompt across model families and found this kind of range, or is the spread here a symptom of the rule rather than of the models?

Links

Paper: 10.5281/zenodo.21740239 (also arXiv:2609.13936) · Dataset: 10.57967/hf/8960

Full confusion matrices are in the paper. Two of the four LLM raters’ per-scene files were lost and their counts were recovered arithmetically from the surviving κ and raw-agreement values; those are marked as derived rather than raw, in the paper and in the repo.

2

I really appreciate that the whole setup — scenes, locked labels, instruction blocks, scoring script — is public down to the file. That’s rarer than it should be.

I’ve got a small tool built around exactly this question — is a categorical predictor actually tracking ground truth, or is the agreement number just an artifact of a skewed base rate. It grades that relationship (chi-square + Cramér’s V, A-F) rather than a single agreement coefficient. It gew out of validating classifiers against physical ground-truth data in a completely different field, but the math doesn’t know or care what the categories mean.

Happy to run your five raters against your locked labels and post the per-rater grades back here, alongside whatever your second annotator turns up …it could be a useful cross-check either way.

No strings, just curious whether it converges with what you find.

Hope the annotator search goes well.

3

Thank you — genuinely. An independent cross-check from outside the project is exactly what this needs, and no strings on my side either: whatever grades come back get reported alongside my own numbers, including if they cut against my reading.

A few things worth knowing before you run it:

  1. There is no second human annotator yet. That request is still open, so for now the comparison is five raters against a single locked human reference. Your check stands on its own and doesn’t depend on it.

  2. Two of the five raters’ counts are derived, not raw. Per-scene label files exist for the rule-based detector and for machine raters C and D. For A and B the per-scene files were lost, and their confusion counts were recovered arithmetically from the surviving per-rule kappa and raw agreement values (unique solution for 11 of 12 cells). Since chi-square only needs the 2x2 table that shouldn’t block anything, but it’s worth flagging next to those two grades.

  3. A question about the edge case. On materialized metaphor the human reference has only 9 positives in 100, and machine rater A returned 0 positives — an empty row and expected counts well below 5. How does the tool grade that: does it fall back to Fisher’s exact test, or mark the cell as ungradeable? I ask because in a 2x2 table Cramér’s V reduces to |phi|, which is itself prevalence-sensitive, and I’d like to be sure we aren’t swapping one base-rate artifact for another.

  4. It would help readers of this thread if you could post the grade boundaries alongside the grades — what separates an A from a C — so the letters can be read without access to the tool.

The feature where your approach may say the most is atmosphere contradiction: the human reference sits at 44/100 there, so the table is well conditioned, and two raters clear chance on kappa (0.184 and 0.269). I’d be very interested to see whether your grades rank the five raters the same way kappa does, or differently.

Everything is in the evaluation/ folder of the dataset. And if you’d like to be acknowledged by name or handle in the dataset README and the next version of the paper, just say which — or say so if you’d rather not be named.

Thanks again.

4

Just a quick heads up, I could only grade three of the 5 raters- the detector, ChatGPT 5.5, and Claude Fable 5. Gemini and Grok don’t have per-scene files in the repo, just the aggreagate numbers from your write up, and my tool needs the actual scene-by-scene calls to build a contingency table.

Grade Boundaries: A : V >.30 (strong); B: > .20 (moderate); C: > .10 (weak); D: >.05 (very weak). F means no signal and we had enough power to trust that. I means no signal but we didnt have enough power, in other words, we can’ t tell.

At n = 100 the tools ower to catch even a small effect (V = .10) is only about 17% so most nonsignificant results in this dataset land I rather than F.

On materialized metaphor — the one you actually asked about — all three raters graded I on chi-square, but I also ran Fisher’s exact on the side, since it doesn’t need the minimum-cell-count assumption chi-square does, and 9% prevalence is exactly the situation where that assumption gets shaky. Fisher agrees with the I’s: p = 1.000, 1.000, and 0.679 for detector, ChatGPT, and Claude. So this isn’t “the test couldn’t tell” — it’s “the test could tell, and there’s nothing there.” None of the three track your human reference on this feature, at least at this sample size.

One wrinkle worth mentioning: on micro-focus, ChatGPT also graded I on chi-square, but that one turned out to be my tool being overly cautious rather than a real non-finding — the table was skewed enough (92:8) that my minimum-expected-cell guard tripped and refused to trust its own p-value. Fisher’s exact doesn’t have that problem and finds a real effect there: p = 0.031, OR = 15. So ChatGPT’s micro-focus labels probably do track your reference — my chi-square just wasn’t the right tool for that particular table.

And on atmosphere contradiction, the well-behaved one: Claude Fable 5 grades a B (V=0.255, p=0.011, Fisher backs it up at p=0.008). ChatGPT doesn’t quite clear significance either way (chi² p=0.101, Fisher p=0.071 ). So for what it’s worth, my grades rank the two the same way your kappa numbers do — Claude ahead of ChatGPT — on this one feature.

If you’d like I can share the script if you want to poke at it by yourself.

Happy to be credited either way, my name or CONFIRM both work. If it’s useful, I wrote up the same grading approach in more depth on Zenodo (10.5281/zenodo.22287620). It applied the chi-square and Cramér’s V grading to 174 RewardBench 2 reward models instead of annotation data, but it’s the same threshold table I used above. Worth flagging, TraceSeis, the company behind CONFIRM, is commercializing it, so that’s a disclosed interest on my end.

5

This is genuinely useful, and one part of it changes something in my write-up, so let me be specific about what I’m taking from it.

Where it supports what I reported. On materialized metaphor, Fisher at p = 1.000, 1.000 and 0.679 reaches the same conclusion my kappa values did, by a route that doesn’t share kappa’s weakness at low prevalence. That matters more than agreement usually does, because “kappa is uninformative at 9% prevalence” was the strongest available objection to my central finding, and your test isn’t vulnerable to it. Same for the ordering on atmosphere contradiction: your grades put Claude ahead of ChatGPT, and so did kappa (0.269 vs 0.184). Two different statistics, same rank.

Where it corrects me. Micro-focus is the interesting one. My framing was that the four inference-requiring features all failed to transfer. Your Fisher result for ChatGPT (p = 0.031, OR = 15) says that is too broad. Looking back, my own kappa for that cell was 0.296 — the highest in the whole study — so the data was already saying it and my summary flattened it. The accurate statement is narrower: materialized metaphor fails across every rater tested; micro-focus shows a real association for at least one rater. I’ll correct that in the next version of the paper and in the dataset README, with attribution.

On the power point. 17% power to detect V = 0.10 at n = 100 is a limitation I had not stated anywhere, and it should be stated. I’d rather phrase the materialized-metaphor result as “no association detectable at this sample size” than as “there is nothing there” — non-significance isn’t absence, even at p = 1.000 — but either way the conclusion for a practitioner is the same, and the power figure belongs in the limitations section.

On the two you couldn’t grade. That one is on me. The per-scene files for those two raters were lost and only the aggregate confusion counts survive, recovered arithmetically. It was declared in the paper, but your run shows the concrete cost: it blocked an independent check. Every future study will ship per-scene files.

Yes please to the script — I’d like to run it myself and, if you’re happy with it, mirror it in the evaluation/ folder alongside my own scoring script, credited to you.

On credit: I’ll cite both your name and CONFIRM, and I’ll state the TraceSeis commercial interest explicitly wherever these results appear. Thank you for flagging it unprompted. I’ll also cite the Zenodo write-up (10.5281/zenodo.22287620) for the threshold table.

Three small questions so I represent your method correctly:

  1. For micro-focus and atmosphere, is the contingency table a plain 2x2 presence/absence against the locked human labels, or is anything collapsed?
  2. Do you report Cramér’s V with a confidence interval? At these table sizes a point estimate alone may overstate precision, and I’d rather include the interval if you have it.
  3. Would you want to be listed as the analyst for this cross-check in the next version, or credited in the acknowledgements only? Either is fine by me.

One thing your analysis doesn’t settle, and I want to be clear I’m not claiming it does: whether the failure on materialized metaphor means the feature is inherently interpretive or my definition is underspecified. That still needs a second independent human annotator. Your result does remove one alternative explanation, though — it isn’t an artifact of the agreement coefficient.