Papers
arxiv:2610.27597

WORTH Benchmark: Evidence Discovery under Not-Fully-Cooperative Information -- A Two-Episode Pilot

Published on Oct 20
· Submitted by
Bo Zou
on Oct 26
Authors:

Abstract

The WORTH Benchmark evaluates autonomous evidence discovery and reasoning in mystery narratives, revealing that models select clues and participants inconsistently due to narrative structure.

Existing reasoning evaluations typically provide task-relevant information to models as available premises, rarely examining concurrently how models discover evidence autonomously and how information presentation within narratives guides or obscures judgment. This paper proposes the WORTH Benchmark: models read original mystery narratives without clue annotations, autonomously select key clue sentences, identify case participants, and explain their reasoning process. The narrative materials are designed by professional authors such that evidence facilitating reasoning coexists with narrative arrangements that delay the revelation of truth. We report a pilot experiment under full-story conditions across two episodes, conducting a total of 70 formal runs on models, of which 67 runs are content-scorable. Scoring separately records hits on author-designated key clue sentences and participant selections; natural language explanations are used for case diagnosis and are not scored independently. The results show that clue hits and participant judgments are not synchronized; across 37 content-scorable runs in Episode 02, 31 runs simultaneously selected the same set of incorrect participants, while adjacent suspicion-oriented and exoneration-oriented sentences were selected 25 times and 0 times, respectively. These phenomena illustrate the diagnostic value of component-wise analysis and suggest that not-fully-cooperative informational relationships may influence evidence selection and judgment. This study only implements full-story conditions for two episodes, and does not establish stable model rankings or infer general mechanisms on this basis; human player records have not yet been cleaned, and this version does not report human-model comparisons.

Community

Paper author Paper submitter

WORTH tests whether models can find evidence on their own, not just reason from evidence they are given. Models read an original mystery story with no clues marked, pick the key clue sentences, and decide who took part in the homicide. The two parts are scored separately, and they do not move together: in Episode 02, 31 of 37 scorable runs picked the same wrong set of people. The stories were written by professional authors, so the evidence and the narrative misdirection sit in the same text.

You can try the Episode 01 task yourself (English version, no models run): https://huggingface.co/spaces/BreadStudio/worth-ep01-en

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2610.27597
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2610.27597 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2610.27597 in a dataset README.md to link it from this page.

Spaces citing this paper 1

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.