Papers
arxiv:2611.23824

Evidence-Based Credit Assignment for Search-Augmented Language Models

Published on Nov 18
· Submitted by
Xingye Wang
on Nov 19
Authors:
,
,
,
,
,

Abstract

Reinforcement learning from terminal answer rewards enables search-augmented language models to interleave reasoning and retrieval without step-level annotations, yet outcome supervision assigns identical credit to every search in a trajectory regardless of whether it retrieved useful information. This limitation is especially severe when all sampled rollouts for a question fail, collapsing group-centered outcome advantages to zero even when trajectories gather markedly different evidence. We present Evidence-based Credit Assignment (ECA), which augments terminal outcome rewards with a training-only auxiliary signal measuring how retrieved documents improve the likelihood of a reference answer. Rather than scoring the full dialogue prefix, ECA queries a frozen snapshot of the current policy under an evidence-only prompt containing solely the question and the deduplicated documents observed so far, isolating the utility of retrieved passages from surface variations in generated thoughts or search queries. Within this framework, we compare two return formulations, ECA-Step (which accumulates step-clipped potential increments) and ECA-Horizon (which preserves the telescoping horizon gain in potential), paired with cross-fold group normalization, novelty gating, and search-local token assignment. Across seven Search QA benchmarks, ECA-Horizon improves strict macro exact match over an outcome-only GRPO baseline by 1.03 and 4.80 percentage points on Qwen2.5-3B- and 7B-Instruct, respectively, outperforming both RLOO and GiGPO. Zero-shot evaluation on MultiHop-RAG further demonstrates strong transfer to multi-hop inference queries.

Community

Paper author Paper submitter

We propose Evidence-based Credit Assignment (ECA), a training-only RL framework for search agents that scores how retrieved documents alone improve reference-answer likelihood—recovering dense step-level search credit even when all rollouts fail and suppressing redundant queries with zero inference overhead.

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Giving every search the same credit whether or not it found anything is paying every sub the same when the job goes wrong, including the one who did his part right. Scoring what each lookup added to the evidence is how a foreman would sort out a bad day. I'd be curious whether the model learns to stop searching sooner once it has enough.

·

Love the foreman analogy—that captures the exact motivation behind ECA!

To answer your question: under a fixed 3-search budget, we found that all RL-trained models still tend to use all three search turns, but what changes dramatically is how they spend those turns.
Under outcome-only group methods like GRPO and RLOO, the model often pads the budget by simply repeating its earlier query (at 3B, episodes with duplicate queries jump to 42.4% under GRPO and 54.1% under RLOO). With ECA and our novelty gate, query repetition drops down to 24.8% (Figure 4). Instead of issuing the same query over and over, the agent learns a progressive retrieval pattern: as we show in the qualitative examples in Appendix E, it uses the first search to uncover an intermediate bridge entity or disambiguate a timeframe, and then conditions its next query on what it just found to hop to the final answer—turning wasted repeat lookups into genuine multi-step exploration.

Sign up or log in to comment

Get this paper in your agent:

hf papers read 2611.23824
Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2611.23824 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2611.23824 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2611.23824 in a Space README.md to link it from this page.

Collections including this paper 0

No Collection including this paper

Add this paper to a collection to link it from this page.