Can Less Context Beat Full Context? A 600-Case Test of Bridge-Aware Routing

Research
1

I consolidated two fixed 300-case runs into a 600-case frozen Qwen2.5-7B-Instruct evaluation on disjoint MuSiQue 2-hop training-split slices.

This is an inference boundary test, not a new QLoRA training result. Qwen and the MiniLM selector were frozen. The selector policy was fixed before these two runs: a guarded semantic core, followed by promotion of two inactive paragraphs that appear to connect the question and selected core evidence.

I compared:

  • Full capped context

  • Semantic core plus three matched random-promotion controls

  • BM25 lexical selection at the same per-example paragraph budget

  • Semantic bridge promotion

  • Gold-evidence-only, as a diagnostic ceiling rather than a fair-cost baseline

Combined results:

Policy F1 EM Input tokens Prefill Latency All gold evidence retained
Full context 0.3381 0.2283 2443 147.9 ms 215.4 ms 100.0%
BM25, matched budget 0.3113 0.2033 2048 125.8 ms 196.7 ms 75.7%
Semantic core + random promotion, mean of 3 seeds 0.3310 0.2233 2024 124.3 ms 195.4 ms 90.6%
Semantic bridge promotion 0.3482 0.2383 2038 125.2 ms 194.8 ms 95.2%
Gold evidence only — diagnostic 0.6254 0.4733 329 35.1 ms 90.6 ms 100.0%

Paired bootstrap comparisons over all 600 cases:

  • Bridge vs mean matched-random: F1 delta +0.0172, 95% interval [+0.0029, +0.0317].

  • Bridge vs BM25: F1 delta +0.0368, 95% interval [+0.0146, +0.0594].

  • Bridge vs BM25: EM delta +0.0350, 95% interval [+0.0117, +0.0583].

  • Bridge vs full context: F1 delta +0.0101, 95% interval [-0.0097, +0.0305].

Relative to full context, bridge promotion used 16.6% fewer input tokens, reduced prefill time by 15.3%, and reduced end-to-end latency by 9.6%. The fixed guard used the base budget for 246 cases, expanded budget for 335 cases, and full fallback for only 19 cases.

My cautious interpretation is:

  1. Generic shortening is useful, but it does not preserve dependency-bearing evidence as reliably as bridge-aware promotion.

  2. Bridge promotion outperformed both the matched random controls and the matched-budget BM25 baseline in this fixed evaluation.

  3. Its difference from full context is not conclusive, so I do not claim that less context is generally better than full context.

  4. The gold-evidence ceiling remains far higher than every realistic policy. That suggests the remaining bottleneck is still evidence selection and downstream composition, not merely reader capacity.

I would welcome criticism of the benchmark design, especially whether a different lexical baseline or an independent long-context multi-hop benchmark should be the next validation step.
This is an expanded follow-up to my earlier exploratory work on long-context selection. Readers interested in the earlier QLoRA pilot, the initial matched-random controls, and how the design evolved can find the prior experiments in the earlier updates of this thread and on my profile.

2

Solid write-up, the paired bootstrap intervals and the matched-random control specifically are what make this credible rather than just another “we shortened context and got a similar score” post. Two things worth pushing on: with F1 deltas this tight (bridge vs full context is +0.0101 with a CI crossing zero), 600 cases might still be underpowered to separate “roughly equivalent” from “slightly worse,” MuSiQue 2-hop dev/test in addition to train-split slices would help confirm this isn’t overfit to whatever quirks exist in the training split specifically. Second, the gold-evidence ceiling (0.63 F1) vs everything else (~0.33-0.35) is the most interesting number in the table honestly, that ~0.28-0.30 F1 gap between gold and your best realistic policy is huge and suggests your selector’s real bottleneck isn’t paragraph-count budget at all, it’s precision of what counts as “core” evidence before promotion even kicks in.

On your actual question, an independent benchmark matters more than a different lexical baseline here since BM25 already did its job as a sanity-check floor. HotpotQA’s bridge-question subset would be a natural next step since it’s explicitly labeled by bridge-entity type rather than inferred, that’d tell you if the promotion policy is actually detecting bridge structure or picking up something correlated with it in MuSiQue specifically.

3

Thank you — this is a very helpful reading of the result.

I agree that the bridge-versus-full comparison should be treated as roughly equivalent within the current uncertainty, not as evidence that bridge is generally better than full context. The 600-case result is enough to separate bridge from the matched random and BM25 controls in this internal protocol, but not to make a broad full-context superiority or non-inferiority claim.

I also agree that the gold-evidence ceiling is the most important remaining diagnostic. The gap between gold-only F1 and the realistic policies suggests that the reader can use the evidence when it is available; the next bottleneck is improving the precision and recall of the initial semantic core, rather than simply changing the paragraph budget.

I will therefore prioritize an independent benchmark before adding more lexical baselines. HotpotQA bridge questions sound especially appropriate because their bridge structure is explicitly annotated rather than inferred from MuSiQue support labels.

For that evaluation, I plan to keep the current policy fixed and report the same chain separately: core evidence retained, evidence restored by promotion, final answer F1/EM, input tokens, and latency. That should make it clearer whether the policy is detecting bridge structure itself or exploiting something specific to the MuSiQue setup.

Thank you again for pointing to the right next boundary.