For now, I tried running an experiment in Colab:
I kept the order you asked for: canonical v1.0 first, modification second.
I also made the first experiment deliberately narrower than a real web-research run: same model, same question, and a fixed evidence packet with no live retrieval. The idea was to separate:
- what the Research Pass does with evidence that is already available, from
- whatever differences happen to come from search/retrieval.
I found one reproducible boundary that looks more like an execution/output-layer effect than an evidence-judgment failure.
Short version
The main run used:
- Qwen3-8B, 4-bit
- Colab L4
- non-thinking mode
- 3 repeats
- canonical Research Pass pinned to this v1.0 commit
- fixed evidence, no web access during the test
I first used five small evidence packets covering things like insufficient evidence, source lineage, conflicting sources, and a polished-but-weak source.
On those deliberately explicit packets, the interesting case was the source-lineage one.
The canonical Research Pass correctly recognized the underlying evidence structure, but under a 256-token output budget it kept expanding the report structure and ran out of output before finishing the answer.
So I isolated that one case and reran it with larger budgets:
| Variant |
Natural EOS |
Hit output limit |
Mean output |
| Canonical RP, 256 tokens |
0/3 |
3/3 |
256 |
| Canonical RP, 512 tokens |
0/3 |
3/3 |
512 |
| Canonical RP, 768 tokens |
0/3 |
3/3 |
768 |
| RP + experimental answer-first/stop rule, 256 tokens |
3/3 |
0/3 |
~189 |
The useful part is that the 256/512/768 canonical runs used the same prompt and paired seed.
For all three repeats, the 512- and 768-token runs reproduced the entire first 256 generated tokens of the 256-token run exactly, then continued.
So this was not just three unrelated generations that happened to be verbose.
By 768 tokens, all three canonical runs had reached the right substantive conclusion:
- D1–D4 were not four independent confirmations;
- they were one derivative source lineage;
- D5 was genuinely independent counter-evidence;
- the claimed 30% reduction was therefore not independently supported by that packet.
But the model then continued through later Research Pass-style sections until it hit 768 tokens.
That suggests two things can be separated here:
evidence reasoning
↓
was basically right
output/report execution
↓
did not naturally stop
I would not generalize that to “Research Pass is too verbose.” In another simple fixed-evidence case, the same canonical prompt answered in about 32 output tokens.
This looks more conditional: on this model and this source-lineage case, the full-report structure exerted more pressure than the instructions to compress/omit low-value sections and stop at decision sufficiency.
The experimental modification was also intentionally small. I appended roughly:
Begin with a "Bottom line" section and answer the actual question
in no more than three sentences.
State:
1. whether the 30% claim has independent support,
2. whether D1-D4 are independent confirmations or one source lineage,
3. what D5 contributes.
Only then continue with additional Research Pass analysis if it is useful.
That condition produced the correct bottom line and naturally terminated in 3/3 runs, using 145–221 output tokens.
There is an important caveat: that is not a pure answer-ordering ablation. It changed three things together:
- answer first;
- short initial answer;
- explicit permission to stop rather than complete a large report.
So I would only treat it as a promising implementation option, not evidence that “answer-first alone” is the causal fix.
A cheap control that might make future A/B results easier to interpret
I would keep your current canonical-first testing protocol exactly as the first stage:
Normal
vs
Canonical Research Pass
If that produces an interesting difference, one extra arm seems unusually informative:
Normal
vs
Compact research instruction
vs
Canonical Research Pass
The compact arm should not secretly reproduce the entire Research Pass. Something roughly at this level would be enough:
Research this carefully using the available tools where useful.
Challenge important assumptions, verify consequential claims,
consider plausible alternatives and counter-evidence, and stop
when further work is unlikely to change the answer.
Keep the effort and final response proportional to the question.
Then the branches become easier to read:
Canonical > Normal
|
+-- Canonical ≈ Compact
| -> much of the gain may be generic "research harder" elicitation
|
+-- Canonical > Compact > Normal
| -> the detailed RP structure may be adding something further
|
+-- Compact > Canonical
| -> look at routing, instruction competition, output shape,
| stopping, or research overhead
|
+-- all roughly equal
-> try a messier question where reframing, vocabulary discovery,
or search-space expansion actually matters
I mean this as a second-stage diagnostic control, not a replacement for your Normal-vs-Canonical test.
Exact fixed-evidence experiment and reproduction notes
The source-lineage case
The question was:
How much independent support is there here for the claim that Alloy K reduces fatigue failures by 30%?
The complete evidence packet was intentionally tiny:
SOURCE D1 — Vendor press release P-19
Claims Alloy K reduces fatigue failures by 30%.
No underlying dataset is included.
SOURCE D2 — Industry Blog A
Repeats the 30% figure and links to Vendor press release P-19
as its source.
SOURCE D3 — Trade Newsletter B
Repeats Blog A's 30% figure; its only evidence link is Blog A.
SOURCE D4 — Vendor FAQ
Repeats the 30% figure and cites press release P-19.
SOURCE D5 — Independent replication report R-8
A separate 200-sample test did not find a statistically significant
reduction in fatigue failures.
The intended evidence distinction was simply:
D1
├── D2
│ └── D3
└── D4
D5 <- independent branch
So four documents did not mean four independent confirmations.
This was useful because it directly exercises several things already present in the canonical Research Pass:
- source lineage;
- independence;
- discovery vs proof;
- counter-evidence;
- contradiction handling;
- stopping / decision sufficiency.
Broader five-case probe
Before focusing on that one case, I ran five fixed-evidence cases with three conditions:
- Normal
- Compact research control
- Canonical RP
The cases covered:
- a simple sufficient fact;
- an event where the evidence did not identify the cause;
- derivative consensus/source lineage;
- preliminary vs final conflicting evidence;
- a polished secondary source conflicting with stronger primary material.
After reading the actual outputs rather than relying on a simple keyword scorer, the substantive result was approximately:
| Arm |
Human-reviewed substantive completion |
Avg input tokens |
Avg latency |
| Normal |
15/15 |
~262 |
~6.5 s |
| Compact |
15/15 |
~315 |
~6.7 s |
| Canonical RP |
12/15 fully completed |
~5,872 |
~16.9 s |
The three incomplete canonical runs were all the same source-lineage case.
This is not a benchmark score and I would not use it to say Normal or Compact is generally equivalent to the Research Pass.
These packets were deliberately explicit. They do not exercise what may be one of the more interesting RP use cases: starting with a messy question and discovering better framing, terminology, disciplines, hypotheses, or search directions.
The simple fact case is also a useful counterexample to a blanket over-research interpretation: canonical RP routed that one to a short answer.
Focused budget probe
I then isolated only the source-lineage case.
| Variant |
Natural EOS |
Limit hits |
Mean output tokens |
Mean latency |
| Canonical 256 |
0/3 |
3/3 |
256 |
~23.4 s |
| Canonical 512 |
0/3 |
3/3 |
512 |
~42.9 s |
| Canonical 768 |
0/3 |
3/3 |
768 |
~63.0 s |
| Answer-first + stop, 256 |
3/3 |
0/3 |
~189 |
~17.5 s |
Within each repeat, canonical 256/512/768 had:
- the same model;
- the same evidence;
- the same system prompt;
- the same sampling seed.
The first 256 generated tokens matched exactly between the 256-, 512-, and 768-token versions in all three repeats.
That makes the interpretation fairly clean:
Increasing the budget repaired answer reachability, but did not repair natural termination.
At 768 tokens the substantive evidence judgment was already there, but the output continued.
The canonical prompt itself contains both pressures:
- a Full Research Pass “should normally report” a long list of possible sections;
- it also explicitly says to compress or omit sections that add no value and to stop at saturation + decision sufficiency.
In this one Qwen3-8B case, the report scaffold seems to have been followed more strongly than the compression/stopping instructions.
That is why I currently think doctrine vs execution/output architecture is the most useful separation here.
It also seems to fit naturally with several ideas you are already tracking in EXPERIMENTAL_HYPOTHESES.md, especially the doctrine/execution separation and the compression/modularization questions.
Exact model-side setup
I used Qwen3-8B in non-thinking mode with the model card’s suggested non-thinking sampling:
temperature = 0.7
top_p = 0.8
top_k = 20
min_p = 0
The fixed-evidence experiment did not give the model browsing tools, so none of these results should be read as a test of RP’s real-world retrieval quality.
Evaluation controls that seem useful for separating causes
One thing I found helpful while trying to interpret the result was to avoid treating “research quality” as one scalar.
For RP testing, four layers seem separable:
| Layer |
Example questions |
| Process |
Did it generate alternatives, check lineage, look for counter-evidence, stop appropriately? |
| Evidence state |
Was the relevant evidence actually available/retrieved? Was it independent? Conflicting? Sufficient? |
| Outcome |
Is the conclusion supported? Are consequential facts right? Did it abstain when necessary? |
| Presentation / cost |
Is the answer visible? Did useful reasoning get buried? Tokens, latency, tool calls? |
That distinction also appears in several nearby deep-research evaluation projects.
LiveDRBench / Characterizing Deep Research is particularly interesting here because it explicitly separates the underlying claim-discovery/search problem from surface-level long-report generation.
DeepResearch Bench II similarly evaluates long-form research across separate dimensions for information recall, analysis, and presentation.
And BrowseComp-Plus uses a fixed corpus specifically so retrieval effects can be separated from the rest of the agent.
That suggests two complementary RP test modes rather than one replacing the other:
LIVE WEB
"What happens for a real user with the whole system?"
FROZEN EVIDENCE
"Given the same evidence, what part of the protocol changes behavior?"
The Colab run above is only the second kind.
Stopping / answerability
The stopping rule also looks worth testing as its own capability, because it can fail in two directions:
Decisive evidence exists
|
+-- find it and stop
| -> useful stopping
|
+-- find it, then keep searching without changing the decision
-> possible over-search / cost boundary
Decisive evidence does not exist
|
+-- report insufficient / not verifiable
| -> calibrated stopping
|
+-- keep searching until a firm-looking answer appears
-> evidence-sufficiency boundary
This is already part of RP’s doctrine, so I do not mean “add a stopping rule.”
I mean it may be useful to test that rule independently.
A nearby benchmark is Over-Searching in Search-Augmented Large Language Models / OverSearchQA, which explicitly studies search behavior on answerable vs unanswerable questions.
A cheap RP version would just use a paired question:
A — answer is actually recoverable
B — crucial evidence genuinely does not exist in the available environment
and observe whether additional research is still decision-relevant.
Misleading evidence
Another compact stress pattern would be:
CLEAN:
two independent indirect evidence chains
NOISY:
the same evidence
+
one polished document that directly states a conflicting answer
Then watch whether the direct-looking source overrides the underlying evidence chain.
DRNOISE uses a related paired setup for deep-research agents.
I would not infer from that paper that Research Pass has the same failure mode. It just looks like a very economical test of RP’s existing:
- source-lineage rule;
- directness-vs-rigor distinction;
- contradiction handling;
- failure search;
- stopping rule.
“Skeptic” vs verification
I would also keep two events separate in test reports:
the model generated a criticism/counter-hypothesis
and
the model found evidence that actually verified or falsified it
The first can still be useful for expanding the hypothesis space, but it is not automatically the second.
A useful background reference here is the survey When Can LLMs Actually Correct Their Own Mistakes?, which distinguishes intrinsic self-feedback from correction supported by reliable external information.
Again, I would use that as a test-design distinction, not as an argument that RP’s Skeptic role is unnecessary.
Long instruction load
There is also adjacent work such as ManyIFEval, which studies simultaneous instruction following.
I would only use that as a reason to measure compression/ablation, not as evidence that a long canonical RP is inherently wrong.
The local result above is much more relevant to RP itself than a general instruction-following benchmark.
A small test matrix if you want to keep the lab lightweight
I do not think every tester needs to run a giant benchmark.
A small branching matrix could preserve the “bring me real failures” spirit while giving useful controls only when they are needed:
| Question |
Cheap comparison |
What it helps separate |
| Does RP help in normal use? |
Normal / Canonical |
user-facing effect |
| Is the detailed doctrine adding value? |
Normal / Compact / Canonical |
generic research elicitation vs RP-specific structure |
| Why did a correct answer get buried? |
Canonical / answer-first / stop-compression |
evidence reasoning vs output execution |
| Can it stop when the answer is unavailable? |
Answerable / genuinely unanswerable |
search effort vs calibrated abstention |
| Can it reconcile tempting bad evidence? |
Clean / one plausible conflicting source |
verification / source reconciliation |
| Is retrieval causing the difference? |
Live web / frozen evidence |
search stack vs protocol |
| Does a change generalize? |
Alpha case / fresh structural analogue |
regression preservation vs broader transfer |
So the default path can stay very small:
1. Normal vs Canonical
2. If something interesting happens, reproduce it
3. Add the smallest control that distinguishes the leading explanations
4. Only then modify the protocol
5. Re-run the original case + one unrelated regression case
That is already very close to the philosophy in your testing protocol.
Later: functional ablation rather than random prompt deletion
If the project eventually gets enough failures to ask “which parts actually matter?”, I would probably ablate by behavioral cluster:
depth routing
starting-state freeze
rival hypotheses / explanation map
vocabulary + cross-disciplinary expansion
source lineage / independence
counter-evidence + failure search
contradiction handling
stopping / decision sufficiency
epistemic output / confidence
That makes a regression easier to interpret than removing arbitrary paragraphs.
Alpha benchmark vs fresh cases
The current Alpha set looks useful as a regression suite.
If future versions start getting tuned repeatedly against those exact public cases, a second lane of fresh community cases or structural analogues could answer a different question:
Alpha
-> did we preserve expected behavior?
Fresh cases
-> does the behavior transfer?
For browse-enabled agents there is also a separate future concern that public benchmark material itself may become retrievable during evaluation; Search-Time Contamination in Deep Research Agents is relevant background there.
I would treat that as future-proofing, not as a problem with the current Alpha set.
So the main thing I got from this small experiment is not “the Research Pass gave the wrong research answer.”
It is almost the opposite: the evidential judgment was already basically there, but the execution/output layer could spend a lot of budget before surfacing and terminating on that judgment.
That seems like a useful case for keeping the research doctrine and the implementation/output contract separately testable.
And because your repo is already treating the canonical version as a frozen baseline and keeping alternatives experimental until they survive comparison and regression tests, this kind of failure seems to fit the lab structure pretty naturally.