A Proposition Decomposition and Contradiction-Driven Reasoning System — Looking for Discussion and Collaborators

Abstract

I am exploring a reasoning system that uses proposition decomposition and contradiction search rather than asking an LLM to directly judge whether two propositions are contradictory.

The basic idea is to decompose propositions into smaller components, identify corresponding propositions through implication relations, and use these relations to search for contradictions. The system is intended to separate probabilistic exploration from a more reliable formal verification process.

I am particularly interested in whether actively searching for contradictions can provide a useful direction for reasoning and exploration under a limited generation budget.

This is still a research design rather than a completed system. I am looking for discussion, criticism, alternative approaches, and collaborators who may be interested in implementing or testing the idea.

Terminology used in this specification

  • Proposition: A statement that can be treated as a logical unit and can be related to other propositions by logical relations.
  • Decomposition: Breaking a proposition into smaller propositions while preserving its logical content, or a specified implication relationship to the original proposition.
  • Implication relation: A relation of the form P → Q, meaning that if P holds, Q follows within the adopted formal system.
  • Identification / correspondence: Determining which decomposed or derived propositions refer to the same subject or content, using their internal structure and implication relations rather than exact string matching.
  • Collision: A search-stage situation in which corresponding content is found in both positive and negative forms through decomposition, identification, and/or implication. A collision is a target for logical verification; it is not itself synonymous with a formally proven contradiction.
  • Contradiction: A formally established state in which the two input propositions cannot be true simultaneously.
  • Candidate: A proposition, decomposition, implication, or derived statement that has been generated or proposed for further verification but has not yet been established as valid.
  • Formal verification: Mechanical or otherwise explicitly formal checking of logical properties within the adopted formal system.

0. Purpose

The central system to be built is a contradiction checker.

The basic I/O is deliberately simple.

Input: Proposition P, Proposition Q
Output: Whether P and Q contradict each other

However, the two propositions are not compared directly as strings.

The propositions are decomposed, and implication relations between propositions are used to identify parts that can be regarded as expressing the same subject or content; the resulting identification is then used to search for contradictions.

Therefore, proposition identification is not merely a function for checking whether a decomposition is valid; it is itself a foundation of contradiction checking.

The central hypothesis is that, compared with generating a large number of candidates without direction, a search that actively pursues collisions with existing propositions may discover contradictions more efficiently under limited generation and computational budgets. This is a hypothesis to be supported or refuted experimentally.


1. Central system structure

       Proposition P        Proposition Q
              ↓                  ↓
       Proposition decomposition / normalization
              ↓                  ↓
       Acquisition of implication / compositional relations
              ↓                  ↓
       Proposition identification / correspondence
                         ↓
                  Collision search
                         ↓
                Formal logical verification
                         ↓
        CONTRADICTION / CONSISTENT / UNKNOWN

If a learning system is connected later, a search loop is added here.

Collision candidate generation
            ↓
        Verifier
            ↓
         Reward
            ↓
  Next candidate generation

The checker itself and the search process that looks for candidates are kept separate.


2. Meaning of “proposition identification”

Here, identification does not mean exact string matching.

A proposition is decomposed into constituent sub-propositions, and the implication relations between them and existing propositions are examined.

Conceptually, for example,

P → A → B → C
Q → A → B → C

means that P and Q share at least the common logical structure A, B, and C.

Likewise,

P → A
Q → B
B → A

means that, although P and Q are superficially different, they become comparable with respect to A.

Therefore, identification and correspondence use:

  • decomposed propositions,
  • implication relations,
  • relations by which higher-level propositions are constructed, and
  • structure separate from the truth value of a proposition’s affirmation/negation.

Truth value and the content/structure of a proposition are treated separately.

For example, p and ¬p should be treated as corresponding positive and negative forms of the same underlying proposition, rather than as completely unrelated propositions.


3. Role of proposition identification

Proposition identification has at least two uses.

3.1 Validity of decomposition

When an original proposition P is decomposed into

P1, P2, ..., Pn

the system checks whether the content reconstructed from the decomposed elements has the same meaning, or the same specified implication relations, as P.

For a simple AND decomposition, for example,

P ↔ (P1 ∧ P2 ∧ ... ∧ Pn)

can be verified.

3.2 Contradiction checking

When comparing P and Q, the propositions derived from both are first put into correspondence.

Then, if a relation such as

P → A
Q → ¬A

is found for the same subject/content, this becomes a candidate for testing whether P and Q can hold simultaneously.

Longer paths are also allowed.

P → A → B → C
Q → D → ¬C
D → C

In this case, P and Q respectively imply C and ¬C, so they form a collision candidate.

Therefore, proposition identification/correspondence is not merely a preprocessing step for contradiction checking; it constructs the comparison space in which contradictions can be discovered.


4. Contradiction checking

The initial version primarily targets propositional logic.

Input:

P
Q

Output:

CONTRADICTION
CONSISTENT
UNKNOWN

Definitions:

  • CONTRADICTION: It has been formally shown that P ∧ Q cannot hold simultaneously.
  • CONSISTENT: It has been shown that there exists an assignment/model in which P ∧ Q can hold simultaneously.
  • UNKNOWN: The current implementation cannot decide the result within its supported scope.

“Failure to find a contradiction” is not treated as “consistent.”

Within domains such as propositional logic where finite search is possible, the system should be deterministic whenever possible.


5. Idea of contradiction search

P and Q are not compared in a single step.

Both are decomposed, and propositions that can be derived from them are expanded step by step.

Conceptually, paths such as

P → A → B → C
Q → D → E → ¬C

are constructed, and the system checks whether positive and negative forms eventually appear for content that can be identified as the same.

The ideal form is to explore combinations of propositions derived from P and Q as broadly as possible and discover collisions such as

A and ¬A
B and ¬B
C and ¬C
...

However, enumerating all consequences leads to combinatorial explosion, so the search process selects promising directions.


6. “Collision-seeking”

Candidate generation is not treated as undirected random search.

For example, given an existing proposition P, the search objective is:

Generate a proposition that may collide with P.

However, the system must consider the possibility that a search may degenerate into simply generating ¬P repeatedly. The appropriate reward structure is itself an experimental variable.

Initial reward candidates:

Collision found              → positive reward
No collision found           → low / 0 reward
Could not be formalized      → low / negative reward

As needed, additional terms such as the number of inference steps to the collision, decomposition validity, and redundancy may be added.

A “non-trivial collision” is not required as a formal definition.

The central comparison is:

Undirected search
vs.
Collision-seeking search

with the collision discovery efficiency compared under the same generation and computational budgets.


7. Proposition decomposition

Decomposition is not simply “the more fine-grained, the better.”

At minimum, the following properties are required.

7.1 Validity

Logical content should be preserved between the original and decomposed forms.

Original proposition
       ↓ decomposition
P1, P2, ..., Pn
       ↓ reconstruction
Equivalent to the original proposition / satisfies the specified implication relations

7.2 Non-redundancy

The system checks whether each component is actually necessary.

If removing an element Pi still allows the original proposition to be fully preserved from the remaining elements, Pi may be treated as a redundant component.

Conceptually, one can verify conditions such as:

P1 ∧ ... ∧ Pn entails P
P1 ∧ ... ∧ ¬Pi ∧ ... ∧ Pn does not entail P

A component is not rewarded merely for making the representation finer.

The current specification does not claim that this problem is necessarily NP-hard. The intended claim is only that searching for minimal sufficient decompositions or minimal evidence sets may cause combinatorial explosion.


8. Proposition graph

Propositions are not treated as mere strings, but as nodes connected by relations.

Example:

P → A
A → B
B → C
Q → D
D → C

This shows that P and Q both imply C.

With negation included,

P → A → B → C
Q → D → ¬C

can produce a collision candidate.

Role of the graph

The proposition graph is not the final truth checker.

Its main roles are to:

  • preserve proposition structure,
  • find propositions that are comparable,
  • trace implication paths, and
  • reduce the search space for collisions.

Final logical checking is performed by a separate formal verifier.


9. Comparability

Not every proposition needs to be compared with every other proposition.

First, implication relations, structure, and correspondence of subjects are used to narrow the set to propositions that may be describing the same thing.

After candidate correspondences are produced, an LLM may be used for semantic comparison of atomic propositions that cannot be formally matched.

Do not insist on complete mechanical determination here. The LLM is used as an auxiliary mechanism for constructing comparison targets, not as the final contradiction checker.


10. Unknown / undecidable within scope

Unknown propositions are not forcibly assigned a truth value.

Decidable within the formal system
    → formal verification

Not decidable within the formal system
    → UNKNOWN

Search and experiments for acquiring external information are future extensions and are not the center of the minimal implementation.


11. Learning / search process

After the checker is completed, an LLM may be connected as a search process.

LLM
 ↓
Generate proposition decompositions / derived propositions / collision candidates
 ↓
Formal verifier
 ↓
Reward
 ↓
Next candidate generation

Multiple candidates are generated and evaluated through the verifier.

At this stage, reinforcement learning such as GRPO may be used.

In production, LoRA / QLoRA may be applied to an existing LLM, and Unsloth or similar tools may be used as the training infrastructure. The training infrastructure itself is not the research target.


12. Minimal research experiment

The first experiment is not intended to determine whether an “intelligent AI” can be built.

It tests the following single point:

Under the same generation budget, can collision-seeking search discover more contradictions than undirected search?

The comparison should be controlled for quantities such as:

Number of generated candidates
Token count
Computation time

Candidate evaluation measures include:

Number of discovered contradictions / number of generated candidates
Number of discovered contradictions / token count
Number of discovered contradictions / computation time

A toy dataset may be used initially, but the production code and toy-specific code should not be designed as separate systems. The same design should allow the model size to be changed while preserving the system being evaluated.


13. Implementation boundaries

LLM is responsible for

  • Natural-language proposition generation
  • Generating proposition decomposition candidates
  • Generating derived-statement candidates
  • Generating collision candidates
  • Proposing comparison candidates for atomic propositions that cannot be formalized
  • Proposing search directions

As mechanically as possible

  • Syntax checking
  • Logical validity of decompositions
  • Verification of implication relations
  • Equivalence checking
  • Contradiction / satisfiability
  • Redundancy checking
  • Core reward calculation

In principle, the LLM is not used to make the final judgment “Do these two propositions contradict?”


14. Theoretical considerations

14.1 Distinguish “consistent” from “unknown”

This is the most important point.

14.2 Distinguish “logical identity” from “semantic similarity in natural language”

Do not treat a formally proven equivalence and an LLM judgment that two statements are similar as the same kind of result.

14.3 Mutual information and logical implication are not the same

Mutual information represents statistical dependence over probability distributions.
Implication represents a relation within a logical system.

The present method is not proposed as a direct alternative definition of mutual information. It is positioned as a method for making comparable information explicit as logical structure.

14.4 State the scope of completeness explicitly

Do not claim that every natural-language proposition can be determined with certainty.

First specify the formal system being targeted, and then seek as much rigor and completeness as possible within that scope.


15. Research policy

Conserve data and reduce hypotheses with small experiments before spending large amounts of computation.

Use intuition to generate hypotheses, but use explicit definitions, predictions, and falsification conditions during verification.

The following supplementary note describes the broader motivation that led me to focus on trial-and-error search and the direction of reasoning.

An Additional Note on Trial and Error in AI

The history of academic discoveries has, in most cases, been driven by trial and error. If there were a universal algorithm capable of solving every unsolved problem, one would expect it to have been discovered during the last 2,000 years. It has not. For details, please search for the P vs NP problem.

Since there is no universal algorithm for problem solving, the only way to approach unsolved problems may be to “search through them the hard way.” Therefore, is it really correct to try to solve unsolved problems with an algorithm called an LLM? Even if trial and error can be performed inside a model, I suspect it would remain very limited.

There is research suggesting that OpenAI’s o1, particularly the o3 series, can exhibit trial-and-error behavior, and I think this is directionally correct. However, many problems remain:

  • Long-term memory
  • Long-term adjustment of the direction of reasoning
  • Diverse trial and error toward more meaningful directions
  • Generating combinations at a more abstract level than random word selection at the output level
  • Efficient search algorithms
  • Changing the direction of future trial and error based on the results of previous trials

Personally, I think the last two are especially important. I have tried to tackle them, but they are quite difficult.

Whenever I think about the problems of AI, extrapolation seems to appear as an issue everywhere. But perhaps humans also extrapolate by performing an enormous amount of search.

I think this could be strengthened by separating the pieces and measuring them independently:


For the central question — whether collision-seeking can find contradictions more efficiently than relatively undirected generation under a limited budget — I think it is worth testing directly as a search-policy hypothesis.

I would not frame it as collision search versus SAT/SMT. If P and Q are already complete formal propositional formulas, checking whether P ∧ Q is satisfiable is the natural control; for example, this is exactly the kind of satisfiability question handled by Z3.

The interesting search problem seems to be the part before that final formal query is available:

  • which decompositions to explore,
  • which implication/bridge relations to pursue,
  • which propositions should be put into correspondence,
  • and which frontier to expand under a limited generation/verifier budget.

So for the first experiment, I would probably keep almost everything fixed and vary only the search policy.

A simple progression could be:

Stage A: fixed symbolic problem
    random / repeated search
    vs BFS-like search
    vs generic best-first
    vs collision-directed search

Stage B: add correspondence uncertainty
    paraphrases
    entity / argument changes
    time / context
    hard same-topic non-conflicts

Stage C: add open-ended LLM generation
    decomposition
    bridge generation
    learned search policy
    RL/GRPO if it is still useful

For Stage A in particular, I would keep the verifier, verification frequency, feedback, stopping rule, and information available to each searcher as controlled as possible.

A few controls seem especially important:

  1. Define “undirected” operationally. Random generation alone may be too weak a baseline; I would include a reasonably strong generic best-first search as well.
  2. Count verifier access as part of the budget. Two systems with the same token count but different verifier calls or different feedback richness are not really receiving the same information budget.
  3. Keep the final stopping authority outside the generator. A model can propose “done”, but CONTRADICTION, CONSISTENT, UNKNOWN, and budget exhaustion should remain distinct formal states.
  4. Keep collision separate from contradiction. I think your specification already makes this distinction correctly.
  5. If you add a correspondence/context gate, I would make it high-recall rather than aggressively selective. Extra false-positive candidates mostly cost verifier work; a false negative can remove the actual contradiction path completely.

A rough pipeline I have in mind is:

candidate proposition / edge
        ↓
high-recall comparability checks
        ↓
soft collision / correspondence ranking
        ↓
top-K frontier candidates
        ↓
hard formal verifier
        ↓
CONTRADICTION / CONSISTENT-within-scope / UNKNOWN
Why I would separate search, semantic matching, and verification

There are several nearby systems that make the component separation look reasonable, even though I have not found one that tests exactly the collision-directed search hypothesis you describe.

Logic-LM is one useful reference point: the LLM produces a symbolic formulation, deterministic solvers perform inference, and solver errors can be fed back into refinement. That supports the broad division of labor, but it does not answer whether collision-directed search is better than a generic search policy.

SymStep is interesting for a slightly different reason. It separates per-step symbolic verification from search guidance, including an MRV-style guidance mechanism. The task is much more structured than an open proposition graph, and MRV is not collision search, but it is a useful example of treating:

"Is this step valid?"

and

"Where should I search next?"

as separate experimental questions.

That is the part I would preserve here.

There is also a useful negative/control perspective from search-heavy theorem proving. Work such as BFS-Prover is a reminder that fairly simple generic search strategies can be surprisingly competitive. I would therefore avoid comparing collision-seeking only against a weak random baseline.

A search result becomes much more informative if the interpretation is something like:

collision > random/BFS
    but collision <= strong generic best-first
        → useful guidance signal exists,
          but not yet evidence of a collision-specific advantage

collision > strong generic best-first
    and both receive the same information
        → much stronger evidence that
          the collision objective/action structure matters

The same separation applies to verifier feedback.

For example, Hypothesis Frontier compares verifier-guided search against repeated generation under matched model/problem/round budgets, while Counterexample Guided Learning in the Large shows in a controlled symbolic induction setting that rich verifier counterexamples can substantially change sample efficiency.

Those tasks are different from contradiction search, but they make one experimental point quite clear: feedback richness is itself a variable.

So I would distinguish things such as:

SEARCH:
    random
    BFS
    generic best-first
    collision-directed

VERIFIER FEEDBACK:
    pass/fail
    failed edge
    counterexample
    unsat core / conflict information

VERIFICATION TIMING:
    every attempted edge
    every k steps
    final only

and change one of these at a time initially.

How I would structure the first collision-search experiment

One subtlety is that a fully visible formal graph can make the search problem disappear.

If all propositions and implication relations are already known, a global solver or closure procedure may simply bypass the intended frontier search. So I think Stage A needs an explicit information contract:

What is initially visible?
What does one expansion reveal?
What costs one budget unit?
What does the verifier return?
When is the search allowed to terminate?

A simple decision tree would be:

Are P and Q already complete formal formulas?
 |
 |-- yes
 |    |
 |    |-- only need contradiction decision?
 |    |       -> direct SAT/SMT control
 |    |
 |    `-- want to study search efficiency?
 |            -> hide/charge for bridge or edge discovery
 |
 `-- no
      -> first isolate the search experiment
         from natural-language/formalization noise

Baselines

I would probably start with:

B0 random/repeated frontier selection
B1 BFS-like
B2 generic best-first
B3 collision-directed

The exact B2 heuristic matters less than having a non-trivial generic search control.

Budget

I would record at least:

  • number of candidate generations,
  • expansions,
  • verifier calls,
  • tokens,
  • wall-clock time,
  • amount/type of verifier feedback.

The primary result could be something like:

verified contradiction discovery rate @ budget

with secondary curves such as:

expansions to first verified contradiction
verifier calls to first verified contradiction
success vs budget

Difficulty calibration

I would also deliberately create several strata:

direct negation
one-hop conflict
multi-hop conflict
hidden bridge
matched distractors
no contradiction

This matters because an experiment can become uninformative in both directions:

too easy  -> every search method succeeds
too hard  -> every search method fails

The middle regime is where a search heuristic can actually be measured.

Logic Haystacks has a useful related idea: its contradiction-evidence retrieval tasks use distractors that are difficult to distinguish from relevant evidence while being formally non-interfering. That seems much closer to a meaningful collision-search stress test than simply adding unrelated filler.

Direct-negation shortcut

Your ¬P shortcut concern also looks testable without complicated reward shaping at first.

I would begin with the simplest restriction:

A rewarded collision must have a verified derivation path from the allowed roots/current state.

Then stratify:

direct negation
one-hop derivation
multi-hop derivation
hidden bridge

If direct negation still dominates after that, then more explicit reward shaping becomes justified.

Correspondence may be a separate bottleneck from search

This is the part where I think the system can easily become difficult to interpret.

A proposition can be:

  • textually similar,
  • semantically related,
  • about the same entity,
  • structurally comparable,
  • logically equivalent,

and those are not the same relation.

I would keep a soft semantic model on the candidate-construction side of the boundary rather than letting it certify correspondence.

Something like:

raw proposition
    ↓
entity / alias normalization
argument-role binding
time / context compatibility
modality / conditional scope
quantifier structure
    ↓
comparability gate
    ↓
semantic correspondence / collision score
    ↓
candidate edge
    ↓
formal verification

Not every one of those fields needs to exist in the first implementation. I mean them more as failure categories than as a required ontology.

There are a couple of useful nearby examples.

Decompose-and-Formalise explicitly treats autoformalization failures as localizable errors and introduces a substitution mechanism to preserve consistent argument-role bindings. That seems relevant to cases where two sentences look very similar but bind entities/arguments differently.

ALICE, in controlled requirements contradiction detection, separately checks things such as variable identity, action inconsistency, condition equivalence, and condition co-occurrence. One of its documented false positives is particularly instructive: two effects looked contradictory, but their triggering conditions were actually mutually exclusive.

That suggests a useful general rule:

opposite-looking effects
        +
same/comparable proposition
        +
compatible conditions/context
        ↓
worth sending to the formal contradiction checker

rather than:

opposite-looking text
        ↓
contradiction

A small sanity check I tried

I tested this boundary with a few small synthetic/controlled experiments. These are only sanity checks, not evidence about your full system.

In a constrained English panel, a small NLI model was surprisingly useful for ranking collision candidates: the real collision was often very near the top.

But the absolute scores were much less trustworthy.

For example, statements with:

  • a different subject,
  • a different time,
  • reversed argument roles,

could still receive extremely high “contradiction” probabilities.

That distinction became even clearer with stress tests.

HANS is specifically designed to expose lexical-overlap, subsequence, and constituent heuristics in NLI systems. The small model I tested failed badly on some of those non-entailment cases.

Likewise, on a sampled ANLI dev set, its contradiction ranking degraded dramatically compared with the controlled examples. ANLI was built adversarially, so that kind of gap is not especially surprising.

A larger NLI model improved those stress-test results somewhat, but did not make the issue disappear.

So my current interpretation would be:

learned semantics may be quite useful for deciding where the verifier should look next, but not for deciding what is logically true.

That actually seems very compatible with your proposed boundary.

High-recall gate rather than high-precision gate

I also tried an oracle-style experiment where entity/role/time/context metadata was already known.

As expected, a structured gate could reduce the number of candidate pairs sent downstream sharply.

The more useful observation came from injecting errors into the gate:

gate false positive
    -> mostly extra verifier calls

gate false negative
    -> potentially deletes the real contradiction path

So if you add such a gate, I would optimize it conservatively for recall and let the formal verifier handle some extra false positives.

The important caveat is that this experiment used known structured metadata. It is an upper-bound/interface test, not evidence that raw natural language can be decomposed that reliably.

Formal verification does not remove the formalization problem

There is another boundary that may become important once natural language enters the system:

valid proof relative to a formalization
        !=
faithful formalization of the original text

Recent work such as Do LLMs Game Formalization? examines exactly this gap. I would read its result carefully: it does not say that models systematically manipulate the formalization whenever given a prover, but it does find distinct forms of unfaithfulness and argues that high compilation/proof-success rates should not automatically be treated as evidence of faithful reasoning.

For your system, I think that argues for keeping these artifacts separately auditable:

original proposition
decomposition
correspondence / bridge relation
formal representation
verifier result

rather than only storing the final solver answer.

One potentially useful later ablation would be:

semantic-fixed:
    decomposition/correspondence is fixed before search

semantic-open:
    search may also propose new bridge/correspondence relations

If the improvement appears only in the second condition, then part of the gain may be coming from changing the representation rather than from the frontier policy itself.

That is not necessarily bad — it is just a different mechanism and probably deserves its own measurement.

Verifier outputs, UNKNOWN, and explanations

I like the fact that your specification explicitly says:

failure to find a contradiction is not the same as consistency.

I would preserve that distinction all the way through implementation.

For example:

VERIFIED_CONTRADICTION
VERIFIED_SAT / CONSISTENT_WITHIN_SCOPE
SOLVER_UNKNOWN
SEARCH_BUDGET_EXHAUSTED
FORMALIZATION_UNCERTAIN

can remain separate internally even if the user-facing API later compresses some of them.

The Z3 basic-command documentation is a useful reference here: sat, unsat, and unknown are distinct solver outcomes.

I would similarly keep search termination separate from logical termination.

For example:

generator says "done"
        ↓
formal/search state checks
        |
        |-- verified contradiction
        |       -> CONTRADICTION
        |
        |-- complete supported search + satisfying model
        |       -> CONSISTENT within the declared scope
        |
        |-- solver unknown
        |       -> UNKNOWN
        |
        `-- generation/search budget exhausted
                -> UNKNOWN / NOT FOUND

For the first experiment, verifier feedback can probably remain minimal.

Something like:

candidate accepted / rejected

is easier to interpret than immediately returning detailed repair hints.

Later, richer feedback could include:

  • failed edge,
  • counterexample,
  • unsat core,
  • repair hint.

But that should probably be a separate ablation, because rich feedback can itself improve search efficiency.

If explanation becomes important, Z3 can return unsat cores, but I would also keep in mind that an unsat core is not automatically a minimal conflict. MUS/minimal-conflict extraction can come later if compact explanations become important.

Failure modes I would log separately

I think this project could benefit a lot from retaining why an attempt failed rather than only recording whether a contradiction was found.

For example:

DECOMPOSITION_DRIFT
FORMALIZATION_UNFAITHFUL

CORRESPONDENCE_FALSE_POSITIVE
CORRESPONDENCE_FALSE_NEGATIVE

ENTITY_BINDING_ERROR
ARGUMENT_ROLE_DRIFT
TEMPORAL_CONTEXT_MISMATCH
MODAL_SCOPE_MISMATCH
QUANTIFIER_SCOPE_MISMATCH

INVALID_IMPLICATION
FABRICATED_BRIDGE

SEARCH_MISS
SHORTCUT_COLLISION
DUPLICATE_CONFLICT

PREMATURE_TERMINATION
SOLVER_UNKNOWN
SEARCH_BUDGET_EXHAUSTED

The exact names do not matter much.

The useful part is that a negative result can then tell you whether the bottleneck was:

representation
correspondence
search
verification
or stopping logic

rather than collapsing everything into “the system did not find a contradiction.”

Things I would postpone until the first experiment is informative

I would personally delay:

  • GRPO / RL,
  • LoRA/QLoRA,
  • multi-agent search,
  • belief revision,
  • paraconsistent extensions,
  • complex MUS-based rewards,
  • large external retrieval,

until the simpler search comparison is interpretable.

Not because those directions are bad, but because otherwise they add several new degrees of freedom before the central hypothesis has been measured.

A larger semantic model is also probably a secondary knob rather than the first fix. In my small stress test, moving from a small to a larger DeBERTa NLI model improved HANS/ANLI behavior but did not remove the structural/distribution-shift failures.

Similarly, post-hoc confidence calibration is useful hygiene, but it solves a different problem. Temperature scaling can improve confidence calibration, but it cannot repair a wrong entity binding, temporal mismatch, or argument-role error.

So I would only add these layers once there is a specific measured failure for them to address.

The result I would find most informative from the minimal experiment is therefore not simply “collision search wins” or “collision search loses.”

I would look for something more conditional:

If collision guidance beats random/BFS
but not strong generic best-first:
    the collision score contains useful search information,
    but the collision-specific objective is not yet established.

If it also beats a strong generic baseline
under the same information/verifier budget:
    that is much stronger evidence for the search idea.

If the advantage disappears with hard matched distractors:
    correspondence/scoring is probably the current bottleneck.

If it survives those controls:
    learning the collision-directed policy becomes much easier to justify.

Even a negative result at that stage would be useful, because it would tell you where the difficulty is instead of mixing the search policy, correspondence model, formalizer, verifier feedback, and training loop into one result.

So at least for the first implementation, I think the most valuable version is the boringly controlled one: make the checker trustworthy, make the information/budget contract explicit, and then see whether collision-seeking changes the search curve.

I’m quite busy at the moment, so I don’t have enough time to write code and run experiments. I may also not be able to reply immediately, but I will definitely reply eventually. If anyone wants to implement this idea, please feel free to proceed without waiting for me.

Also, thank you for your reply. As I understand it, what you are saying is, roughly speaking, about designing experiments in a way that allows us to interpret the experimental results uniquely and accurately. Please correct me if I misunderstood.

I previously had Claude’s Deep Research look into this, and I found that model general-purpose performance has a strong correlation with the computational resources used. So, are you essentially asking whether it is fundamentally possible, in theory, to obtain general-purpose capabilities that exceed the computational resources invested? I suspect this may be the reason for emphasizing the computational budget, but I’m not sure. What do you think?

I agree that the minimal experiment should probably be the highest priority. I tried to prepare an environment for it, but this part has turned out to be quite difficult. I would like to explain the details below.

The first thing that comes to mind is automatically generating propositional logic problems to create the data. However, I think that in completely formal data, it may be impossible to perform the kind of “useful shortcut reasoning” I have in mind, so we may not be able to observe a change in the curve. Shortcut reasoning seems to involve non-formal aspects such as analogy, right? At the same time, I don’t think I can make that claim without actually running the experiment.

Otherwise, there is a possibility that we would simply be feeding combinations of propositions into a black box that only tells us whether they are correct or incorrect, and then searching through them. So I think there needs to be some kind of underlying structure in the data, such as mathematics or the universe.

What I have been thinking about for some time is an automatically generated search environment. For example, consider mathematics. The chain rule itself is simple, but by canceling terms in the numerator and denominator, information is irreversibly lost. If you try to differentiate a complicated composition of functions without knowing the chain rule, an enormous amount of trial and error is required. In other words, there may be a way to generate artificial data by starting from axioms → deduction → removing information.

I previously tried to create artificial data based on this idea, but maintaining consistency across the entire dataset proved to be extremely time-consuming.