Quick recap of ByteLex's Cross-Tokenization and the faults leading to successes

Show and Tell
1

We built a map of byte-space out of 11 tokenizer vocabularies — and it predicted which made-up words our model would flub

This is a Claude Opus 5.0 summary of the bytelex week-long failures leading to a single success that cascaded into a series of successful prototypes.

A show-and-tell about byte-level models, tokenizer-free structure, and why we think bytes are the right shared substrate for models that need to talk to each other.

TL;DR

  • We train small byte-level language models — no tokenizer, 256 symbols, that’s the whole alphabet.
  • Separately we maintain a library that extracts vocabularies from other people’s tokenizers and studies them as pure structure (no corpus, no weights, no model).
  • Those two things met this week. We turned 11 tokenizer vocabularies into a single fixed coordinate space over 3-byte windows, with no corpus statistics anywhere, and asked it to predict which invented words our trained byte model would get wrong.
  • Spearman correlation with the model’s measured per-word accuracy: −0.98 (n=8 words). A corpus-derived statistic on the same words scored −0.62.
  • Then we used the map to generate training data aimed at the predicted weak spots. The failure pattern flattened, and the fix transferred to words that were never in any training set: held-out accuracy went from .53 / .40 (two seeds) to .84 / .79.
  • The map costs ~200 ns per lookup and occupies 0.58% of its own address space. It builds in seconds.

Links at the bottom. Everything below is measured; where a number is shaky I say so.

The setup, in plain terms

The model. A 237M-parameter byte-level language model trained on about 16B tokens of mostly-English text. It reads raw UTF-8. Nothing is tokenized. One detail matters for later: its input embedding for position t is the sum of three lookups — the current byte, the previous byte, and the one before that. So the thing the model actually “sees” at each step is, structurally, a 3-byte window. Hold that thought.

The adapters. We don’t retrain the model to teach it a behavior. We attach a small module (about 8.5M parameters, ~4,000 training steps, roughly two hours on a single RTX 4090, ~4 GB of VRAM) and train only that, with the base model frozen. It detaches cleanly — with the adapter switched off, the model’s outputs are bit-identical to the untouched original. Cheap experiments, no risk to the base weights.

The library. A separate, model-free thing: it pulls the vocabulary out of any tokenizer (we’ve published extractions for twelve: GPT-2, cl100k, o200k, Qwen, DeepSeek-V3, Llama-3.1, Mistral, SmolLM2, T5, XLM-R, BERT, ByT5) and analyzes the byte structure of the tokens themselves. One early result from it: somewhere between 27% and 36% of every BPE vocabulary we looked at consists of tokens that are internally divided — multiple byte-units wearing one token ID. Run the same analysis on ByT5, where each token is literally one byte, and you get exactly zero internal structure, which is the control working correctly.

The experiment that failed in a useful way

We wanted to know whether a small adapter could learn to actually chain reasoning steps rather than pattern-match. The task:

If someone is harl, then they are blim.
If someone is blim, then they are quen.
If someone is quen, then they are torv.
If someone is torv, then they are vell.
If someone is vell, then they are sook.
Wren is harl. What follows?

Reply with the final answer only: sook. The words are invented, so there’s no world knowledge to lean on — only the chain. Five steps. Chance among the listed words is about 1/6.

First attempt, with a purpose-built recurrent module bolted on: 63%. Looked great. Then we shuffled the order of the rules — which changes nothing logically — and it fell to 17%, i.e. chance. The model had learned that on our nicely-ordered prompts, the answer is always the consequent of the last rule listed. Worse, a size-matched non-recurrent control scored 87% on the same shortcut. Our fancy module was a worse pattern-matcher than a plain one.

The honest read: the task was broken, not the architecture. So we changed one thing — every training example now renders its five rules in a fresh random order — and threw the custom module away, using the ordinary adapter.

setup in-order shuffled in-vocabulary check
bolted-on recurrent module, ordered data .633 .167 (chance) .993
its size-matched control, ordered data .867 .207 (chance) 1.000
plain adapter, shuffle-blocked data (seed 1) .527 .553 1.000
plain adapter, shuffle-blocked data (seed 2) .400 .493 .980

The shuffled column is the one that matters. It’s now as good as or better than the in-order column, meaning the ordering shortcut is gone and something order-invariant is doing the work. And this was measured on invented words the adapter had never trained on — the training vocabulary and the exam vocabulary were disjoint. No scaffold, no “let’s think step by step”, no visible chain — the answer comes out directly.

Lesson we keep relearning: when a model looks smart, try scrambling something that shouldn’t matter.

Where the remaining errors actually live

Roughly half the answers were still wrong, so we took every miss apart.

The result surprised us. 73% of wrong answers were correctly-spelled words from the prompt’s own list — just the wrong one. The other 27% were the right word overrun into English (“prin” → “print”, 15 times out of prin’s 18 misses). And one word swallowed everything: 63% of all wrong answers were the single word “sook.”

Then the teacher-forced loss, byte by byte:

position loss on the correct answer
first byte, items the model got right 0.06
first byte, items the model got wrong 4.41
every byte after the first, all items exactly 0.00

The entire decision is the first byte. Once the first letter is committed, the rest of the word is free. That reframed the problem completely: it isn’t a reasoning failure and it isn’t a spelling failure, it’s the English byte-prior hijacking the answer at the moment of commitment, whenever the chain computation isn’t confident.

And it was strongly word-specific:

invented word model accuracy
harl 18/18 (100%)
sook 14/15 (93%)
vell 15/20 (75%)
blim 14/24 (58%)
mund 9/19 (47%)
quen 6/21 (29%)
torv 3/16 (19%)
prin 0/18 (0%)

Which raises the obvious question: could we have predicted that ladder in advance?

The map

Here’s the idea, and it’s simple enough to be suspicious of.

Take the 3-byte window as an address. There are 256³ = 16,777,216 possible addresses — a complete, fixed, deterministic space that exists before any data does. Any byte string is a path through overlapping addresses. sook is two steps: soo → ook.

Now paint weight onto that space using nothing but tokenizer vocabularies. No corpus. No text sampling. No model weights. BPE vocabularies are merge-ordered, so a token’s ID is already a frequency proxy — that’s the only numeric input we use, and in the strongest variant we don’t even use that, just occupancy summed across tokenizers.

For each address you can then compute small things: the entropy over what byte comes next, the share taken by the single most dominant continuation, and whether word-like units are ever observed ending there. That last one is the interesting one — call it “can a unit plausibly stop here?”

The prediction: a made-up word is hard for a byte model precisely when English refuses to let it end. torv and quen and prin sit at addresses where one continuation dominates completely — the byte prior wants to keep going (print, printer), so the model overruns or bails to a safer word.

Test against the measured ladder above:

where the weights came from correlation with model accuracy did it name the worst three?
corpus statistics (8.2 MB of English text) −0.62 yes
one tokenizer, merge-rank weighted −0.79 yes
all 11 tokenizers, occupancy consensus −0.98 yes

Every variant we tried — five of them, including two crude ones — identified the same worst three words, exactly. Two different per-address statistics (dominance and “can it end here”) independently picked out the same trio.

Fairness notes, because this is the part most likely to be oversold: n = 8 words. A Spearman of −0.98 over eight points is a suggestive number, not a law. The rank-exactness of the worst three across five independent weighting schemes is the more robust part of the signal. And the real test isn’t correlation at all — it’s whether acting on the prediction changes anything.

Acting on it

If the map knows which words are hard, it can mint them. So we had it generate a training vocabulary: 48 invented words, half deliberately easy to end, half deliberately hard, screened against the word surfaces of all 11 tokenizers (273,689 of them) so nothing real slipped in, and screened against each other so no two words shared structure. Then we retrained the adapter with the same recipe and everything else identical. Difficulty as a dial, not an accident.

The exam was the old word list — never trained on in either run:

training vocabulary held-out accuracy, seed 1 / seed 2 failure-ladder correlation
12 hand-picked invented words .527 / .400 −0.98
48 map-generated (24 easy / 24 hard) .840 / .793 +0.05 / −0.21

Two things happened at once, on both seeds. Accuracy on words the model had never seen rose by about 35 points. And the failure ladder flattened to noise — the thing the map predicted so well became unpredictable, because it had stopped happening. The weakness was trainable, and training it away on one set of hard words transferred to a different set of hard words.

That’s the result we’re actually excited about. Not “we got a higher number,” but: a statistic computed from tokenizer vocabularies, with no corpus and no model, identified a behavioral weakness in a trained network precisely enough to design the cure.

It also found the next problem, which is what a good instrument does. In the new exam we’d deliberately included pairs of words sharing a two-letter opening. When the answer’s look-alike partner is elsewhere in the same prompt, accuracy is .26; when it isn’t, .80. Same 22/85 split on both seeds. So the model commits at byte one and then fails to discriminate at bytes two and three. That’s the current work.

One honest wart: the harder training data made the adapter noisier on unrelated text — about three times the drift of the previous run, past the threshold we hold ourselves to. There’s a knob for that (a loss term that penalizes the adapter for changing the base model’s behavior on text that isn’t its job), and the fix run is on the card right now. First seed came back clean — drift back under the bar and accuracy slightly up — but one seed is one seed, so treat that as in-flight, not a result.

What this is really for: a shared substrate

Here’s the larger reason we care, beyond invented words.

Every text model consumes bytes eventually. A tokenizer is just a chopping strategy laid over the same underlying stream. Models with different tokenizers can’t easily exchange structure — you can’t line up their vocabularies, you can’t say what one knows that the other has never encountered, and when you try to pass supervision between them, some of it silently evaporates at the boundaries.

But if you express every vocabulary as a field over one fixed address space, those questions become arithmetic:

occupied addresses of which Chinese-lead
Qwen’s vocabulary 59,372 8,574
an English-centric pair 12,798 106
shared between them 10,973 —
in Qwen, absent from the English side 48,399 —
in the English side, absent from Qwen 1,825 —

That “48,399” is a number you can compute in seconds, before training anything: it is exactly the region a Chinese-capable teacher inhabits that an English-trained byte student has never witnessed. And it’s not an approximation — CJK characters encode as exactly three UTF-8 bytes, so at character boundaries one Chinese character is precisely one address. We verified that a specific two-character string resolves to two live addresses in Qwen’s field.

The practical upshot: when a teacher model supervises a byte-level student across a tokenizer mismatch, the portion of its signal landing in never-witnessed territory stops being invisible loss and becomes a measured, reportable quantity — something you can pad for, disclose, or build curriculum against.

And the space is cheap enough to actually use. The 11-tokenizer consensus occupies 96,563 addresses — 0.58% of the 16.7M. Everything else is empty and costs nothing: absence is just a missing dictionary key, so a lookup on empty space returns zero and takes no memory. Lookups run at ~200 ns (100,000 of them in 0.02 s). A full field builds from a vocabulary file in 1–12 seconds.

The last piece is the one that made us take this seriously rather than treating it as bookkeeping. Remember the model’s input embedding: current byte plus the two before it. That is a 3-byte window. The map’s coordinate system and the model’s input composition are the same space, arrived at independently — one from architecture, one from vocabulary analysis. So a prediction made in map coordinates is a prediction about something the model genuinely computes over, which is probably why any of this worked.

What we’re not claiming

  • One model family, one scale. 237M parameters, byte-level, English-dominant training. Nothing here has been shown to transfer to a large tokenized model.
  • One task family. Invented-word rule chaining is a toy, chosen because it has no world knowledge to leak. It is not “reasoning” in any broad sense.
  • Two seeds. Everything above replicates across two seeds, which is our minimum bar, not a strong one. The absolute levels move noticeably between seeds (.53 vs .40 in the early runs); the direction is what replicates.
  • n = 8 on the headline correlation. The convergent agreement across five weighting schemes is the sturdier evidence; the −0.98 itself deserves more words.
  • The cross-tokenizer supervision channel is designed, not demonstrated. Coverage arithmetic works today. Routing actual teacher signal through the map is the next build, and it might not pay.
  • Multilingual competence is not claimed anywhere. The point of the coverage table is that we can measure what’s missing, not that we’ve filled it.

Where it lives

Happy to go deeper on any part of this — especially if you’ve tried cross-tokenizer alignment and hit walls we haven’t hit yet, or if you think the eight-word correlation is doing more work than it should. That last one is a fair hit and we’d rather hear it now.

2

I’ve nailed down a way to have a 99.6% probability of predicting which words are problem words, tail cases, edge cases, and overlapping words now simply by combining the tokenizers together.

Seems I can almost entirely predict the outcome of a train before I even begin. That was unexpected.

3

Hmm… from what I can see from here, maybe something like this?:thinking::

I think there are a few different claims tangled together here, and separating them makes the result look cleaner rather than weaker.

My short version would be:

  • The rho ≈ -0.98, n=8 result looks useful as an exploratory diagnostic, but I would not treat those eight words as an independent confirmation set.
  • The later 48-word run is more interesting to me than the original correlation, because it moves from “this atlas statistic tracks an already-observed failure ladder” to “use the atlas to construct new training material, then see whether behavior changes.”
  • The remaining attribution question is narrower: how much of that improvement came specifically from difficulty-targeted selection, versus simply moving from 12 hand-picked words to a much broader/more diverse 48-word lexicon?
  • The new look-alike/prefix failure looks like a very good next diagnostic target. I would probably chase that before inventing another broad metric.
  • For cross-tokenizer routing, I think the next boundary is not only “how much byte-space overlaps?” but “how much teacher probability mass can actually be transferred without being dropped, duplicated, or assigned to the wrong student continuation?”

So if I had to pick a default path, it would be something like:

What do you want to establish next?

Does the atlas predict difficulty on unseen words?
    -> freeze the current metric
    -> predict a fresh word set before evaluating it

Did difficulty targeting specifically cause the 48-word improvement?
    -> same-size / same-generator / same-screening 48-word control
    -> but do not select by atlas difficulty

What is the new residual failure?
    -> paired look-alike insertion/removal
    -> inspect byte-position margins / continuation ambiguity

Can ByteLex become an actual cross-tokenizer supervision channel?
    -> structural coverage
    -> teacher probability mass captured
    -> residual / dropped mass
    -> downstream KD result

That would let the different claims strengthen independently instead of making one statistic carry all of them.

Why I would separate the 8-word correlation from the later intervention

The n=8 part by itself is not what bothers me most.

The more important distinction is discovery vs. confirmation.

As I understand the sequence, the eight-word failure ladder was already visible from the E-G2 diagnostics, and then the atlas fields/statistics were compared against that known ladder. That makes the very strong Spearman result genuinely interesting, but still exploratory: the same observations helped identify which atlas view looked predictive.

That is different from:

  1. choosing the metric first,
  2. receiving a new set of words,
  3. predicting their ordering without seeing model performance,
  4. then evaluating the prediction.

So I would not read rho ≈ -0.98 as “the atlas has already demonstrated out-of-sample predictive accuracy.” I would read it as:

“There is a surprisingly strong candidate signal here, and it produced a concrete intervention hypothesis.”

The nice part is that the work did not stop at the correlation.

The progression into the 48-word intervention is what makes this more convincing than a pure post-hoc story. The public ByteLex artifacts, ByteLex code, and Mini-Beatrix training materials show the atlas/tooling and generated lexicon work feeding into the subsequent run rather than being invented only after the run succeeded.

I would still avoid calling the original eight-word statistic “confirmed,” but I would give the subsequent intervention real credit.

The cheapest confirmation now seems almost embarrassingly simple:

  • freeze the current field/statistic;
  • optionally freeze the easy/trap bands too;
  • generate a fresh set of invented words that was not used to choose the metric;
  • record predicted rank or difficulty bucket;
  • only then run the model evaluation.

No new training run is required for that.

Even a fairly modest fresh set would answer a much cleaner question than squeezing more inference out of the original eight points.

What the 48-word run establishes — and what it still mixes together

I think the 48-word result supports something like:

Atlas-informed lexical intervention changed the failure behavior in the intended direction.

That is already useful.

I would be a little more careful with the stronger statement:

Atlas difficulty targeting itself caused the gain.

Going from the original setup to the 48-word setup changes several things together:

  • 12 words → 48 words;
  • broader lexical coverage;
  • more surface diversity;
  • generated rather than manually chosen material;
  • screening/distinctness constraints;
  • explicit easy/trap structure;
  • atlas difficulty targeting.

So one small control could make the causal story much easier to read:

generate another 48-word set with the same generator, size, novelty/distinctness screens, training recipe, and evaluation, but do not use the atlas difficulty score when choosing the words.

If:

targeted-48 > difficulty-blind-48

repeats across seeds, then the case for difficulty targeting itself gets much stronger.

If instead:

targeted-48 ~= difficulty-blind-48

that would still be useful. It would suggest that lexical breadth/diversity or the generation/screening process was doing much of the work.

Either outcome helps; it is not really a pass/fail control.

And I would not necessarily expand this into a giant factorial experiment. If compute or attention is limited, one matched blind-48 control seems like a very high-information next run.

The look-alike failure may be the most useful next fault to instrument

The new look-alike behavior actually caught my attention more than another global atlas correlation would.

If the old word-specific ladder became much flatter, but failures now concentrate when a plausible look-alike competitor is present in the prompt, that sounds like a cleaner intervention target.

I would try to distinguish:

intrinsic word difficulty

from:

context-induced competitor confusion

with a paired evaluation.

For the exact same target/problem:

A: target appears, no look-alike competitor
B: same item, insert the look-alike competitor

Then optionally:

C: insert a different control word with similar length/frequency
D: swap which look-alike is present

That gives a direct perturbation rather than another correlational statistic.

A few measurements could then make the failure much more local:

  • correct-target vs. competitor logit margin at byte 1;
  • the same margin at byte 2 and byte 3;
  • continuation entropy after the shared prefix;
  • shared byte-prefix length;
  • atlas path overlap between the two surfaces;
  • whether the wrong completion is specifically the inserted look-alike rather than an unrelated word.

The interpretation then branches fairly naturally:

partner insertion alone causes the collapse
    -> contextual competition is implicated

the target is still bad without the partner
    -> intrinsic word/path difficulty remains

the margin is already bad at byte 1
    -> early commitment remains plausible

byte 1 is healthy, then byte 2/3 collapses
    -> discrimination after commitment may be the better description

That seems especially useful because it follows the same pattern that worked earlier:

find the residual fault → localize it → design the next measurement around that fault.

I would probably prefer that over immediately adding more global atlas features.

Cross-tokenizer: I would separate address coverage from supervision fidelity

The cross-tokenizer direction seems plausible to me, but I think there are two different layers here.

The first is structural:

Which byte-space addresses / paths are represented by each tokenizer vocabulary?

ByteLex seems well suited to making that geometry inspectable.

But I would be careful not to turn:

“this address appears in the Qwen-derived field but not the English-tokenizer-derived field”

directly into:

“the byte student has never seen this region during training.”

Those are different statements.

Tokenizer-vocabulary coverage is structural evidence. Actual student exposure depends on the training corpus and model history.

For an actual distillation channel, I think the useful accounting becomes:

address coverage
    ↓
teacher probability mass covered by those addresses
    ↓
mass transferred exactly / approximately / residually
    ↓
ambiguity of each route
    ↓
actual distillation loss / downstream gain

That distinction also shows up in recent cross-tokenizer distillation work.

A few useful comparison points:

  • Cross-Tokenizer Distillation via Approximate Likelihood Matching tackles cross-tokenizer distribution matching directly and includes transfer toward byte-level tokenization.
  • Byte-Level Distillation is especially relevant here: it explicitly uses bytes as the common interface between mismatched teacher/student tokenizers.
  • SimCT frames one failure mode as lost supervision when exact shared-token matching discards positions that tokenize differently.
  • Byte-Prefix Marginalization makes probability-mass preservation explicit, including residual mass that does not map cleanly.
  • SimpleOPD takes a more conservative shared-text-space route and only aligns predictions occupying identical text spans.
  • NVIDIA NeMo RL now has an implemented xToken off-policy distillation path, using a precomputed projection matrix to bridge tokenizer vocabularies.

I do not think these make ByteLex redundant.

If anything, they help separate what ByteLex might uniquely contribute.

The byte-level common substrate itself already has independent support as a sensible CTD interface. What ByteLex may add is a structural diagnostic/routing layer over that substrate: identifying where coverage is weak, where mappings are ambiguous, where continuation structure differs, and perhaps where teacher mass is likely to need a nontrivial route.

So before a large cross-tokenizer training campaign, I would probably build a small accounting table like:

quantity meaning
address coverage structural overlap
teacher mass captured how much supervision reaches mapped regions
exact-route mass directly transferable
approximate-route mass requires alignment/marginalization
residual/dropped mass not safely assigned
route entropy ambiguity of the mapping
downstream delta whether any of this helps the student

Then “coverage improved” and “distillation improved” cannot accidentally become the same claim.

That seems like a particularly useful boundary to keep if the routing prototype grows.

One smaller point about the 3-byte match

The 3-byte correspondence is interesting, but I would keep it in the “suggestive mechanism” bucket for now.

The Mini-Beatrix-2s model card describes raw UTF-8 byte input with a byte-trigram embedding.

The atlas also works over 3-byte cells.

That gives both systems a matching local width, which makes the empirical relationship less mysterious.

But they are still not literally the same representation:

  • the model composes learned byte-level representations;
  • the atlas treats a trigram as a coordinate/cell carrying statistics derived from tokenizer vocabularies.

So I would be comfortable saying:

“the matching 3-byte support is a plausible clue”

but not yet:

“the atlas works because it matches the model’s trigram embedding.”

A 2/3/4-byte atlas ablation could eventually probe that, but I would rank it below the fresh-word confirmation and the paired look-alike test because those answer more immediate questions at lower conceptual cost.

So overall, I think the most interesting part of this is not actually the headline rho.

It is the workflow:

ordered success
    -> shuffle exposes shortcut
    -> residual failures remain
    -> localize them to specific words / byte positions
    -> find an external structural signal
    -> use that signal to change training material
    -> old failure mode flattens
    -> a new, narrower failure becomes visible

That is a pretty useful debugging pattern even if the eventual explanation of why the atlas works changes.

If I were choosing only one next step for each claim:

  • prediction claim: freeze the metric and test fresh words;
  • intervention attribution: matched difficulty-blind 48;
  • current residual failure: paired look-alike perturbation;
  • cross-tokenizer routing: track probability mass, not only address coverage.

None of those requires changing the overall direction of the project. They mostly separate claims that are currently sitting on top of each other, so whichever parts survive become easier to reuse later.

5

We already pushed to 96 words and had fair results. The current version should be trained with that behavior.

The upcoming V3 train is going to be 24 blocks, structurally identical, and the training process is going to be 4x more tokens and considerably different, so it needs to begin sooner than later.

Also I’m out of Fable for now, so I’ll need to wait until Friday 10 pm to ask fable the answer. I can ask Opus and relay the response, but Fable is more updated on the research.

6

This is one of the most interesting cross-tokenizer structural analyses I’ve seen, especially because it aligns directly with the 3‑byte receptive field of byte-level LMs. Since you asked for deeper technical discussion, here’s a compact expert-level breakdown plus a NumPy-optimized prototype that mirrors the ByteLex approach.

Technical notes (expert-level)

The key insight is that a byte-level LM with a 3-byte input composition implicitly defines a fixed coordinate space of size 256³. Any UTF‑8 token from any tokenizer can be projected into this space via overlapping 3-byte windows. This allows you to build a shared substrate across tokenizers without corpus statistics or model weights.

Two structural metrics emerge as predictive:

  • Dominance: max continuation probability at a 3-byte address. High dominance = English strongly prefers to continue (print → printer).
  • Endability: probability that a token ends exactly at that address. Low endability = “English refuses to stop here.”

Words whose byte-path crosses high-dominance / low-endability regions are systematically harder for byte LMs. The fact that this correlates at −0.98 with model accuracy (n=8) is less important than the rank stability across five weighting schemes and the transferability when training adapters on map-generated hard words.

The cross-tokenizer coverage arithmetic is also compelling: you can quantify exactly which regions a teacher model occupies that a byte-level student has never witnessed. This turns cross-tokenizer supervision from an opaque process into a measurable channel.

Below is a NumPy-optimized prototype that:

  1. Builds a 3-byte map from multiple tokenizer vocabularies
  2. Computes dominance and endability
  3. Merges 11 tokenizers into a consensus map
  4. Computes correlation with a real model’s per-word accuracy

It’s a minimal but faithful version of the ByteLex idea.

NumPy-optimized byte-map builder (multi-tokenizer)

import numpy as np
from collections import Counter, defaultdict
from scipy.stats import spearmanr

def utf8_bytes(s):
    return np.frombuffer(s.encode("utf-8"), dtype=np.uint8)

def three_byte_windows(b):
    if len(b) < 3:
        return np.empty((0,3), dtype=np.uint8)
    return np.stack([b[i:i+3] for i in range(len(b)-2)], axis=0)

def build_map_for_tokenizer(vocab):
    """
    vocab: dict {token: weight}
    Returns:
        occupancy: Counter of 3-byte windows
        continuation: dict {window: Counter(next_byte)}
        endability: Counter of window endings
    """
    occupancy = Counter()
    continuation = defaultdict(Counter)
    endability = Counter()

    for token, weight in vocab.items():
        b = utf8_bytes(token)
        windows = three_byte_windows(b)

        for i, w in enumerate(windows):
            w_tuple = tuple(w.tolist())
            occupancy[w_tuple] += 1

            if i + 3 < len(b):
                continuation[w_tuple][int(b[i+3])] += 1

            if i == len(windows) - 1:
                endability[w_tuple] += 1

    return occupancy, continuation, endability

def merge_maps(maps):
    """
    Merge maps from multiple tokenizers.
    maps: list of (occupancy, continuation, endability)
    """
    occ = Counter()
    end = Counter()
    cont = defaultdict(Counter)

    for o, c, e in maps:
        occ.update(o)
        end.update(e)
        for w, cnt in c.items():
            cont[w].update(cnt)

    return occ, cont, end

def compute_metrics(occ, cont, end):
    metrics = {}
    for w in occ:
        occ_w = occ[w]
        cont_w = cont[w]

        if cont_w:
            total = sum(cont_w.values())
            dominance = max(cont_w.values()) / total
        else:
            dominance = 0.0

        endability = end[w] / occ_w if occ_w > 0 else 0.0

        metrics[w] = (dominance, endability)

    return metrics

def score_word(word, metrics):
    """
    Score a word by averaging dominance and endability over its windows.
    """
    b = utf8_bytes(word)
    windows = three_byte_windows(b)
    if len(windows) == 0:
        return (0.0, 0.0)

    doms, ends = [], []
    for w in windows:
        w_tuple = tuple(w.tolist())
        if w_tuple in metrics:
            d, e = metrics[w_tuple]
            doms.append(d)
            ends.append(e)
        else:
            doms.append(0.0)
            ends.append(0.0)

    return (float(np.mean(doms)), float(np.mean(ends)))

def correlate_with_model(words, model_accuracy, metrics):
    """
    words: list of invented words
    model_accuracy: list of accuracy values
    metrics: byte-map metrics
    """
    predicted = []
    for w in words:
        d, e = score_word(w, metrics)
        # Hardness proxy: high dominance + low endability
        hardness = d - e
        predicted.append(hardness)

    return spearmanr(predicted, model_accuracy)
7

Thanks. This is the most useful kind of reply. Several of your asks were run before or shortly after your post landed, so here is where each one stands. The full tables are in the update below.

Freeze the metric, then predict fresh words. We ran a version of this, and your discovery-vs-confirmation line held up. Before training on 96 freshly minted words (all screened against every earlier word), we registered the forecast: a predicted band for each of 120 word/exam pairs, six yes/no calls, and a fixed scoring rule. The rule: per-word error must beat a base-rate guess (each exam’s average applied to every word) on both seeds.

  • It beat base rate on both seeds, but only barely: per-word error .258 / .245 vs .264 / .253.
  • Only 14% / 11% of words landed inside their predicted band, so the bands were overconfident.
  • 3 of 6 calls were right, below our own 4-of-6 pass bar.

So the −0.98 stays what you called it, an exploratory signal. The forecast is not validated. The one structural call that held: the words predicted hardest did score lowest (.562 vs .858 for the easy ones).

A second result bears on this. The statistic stops predicting once you train against it: −0.07 to −0.16 across ten adapters trained on atlas-picked hard words, against −0.98 on the untreated one. Only the extreme tail keeps a slope. Difficulty has to be re-measured after every training stage; it isn’t a fixed property of a word.

The difficulty-blind 48-word control. Not run yet. Our later data makes your concern sharper, because breadth and exposure clearly do a lot of the work. At the same 4,000 training steps, going from 48 to 64 words (plus look-alike pair rows) took the original 8-word exam from .867 / .847 to .960 / .907. At 96 words it fell back to .833 / .793. So a blind control needs matched exposure per word as well as matched size, generator and screens. It’s on the list.

The paired look-alike test. We trained commitment directly, with rows that contain full look-alike pairs. Accuracy with the partner in the prompt rose with the dose:

  • no pairs: .26–.35
  • 4 pairs: .41–.44
  • 8 pairs: .45–.51
  • 12 pairs: .435 / .447 (it stalled)

With the partner absent, accuracy stays near .89. That is still the split across different items, not your same-item A/B/C/D design with byte-1/2/3 margins. We agree that’s the right next step to localize the fault, and we haven’t run it.

Coverage vs exposure; probability mass, not addresses. Agreed. “Never witnessed” overstated it: vocabulary coverage is structural, and what the student actually saw depends on its corpus. The supervision channel is still designed, not built. When it exists we’ll report mass captured, exact, approximate and residual, plus route entropy, against a downstream result, as you laid it out. Thanks for the references; byte-level distillation and byte-prefix marginalization are closest to what we have in mind.

The 3-byte match. Fair: a plausible clue, not a mechanism. A 2/3/4-byte version of the atlas is cheap to score against the per-word accuracies we’ve already logged, so it doesn’t need a training run. It sits behind the two tests above.

8

Thanks for building it. The overall shape is right. One difference matters if anyone uses it to reproduce the numbers:

  • Where the score is read. The predictive statistic comes from a single address: the word’s last 3 bytes. It is the share taken by the most dominant next byte at that address, which answers “does English insist on continuing after this word ends?” prin fails because after rin one continuation dominates (print, printing…). Averaging dominance and endability over every window dilutes that single reading with the inside of the word, which is mostly unremarkable.
  • The consensus weighting matches yours: each vocabulary contributes a flat count per token, summed across the 11. Merge-rank weighting was the weaker variant (−0.79 with one tokenizer).
  • It’s a moving target. Once an adapter is trained on atlas-picked hard words, the same statistic stops predicting its errors (−0.07 to −0.16). A static scorer like this is a before-training tool only. Details are in the update below.

The library version is pure stdlib, with about 200 ns per lookup: WeightField.consensus and WeightField.word_end_caveat in geolip-bytelex 0.2.1 (GitHub - AbstractEyes/geolip-bytelex: A universal tokenization to byte translation comparator lexicon system. · GitHub).

9

Update: five more runs, a forecast registered before training, and a live demo

Drafted by Claude (Opus 5) from the run ledgers. Every number is measured, on two seeds unless stated.

A correction first: the base model saw 16.1B bytes, not tokens.

1. A forecast registered before training worked only partly.
Before training an adapter on 96 freshly minted words (36 easy, 36 hard, 12 look-alike pairs, all screened against every earlier word), we wrote down a predicted band for 120 word/exam pairs, six yes/no calls, and the scoring rule. The rule: per-word error must beat a base-rate guess on both seeds.

seed 1 seed 2
per-word error, forecast .258 .245
per-word error, base-rate guess .264 .253
words inside their predicted band 14% 11%
  • It beat base rate on both seeds, barely, and its bands were far too confident.
  • 3 of 6 calls were right: trained-word accuracy, closure on unseen words, and “the hardest words will score lowest” (.562 vs .858).
  • It missed the other three. Our own pass bar was 4 of 6, so the forecast is not validated. What survives is a weak per-word prior plus the structural checks.

2. Difficulty moves once you train against it.
The statistic that tracked the first adapter’s failures at −0.98 reads −0.07 to −0.16 across ten adapters trained on atlas-minted words. Only the extreme tail keeps a slope: the hardest words score .53–.65, everything else .70–.96. The failures that remain are mostly look-alike pairs, which that statistic doesn’t measure. Any difficulty estimate has to be re-fit after each training stage, and the next pool minted at the current frontier.

3. Breadth pays only with enough exposure.
Same recipe and the same 4,000 training steps each time; accuracy on the original 8-word exam, which was never trained on:

words in the training pool seed 1 seed 2
48 .867 .847
64 (incl. 8 look-alike pairs) .960 .907
96 (incl. 12 pairs) .833 .793

At 96 words the curve bends back: with fixed steps, each word is seen too few times. What buys transfer is exposures per word, not word count. Look-alike commitment behaves the same way: partner-in-prompt accuracy goes .26–.35 → .41–.44 (4 pairs) → .45–.51 (8 pairs), then stalls at 12 pairs (.435 / .447). Training pools are now sized by exposures per word.

4. The drift fix held on both seeds.
The “stay quiet off-task” term from the first post (then one seed in flight) keeps general-text drift at +.0054 / +.0065 bits per byte, under our .012 bar, while held-out accuracy rose (.867 / .847, up from .840 / .793). On the 96-word pool the same setting reads +.0113 / +.0121, right at the bar, so the term has to scale with pool size.

5. Training a behavior into the whole model instead of an adapter is expensive.
We fine-tuned every parameter of the base model on three-digit subtraction, the task adapters had failed on.

  • Held-out accuracy stayed at or below .03.
  • General-text loss went from 1.11 to 1.85–2.45 bits per byte. The model’s prior was destroyed for a procedure it still couldn’t do.
  • By comparison, an adapter learns the same turn-ending behavior for about +.002.

Behaviors stay in adapters.

6. Two adapters trained together stay separable, because of the quiet term.
We loaded a trained chain adapter, put a fresh turn-ending adapter over it, and trained both under one optimizer, each with its own quiet term. With the turn-ender switched off, the chain adapter scores .833 / .840, against .867 / .853 on its own. The same setup without the quiet term had scored .013. So the quiet term is more than drift control: it is what keeps co-trained adapters detachable. One pre-registered line missed by a hair, domain selectivity at 2.44 against a 2.5 bar on one seed.

7. Try it.
The demo Space now serves every adapter from these experiments, 68 configurations including stacked, routed and recurrent ones. Each is labelled with its measured score and how settled it is. It opens on the best measured one, the 64-word look-alike adapter (.960 on the original words, .640 on unseen ones). With an adapter attached, each message is answered on its own; Core only keeps the whole conversation.

Next. A larger base model: 24 blocks, the same structure, about 4× the training data, with curriculum pools sized by exposures per word. From this thread: the difficulty-blind control at matched exposure, and the same-item look-alike test.

I expect this experiment to yield the necessary measurements needed to reduce the word error further. For now we’re working at around a 20-25% word error chance depending on the structure.

Claude Opus didn’t recognize the details of the tokenizer standalone testing as it was on fable and I don’t think fable marked it into the system correctly, so I’ll need to wait until Friday 11pm gmt -8 to get the necessary update from Fable to make sure the numbers are lined up and correct.

There is definitely newer information than this.

10

Below is the 3‑byte structural map code I used. It’s designed to isolate UTF‑8 byte‑level regularities inside the tokenizer’s vocabulary, without assuming any semantic structure or downstream behavior. The goal is to quantify purely structural signals (occupancy, continuation bias, and endability) that might correlate with per‑token difficulty or stability across training stages.

The pipeline is intentionally simple:

• Convert each token to its raw UTF‑8 byte sequence.
• Extract all consecutive 3‑byte windows.
• Accumulate three statistics:
– occupancy: how often each 3‑byte window appears
– continuation: conditional distribution of the next byte after each window
– endability: how often a window appears at the end of a token

These three quantities form the basis for dominance and endability metrics:
dominance = max(next_byte_count) / sum(next_byte_counts)
endability = endability_count / occupancy_count

The code:

import numpy as np
from collections import Counter, defaultdict

def utf8_bytes(s):
return np.frombuffer(s.encode(“utf-8”), dtype=np.uint8)

def three_byte_windows(b):
if len(b) < 3:
return np.empty((0,3), dtype=np.uint8)
return np.stack([b[i:i+3] for i in range(len(b)-2)], axis=0)

def build_map_for_tokenizer(vocab):
occupancy = Counter()
continuation = defaultdict(Counter)
endability = Counter()

for token, weight in vocab.items():
    b = utf8_bytes(token)
    windows = three_byte_windows(b)

    for i, w in enumerate(windows):
        w_tuple = tuple(w.tolist())
        occupancy[w_tuple] += weight

        # continuation
        if i + 3 <= len(b) - 1:
            next_byte = int(b[i+3])
            continuation[w_tuple][next_byte] += weight

        # endability
        if i == len(windows) - 1:
            endability[w_tuple] += weight

return occupancy, continuation, endability

This structure is intentionally agnostic to the model’s training dynamics. It only captures the tokenizer’s byte‑level geometry. The idea was to check whether any of these structural biases (especially dominance and endability) correlate with per‑word exam accuracy, difficulty slopes, or the stability of difficulty across training stages.

If you run a 2‑byte / 3‑byte / 4‑byte variant, the same counters generalize directly. It’s cheap to compute and can be scored against existing per‑word accuracies without additional training runs.

11

Without checking Claude I can say unequivocally that full UTF-8 is on the horizon. I’ve run multiple experiments on it when handling SVAE prototypes, which by the way isn’t a traditional SVAE. I would like to call them encoder/decoder miniature prototypes to the splat attention really.

It might be wise to revisit the Alexandria build with some new information.

I can say for certain we measured trigram in multiple ways, 2gram, 4gram, and 5gram are on the todo. I have a personal bias towards 5gram because of the patterns I’ve seen in physics such as the Saturn penta and multiple penta that present themselves such as the natural biological form arms, legs, and head - and so on. I was drawn to the shape originally as a default structure, so I’m biased as a person.

The AI keeps me honest and centered with the honest mathematical outcomes and data. Discovery of the SVAE convergence would have been dismissed by opus 4.7 if I didn’t catch it. It was labeled as too easy, when I knew something was clearly different with the variants, so I ran the sweep that created the originals.

In any case, I’m certain that a trie map can be forged to create a lookup spectra for multiple tokenizers simultaneously. I’ve seen the results on countless structures that prove this can converge, so it’s just a matter of aligning the mathematics at this point.

12

The tokenizer geometry and multi-tokenizer trie idea could reveal some useful patterns once the math lines up.

13

The tokenizer fusion has a prototype now. It has a solid entropy outcome as well, we’re looking at some real potential here. The geometric forms are aligning, something that standard entropic decay fails to articulate as shown with the control subject so far.

I’ll need to retrain a full MHA control variant for version 3 so we get a solid baseline here. I would like some real competition.

As it stands, the V3 will take roughly 2 weeks to complete on a single rtx 6000, so I’m going to do some rentals to see if I can speed things along. I’m thinking a heap of A40s might do the job and be cheaper. We’ll see what happens.

image
image1182×274 28 KB

Claude and I agreed on a standard unit of measurement, seemed fitting to the dynamic nature of the model and the modularization of Beatrix in general.

These aren’t a token, but rather an entropic measurement of differentiation and byte requirement per corpus. Full corpus precalc for the 47b byte multi-unit hybrid structure is beginning now. The entropy and weighted behavior is essentially allowing us to form our own full-model atlas and lexical entropy at runtime. This is directly utilizing the under the hood codebook system present throughout the entire model.

The model will converge, so lets see how well she steers when she’s done.

14

image
image1179×385 33.4 KB

Looks like we have some engineering solutions. It’s a bit of bits per byte increase but the yield is going to be interesting.

More tests needed, neither Claude nor I am satisfied that the model will properly converge using the refined byte-fusion token gate, but the tests are promising and yielding some substantial gains. She may need to be wider or deeper, potentially additional splat heads.

The lexicon fusion wasn’t stable yet. We’ll need to shelve the fusion for V3, as much as i wanted the speed I’ll need to align the formulas closer before I attempt a full train with it. As much as I want to obsess over it for 2 weeks to get proper byte fusion down for V3, I’ll shelve it for a permanent implanted arm later to improve the inference speed.

V3 will have the atlas cleanly represented, and that will have a substantial series of bytelex representative structures capable from it. The hope is to unlock the floodgates of potential, training the new deeper model will enable many new behaviors that the current 20 block variant cannot support.

I was thinking about 24 but now I’m thinking 32 might be in order rather than 24. The stability is showing depth improves the model substantially. The 2s depth didn’t fully exhaust the internal space according to the assessments. A deeper variant may not yield beyond block 22 for some, however for others the penultimate layer will yield substantial information for others. Especially routed deeper byte-agnostic structures.

This next version will not have the fusion system, however it will have the depth required to get the real answers for it’s gate limits, whether it be a vector for scale, or a potential for sizing.

In any case, it will take about 2 days to fully prep the test cases for V3 before I train her.