For now, I gave it a quick run:
I was able to load the public checkpoint and do a small matched evaluation against two unusually convenient public models in almost the same parameter range.
On exactly the same 65,643 normalized raw target bytes sampled from the TinyStories validation set, I got:
So, for the immediate goal you mentioned — getting this checkpoint into the range of a decent TinyStories competitor — there is a measurable TinyStories predictive-fit gap to those two TinyStories-specialized models.
I would not read that table as an architecture ranking, though.
The important confound is already in your post: Prism was trained on a mixed corpus, while those two comparison checkpoints were trained specifically on TinyStories. The narrow claim I think the measurement supports is simply:
On this TinyStories held-out sample, the current mixed-corpus Prism checkpoint predicts the text less efficiently than these two ~100K TinyStories-specialized checkpoints.
That is a useful target for the next few days of grinding, but it does not isolate Prism vs Transformer vs Mixer as architectures.
I also went back through your Zenodo PrismFormer paper, and that changes the framing in an important way: you have already done a parameter-matched language-model control there. In §4.5, the language experiment compares a 249,920-parameter PrismFormer against a 259,776-parameter dense transformer, with shared vocabulary, context, training order, LR schedule, and approximately matched parameter budget. Over five runs, the paper reports:
| Model |
Next-char accuracy |
bits/char ↓ |
| PrismFormer |
41.0% ± 1.9% |
3.399 ± 0.081 |
| matched transformer |
21.7% ± 3.6% |
4.652 ± 0.273 |
| unigram baseline |
19.1% |
5.322 |
That is answering a much cleaner matched-condition architecture question than my TinyStories probe.
At the same time, §6 of the paper is explicit that the internal dense-transformer control is not meant to be the last word: it is gradient-checked and does fit the training data, but it has no layer norm, no warmup, and no tuned optimizer, and the paper itself lists comparison against a well-tuned standard transformer as future work.
That makes the new ~100K TinyStories models interesting for almost the opposite reason: they are stronger, more conventional external reference points, but the data/training conditions are no longer matched.
So I think the two comparisons complement each other rather than contradict each other:
- Zenodo §4.5: much cleaner matched-data/matched-budget control, but against the deliberately simple internal transformer baseline.
- Current TinyStories probe: stronger public TinyStories-specialized reference models, but no longer a controlled architecture comparison.
For the immediate TinyStories goal, I think the lowest-cost path is therefore just to keep the thing you are already doing and add two stable measurements alongside it:
- a fixed TinyStories held-out bits-per-byte (BPB) number for each checkpoint;
- a small fixed set of generation prompts.
TinyStories already publishes both the validation data and evaluation prompts, so this does not need to turn into a large benchmarking project.
What I compared, and a few possible next branches
What I actually ran
For the public Prism checkpoint I tested, I could reproduce the model as:
| Property |
Value |
| Parameters |
107,776 |
| Vocabulary |
256 |
| Dimension |
96 |
| Layers |
6 |
| Context |
192 |
| Serving iteration/pass setting |
2 |
The public vocabulary is the 256-entry setup expected by this checkpoint, using the base character vocabulary plus learned subword merges.
The awkward part in comparing this checkpoint with MicroT/MicroMixer is that Prism is using subwords while those two models are byte-level. A token-level perplexity comparison would therefore be misleading: one Prism token and one byte token can represent different amounts of source text.
Hugging Face’s perplexity documentation also notes that perplexity depends on tokenization.
So for the comparison above I worked back to raw text:
- take deterministic context-safe segments from
TinyStories-valid.txt;
- tokenize each segment with Prism;
- use Prism’s first complete token as the conditioning prefix;
- recover the exact remaining raw bytes;
- score those same remaining bytes with Prism, MicroT, and MicroMixer.
The final common target was:
- 368 segments;
- 65,643 target bytes (64.104 KiB) per model;
- identical raw target bytes for all three models.
I also checked that the Prism tokenizer round-tripped all of those segments exactly, that every segment stayed within the 192-token Prism context, and that the cached inference path agreed with full recomputation on the sanity check.
This removes the obvious tokenizer-boundary mismatch.
It does not make the training conditions equivalent.
That distinction is important here.
There are now two different control questions
After reading the paper, I would separate the controls like this.
1. Does Prism itself have an interesting language-model signal at matched size and matched conditions?
Your paper already addresses this.
The §4.5 experiment uses a small 1,670-character language corpus and a character-level next-token task. The PrismFormer and dense transformer share vocabulary, context, training order, LR schedule, and matched parameter budgets. The reported PrismFormer result is substantially better under that experiment.
The paper is also careful about the scope: both models are weak in absolute terms, and it describes the result as a matched-size comparison rather than an absolute language result.
I think that is the right interpretation.
2. How does the current 107K checkpoint compare to reasonably strong public models in the same size neighborhood?
That is a different question, and that is where the two TinyStories models are useful.
MicroT-test1-100K-TinyStories is a 97,872-parameter decoder-only Transformer with causal attention, RoPE, RMSNorm, tied input/output embeddings, a 256-byte vocabulary, and TinyStories-specific pretraining.
MicroMixer-4-100K-TinyStories is a 95,084-parameter attention-free model, also byte-level and TinyStories-pretrained.
That gives a useful three-way neighborhood:
| Model |
Params |
Tokenization |
Training |
Aligned BPB ↓ |
| Prism |
107,776 |
subword, vocab 256 |
mixed corpus |
2.0137 |
| MicroT |
97,872 |
byte, vocab 256 |
TinyStories |
1.8211 |
| MicroMixer |
95,084 |
byte, vocab 256 |
TinyStories |
1.6860 |
I would treat these as external landmarks, not as a controlled architecture experiment.
The fact that the gap persists over ~64 KiB is useful for tracking the current checkpoint. The reason for the gap remains underdetermined: architecture, training distribution, optimizer/training recipe, amount of useful TinyStories exposure, or some combination of those can all contribute.
Why BPB rather than raw token perplexity
For this particular comparison, raw token perplexity is awkward because the tokenizers differ.
BPB asks, roughly:
How many bits of model loss are needed per byte of the original text?
That gives the three models a common raw-text denominator even though they package that text into tokens differently.
It solves one comparison problem — different token packing.
It does not solve unrelated ones such as:
- different training corpora;
- different optimization;
- different numbers of training tokens;
- different sequence lengths during training;
- architectural differences.
So I would use the BPB value as a checkpoint-tracking number, not as an all-purpose score.
The generation roughness does not look uniquely Prism-specific
The repetition and eventual drift in the sample from the post initially looked like something worth isolating, but the nearby baselines are useful context here too.
Both public ~100K TinyStories cards contain generations that can start as recognizable children’s-story prose and then become repetitive or semantically unstable later.
For example, the MicroT and MicroMixer cards explicitly show some continuations as “drifts” after an initially coherent start.
Hugging Face’s generation-strategy documentation also notes the general tendency of greedy decoding to become repetitive on longer sequences, and there is a Transformers issue specifically about stopping repetition loops in small-model evaluation.
I also tried the Prism checkpoint with a deliberately plain greedy diagnostic path rather than treating the browser-side confidence/repetition handling as part of the weights. The raw model still produced recognizable TinyStories-like continuations, together with the expected roughness and repetition.
So I would not remove the nicer serving logic. I would just keep the measurements separated:
| Lane |
What it tells you |
| held-out BPB |
predictive fit of the weights |
| fixed prompt + simple fixed decoder |
diagnostic generation behavior |
| normal Prism decoder / guards |
actual demo behavior |
That makes it easier to tell whether a later improvement came from training or from decoding.
One small comparison trap I noticed: I would not equalize generation by token count between Prism and the byte-level models. Ninety-six Prism subword tokens can represent substantially more text than ninety-six byte tokens.
If comparing generations, equal raw-byte/character length — or at least reporting it alongside token count — is cleaner.
If the immediate goal is “TinyStories competitor”
This looks like the simplest branch.
You could keep a fixed validation sample and log:
- BPB at selected checkpoints;
- generations from the same fixed prompts;
- optionally the normal guarded/demo output beside a plain diagnostic decode.
That gives a quantitative curve plus an intuitive one.
If the BPB moves from ~2.01 toward the ~1.8 region while generations become more stable, that would be fairly easy progress to see without trying to reduce everything to one score.
If later you specifically want to extend the architecture question from the Zenodo experiment into this TinyStories setting, the clean experiment would be something like:
- same TinyStories data;
- same tokenizer, if practical;
- same training exposure;
- similar parameter budget;
- same evaluation bytes;
- same diagnostic decoding conditions;
- Prism vs a well-tuned modern Transformer.
That would be the TinyStories analogue of the matched-control idea already used in the paper, while also addressing the stronger-baseline limitation that §6 already calls out.
I would regard that as an optional research experiment, though, not something required just to keep improving the browser model.
If the mixed-corpus model is actually the main goal
Then TinyStories BPB answers only one narrower question:
How well does this checkpoint model TinyStories-like text?
It does not tell you whether the mixed model is improving overall.
Since you mentioned that the available corpus weighting seems to have pulled generation heavily toward the TinyStories-like distribution, another low-cost branch would be to keep a few tiny held-out slices for the other important parts of the corpus.
If convenient, logging the actual source exposure in bytes/tokens during training would also make the mixture much easier to reason about.
I would treat the corpus weighting as something to separate experimentally, rather than assuming it is the cause of the current behavior.
A note on “842 KB” and “smallest”
There is also one measurement/detail that may save confusion if the “smallest” part attracts comparisons.
The checkpoint I fetched was 862,244 bytes.
With 107,776 parameters represented by 64-bit values:
107,776 × 8 = 862,208 bytes = 842 KiB
So the advertised 842 KiB is essentially the FP64 parameter payload.
That is useful information, but I would keep it conceptually separate from the parameter count:
- 107,776 parameters — architectural/model-size measure;
- 842 KiB FP64 weights — serialized weight footprint;
- runtime memory — another measure again.
Quantization can radically change a file-size comparison without changing the architecture’s parameter count, so defining which “smallest” axis you mean will prevent an otherwise interesting architecture experiment from turning into a serialization-format record argument.
The TinyStories connection is unusually appropriate here
The original TinyStories paper is almost exactly about the broader question behind this experiment: how small can language models get and still produce coherent English?
It demonstrates surprisingly capable small models, including shallow architectures, but it does not directly establish what should happen at ~100K parameters.
That makes the public ~95-98K models useful empirical neighbors, while your Zenodo language experiment supplies a different kind of evidence: a controlled matched-size comparison internal to PrismFormer.
I think keeping those two evidence types separate makes the picture stronger:
- the Zenodo experiment says something about Prism under controlled matched conditions;
- the TinyStories models say something about the performance level currently reachable by other public models around the same tiny parameter budget;
- the current 107K checkpoint shows where this particular mixed-corpus Prism run sits between those two contexts.
So after reading the paper, I would narrow my suggestion rather than expand it.
You already have the matched-control idea in the research work, and you already explicitly identified the need for a stronger modern Transformer baseline there.
For the model in this thread, I think fixed TinyStories BPB + fixed generations is enough to give the current grinding a useful target.
If the goal later becomes an architectural claim rather than improving this particular tiny LM, then a matched TinyStories run against a well-tuned ~100K Transformer would be a natural continuation of the question you already left open in §6.
And if the mixed-corpus behavior remains the more interesting part, I would leave that as its own branch rather than optimizing the whole project into becoming only a TinyStories model.