Eikos
Collection
Open, calibrated, single-pass typed-decision models (4B & 27B) for finance and trading • 10 items • Updated • 9
Eikos-4B — open typed-decision model (System-1)
Eikos-4B answers typed decisions about a given state in a single forward pass, with calibrated probabilities over a bounded set of options. It does not generate text at decision time.
Try it in the browser: Eikos-4B demo on Hugging Face Spaces (ZeroGPU; all three question types, up to 100 options in one pass).
- Question types:
noul/boolean(yes/no with probability),choice(one of N options) andscore(ordinal levels). - Outputs: the full probability distribution plus a confidence value, so callers can abstain below a threshold.
- Focus: global finance, trading and trade finance.
- Apply a stated rule, policy or rulebook to a case: order limits, margin, wash sales, price bands, payments limits, KYC/AML, lending policies, documentary credits, Incoterms® stage mapping, VAT/GST.
- Entity-level financial sentiment, answer verification over financial tables, temporal and numeric checks.
- Scope limit: it applies rules to the facts it is given. It does not predict prices.
- Languages: trained in English and Portuguese. Spanish was held out entirely and is used as a zero-shot test.
- Runs anywhere: one self-contained checkpoint in the official Qwen layout, with no adapters, no external LLM
and no API.
- vLLM, with prefix caching for many questions over the same state and for agent sessions;
- PyTorch / transformers;
- Apple Silicon via MLX or MPS.
- Images (serve v1.3): screenshots, charts, documents and photos, read zero-shot through the base model's vision encoder, with the same one-pass readout (GPU builds with vLLM). See Images (serve v1.3).
Eikos (εἰκός, "the probable") comes in two sizes: Eikos-4B (this model) and Eikos-27B, each with FP8 and INT4 builds for GPUs and MLX builds for Apple Silicon (4B: 8-bit and 4-bit; 27B: 4-bit). Training data:
caiovicentino1/eikos-decisions. Code:https://github.com/caiovicentino/eikos.Numbers below are from our own evaluation harness (see Evaluation). Official JevBench runs are requested, for this model and for Eikos-27B-FP8. The organizers run the models themselves.
How it works
Prompt. The state (evidence), the question (criterion) and the lettered options go into a structured prompt, following the SemIf format.
Readout. The model reads the next-token logits only over the option letters. A softmax with a fixed temperature gives the distribution. Options are labelled A–Z and then AA, AB, … (each label a single token): up to 100 options are read in the same forward pass (serve v1.2,
max_one_passindecision_config.json; beyond ~100 the 4B does better through a tournament whose final round is one pass).Calibration. Temperature T = 1 ships by default (see Calibration).
Training. Distillation from a strong teacher (GLM-5.3-Flash, maximum reasoning effort) on:
- generated and blindly labeled decision items;
- exact programmatic items;
- compositional rules;
- long-context dossiers (6k–32k tokens).
Training-only signals, all switched off at inference:
- soft cross-entropy to the teacher's probabilities, with option-order permutation;
- an auxiliary loss on the item's short rationale (written with the item, explaining the gold answer);
- EN↔PT view consistency (the same item in two languages must give the same answer).
The released 4B is a weight average ("model soup") of two fine-tunes that share data and seed: one with view consistency, and one that also adds a light JEPA-style latent objective (weight 0.2: mid-layer view invariance, latent rationale prediction and a state–option energy head). The average won on our internal dev set and on the held-out family, topic and language.
Quick start
vLLM (production: batching + prefix cache)
Requires vLLM ≥ 0.30.0. Older builds return wrong answers when several long requests are batched together on this hybrid (Gated DeltaNet) architecture; we measured drops of 3–6 points on long, shared-document items with vLLM 0.11. With 0.30 the batched results match the PyTorch reference.
bash serve_vllm.sh <MODEL_DIR> 8001 # vLLM engine: letter readout + hybrid prefix cache (checks vLLM >= 0.30)
python serve.py --model <MODEL_DIR> --vllm-url http://127.0.0.1:8001 --port 8000 # TypeSafe-compatible API on :8000
serve_vllm.sh runs vllm serve with the flags the readout needs: --served-model-name decider,
--enable-prefix-caching --mamba-cache-mode align, --logprobs-mode processed_logprobs --max-logprobs 600,
--dtype bfloat16 and --max-model-len 16384 (change it with MAX_MODEL_LEN).
It also sets --max-num-seqs 64 (change it with MAX_NUM_SEQS): each request in flight holds one hybrid
cache block, and vLLM's default of 1024 needs more blocks than a 27B build leaves free on one 80–96 GB GPU.
HTTP API (TypeSafe-compatible)
curl -s localhost:8000/v1/systemone -d '{
"state": "Order ticket #A-2231. Retail client, moderate risk. BUY 1,500 XYZ at market. Equity USD 48,000. Last price USD 41.20. Rule 4.2: a single order may not exceed 50% of equity without written supervisor approval. Approvals on file: none.",
"questions": {
"allowed": {"type": "noul", "instructions": "Under rule 4.2, can this order be executed as submitted?",
"criteria": {"true": "complies with rule 4.2", "false": "breaches rule 4.2"}},
"action": {"type": "choice", "instructions": "What should the desk do?",
"criteria": {"execute": "send as submitted", "request_approval": "hold and ask a supervisor",
"reduce_size": "cut the order to the allowed size", "reject": "refuse the order"}},
"risk": {"type": "score", "instructions": "How risky is this ticket for the firm?",
"criteria": ["low", "moderate", "high", "critical"]}
}}'
- One pass for the whole request: all questions are answered together, and the shared state is processed once through the prefix cache.
- Response per question:
noul:probability(of "yes"),valueandconfidence;choice:choice,probabilitiesandconfidence;score:score,probabilities,expectedandconfidence.
Agent sessions (incremental state + cache)
POST /v1/sessions {"state": "..."} -> {"session_id"}
POST /v1/sessions/<id>/append {"text": "..."} -> grows the state
POST /v1/sessions/<id>/systemone {"questions": {...}} -> answers over the current state
DELETE /v1/sessions/<id>
On a hybrid (linear-attention) model the state is cached at the end of the context and forked for every question. In our tests a 15-step agent session gave answers identical to sending the full request each time.
Local (Apple Silicon)
python mlx_decide.py <MODEL_DIR> # MLX backend: same readout and calibration as PyTorch
python local_demo.py <MODEL_DIR> mps # PyTorch MPS
Images (serve v1.3)
Eikos also decides about images: screenshots, charts, documents and photos. Our training was text only and left the
base model's vision encoder untouched; serve.py v1.3 passes the images to it and reads the answer with the same
one-pass letter readout, so every option still gets a probability and no text is generated. This needs the vLLM
backend (serve_vllm.sh) and works with all six GPU builds; the MLX builds are text only.
import base64, requests
img = "data:image/png;base64," + base64.b64encode(open("screen.png", "rb").read()).decode()
r = requests.post("http://127.0.0.1:8000/v1/systemone", json={
"state": "",
"images": [img],
"questions": {
"done": {"type": "noul", "instructions": "The upload finished without errors."},
"next": {"type": "choice", "instructions": "Which button saves the file?",
"criteria": {"save": "the Save button", "cancel": "the Cancel button", "none": "no such button"}},
},
})
print(r.json()["answers"])
- Formats:
"images"takes data URIs or base64 strings. multipart/form-data works too (the JSON in arequestfield and the images asimagefiles), and so do image data URIs inside thestate, in a string, a dict or list value, or chat-style content parts: they are taken out in order and replaced with[image N]. - Limits: up to 4 images per request; images above 3840×2160 pixels are scaled down; image URLs only with
serve.py --allow-image-urls. A 4K screenshot is about 8,300 tokens, within the 16k windowserve_vllm.shsets by default (MAX_MODEL_LENchanges it). - Every question in a request reads the same images. Sessions stay text only.
Zero-shot accuracy on public sets, used for evaluation only. GUI grounding is asked as "which part of the screen holds the element for this action?": a 3×3 grid described in words (no marks), or a labelled 4×4 grid drawn on the screenshot.
| Set | Items | Chance | Eikos-27B-FP8 | Eikos-4B |
|---|---|---|---|---|
| MME (yes/no) | 2,374 | 50% | 89.8% | 87.6% |
| MMStar | 1,493 | 25% | 72.3% | 66.9% |
| SEED-Bench-2-Plus (charts, maps, web pages) | 2,277 | 25% | 74.5% | 72.5% |
| ScreenSpot-v2, no marks | 1,272 | 11% | 73.7% | 57.5% |
| ScreenSpot-v2, 4×4 grid drawn | 1,272 | 6% | 60.9% | 53.9% |
| ScreenSpot-Pro, no marks (500 of 1,581) | 500 | 11% | 65.0% | 51.8% |
- Against the base model: in the same one-pass format, Eikos-27B-FP8 is better than Qwen3.8-27B (the bf16 base model) on all six sets (300 items each) and better calibrated: MME 90.0% vs 73.7%, MMStar 70.7% vs 65.3%, ScreenSpot-Pro 65.7% vs 59.0%. The text-only decision training carried over to images. This compares both models in this format, not the base model at its best with generated answers. Eikos-4B was not compared with its base model.
- Every GPU build on the same 300 items per set (the FP8 and INT4 builds keep the vision tower in bf16):
| 300 items per set | Eikos-4B | Eikos-4B-FP8 | Eikos-4B-INT4 |
|---|---|---|---|
| MME (yes/no) | 89.7% | 89.3% | 87.3% |
| MMStar | 67.0% | 67.3% | 66.7% |
| SEED-Bench-2-Plus | 73.7% | 74.0% | 71.7% |
| ScreenSpot-v2, no marks | 55.7% | 56.3% | 55.0% |
| ScreenSpot-v2, 4×4 grid drawn | 53.3% | 54.3% | 54.0% |
| ScreenSpot-Pro, no marks | 52.7% | 51.7% | 52.0% |
| Same answer as bf16: all / confident (≥0.9) | — | 95.2% / 100.0% | 88.9% / 99.2% |
- Limits: these are multiple-choice and yes/no decisions about a visible screen or picture. Pointing at an exact element without a grid or marks, multi-step reasoning about a screen and very small text were not measured. Calibration on images was not fitted separately: ECE is 0.03–0.10 per set, and both sizes are too confident on text-rich images (SEED-Bench-2-Plus: the 27B is 84% confident on average for 74.5% accuracy).
Evaluation
All numbers are our own measurements with a single harness. We never trained on any evaluation set. Spanish, one task family (trade-offs) and one topic (healthcare administration) were held out of training entirely.
| Benchmark (our harness) | 4B final | 27B final | Jev (reference) |
|---|---|---|---|
| JevBench public — original / hard | 91.7 / 72.1 | 100.0 / 82.9 | 98.6 / 73.0 (official leaderboard) |
| JevBench hard — ECE (lower is better) | 0.049 | 0.051 | — |
| DecisionBench (OOD) — medium / hard | 77.1 / 66.9 | 88.4 / 78.5 | 89.1 / 69.3 |
| General battery (9 human-labeled tasks) | 76.0 | 82.5 | 84.1 |
| Finance (CUAD, financial sentiment, FinQA-judge) | 74.7 | 85.3 | 79.9 |
| Financial sentiment (Twitter Financial News) | 79.3 | 87.7 | 68.3 |
| Central-bank stance (WCB, balanced acc.) | 38.4 | 44.5 | 58.6 |
| FinDVer (claim verification) | 74.0 | 78.0 | 73.3 |
| Trade rules — seen templates / unseen rules | 73.4 / 76.0 | 85.1 / 88.0 | 78.2 / 79.4 |
| Compositional rules — same type / new domain / rulebooks* | 95.8 / 91.7 / 91.7 | 95.8 / 94.3 / 95.3 | 83.4 / 84.3 / 88.0 |
| Held-out Spanish (never trained), internal | 90.9 | — | — |
| Long context: decision hidden in 64k tokens of text (mean of 3 positions) | 74.2 | 88.3 | — (API limit 32k) |
| Robustness: same answer after option reversal / paraphrase | 89.9 / 89.9 | 94.9 / 97.5 | — |
| RuleArena NBA (balanced acc.) | 0.50 | 0.50 | — |
* generated by the same rule generator family used in training (different rules and cases); see notes.
How to read the table:
- JevBench: public items only, not the official leaderboard run.
- Jev: measured through its API on our batteries, not distilled from.
- Laya (charts only): run with its official package; its JevBench scores are within one item of its official leaderboard results.
- Charts: 7,140 items common to all four systems. The confidence chart shows, for every confidence threshold, how many decisions a system would take on its own and how often those decisions are wrong.
- WCB (central-bank stance): reported as balanced accuracy, because the classes are imbalanced.
- Rules suites: "rules / new domain / rulebooks" come from the same generator family used in training, so
they measure learning of that generator. Transfer to rules the model never saw is measured by "Trade (unseen
rules)" (Incoterms®, documentary-credit presentation, EU VAT) and by RuleArena.
- "New domain" has two parts. 492 items come from a domain absent from training (insurance): 27B 95.1, 4B 91.5. The other 108 items use a rule combination held out of training (an exception over a business-day window): 27B 91.7, 4B 90.7. That combination appears in only 6 of the ~6,000 rule rows used in training.
Long context
The case file is hidden inside unrelated text at the start, middle or end of the prompt (80 public JevBench decisions per point; released checkpoints served with vLLM 0.30).
Release builds
Final evaluation of every published build, run on the exact files in these repos: 7 suites, 7,371 items, vLLM 0.30 with batching and prefix cache on (MLX builds: MLX's CUDA backend on Linux). Batched vLLM is not bit-for-bit deterministic across runs (1–2 items per suite can change), so the bf16 rows differ slightly from the Evaluation table above, which uses our reference harness. "≥0.90" = share of decisions the model would take on its own at ≥90% confidence, and the real error rate among them.
| Build | Size | JevBench orig / hard | DecisionBench med / hard | General | Finance | Unseen trade rules | ECE | ≥0.90: decides / error | Gate vs bf16 |
|---|---|---|---|---|---|---|---|---|---|
| Eikos-27B | 55.6 GB | 100.0 / 82.0 | 89.1 / 78.2 | 82.6 | 85.4 | 87.4 | 0.043 | 44.1% / 2.5% | reference |
| Eikos-27B-FP8 | 31.2 GB | 100.0 / 83.8 | 88.7 / 78.8 | 82.8 | 85.3 | 86.9 | 0.042 | 44.2% / 2.6% | pass |
| Eikos-27B-INT4 | 19.4 GB | 100.0 / 82.9 | 88.4 / 78.5 | 82.5 | 84.9 | 88.6 | 0.041 | 43.6% / 2.4% | pass |
| Eikos-27B-MLX-4bit | 15.1 GB | 100.0 / 83.8 | 88.4 / 78.2 | 81.8 | 84.9 | 85.7 | 0.041 | 42.9% / 1.9% | pass except agreement |
| Eikos-4B | 9.3 GB | 91.7 / 73.9 | 77.1 / 66.6 | 75.7 | 74.7 | 76.0 | 0.033 | 34.7% / 2.5% | reference |
| Eikos-4B-FP8 | 5.8 GB | 90.3 / 73.0 | 78.5 / 66.6 | 75.8 | 75.2 | 76.0 | 0.031 | 34.8% / 2.5% | pass |
| Eikos-4B-INT4 | 4.0 GB | 91.7 / 72.1 | 77.1 / 66.2 | 75.6 | 75.7 | 76.0 | 0.033 | 32.9% / 2.0% | pass except agreement |
| Eikos-4B-MLX-8bit | 4.5 GB | 91.7 / 72.1 | 78.2 / 66.9 | 76.0 | 74.4 | 76.6 | 0.033 | 34.6% / 2.5% | pass |
| Eikos-4B-MLX-4bit | 2.4 GB | 95.8 / 73.0 | 78.8 / 64.8 | 75.3 | 76.0 | 76.0 | 0.021 | 32.5% / 2.7% | pass except agreement |
Release gate, fixed before looking at results: accuracy within 1 point of bf16, ECE within 0.01, and at least 97% of answers unchanged.
| Build | Accuracy (bf16) | Same answer as bf16: all / confident (≥0.9) | ECE (bf16) | Changed answers on items where bf16 was unsure (<0.7) | Gate |
|---|---|---|---|---|---|
| Eikos-27B-FP8 | 83.4 (83.4) | 98.8% / 100.0% | 0.042 (0.043) | 99% | pass |
| Eikos-27B-INT4 | 83.3 (83.4) | 97.8% / 100.0% | 0.041 (0.043) | 96% | pass |
| Eikos-27B-MLX-4bit | 83.1 (83.4) | 96.6% / 99.9% | 0.041 (0.043) | 92% | pass except agreement |
| Eikos-4B-FP8 | 76.9 (76.7) | 97.3% / 100.0% | 0.031 (0.033) | 99% | pass |
| Eikos-4B-INT4 | 76.6 (76.7) | 95.5% / 100.0% | 0.033 (0.033) | 94% | pass except agreement |
| Eikos-4B-MLX-8bit | 76.8 (76.7) | 98.7% / 100.0% | 0.033 (0.033) | 99% | pass |
| Eikos-4B-MLX-4bit | 76.4 (76.7) | 92.5% / 99.8% | 0.020 (0.033) | 93% | pass except agreement |
- Quantization cost: no build loses more than 2 points on any metric of any suite, and calibration (ECE) stays within 0.01 of bf16 or better.
- Eikos-4B-INT4 keeps accuracy and calibration but changes 4.5% of answers, above the 3% the gate allows. Almost all of them (94%) are on items where bf16 itself was unsure (confidence below 0.7); on decisions bf16 takes with confidence ≥0.9, it gives the same answer on 100% of items. We publish it with this note.
- Mac builds (MLX): Eikos-4B-MLX-8bit passes the gate. Eikos-27B-MLX-4bit and Eikos-4B-MLX-4bit keep accuracy and calibration but change 3.4% and 7.5% of answers. Most of those changes (92% and 93%) are on items where bf16 itself was unsure; on decisions bf16 takes with confidence ≥0.9, they agree on 99.9% and 99.8% of items. We publish them with this note. The small JevBench-original split (72 items) moves by a few items between builds.
Calibration
- Why T = 1: our held-out calibration sets turned out easier than real hard data. Every temperature fitted on them was below 1 and made the model over-confident on hard items. Absent a realistic calibration set, we ship T = 1.
- Changing it:
calib.jsonholds the temperature, and callers can re-fit it on their own data.
Parallelism and speed
- Same state, many questions: the hybrid prefix cache processes the state once.
- 200 questions over a 3.3k-token state ran 21× faster with vLLM prefix caching than without it.
- Throughput: batched decisions reach ~51 decisions/s on a shared GPU, with results identical to PyTorch.
- Full GPU (RTX PRO 6000, vLLM, hybrid prefix cache), 200 questions over one state:
- 4B: 161 decisions/s on a 751-token state and 197 decisions/s on a 3.3k-token state;
- 27B: 44 decisions/s on a 3.3k-token state.
- Our own PyTorch cache path: 1,500 questions over one state in 8.7 s (4B).
- Apple M4 16 GB, MLX bf16: ~0.4–0.8 s per decision, and 3 questions over one state in ~1.1 s.
Limitations
- Single pass means no multi-step reasoning. Long chains of arithmetic across many rules are out of reach; on RuleArena (NBA salary-cap rulebooks, ~25k tokens) the model does not discriminate. An optional "verify" mode (a short reasoning budget before the letter) exists in the server; it is off by default and not part of any reported number.
- General world knowledge is bounded by model size. Both sizes trail large frontier systems on knowledge-heavy tasks such as MMLU-Pro.
- Monetary-policy stance (hawkish/dovish) is a known weakness of both sizes: balanced accuracy 38.4 (4B) and 44.5 (27B), against 58.6 for Jev.
- Context: trained on inputs up to 32k tokens (4B) or 12k (27B). The base supports longer inputs, but accuracy beyond the trained range degrades gradually.
- Not advice: it is not legal, tax or investment advice. It applies the rules it is given and does not know your jurisdiction's current law.
Training data and licenses
See NOTICE.
- Model and code: MIT for our contributions. The base model is Apache-2.0 (
LICENSE-Qwen). - Third-party data, with attribution: FinEntity (ODC-BY 1.0), TAT-QA (CC BY 4.0), GSM8K train (MIT).
- Prompt format: SemIf (MIT).
- Trademark: Incoterms® is a trademark of the ICC. No ICC or regulator text was used.
Citation
@misc{eikos2026,
title = {Eikos: open, calibrated, single-pass typed-decision models for finance and trading},
author = {Caio Vicentino},
year = {2026},
url = {https://github.com/caiovicentino/eikos}
}
- Downloads last month
- 534