UltraFinBERT-150M

UltraFinBERT-150M is an English financial sentiment classifier obtained by full fine-tuning dmis-lab/UltraBERT-150M on the unanimous-annotation subset of FinancialPhraseBank. It predicts the financial polarity of a sentence from an investor-oriented perspective: negative, neutral, or positive. The model has 151,034,883 parameters and is distributed as FP16 Safetensors weights, with the tokenizer and custom UltraBERT implementation required for loading.

Model details

• Base checkpoint revision: 4cc8ca415eb1958184c491297fe5aaa94c4a8ad2.
• Architecture: UltraBERT encoder with a three-class sequence classification head.
• Training method: full parameter fine-tuning; standard three-class cross-entropy.
• Label mapping: 0 = negative, 1 = neutral, 2 = positive.
• Fine-tuning and evaluation maximum input length: 256 tokens.
• The upstream encoder supports longer context, but this classifier was trained and evaluated on short sentences; long-document performance was not evaluated.
• Custom code: required (trust_remote_code=True).

Training data and split

The source is the provided FinancialPhraseBank AllAgree data. Input source: provided TXT file. Encoding: cp1252. Source SHA-256 fingerprint(s): 5fb370b3005b9a2759f708849a691ad2e4d2593d4c995c1afde229398a516594. The optional TXT fallback is parsed at the last @ separator on each non-empty line.

• Original examples: 2264.
• Duplicate rows removed after NFKC, whitespace and case normalization: 5.
• Contradictory-label rows excluded: 0.
• Retained examples: 2259.
• Split method: Stratified 70/15/15 split generated from TXT with seed 42 (fallback mode).
• Partition sizes: 1581 train / 339 validation / 339 test.

Text is preserved in its original form after outer whitespace trimming. Normalized exact duplicates cannot cross the three splits. Semantic or article-level duplicates are not identified; no article identifiers are supplied. The test split is held out from this fine-tuning run. Exposure during upstream pretraining has not been audited. This is a custom split, not an official FinancialPhraseBank test partition. The split manifest contains hashes and labels, not the raw sentences.

Fine-tuning procedure

• AdamW: beta1 0.9, beta2 0.98, epsilon 1e-6.
• Weight decay: 0.1, using Trainer's standard decay parameter grouping.
• Linear learning-rate decay, 5% warm-up, rounded up to a full optimizer step.
• Learning-rate search: 5e-6, 8e-6, 1e-5, 3e-5, 5e-5, 8e-5.
• Selected learning rate: 8e-06.
• Maximum epochs: 10; early stopping patience: 3 validation checks.
• Per-device batch: 8; gradient accumulation: 4; effective batch: 32 on one GPU/process.
• Gradient clipping: 1.0; gradient checkpointing enabled; attention: SDPA.
• Training compute precision: bfloat16, with FP32 trainable parameters.
• Export precision: FP16. Evaluation compute precision: torch.float16.
• Selection: maximum validation macro F1; validation loss breaks ties across trials. Each trial restores its best validation checkpoint. No final refit on the test or validation split is performed. The exported weights are evaluated directly.
• One training seed is used. Training-seed variability has not been measured.

Learning-rate search (validation only)

learning_rate	validation_macro_f1	validation_accuracy	trained_epochs
8e-06	0.988832	0.9941	8

Post-export validation macro F1: 0.9888. Change from the restored pre-export checkpoint's validation macro F1: +0.000000; compute precision also differs.

Evaluation and checkpoint comparison

All three checkpoints are evaluated on exactly the same local test sentences, with their own tokenizer, right padding, the same 256-token truncation cap, and probabilities aligned by the label names in each model's configuration. Reference checkpoints are evaluated as published, without additional fine-tuning. The reference revisions are recorded in experiment.json and test_comparison.json.

model	accuracy	macro_f1	weighted_f1	macro_f1_ci95_low	macro_f1_ci95_high
beethogedeon/UltraFinBERT-150M	0.9646	0.9616	0.9642	0.9373	0.9815
ProsusAI/finbert	0.9823	0.9775	0.9824	0.9591	0.9945
beethogedeon/Modern-FinBERT-large	0.9823	0.9749	0.9821	0.9526	0.9916

Comparison caveat: both ProsusAI/finbert and beethogedeon/Modern-FinBERT-large report fine-tuning on FinancialPhraseBank. Their original training partitions are not verified against this test, so overlap is possible. These scores are a descriptive checkpoint comparison, not a contamination-controlled ranking or evidence of superiority on unseen data. Use an independently collected external test corpus for stronger conclusions.

Confidence intervals are 95% percentile intervals from 2000 class-stratified bootstrap resamples of the held-out test predictions. They describe test-sampling uncertainty, not variation across training seeds. Per-class reports, log-loss, balanced accuracy, truncation rates and confusion matrices are provided in the repository.

Usage

Install the Transformers version used in this experiment:

pip install transformers==5.16.1 'torch>=2.6' safetensors

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "beethogedeon/UltraFinBERT-150M"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
dtype = torch.float16 if device.type == "cuda" else torch.float32
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
tokenizer.padding_side = "right"
model = AutoModelForSequenceClassification.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
    attn_implementation="sdpa",
).to(device).eval()

texts = ["Operating profit increased compared with the previous year."]
inputs = tokenizer(texts, padding=True, truncation=True,
                   max_length=256, return_tensors="pt").to(device)
with torch.inference_mode():
    probabilities = model(**inputs).logits.float().softmax(dim=-1)
for row in probabilities.cpu():
    print(dict(zip([model.config.id2label[i] for i in range(3)], row.tolist())))

For reproducible deployment, pin revision to a specific commit from this repository. Loading custom code executes repository Python code; review it first. CPU inference casts the stored FP16 weights to FP32. This does not change the FP16 format of the distributed checkpoint.

Intended uses and limitations

Intended for research and non-commercial financial sentence sentiment analysis consistent with the applicable license. This model estimates textual sentiment; it is not a trading signal, a forecast, or financial advice.

The dataset is small and imbalanced, concentrated in English financial news and the Finnish listed-company context. Unanimously annotated examples may be easier than ambiguous real-world news. Performance on other markets, current events, sarcasm, multilingual inputs and long documents is not established. Softmax probabilities are not calibrated confidence guarantees. Evaluate on representative data before using the model in a downstream application.

Reproducibility

See experiment.json, requirements.txt, data_audit.json, split_manifest.csv, selection.json, learning_rate_search.csv, fp16_validation.json, test_comparison.json, and the classification reports. The original text corpus, optimizer state, authentication secrets and text-level prediction logs are not included in this model repository.

License and attribution

The original UltraBERT checkpoint is published under Apache-2.0. FinancialPhraseBank is distributed under Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported (dataset card, license). The fine-tuned weights in this repository are released under CC BY-NC-SA 3.0. Upstream code retains its original licensing and attribution. Commercial use requires appropriate rights for the data and any other applicable components. No license is inferred from the licenses of the reference models.

Please credit the authors of UltraBERT and FinancialPhraseBank:

• Jungwoo Park, Taewhoo Lee and Jaewoo Kang. UltraBERT: Beyond the Limit of BERT Pre-Training (2026). https://arxiv.org/abs/2610.15028
• Pekka Malo, Ankur Sinha, Pekka Korhonen, Jyrki Wallenius and Pyry Takala. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology (2014). https://arxiv.org/abs/1307.5336

Reference model cards: ProsusAI/finbert and beethogedeon/Modern-FinBERT-large.