Measured on device on earlier files (edge-compat): Galaxy S26 Β· LiteRT-LM 0.16.0 Β· GPU Β· decode 35.6 tok/s Β· prefill 1239 tok/s Β· TTFT 190 ms Β· all 914 ops delegated (2026-08-24); Galaxy S26 Β· LiteRT-LM 0.16.0 Β· CPU Β· did not run: degenerate_output (2026-09-05). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/paddleocr-vl-1.6/CARD.md. This file's Galaxy S26 rows are in the Galaxy S26 section below.

PaddleOCR-VL-1.6 β€” LiteRT-LM (on-device document-parsing / OCR VLM)

PaddlePaddle/PaddleOCR-VL-1.6 converted to the LiteRT-LM (.litertlm) format for on-device document AI with Google's LiteRT-LM runtime β€” the first document-parsing / OCR-specialist VLM in this format.

This repo has two files with the same vision sections, byte for byte: PaddleOCR-VL-1.6.litertlm keeps the text decoder's weights in fp16, and PaddleOCR-VL-1.6_decoder_int8.litertlm (added 2026-11-17; "the int8 file" from here on) has the weights of the decoder's linear layers in int8 and is 350 MB smaller. For the CPU, which is what the litert-lm CLI uses when no backend is given, take the int8 file: on a Galaxy S26 CPU it decoded 2.5–2.9 times as fast and peaked at 1.99 GB of memory against 4.21 GB. On the Galaxy S26 GPU PaddleOCR-VL-1.6.litertlm prefills 1,024 tokens faster and the int8 file decodes faster; those numbers, the Pixel 11 Pro and what was not measured are in Which file to use.

On 2026-11-15 PaddleOCR-VL-1.6.litertlm was replaced with a file whose prefill/decode graph is rewritten for the GPU, with every weight unchanged. Outside the dated entries under Conversion notes, "this file" means that file and "the previous file" the one it replaced. On a Galaxy S26 GPU this file decodes 65.8 tokens/s and prefills 1,024 tokens at 1,810 tokens/s (LiteRT-LM v0.19.0). Run back to back with the previous file, it prefilled 16–18 % and decoded 12–19 % faster in the first two of each run's three iterations; in the third the previous file was slower still, and during that iteration of its runs the phone lowered the GPU clock cap (see the Galaxy S26 section). This file and the int8 file were run only with LiteRT-LM 0.19.0 to 0.19.2; earlier runtimes were not tried with either.

PaddleOCR-VL is the SOTA open document-parsing model (OmniDocBench v1.6 96.33%, ahead of both open and closed alternatives at any size) in a phone-sized package: a NaViT-style dynamic-resolution SigLIP vision encoder + the tiny ERNIE-4.5-0.3B decoder, ~0.9B parameters total, supporting 109 languages. It is a task-prompted model β€” you select what it does with the text prompt:

Prompt Task
OCR: text recognition (page / paragraph / line)
Table Recognition: table β†’ structured cells (<fcel>/<nl> format)
Formula Recognition: formula β†’ LaTeX
Chart Recognition: chart parsing
Spotting: text + `<
Seal Recognition: seal/stamp text
Files PaddleOCR-VL-1.6.litertlm (1.39 GB); the int8 file (1.04 GB)
Vision SigLIP-so400m-class NaViT encoder (27L, hidden 1152) made static 560Γ—560 β†’ 1600 patches β†’ 2Γ—2 merge β†’ 400 image tokens, int8 weights
Adapter LN β†’ 2Γ—2 spatial merge β†’ MLP (4608β†’1024), int8
Decoder ERNIE-4.5-0.3B (18L, hidden 1024, GQA kv2, head_dim 128), fp16 weights (+ the fp16 embedding table in its own embedder section); in the int8 file the weights of the decoder's 127 linear layers (seven in each of the 18 layers and the output head) are int8 per output channel, and the embedding table is the same fp16 section
Context (KV cache) 4096
Image input resized by the runtime to 560Γ—560 (mean/std-0.5 normalization baked into the encoder)
Base model PaddlePaddle/PaddleOCR-VL-1.6 (Apache-2.0)

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical.

How to use

1. Install the runtime

pip install litert-lm

2. Run it in one command β€” this downloads the file, encodes your page and transcribes it:

litert-lm run --from-huggingface-repo litert-community/PaddleOCR-VL-1.6 PaddleOCR-VL-1.6.litertlm \
  --attachment page.png \
  --prompt "OCR:"

The same command with the int8 file:

litert-lm run --from-huggingface-repo litert-community/PaddleOCR-VL-1.6 PaddleOCR-VL-1.6_decoder_int8.litertlm \
  --attachment page.png \
  --prompt "OCR:"

Run with litert-lm 0.19.2 on a Mac (2026-11-17), this command printed, byte for byte, the text that this file prints on the test page below. The file was already in the CLI's cache folder, fetched from this repo beforehand (the same sha256), so the CLI did not download it.

This is a task-prompted model: the prompt selects the task (OCR:, Table Recognition:, and the others listed above), so a generic instruction such as "describe this image" returns nothing. On a 960Γ—540 test page with a heading, one paragraph of figures and a contact line (dark text on white), this file prints:

Quarterly Report 2026

Revenue increased by 18.4% to $2,315 million in Q2 2026, driven by strong demand for on-device AI accelerators. Operating margin improved from 21.3% to 24.7%, while R&D spending reached $412 million, up 9% year over year. The board approved a dividend of $0.85 per share, payable on August 15, 2026 to shareholders of record.

Contact: ir@example.com | +1 (555) 010-2345

This text came from litert-lm 0.19.2 on a Mac (2026-11-15), in two runs of this file with only the flags above, byte for byte; the previous file printed the same bytes, and so did this file with --backend gpu and with --backend gpu --vision-backend gpu. Each file was given by its local path in place of --from-huggingface-repo …: this file as a copy with the same sha256 as the file in this repo, the previous file as downloaded from this repo. After this file was uploaded, the command exactly as written above (litert-lm 0.19.2, Mac) downloaded it (the same sha256) and printed the same text after its download message. This file sets no default backend; for such a file the CLI of litert-lm 0.19.0 to 0.19.2 picks the CPU for the decoder and the vision encoder unless the CLI's config sets a backend for the model (read from the CLI's code; the same for the previous file, and for the int8 file, whose metadata section is the same).

If run stops right after the download with no error message from the CLI (a bus error; exit status 138 on a Mac) or with an error log that contains too large for file size, the downloaded file may be incomplete. The litert-lm CLI (0.14.0 to 0.19.2, and litert-lm-nightly 0.20.0.dev20261112) can keep an incomplete download in ~/.litert-lm/cache/huggingface/litert-community/PaddleOCR-VL-1.6/ and uses it again on later runs without checking its size. Delete that file and run the command again.

If you downloaded PaddleOCR-VL-1.6.litertlm with --from-huggingface-repo before it was replaced on 2026-11-15 (14:09 UTC), delete ~/.litert-lm/cache/huggingface/litert-community/PaddleOCR-VL-1.6/PaddleOCR-VL-1.6.litertlm, then run the command again. Without deleting it, the litert-lm CLI (0.14.0 to 0.19.2) keeps running the previous file: it uses a file it finds there without checking it against this repo.

litert-lm serve exposes the same model as a local OpenAI-compatible API. That was not tried with this file or the int8 file, and on desktop both of them were run on macOS only.

Which file to use

The two files have eight sections each; only the one that holds the text decoder's graph and weights differs. The other seven sections are byte-identical: the vision encoder and the adapter with their two weight sections, the embedding table, the tokenizer and the metadata. On 2026-11-16 the two files were run back to back on each phone, in the order this file, the int8 file, the int8 file, this file, with the program and the workload of the Galaxy S26 and Pixel 11 Pro sections below: litert_lm_advanced_main built from the LiteRT-LM v0.19.0 source, 1,024 prefill tokens and 256 decode tokens, three iterations per run. In these runs this file's side was a copy that holds this file's eight sections, byte for byte, under a newly written file header.

Each row reads this file β†’ the int8 file. GPU values are the median of iterations 2–3 over a file's two runs. CPU values are the median of each iteration over a file's two runs, written iteration 1 / 2 / 3, because the phone lowered its CPU clock caps during every counted CPU run.

Galaxy S26 GPU Galaxy S26 CPU (4 threads) Pixel 11 Pro GPU
Decode, 256 tokens 66.0 β†’ 111.7 tok/s (Γ—1.69) 25.6 / 20.8 / 20.5 β†’ 74.2 / 57.5 / 52.1 tok/s (Γ—2.5–2.9) 24.5 β†’ 29.2 tok/s (Γ—1.19)
Prefill, 1,024 tokens 1,835 β†’ 1,566 tok/s (Γ—0.85) 460 / 406 / 350 β†’ 716 / 673 / 483 tok/s (Γ—1.4–1.7) 1,151 β†’ 1,233 tok/s (Γ—1.07)
Time to first token, 1,024 tokens 0.57 β†’ 0.66 s 2.29 / 2.57 / 2.99 β†’ 1.44 / 1.54 / 2.14 s 0.93 β†’ 0.87 s
Engine start 5.05 β†’ 1.90 s 2.20 β†’ 1.05 s 5.33 β†’ 6.42 s
Peak memory (VmHWM) 1.77 β†’ 1.08 GB, plus GPU driver memory 1.51–1.52 β†’ 1.27 GB 4.21 β†’ 1.99 GB 1.09–1.10 β†’ 1.14–1.15 GB (GPU driver memory not read)
Test page, OCR: (prefill + decode time) 2.63 β†’ 1.78 s 8.72 β†’ 3.91 s 7.29 β†’ 6.68 s
Test table, Table Recognition: (prefill + decode time) 2.06 β†’ 1.46 s 7.11 β†’ 3.11 s 5.44 β†’ 5.68 s
Cache files the runtime writes next to the model (timing runs) 5.7 β†’ 6.4 MB 1.44 GB β†’ 364 MB 6.0 MB β†’ 7.7 MB and a 362.7 MB GPU weight cache

The two document rows are one run per file: the time the runtime reports for the prefill (411 tokens) plus the time it reports for the decode (147 tokens for the page, 108 for the table). Engine start and the image encoding are not in them. On each phone and backend of the table, both files printed the same text for the test page and the same text for the test table; the page text is the block under How to use.

  • CPU: the int8 file. On the Galaxy S26 CPU it is ahead in every row of the table; on the test page the whole run took 8 s against 15 s. With no backend given, the litert-lm CLI runs both files on the CPU (see How to use).
  • Galaxy S26 GPU: it depends on the prompt. This file prefills faster, also on the 411 tokens of a document (0.28 s against 0.32 s). The int8 file decodes faster, and on both test documents its prefill and decode together took less time. With the int8 file the GPU is still the faster backend of this phone on the text path (decode 111.7 tok/s against 52.1–74.2 tok/s on the CPU).
  • Pixel 11 Pro GPU: a smaller gain. The int8 file decoded and prefilled faster, but its engine start was slower and the runtime wrote a GPU weight cache file next to it (the sizes and times are in the table). On the test table its decode was slower in its one run (20.35 against 21.38 tok/s).
  • The accuracy evidence differs. This file's fp16 decoder weights have a teacher-forced check against the original model (correlation 1.0000, top-1 10 of 10, July 2026). The int8 file has no such check. Its evidence is its replies: byte-identical to the replies of the copy with this file's eight sections on 10 synthetic test documents on the Mac CPU and GPU (litert-lm 0.19.2), and on 2 of them on the Galaxy S26 (GPU and CPU) and the Pixel 11 Pro (GPU). On a text prompt outside the model's task (it asks for a Python function), the two files' Mac CPU replies differ (274 B and 494 B, neither of them a function); on the Mac GPU they are the same.
  • Not measured with the int8 file: iPhone and the Swift package, Google AI Edge Gallery, the Pixel 11 Pro CPU, any Mac speed, litert-lm serve, runtimes before LiteRT-LM 0.19.0, documents other than the synthetic test documents, the task prompts other than OCR: and Table Recognition:, and conversations of more than one turn.

The runs behind each number: MEASUREMENTS.md.

Performance

The speeds of this file and of the int8 file were measured on two phones (Which file to use above, and the Galaxy S26 and Pixel 11 Pro sections below), not on a Mac. Measured on an earlier file: litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine. These figures cover the text path; the vision encoder runs once per image and is not included.

Device Backend Prefill (256) Decode TTFT
Apple M4 Max (macOS) CPU 653 tok/s 53.2 tok/s 0.50 s
Apple M4 Max (macOS) GPU (Metal) 4391 tok/s 208.3 tok/s 0.07 s

When this table was written, each backend in it was checked on that earlier file by generating output, not only by reading the benchmark tool's output. With litert-lm 0.19.2 on the Mac, this file and the previous one gave the same bytes for OCR: on the test page above and for Table Recognition: on a test table, on the CPU (1 thread, greedy decoding with top-k 1) and on the GPU (the decoder and the vision encoder both on the GPU). No Mac speed was measured with this file or with the int8 file.

Accuracy note

This note was written for an earlier file and was not re-measured with this one; its sentences about the int8 file are from 2026-11-16.

Device-verified on a Pixel 8a (Google AI Edge Gallery 1.0.15, CPU backend, an earlier file):

  • Page OCR (OCR: on a synthetic report page): perfect transcription in 22 s β€” every figure, the e-mail address and the phone number come out exactly ("Revenue increased by 18.4% to $2,315 million …").
  • Table recognition (Table Recognition:): every cell correct including the Total row, in 17 s (Product / Units / Revenue … Total 24,510 $1,207,000).

Also validated on the desktop LiteRT-LM runtime (macOS CPU): table output is byte-identical to the full-precision eager reference, including the structured <fcel>/<nl> cell tokens (the file's SentencePiece vocabulary carries all 1 019 added tokens at their exact ids, so they detokenize correctly on-device).

  • Vision tower: tflite vs the reference implementation corr 1.0 (fp32) / 0.9975 (shipped int8), zero FLEX/CUSTOM ops (GPU-clean).
  • Decoder: extracted as a standalone Llama-layout model β€” bit-exact fp32 logits vs the original; shipped fp16 weights match the original under teacher forcing at corr 1.0000, top-1 10/10. Of the decoders exported in July 2026 and not published, one had int4 weights: top-1 6 of 10, correlation about 0.85, recorded as "transcription collapses"; this was not measured again. Another was exported with the dynamic_wi8_afp32 recipe (int8 weights): it matched the reference model's top-1 token in 8 of 10 teacher-forced comparisons (correlation 0.978), and its output in the LiteRT-LM runtime was recorded as corrupt. The int8 file is not that export: it is this file with only the 127 linear-layer weights of the text decoder in int8 per output channel, and it has no teacher-forced check; what it was checked with is under Which file to use.

On-device demo (Pixel 8a, CPU, an earlier file)

Page OCR β€” OCR: (22 s) Table Recognition β€” Table Recognition: (17 s)

Captured in Google AI Edge Gallery on a Pixel 8a (CPU backend). The Gallery UI renders the model's structured table markup (<fcel>/<nl>) as plain text.

M-RoPE note. The base decoder uses Qwen2-VL-style 3-D M-RoPE. The LiteRT-LM fast_vlm contract supplies plain sequential positions β€” for text tokens this is mathematically identical, and for image tokens an A/B eager test (true M-RoPE vs 1-D) showed no quality loss on OCR/table tasks (raster reading order survives 1-D positions).

Scope. This is the recognition component of the PaddleOCR 3.x pipeline. The official server pipeline puts a layout-detection model (PP-DocLayout) in front for full-page multi-element parsing; standalone, this file handles pages, paragraphs, text lines, tables, formulas and charts directly β€” best on single elements or simple pages, exactly like the upstream transformers usage.

Aspect ratio. The runtime resizes input to a fixed square, so extreme aspect ratios (very wide single lines) get distorted. For best results feed roughly page/paragraph-shaped crops; the deployed 560Β² contract transcribed a 16:9 test page perfectly.

One image per chat. Like the other fast_vlm files, send each document in a fresh conversation β€” a second image in the same chat degrades (context bleed from the first turn was observed on CPU).

Galaxy S26 β€” GPU and CPU

Each run prefills 1,024 tokens and generates 256 tokens, three times (three iterations): the runtime's benchmark mode puts the prompt file's one sentence ("Explain what on-device AI means in simple terms.") in the chat template as a user message and pads the tokens to 1,024 with token id 0. Iteration 1 is the first pass after the engine starts. The GPU row is the median of iterations 2–3 over two runs, so it leaves iteration 1 out (this file's iteration 1: prefill 1,788–1,821 tok/s, decode 62.4–63.0 tok/s). On the CPU the median prefill and decode speeds drop from iteration 1 to iteration 3, so each iteration has its own row: the median of that iteration over two runs.

Backend Prefill, 1,024 tokens Decode Time to first token Peak memory
GPU (LiteRT GPU, every node of the prefill and decode graphs), iterations 2–3 1,810 tok/s 65.8 tok/s 0.58 s 1.77 GB + 1.52 GB GPU driver
CPU (XNNPACK for most nodes), iteration 1 499 tok/s 26.5 tok/s 2.09 s 4.20 GB
CPU (XNNPACK for most nodes), iteration 2 442 tok/s 21.2 tok/s 2.37 s 4.20 GB
CPU (XNNPACK for most nodes), iteration 3 333 tok/s 20.9 tok/s 3.13 s 4.20 GB

Samsung Galaxy S26 (model SM-S942Q, chipset SM8850, Android 16), litert_lm_advanced_main built from the LiteRT-LM v0.19.0 source (commit 771b06e1; the v0.19.0 tag is on the next commit, which changes only Package.swift and README.md), --benchmark with 1,024 prefill tokens and 256 decode tokens, context 2,048, 2026-11-15. Each run is a new process. The rows cover the text path.

GPU runs: four, in the order previous, this, this, previous (GPU runs 1–4). Each started at Android's thermal status 0 (none), with no CPU clock cap, the GPU clock cap at 1,300 MHz and GPU thermal level 0 (the GPU driver's own thermal level; 0 = no limit). In both of the previous file's runs (22 and 23 s long) the phone lowered the GPU clock cap from 15–18 s in, to 726 MHz by 20–21 s in (thermal level 7); iteration 3 began 16 s in. This file's runs took 19 s each: every sample during them (about every 2 s, the last 17–18 s in) read the cap at 1,300 MHz and level 0, and the reading taken 0–2 s after each of the four runs ended read level 7. The phone also lowered the caps of its two CPU clock domains (policy0 / policy6), to 1.79 / 1.75 GHz in GPU run 1 and to 2.11–2.23 GHz in GPU runs 2–4. GPU runs 1 and 2 ended at thermal status 1 (light), runs 3 and 4 at thermal status 0. In the GPU runs a small extra graph named main (5 nodes) has 1 node on XNNPACK (the CPU), with both files.

CPU runs: four counted runs in the order previous, this, this, previous (CPU runs 1–4), after one warm-up run of this file that is not counted. The runtime used its default of 4 threads. The warm-up and CPU runs 1–3 started at thermal status 1, run 4 at thermal status 0, all without a CPU clock cap. The phone began to lower the CPU clock caps 2–4 s into each run; they reached 1.56 / 1.50 GHz in the warm-up and in CPU run 1 (36 and 43 s in) and 1.79 / 1.75 GHz in CPU runs 2–4 (24–27 s in). Each run took 44–47 s. No document was read on this phone's CPU in these runs; on 2026-11-16 the copy with this file's sections and the int8 file both read the two test documents on this phone's CPU (Which file to use).

Peak memory is per run, so the three CPU rows share one value. It is the process's VmHWM: on the CPU 4.20 GB in this file's two counted runs (4.09 GB in the warm-up run) and 4.36 GB in the previous file's two; on the GPU 1.76–1.77 GB with this file and 1.76 GB with the previous one. VmHWM does not count memory the GPU driver holds, so the GPU row's value adds the rise of the phone's GPU allocation counter (kgsl_page_alloc) during each of this file's GPU runs (1.52 GB; 1.47 GB in the previous file's). With a document and the vision encoder on the GPU (below), the process peaked at 1.92 GB (the page) and 1.86 GB (the table), and the counter rose by 2.35 GB in each. Engine start (the runtime's init time, the median of a file's two runs) took 5.13 s with this file and 5.35 s with the previous file on the GPU, and 2.82 s and 2.67 s on the CPU.

With the decoder and the vision encoder on the GPU (--vision_backend=gpu), this phone read the test page under How to use with OCR: and a test table with Table Recognition:: each reply held every key string of its document (one run each): 10 strings of the page (the heading, six figures, the date, the e-mail address and the phone number) and the 15 cells below the table's header.

On the GPU, in iterations 1 and 2 of each run this file prefilled Γ—1.160–1.177 and decoded Γ—1.118–1.193 as fast as the previous file (four runs in the order previous, this, this, previous; each pair and iteration in MEASUREMENTS.md). Each run's mean of iterations 2–3: prefill 1,280, 1,816, 1,806 and 1,293 tok/s, decode 52.7, 65.8, 65.5 and 53.5 tok/s, so from the previous file to this one prefill Γ—1.419 and decode Γ—1.249 in the first two runs, prefill Γ—1.396 and decode Γ—1.225 in the last two.

Those wider gaps come from iteration 3. In both of the previous file's runs it prefilled more slowly than iteration 2 (1,012 and 1,039 against 1,548 tok/s) and decoded more slowly (50.2 and 51.1 against 55.1 and 55.9 tok/s), and during it the phone lowered the GPU clock cap (above); this file's three iterations prefilled 1,788–1,822 tok/s. The table's statistic (the median of iterations 2–3 over a file's two runs) gives the previous file 1,293 tok/s prefill and 53.1 tok/s decode, this file 1,810 and 65.8.

On the CPU this file's medians were above the previous file's in each iteration (prefill 10.8 %, 11.1 % and 0.5 % above, decode 2.4 %, 3.6 % and 4.6 % above, in iterations 1–3). The previous file's two runs differ by 25.5 % in iteration 1 prefill (399.03 and 500.76 tok/s), and in the last two runs alone this file prefilled Γ—1.009 and decoded Γ—0.996 as fast in iteration 1. The runs behind each number, and the S26 section this card carried for an earlier file: MEASUREMENTS.md.

On the Galaxy S26 the GPU decodes 2.5Γ— as fast as the CPU's iteration 1 and 3.1Γ— as fast as its iterations 2 and 3, and prefills 3.6Γ—, 4.1Γ— and 5.4Γ— as fast (iterations 1, 2 and 3), so this repo's litertlm_manifest.json recommends this file on gpu for the Android device class it calls flagship. That recommendation rests on the Galaxy S26 alone (on the Pixel 11 Pro the GPU decoded 24.7 tok/s and the CPU was not measured), and its speeds cover the text path; the page and the table above were read with the vision encoder on the GPU too. A reader that follows the manifest's rules, such as the reference readers of hf-to-litertlm, picks this file on the GPU for an Android request that names the GPU or no backend, and the int8 file for one that names the CPU. With no platform or another platform no recommendation matches, so it picks the smaller file, the int8 file.

GPU wiring, including the Gallery import toggle: GPU guide.

Pixel 11 Pro β€” GPU

On a Pixel 11 Pro (the same build and workload, 2026-11-14) every node of the prefill and decode graphs runs on the GPU with both files. Four runs went previous, this, this, previous, each started at thermal status 0 (each run's mean of iterations 2–3): prefill 1,105, 1,151, 1,173 and 1,101 tok/s, decode 21.9, 24.7, 24.6 and 22.4 tok/s, so prefill Γ—1.041 and decode Γ—1.132 in the first two runs, prefill Γ—1.065 and decode Γ—1.098 in the last two. With the decoder and the vision encoder on the GPU, both files read the test page and the test table with every key string (one run each). The Pixel 11 Pro CPU was not measured. The runs: MEASUREMENTS.md.

Run on iPhone / macOS

Use swift-litert-lm, a community Swift package over the LiteRT-LM runtime. Load PaddleOCR-VL-1.6.litertlm with the vision tower enabled (Modality.textImage), attach a document photo, and send one of the task prompts above (e.g. OCR:).

Vision-only file (no audio tower): bring the engine up with the vision modality only β€” requesting .all fails at session creation on files without an audio section.

These steps were written for an earlier file and were not run with this one or with the int8 file.

Run on Android β€” Google AI Edge Gallery

Install a recent Google AI Edge Gallery, download PaddleOCR-VL-1.6.litertlm, import it (tap +), attach a document image and prompt OCR:. The file carries the tokenizer, template and both towers.

These steps were written for an earlier file and were not repeated with this one or with the int8 file, and which backend Gallery gives the vision encoder was not checked. On a Galaxy S26 and a Pixel 11 Pro, LiteRT-LM's litert_lm_advanced_main (built from the v0.19.0 source) read the test page and the test table with this file, with the decoder and the vision encoder on the GPU (see those sections).

Conversion notes

  • LiteRT-LM fast_vlm file: VISION_ENCODER ([1,560,560,3]β†’[1,1600,1152]) + VISION_ADAPTER ([1,1600,1152]β†’[1,400,1024]) + single-token EMBEDDER + PREFILL_DECODE (embeddings-input).
  • Static NaViT rewrite: PaddleOCR-VL's encoder is dynamic-resolution (packed patches, interpolated position embeddings, 2-D rotary over h/w ids) and does not torch.export. The LM calls it with full attention (window_size=-1), so the static graph is: whole-image patch Conv2d (the processor packs patches in pure raster order β€” no gather needed), precomputed bilinear-interpolated position embedding, precomputed 2-D rope cos/sin for the fixed 40Γ—40 grid.
  • The projector's 2Γ—2 spatial merge is done GPU-safe with 4 strided slices + concat (all tensors ≀4D) instead of the literal 6-D rearrange.
  • Decoder: the ERNIE-4.5-0.3B inside the VLM is layout-identical to Llama β€” re-hosted as a standalone LlamaForCausalLM (state-dict 1:1, bit-exact logits) and exported with the standard litert-torch path, cache 4096. The int8 file's decoder graph is this file's graph with its 127 linear-layer weights rewritten to int8 after the export (the 2026-11-17 entry below).
  • Tokenizer: the base SP model lacks the 1 019 added tokens (<|IMAGE_START|>, <|LOC_0|>…<|LOC_1000|>, <fcel>/<nl> table tokens…). They are appended as USER_DEFINED SentencePiece pieces at their exact ids (padded to vocab 103 424) so table/spotting output detokenizes correctly on device.
  • Prompt template (baked into the file): <|begin_of_sentence|>User: <image>PROMPT\nAssistant:\n, stop token </s>.

Each dated entry below was written on its date: in it, "this file" is the file published that day, "the previous file" is the one it replaced, and "this card" is the card as it was then. The 2026-11-17 entry added the int8 file and replaced no file, so it names both files.

2026-08-30 β€” tokenizer section replaced (weights unchanged)

The tokenizer in PaddleOCR-VL-1.6.litertlm is the vendor's SentencePiece model, whose special tokens are typed as control symbols; SentencePiece never matches a control symbol in text, so the </s> the chat template writes after each history turn reached the model as several tokens instead of the single </s> id. Those symbols are now user-defined pieces with the same ids, so a multi-turn prompt tokenizes as the upstream tokenizer does. Single-turn prompts were unaffected.

Tokenizer-only change: every section of the file except the tokenizer is byte-identical to the previous file (verified by sha256 per section), so the weights, the graph and the chat template are unchanged and the speed and memory numbers on this card still describe this file per token β€” only the file's own sha256 differs. Verified on the LiteRT-LM runtime: one user message ("What is the capital of France?") as the chat template renders it, 7 test strings, the 223 standalone characters U+00A1–U+017F and every special token except <unk> now tokenize identically to the upstream tokenizer, and the four ASCII-only test questions answer byte-identically to the previous file (same ids in, same tokens out). In the same checks, that user message, the 7 test strings and the 223 characters got the same token ids from the previous file as from this one, and two test prompts with accented letters and an emoji got the same replies from both files (each check reply was limited to 48 tokens). What changed is how 11 special tokens tokenize when they are written as text, </s> among them. When this entry was written, the accuracy figures on this card and its on-device rows had been measured on earlier files unless a row said otherwise, and the 2026-08-25 Galaxy S26 GPU check (now in MEASUREMENTS.md) had not been re-run on the 2026-08-30 file (the runtime's tokenizer code is the same on macOS and on device; the weights and graph are byte-identical). If you downloaded before 2026-08-30 (05:53 UTC), download again; with the litert-lm CLI, first delete the cached file as described under the command in How to use.

2026-11-15 β€” GPU graph (weights unchanged)

The prefill/decode graph now has the GPU form litert-torch writes with --apply_gpu_composites: one odml.cache_update per layer writes the KV cache, and odml.runtime_bmm runs the attention products. The file already had prefill signatures of 128, 512 and 1,024 tokens and keeps them. Every weight, and every section outside that graph, is byte-identical to the previous file. The file header's file-format version went from 1.7.0 to 1.9.0. The whole file went from sha256 39fb2d2c… (1,390,313,472 B) to 94cf9769… (1,390,542,848 B). The previous file stays at revision ab87bee.

On a Mac the two files gave the same replies byte for byte in these runs:

  • OCR: on the test page under How to use and Table Recognition: on a test table, on the CPU (litert-lm 0.19.2, 1 thread, greedy decoding with top-k 1) and on the GPU (the decoder and the vision encoder both on the GPU); each of the 8 replies held every key string of its document.
  • 4 text prompts on the CPU (litert-lm 0.19.2, 1 thread, greedy decoding with top-k 1).
  • The command under How to use (see there).
  • With litert-lm 0.19.0 to 0.19.2, on the CPU and on the GPU: a text prompt that asks for a Python function (neither file wrote one) and a picture with "Describe this picture in one sentence." (neither file described it).

8 short questions (each ends with "Answer briefly."): the expected word or number was in 1 of the 8 replies with both files, on the Mac CPU and on the Mac GPU; open questions are not this task-prompted model's job.

Galaxy S26 GPU (LiteRT-LM v0.19.0, four runs in the order previous, this, this, previous): in iterations 1 and 2 of each run this file prefilled Γ—1.160–1.177 and decoded Γ—1.118–1.193 as fast as the previous file; in iteration 3 the previous file was slower (see the Galaxy S26 section), so each run's mean of iterations 2–3 gives prefill Γ—1.419 and Γ—1.396 and decode Γ—1.249 and Γ—1.225 per pair. On a Pixel 11 Pro GPU (same build and workload; each run's mean of iterations 2–3) prefill Γ—1.041 and Γ—1.065, decode Γ—1.132 and Γ—1.098. Both phones read the test page and the test table with this file, with the vision encoder on the GPU. Procedure: REPRODUCE.md.

2026-11-17 β€” the int8 file added (PaddleOCR-VL-1.6.litertlm unchanged)

The int8 file is not a new export. Its prefill/decode graph is the graph of PaddleOCR-VL-1.6.litertlm with the weights of the linear layers rewritten to int8. That section is the only one that differs (725,540,096 β†’ 375,345,152 B); the other seven sections are byte-identical, and the file header's file-format version is 1.9.0 in both. The whole file is 1,040,351,232 B (sha256 5d415de3…), 350,191,616 B smaller than PaddleOCR-VL-1.6.litertlm.

In that graph the decoder has 127 linear-layer weights: seven in each of the 18 layers (the four attention projections and the three of the MLP) and the output head. In PaddleOCR-VL-1.6.litertlm each is an fp16 constant that a DEQUANTIZE op turns to float. In the int8 file each is an int8 tensor that the fully connected op reads directly, quantized per output channel with zero point 0: the scale of a channel is its largest absolute weight divided by 127, each weight goes to the nearest step, and no weight was clipped. The relative RMS error of the int8 weights against the fp16 values is 1.27 %. The embedding table is not in this graph and stays fp16. Procedure: REPRODUCE.md.

Checks of the int8 file against PaddleOCR-VL-1.6.litertlm's sections: the same replies byte for byte on 10 synthetic test documents on a Mac (litert-lm 0.19.2, CPU and GPU, greedy decoding with top-k 1), and the same text for the test page and the test table on a Galaxy S26 (GPU and CPU) and a Pixel 11 Pro (GPU). With litert-lm 0.19.0, 0.19.1 and 0.19.2 on the Mac, a text prompt that asks for a Python function got different replies from the two files on the CPU and the same reply on the GPU (neither file wrote one), and a picture with "Describe this picture in one sentence." got the same 1-byte reply from both on both backends. Speed and memory on the two phones: Which file to use; the runs: MEASUREMENTS.md.

Known runtime issues

External weights (since 2026-11-09). In PaddleOCR-VL-1.6.litertlm, the vision encoder and the vision adapter keep 437 MB of their 447 MB of weights and other constants in TFLiteWeights sections at the end of the file (LiteRT's external weights) instead of inside their model graphs; the decoder and the embedder keep theirs inside their graphs. On a Pixel 8a, the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on a Qwen3-0.6B file with external weights (LiteRT-LM#3986). The int8 file has the same two TFLiteWeights sections, byte for byte, and was not tried with the 0.18 Android builds. The file from before 2026-11-09, with every weight inside its graph, stays at revision 51dc085.

From the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because each file of this repo holds the embedding table in its own section and its decoder takes embeddings, so later answers in a conversation can differ from the answer the model gives to the same messages in one prompt (read in the source of LiteRT-LM 0.16.0 to 0.19.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on this repo's files). One question per conversation avoids it, as does a new conversation that starts with the earlier messages and the new question.

On LiteRT-LM 0.18.1 and 0.19.1, once a message fails because it does not fit in the context window, every later message in that conversation fails as well (the next one with Task failed with state: 7). Start a new conversation after such an error. This was measured with other models' files, not with this one.

License

Apache-2.0, inherited from the base model PaddlePaddle/PaddleOCR-VL-1.6.

Downloads last month
837
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for litert-community/PaddleOCR-VL-1.6

Quantized
(28)
this model

Collection including litert-community/PaddleOCR-VL-1.6