OMP NInfer
Benchmarks
Measured, receipted, reproducible
Each result names its release and measurement scope: lifecycle qualification and published-route acceptance are distinct. Warm/cold request pairs compare retained state with a fresh cold build; they do not measure restart duration or compare against other runtimes’ in-process prefix caches.
v0.7.3: client repin, no new throughput measurement
OMP 18.2.3 client and documented-route acceptance is fresh; runtime measurements below are carried v0.7.2 evidence, not remeasured throughput. Runtime image/package bytes, model, serving arguments and memory floors are unchanged. Eligibility is RTX 5090 on Windows 11 + Docker Desktop/WSL2 and RTX 4090 native Windows 11; RTX 3090 is deferred. Release state, client proof and historical RTX 3090 route.
Historical runtime qualification: v0.7.2
| Scope | Measurement |
|---|---|
| RTX 5090 published image | Exact 130,048-token retrieval at 2,151.7 tok/s wall; 139.8 tok/s server decode (132.57 wall) over 2,048 tokens |
| RTX 5090 target-session restore | 125,891 / 125,892 input tokens; both resume with cached continuation state in 5.96 s and 23.57 s |
| RTX 5090 measured limits | Three unsaved predecessor sessions in the combined process; 2 of 8 continuations/forks lose warm reuse in the separate control |
| RTX 4090 exact-source qualification | 15 phases passed, including exact 130,048-token retrieval; C1 decode 153.431 tok/s and prefill 2,113.995 tok/s |
Single qualification samples, not a GPU ranking or a throughput gain attributed to reclaim. The two target restores do not prove loss-free shutdown for every session. RTX 5090 receipt · RTX 4090 receipt · Separate public-route acceptance.
Historical qualification: v0.7.1
| Scope | Measurement |
|---|---|
| RTX 5090 qualified v0.7.0 configuration | Exact 130,048-token retrieval at 2,203.0 tok/s; 133.03 tok/s wall decode over 2,048 tokens |
| RTX 4090 two sessions at the ceiling (v0.7.1) | Both checkpoint (4.99 GB each), a graceful stop saves both, and both resume exactly in 5.63 s and 7.78 s where the previous pool lost them to HTTP 404 |
| RTX 5090 two sessions at the ceiling | 0 of 8 continuations and forks lost; about 125,900 cached tokens reused in 1.6–3.5 s, against a 58 s root re-prefill on the v0.6.10 pool |
| RTX 4090 native qualification | Exact retrieval in 91.2377 s; 153.464 tok/s C1 decode; 87.58865% MTP acceptance |
These v0.7.1 measurements retain their original scope; they do not qualify the new binaries. Different workloads are not a GPU ranking or a head-to-head comparison against vLLM or llama.cpp. Historical facts and receipts.
Historical restart measurements: v0.4.0
| Claim | Measurement (RTX 5090 · v0.4.0) |
|---|---|
| Restart recovery | 109,589 retained tokens served; 24.8 s end-to-end including first-touch restoration of the 7.95 GB checkpoint; 56.6 s model reload recorded separately |
| Post-restoration request | 0.778 s server-side time to first token after restoration, not checkpoint restore or restart time |
| Separate cold/warm request pair | 47.920 s cold wall time vs 1.790 s warm follow-up wall time at a 109,594-token session |
| Decode | 144.80 tok/s agent-shaped gate (44.10% MTP acceptance); 152.2 tok/s at 83.3% on retrieval |
| Long context | Exact needle retrieval at a 130,448-token prompt; 131,072 ceiling |
These historical v0.4.0 samples were measured on one RTX 5090. Comparing the 0.778 s post-restoration TTFT to 47.920 s cold wall time mixes timing boundaries; neither is a subsecond reboot or a 61× restart speedup. Restart receipt · Separate request-pair receipt.
Method, per-lane numbers, and historical sections: docs/BENCHMARKS.md · raw receipts: docs/measurements/
Updated 2026-09-18: v0.7.2 runtime evidence is carried into the client repin; historical v0.7.1 and v0.4.0 results retain their original scope.