Durable local inference for coding agents

Resume long local Qwen coding sessions after an inference-server restart.

OMP NInfer resumes from saved model state. Qwen3.8 27B is the model, Neroued’s NInfer is the inference engine, and Oh My Pi is the coding agent. This project packages their integration, explicit continuation, and durable checkpoints into exact, qualified releases.

Before installing — v0.7.3 eligibility

RTX 5090: documented Windows 11 + Docker Desktop/WSL2 runtime route.
RTX 4090: documented native Windows 11 route.
RTX 3090 is deferred for v0.7.3. Its historical v0.7.2 route remains on OMP 18.0.9, not qualified with the new client.
Exact pinned client, runtime, model, and profile required; other deployments are not supported by implication. Check the guide’s memory, disk, and download prerequisites.

v0.7.3 manifest · acceptance and release state

Get started with v0.7.3 Download v0.7.3

v0.7.3 repins the client to OMP 18.2.3. Runtime bytes, model, serving settings and memory floors stay at v0.7.2; performance evidence is carried, not remeasured. Do not bypass the ready gate.

What the historical restart receipt measured

On one RTX 5090, v0.4.0 served 109,589 retained tokens after a Docker restart. The receipt records 24.8 s end-to-end including first-touch restoration of a 7.95 GB checkpoint and a separately measured 56.6 s model reload.

0.778 s is server-side time to first token after restoration, not restore or restart time. A different request pair measured 47.920 s cold wall time vs 1.790 s warm follow-up wall time. These are single-machine samples with different timing boundaries, not a restart speedup ratio or a subsecond reboot. Historical restart receipt · method and separate request-pair receipt.

Checkpoints already leave the machine

Export verified session checkpoints, keep replicas on another host or NAS, and import them back to local storage for restore. Historical recovery after loss of local state was exercised on the three 2026-09-05 GPU profiles; host-to-host transport and NAS replication were measured separately.

Restore needs the same runtime binary, model, profile, and session credentials. This is not arbitrary GPU-to-GPU migration or automatic failover. Network shares are replica storage, not live checkpoint roots. Scope, usage, and receipts.

Use it when

You use Oh My Pi and own an eligible RTX 5090 or RTX 4090 setup.
Qwen3.8 27B is the model you want, private and on your own hardware.
Sessions are long-lived, and restart recovery matters more than model breadth.

Use something else when

You want a broad model catalog (Ollama), a desktop GUI (LM Studio), maximum portability (llama.cpp), or multi-user serving (vLLM). The comparison is candid about this — the narrow target is what makes the recommendation credible.

How it fits together

Oh My Pi (client) → OMP NInfer (qualified appliance + releases) → NInfer (engine) → Qwen3.8 27B (model). Loopback-only, bearer-authenticated, fail-closed instead of cloud fallback, every byte hash-pinned. Full detail: canonical facts.

Oh My Pi owns tools, coding workflow, and subagents. Neroued's NInfer and its GPU ports supply the hardware-tuned engine; this project's runtime adds explicit continuation state and durable checkpoints. Ordinary in-process prefix caching already reuses matching input in other runtimes; that is not new here. Engine throughput and avoiding repeated prefill are separate benefits, not a blanket speed claim over other backends. See the comparison and measured results.