On-Premise Inference, a bifurcated strategy

The premise was a budget constraint: a single consumer-class GPU with 24 GB of VRAM, running a private coding assistant with no calls leaving the machine. The question was whether one model could optimally serve both fill-in-the-middle completion and deep reasoning. After benchmarking five coding models across both task shapes, the answer is no, and the more useful answer is the split that replaces it.

Hypothesis

Autocomplete and reasoning are not points on a quality spectrum; they are different products. Completion is dominated by time-to-first-token. A suggestion that arrives after the keystroke is worthless no matter how good it is. Reasoning is dominated by capacity. Depth of context and correctness matter far more than the half-second of latency. A model tuned to win one of these necessarily compromises the other. The hypothesis: on fixed hardware, two specialised models beat one generalist, because they let you spend the VRAM budget where each task actually needs it.

Method

1. Fix the envelope. 24 GB VRAM, single GPU, fully local. Every candidate has to fit, including its KV cache at the context length the task demands.
2. Split the workload. Define two benchmark suites: a fill-in-the-middle (FIM) suite scored on latency and accept-rate, and a reasoning suite scored on multi-file refactor correctness.
3. Sweep five models. Run each candidate through both suites at quantisations that fit the envelope, recording time-to-first-token, throughput, and quality per task.
4. Route. Stand up a thin local router that sends completion traffic to the fast model and reasoning traffic to the high-capacity model, swapping the heavyweight in only when a deep request arrives.

Where it stands

The split holds. A small FIM-tuned model wins the completion suite on time-to-first-token by a margin no larger generalist could close, while a high-capacity reasoning model, swapped in only on demand, handles refactors the fast model simply gets wrong. Crucially, both fit the 24 GB envelope because they are never resident at full size simultaneously; the router treats VRAM as the scarce resource it is.

The open engineering cost is the swap. Loading the heavyweight on the first deep request adds a one-time stall, and hiding that latency, through speculative pre-warming or keeping a quantised reasoning model warm, is the next milestone.

Open questions

• QCan the swap stall be hidden entirely, or is a second smaller card the cheaper answer than software gymnastics?
• QHow far does the FIM model's accept-rate drop at aggressive quantisation before users notice?
• QDoes this routing pattern generalise to running the R·017 alpha-search loop fully on-premise?