One program that serves a model as fast as your hardware allows
VEYMEER loads a GGUF model, serves it over the OpenAI API, and spends everything the machine has on getting the answer out sooner. It changes nothing about the answer. This page is what it does; the benchmarks are how well.
It writes ahead, and keeps only what the model would have written
A GPU answering one person has capacity to spare. VEYMEER uses it to work several words ahead and check them together, so one step of the model can yield more than one word.
What it keeps is exactly what the model would have written one word at a time. Where it cannot get ahead, it costs nothing: the answer comes at the plain speed.
When its guesses disagree -- the draft model says one thing, earlier text another -- it does not have to choose: it checks them side by side in the same step and follows whichever one the model takes.
- On by default, with nothing to set up
- Several guesses checked in one step, not one at a time
- Works with sampling, penalties, structured output and tools
- A request can turn it off for itself; a setting turns it off for all
A small model drafts, the large one decides
Chat, explanations and code written from scratch are where a model is slowest. When a much smaller model of the same family is installed, the engine has it draft the next few words and the model you asked for checks them at once.
The engine finds a suitable draft model among the installed ones by itself, uses it when there is room, and says which in its log. The answer is still the large model's, word for word.
- Qwen2.5 0.5B drafting for Qwen2.5 7B: chat goes from 34.4 to 49.5 tokens a second
- Picked automatically, or named, or switched off
Rewriting what is already on the page is nearly free
Fixing a function, applying a change to a file, quoting from a document: most of the answer already exists in the prompt. The engine recognises it and writes it as fast as it can be checked.
This is the work a coding assistant does all day, and where the difference is largest.
The same answer, whatever the engine did to get it
Speed that changes the answer is a different product. VEYMEER's default changes nothing, and says so in a way you can check.
The model's answer, word for word
At temperature 0 the answer equals plain one-word-at-a-time decoding. In our benchmark that holds for 52 of 52 answers, compared token by token.
Trade a little, knowingly
Two lower settings halve the memory a conversation takes and read prompts about a fifth faster. An answer can then differ only where the model was nearly undecided between two words.
The same run, again
A seed reproduces a sampled answer. A stricter setting makes every answer independent of what the engine did before, for testing and audits.
Answers that always have the form you asked for
Hold an answer to a JSON schema, a grammar or a regular expression and it always fits: the model only ever chooses among words the form allows. Tool calls arrive as OpenAI's tool_calls, with arguments that fit the function's schema.
A form is also a head start. Wherever it leaves no choice, the engine already knows the words, so a structured answer comes faster than a free one.
response_format: JSON mode and JSON schema, as OpenAI defines themgrammar(GBNF) andregextools,tool_choice, parallel calls- Log probabilities and the likeliest alternatives, on request
Not reading the same thing twice
Before a model can answer it has to read the prompt. The engine keeps what it has read, and can be told to read ahead.
Seen before, not re-read
A conversation's history, a system prompt, a file sent again: only what is new gets read.
0.06 s to the first word on a prompt seen before
Read before the question exists
An application sends the open file or the instructions to /prime while the person is still typing.
0.11 s to the first word after reading ahead
Asked again, answered at once
The engine remembers what it has written and checks it against the model instead of writing it anew.
545.2 tok/s on a question asked again
Idle time, spent
Between your questions the engine prepares for the likely next ones, and stops the moment a real request arrives.
On by default; one setting turns it off
A model larger than the GPU still runs on the GPU
When a model does not fit, the engine keeps what fits on the GPU, holds the rest in system memory, and brings it over as each pass reaches it. A model larger than memory too is read from disk the same way.
Nothing has to be configured, and no layer count has to be guessed. Reading a prompt stays fast; writing slows with how much has to travel, and a draft model wins much of it back.
- Three tiers, chosen automatically: GPU memory, system memory, disk
- The same answers as when everything fits
- The rest of the machine keeps its memory
- The context sized by itself: as long as the model was trained for, as far as the machine holds it
The 7B model with 2.4 GB of GPU memory left to it, so 17 of its 28 layers do not fit. Mean of three prompts. llama.cpp with 12 layers on the GPU, the most that fit.
A server you can leave running
The same program that answers one person at a desk serves a team behind a key.
The OpenAI API
Chat and text completions, streaming, tools, structured output, log probabilities. Existing clients and SDKs work unchanged.
Settings that change while it runs
The model, the context length, the precision, the draft model: change them over the API. Running answers finish, waiting ones are answered by the new configuration, and a bad change is rolled back.
Any installed model, by name
With model switching on, a request names the model it wants and the engine loads it. One engine serves every model on the machine.
Keys, TLS and a queue
API keys, HTTPS, request and queue limits, timeouts. The engine refuses to listen beyond the machine without keys unless told to, on purpose.
Health and metrics
Liveness and readiness probes, Prometheus metrics, text or JSON logs, the speed of every request in its response.
A system service
One command registers it to start with the machine and to restart if it fails. Stopping it lets running answers finish.
Everything from a terminal
Download a model, check the machine, talk to a model, measure it, change a running engine's settings, sign in: one command each, each with --help.
veymeer pulldownloads a model from the Hugging Face Hub, resuming and verifyingveymeer doctorsays what would stop the engine here, before it doesveymeer benchmeasures a model on this machineveymeer chattalks to a model, through a running engine or on its own
The models you already have
GGUF files, named the way Ollama names them or by path. Models Ollama downloaded are found where they are.
Llama, Qwen, Gemma 4, Mistral and Mixtral, Phi, gpt-oss, DeepSeek MoE, Granite, OLMo, StarCoder and more, mixtures of experts included.
From the 1.5-bit IQ1 formats to full 32-bit floats, MXFP4 and the ternary formats included.
ChatML, Llama 3, Mistral, Gemma, Gemma 4, Phi and harmony, each with its own way of offering tools.
See how it does on the same model you run
The benchmarks compare it with llama.cpp and Ollama on one model file and one GPU, and say where it is behind.