Serve a model in three commands

Quick start

Check the machine, see which models are on it, and serve one. Models that Ollama already downloaded are found where they are.

veymeer doctor                       # what this machine offers, and anything in the way
veymeer models                       # the models on this machine
veymeer serve --model qwen2.5:7b     # serve one at http://127.0.0.1:8080/v1

Any OpenAI client works against it. Nothing about the client changes except the address.

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8080/v1", api_key="not-needed-locally")

stream = client.chat.completions.create(
    model="qwen2.5:7b",
    messages=[{"role": "user", "content": "Explain what a mutex is."}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen2.5:7b", "messages": [{"role": "user", "content": "Hello"}]}'

Or skip the server and talk to a model in the terminal:

veymeer chat --model qwen2.5:7b

The command line

Every command takes --help.

The output of the engine on our test machine, replayed. Yours will name your own hardware.

The API

OpenAI's routes and fields, plus a few of the engine's own. Streaming uses server-sent events and ends with data: [DONE]. Errors are JSON with the usual status codes.

Reading ahead

The first word of an answer waits for the prompt to be read. An application that knows the beginning of a prompt before the question exists, such as an open file or a long instruction, can send it ahead. The request that follows only reads what is new.

curl http://127.0.0.1:8080/prime \
  -H "Content-Type: application/json" \
  -d '{"messages": [{"role": "system", "content": "...a long instruction..."}]}'
# {"tokens": 1840, "read": true}

Per-request switches

Each response carries a veymeer object with the speed it was written at. A request can turn features off for itself in its own veymeer object: "remember": false, "ahead": false, "draft": false, "speculate": false. None of them changes the answer.

Structured output and tools

An answer can be held to a form, and then it always has that form.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen2.5:7b",
    "messages": [{"role": "user", "content": "Invoice 1042, 3 items, 59.90 EUR, due 2026-11-01"}],
    "response_format": {"type": "json_schema", "json_schema": {"strict": true, "schema": {
      "type": "object",
      "properties": {
        "number": {"type": "integer"},
        "total":  {"type": "number"},
        "due":    {"type": "string", "format": "date"}
      },
      "required": ["number", "total", "due"]
    }}}
  }'

Give max_tokens room: an answer that runs out of tokens before its form is complete ends with finish_reason: "length". A schema keyword that cannot be enforced is refused when strict is set, instead of being ignored.

Settings

Every setting can be given in a settings file, as an environment variable (VEYMEER_<SETTING>) or as a flag (--<setting>), later ones overriding earlier ones. veymeer config prints the settings a serve would run with.

Changing settings while it runs

A running server takes changes over its API, or from the command line. A change that needs the engine rebuilt lets running answers finish, queues new ones, and is rolled back if the engine does not start with it. Web pages cannot change settings, since any site you open could send requests to your machine: only programs, pages served from this machine, and sites named in cors_origins can.

veymeer settings set ctx=8192 precision=fast --persist

curl -X PATCH http://127.0.0.1:8080/v1/settings \
  -d '{"model": "llama3.1:8b", "draft_model": "off"}'

Models

A model is named the way Ollama names it (qwen2.5:7b) or by the path of a .gguf file. Named models are looked up in the models folder: --models-dir, else $OLLAMA_MODELS, else Ollama's own folder. veymeer models lists every model found and says ok or why it cannot run.

A draft model has to share the large model's vocabulary: a small model of the same family, such as Qwen2.5 0.5B for Qwen2.5 7B, or Llama 3.2 1B for Llama 3.1 8B. The engine refuses a pair that does not match.

Signing in

The Individual tier needs no account. Signing in brings a paid plan to a device.

veymeer login      # opens the browser; sign in there and approve this device
veymeer license    # who is signed in, and the tier in force
veymeer logout     # sign this device out

The engine prints a short code and opens the device page of this site. You sign in there, with your second factor if the account has one, and approve the device. The engine then holds a licence for that device and renews it by itself while it serves. A machine without a browser can use veymeer login --no-browser and approve from any other device.

The first time the engine is run at a terminal it asks whether to continue without an account or to sign in. A program that runs the engine for you, such as an editor, a service or a script, is never asked.

Clusters

One model can run across machines on the same network.

# on every machine that lends its hardware
veymeer node --beacon

# on the machine that serves
veymeer serve --model llama3.1:8b --discover --cluster-key KEY
# or name the nodes
veymeer serve --model llama3.1:8b --nodes KEY@192.168.1.20,KEY@192.168.1.21

Connections are encrypted and both ends must hold the join key. Treat the key like a password. How many other machines a cluster may have depends on the tier; what we have and have not measured is on the hardware page.

What the engine sends

Serving a model, the engine talks only to the clients that call it, and to the machines of a cluster if you set one up. Beyond that it makes a network request of its own in three cases, each of which you start or can see:

The identifier is a one-way hash of the operating system's installation id. It is stable for an installation and means nothing anywhere else; no hardware address is read or sent. The engine never sends your models, your prompts, your answers, or how much you use it.

Troubleshooting