Local inference & models

What the local model is used for

The language model is used for onboarding and explanation: reading a photo, drawing or document while planning a site, and answering a question with evidence attached. It is never part of the sensing path. Counts, machine states, alarms and defect verdicts all come from the deterministic detectors and the vision worker — see Detectors and Vision. If the model server is down, detection and alerting keep working; only onboarding assistance and the Ask feature are affected.

Serving

rw-infer exposes an OpenAI-compatible API over one of three backends, chosen per site:

• llama.cpp server — the default on unified-memory boxes and smaller GPUs (GGUF weights).
• vLLM or SGLang — for production GPU servers, where prefix caching suits the repeated-turn pattern of an agent workflow.

A model profile file declares the backend, context length, tool-call format and concurrency for one model. rw infer args <profile> renders that profile into the flags the chosen backend actually needs, so the backend container itself never needs the rw tool or a network connection to start.

uv run rw models pull <profile-or-repo> --dest ./.rw/models
uv run rw infer args <profile> > .rw/infer-args/rw-infer.args
RW_PROFILE=<profile> docker compose --profile demo up -d --wait rw-infer rw-embed

rw models pull is the one explicit, admin-run step that reaches the network — it downloads weights once, ahead of time. Nothing in the running rw-infer/rw-embed containers makes an outbound call: they read weights already on disk.

Choosing a model

ReduceWaste is not tied to one model. The profile layer supports any model that passes the project’s own evaluation suite for the capabilities it needs (tool use, grounding, and — where vision-based onboarding matters — image and document understanding). Model choice, and the exact tier recommended for a given site size, is an active area of evaluation; this page intentionally does not name a single required model, since that recommendation changes as new options are evaluated.

Fine-tuning

Two separate things are possible, both optional:

• A general, publicly released tuning pass trained on public industrial datasets and synthetic scenarios from the simulator, intended to improve the shared model for everyone.
• A site-specific adapter trained on that site’s own box, from its own confirmed findings. This kind of adapter never leaves the site it was trained on.

Neither is required to run ReduceWaste; both operate on top of the same local inference server described above.