Vision

Camera-based counting and defect detection run as their own worker, independent of both the signal detectors and the local language model.

Counting and tracking

Each configured camera is processed by a per-camera detection-and-tracking pipeline that produces line or zone counts. Those counts are published back as ordinary signals — the same way a current-clamp recipe’s signal is — and as vision events, so they flow into the same historian and the same dashboards as every other sensor.

Defect detection

Defect detection is trained on good parts only: during a short “teach normal” pass in the app, the system learns what a normal part looks like, and anything that deviates is flagged without needing a labelled set of known defects ahead of time.

Why this is not the language model

Counts, line/zone states and defect verdicts all come from this deterministic vision worker, never from the local language model (see Local inference & models). The model is used only during onboarding, for example to help interpret a photo of a cell layout while planning where a camera should go — it is never the thing producing a live count or a defect verdict.

Hardware and privacy

Vision compute scales independently of the rest of the system — see Hardware sizing for how many cameras fit on a shared box versus needing their own. Person blurring at the edge is on by default, and video clips are retained only around flagged events, following the site’s own retention policy — raw video is not kept indefinitely by default.

Accelerators and images

The vision worker is the same code on every box; only the inference runtime build in its image differs, and RW_VISION_PROVIDERS sets the order it tries (TensorRT, CUDA, OpenVINO, CPU by default — anything the image lacks is skipped, the CPU is always the last resort).

Image	Hardware	Where it comes from
rw-vision (CPU)	any x86_64 or arm64 box	published
rw-vision OpenVINO (services/rw-vision/Dockerfile.openvino)	Intel CPUs, iGPUs, NPUs (x86_64)	published; open source only (OpenVINO is Apache-2.0)
rw-vision-gpu (services/rw-vision/Dockerfile.gpu)	x86_64 + NVIDIA GPU, GB10 / DGX Spark class (arm64), Jetson Thor or JetPack 7	built on site
rw-vision-gpu (services/rw-vision/Dockerfile.jetson)	Jetson Orin on JetPack 6 (L4T r36.4)	built on site

The GPU images are never published: they contain NVIDIA’s CUDA, cuDNN and TensorRT libraries, which are proprietary and outside the project’s open-source licence policy. A site builds them itself from the Dockerfiles in the repository, accepting NVIDIA’s licence terms; the project’s CI only checks that they build.

# OpenVINO (Intel): device CPU by default; GPU/NPU need the device node in an override file
docker compose --profile openvino up -d --wait rw-vision-openvino
RW_VISION_OPENVINO_DEVICE=GPU docker compose --profile openvino up -d rw-vision-openvino

# NVIDIA: build locally (needs internet for NVIDIA's packages), then run
docker compose --profile gpu build rw-vision-gpu
RW_VISION_GPU_DOCKERFILE=services/rw-vision/Dockerfile.jetson \
  docker compose --profile gpu build rw-vision-gpu      # Jetson Orin, JetPack 6
docker compose --profile gpu run --rm --entrypoint python rw-vision-gpu -m rw_vision.trt_build
docker compose --profile gpu up -d --wait rw-vision-gpu

TensorRT compiles each model for the exact GPU the first time it is used — minutes per model. rw_vision.trt_build does that ahead of the cameras and keeps the result in the rw-vision-trt volume, one folder per GPU, TensorRT and runtime version, so later starts take under a second. After a driver, TensorRT or image upgrade a new folder is built; old folders can be deleted.

The Jetson Orin image compiles its inference runtime from source during the build (about one to two hours on the device), because no ready-made build exists for JetPack 6 with the Python version the worker needs.

Measuring a box

uv run rw eval --suite vision --out evals/results/vision-$(hostname)

runs the three vision checks on the box it is started on, with the same RW_VISION_* settings as the worker: count error on three recorded conveyor feeds (gate ≤ 1 %), defect detection quality on the bundled sample set (image AUROC ≥ 0.95), and how many 15 fps cameras the box sustains with counting and defect inspection (gate ≥ 4 on the reference GB10 box; needs docker). Every result names the hardware and runtime it was measured on.