# Engines and metrics

The SDK collects engine health and metrics before connecting, then polls them outside your quote callbacks. Your policy reads the most recent valid snapshot without making a network request for each quote.

## llama.cpp

```ts
import { llamaCpp } from "@canopy/provider";

const engine = llamaCpp({
  baseUrl: "http://127.0.0.1:11434",
  model: "gpt-oss:20b",
});
```

The contract target is llama.cpp `server-vulkan-b10795`, running one model with `--metrics` enabled. This adapter has been exercised locally.

| Metric                 | Meaning                              |
| ---------------------- | ------------------------------------ |
| `runningRequests`      | Requests currently processing        |
| `waitingRequests`      | Requests waiting in the engine queue |
| `promptTokensTotal`    | Raw cumulative prompt token count    |
| `generatedTokensTotal` | Raw cumulative generated token count |

Counters can decrease after an engine restart. If you calculate rates, discard a sampling window when a counter decreases. The SDK does not derive rates.

See the [llama.cpp metrics documentation](https://github.com/ggml-org/llama.cpp/blob/b10795/tools/server/README.md#get-metrics-prometheus-compatible-metrics-exporter) for the pinned contract.

## vLLM

```ts
import { vllm } from "@canopy/provider";

const engine = vllm({
  baseUrl: "http://127.0.0.1:8000",
  model: "your-served-model",
  labels: { engine: "0" },
});
```

The contract target is vLLM `0.17.1`, V1, with one selected model and engine. Live GPU execution validation is still pending. Do not assume that arbitrary versions or data-parallel deployments satisfy this contract.

| Metric              | Meaning                                       |
| ------------------- | --------------------------------------------- |
| `runningRequests`   | Requests currently running                    |
| `waitingRequests`   | Requests waiting in the engine queue          |
| `kvCacheUsageRatio` | KV cache usage as a fraction from zero to one |

The adapter selects the `model_name` label from `model`. Add `labels` to select exactly one engine when needed. Ambiguous samples fail validation. The SDK does not add worker percentages together or substitute zero for missing metrics.

See the [vLLM 0.17.1 metrics documentation](https://docs.vllm.ai/en/v0.17.1/usage/metrics/).

## Endpoint configuration

Both adapters accept these options:

| Option       | Description                                                                  |
| ------------ | ---------------------------------------------------------------------------- |
| `baseUrl`    | Engine origin used for `/health` and `/v1/models`, and by default `/metrics` |
| `model`      | Exact model ID returned by the engine's discovery endpoint                   |
| `apiKey`     | Optional bearer credential for engine collection                             |
| `metricsUrl` | Optional override for the metrics scrape URL                                 |

These are the agent's collection settings. Set Canopy' inference endpoint and its credentials separately in Provider config.

Use a metrics endpoint for the engine you actually serve. A load-balanced scrape that combines unrelated workers does not meet the single-engine contract.

## Freshness and failures

| Provider option    | Default | Purpose                                  |
| ------------------ | ------- | ---------------------------------------- |
| `pollIntervalMs`   | `1000`  | Delay between collection cycles          |
| `requestTimeoutMs` | `3000`  | Collection and ordinary callback timeout |
| `maxMetricsAgeMs`  | `5000`  | Maximum age of a usable snapshot         |

A collection failure immediately invalidates the snapshot. When collection fails or the sample becomes stale, callbacks receive both `metrics: undefined` and `collectedAt: undefined`.

Your policy decides whether to decline. The SDK does not override an otherwise valid acceptance because metrics are missing or the engine is busy. Startup still requires successful health, model discovery, and metrics validation.

Metrics stay in the provider process. Engine counts can include traffic outside Canopy and can overlap with `activeRequests`. See [Pricing and admission](/providers/policies) before combining them.