# How Canopy serves a request

Canopy sits between an inference user and a group of inference providers. Its job is to keep the
user's integration stable while finding a provider that can serve the requested preset.

## The request journey

1. **The client sends a familiar request.** The application uses the normal OpenAI Chat Completions
   format and places an Canopy preset in `model`.
2. **The preset sets the boundaries.** Canopy resolves the preset for the organization and finds
   provider models mapped to its canonical model.
3. **Providers are checked for eligibility.** A provider must be approved, active, connected, and
   configured for the requested endpoint and model.
4. **Eligible providers receive an RFQ.** The RFQ includes the information needed to estimate the
   work, such as model, size estimates, and request capabilities. It does not include prompt
   content.
5. **Providers quote or decline.** Quotes are valid only for a short period. Canopy ignores late,
   expired, duplicate, or mismatched quotes.
6. **A route is selected.** The current MVP selects the lowest valid expected cost. Provider
   eligibility and the preset still come first.
7. **Only the selected provider receives the request.** Canopy sends the complete request with the
   provider's upstream model ID and streams the response back to the client.
8. **The route can become reusable.** A successful request may create or refresh a time-limited
   commitment for a matching prompt prefix.

The user does not need to know which provider won. The provider does not receive work until its
quote is selected.

## Why prefix reuse matters

Long-running conversations often repeat the same system instructions, history, tools, or other
prefix content. A provider may offer a lower price for input it can reuse from its cache. Canopy can
remember that commercial commitment and use it for later requests with the same beginning.

Prefix reuse can provide:

* less repeated work for the provider;
* more stable pricing during a conversation; and
* a better chance of keeping a warm provider cache.

Canopy uses privacy-preserving matching for this purpose and does not retain raw prompt text.
Reuse stays within the request's organization, preset, and model. Applications can use the normal
request fields that identify a user or cacheable session to narrow reuse further.

For cache pricing to affect an offer's cost estimate, the quote must include a cached input rate and
cache terms. The TTL must meet the routing policy's required lifetime, the reusable prefix must meet
the minimum cacheable token count, and the cached rate must meet the required discount below the
fresh input rate. Otherwise, estimation uses the fresh input rate. Valid wire fields alone do not
qualify an offer for cache pricing.

A commitment is usable only while its terms, cache lifetime, and provider eligibility remain valid.
A changed preset or prompt prefix starts a new route decision when the existing commitment no longer
matches.

## What users should expect

* A preset identifies the model and route options available to the application.
* The full request goes to one selected provider, not to every provider that quotes.
* Successful provider responses keep the expected OpenAI-compatible response format.
* A provider failure before a response is selected returns an error in the normal API flow.
* A provider failure after streaming starts ends that stream. Canopy does not silently switch
  providers halfway through a response.
* The current MVP does not provide billing, provider settlement, or automatic provider failover.

See [get started as an inference user](/consumers) for request examples.

## What providers should expect

Providers maintain an outbound WebSocket connection and answer RFQs before their deadlines. They can
change standing prices and decide which model capabilities and workloads to quote.

An RFQ is deliberately smaller than an inference request. It exposes cost and capability signals,
not the user's prompt. The selected provider then receives the complete request at its configured
endpoint.

See the [provider overview](/providers), [SDK getting started guide](/providers/getting-started),
or [WebSocket protocol reference](/providers/protocol).

## When a new decision is made

Canopy starts a new auction when there is no usable commitment for the request. This happens when:

* no commitment matches the prompt prefix;
* the commitment or cache lifetime has expired;
* the preset or model has changed; or
* the committed provider is no longer eligible.

A connection failure invalidates that provider's active commitments. Client cancellation does not
invalidate a provider, but it also does not create a new commitment.