Canopy

Inference without the provider lock-in

Canopy gives applications one OpenAI-compatible endpoint in front of multiple inference providers. You keep the client, request format, and streaming behavior you already use. Canopy helps you find an eligible provider for each request instead of making your application integrate with every provider separately.

Why use Canopy?

• Keep your existing client. Change the base URL, API key, and model value. No client plugin or custom request flow is required.
• Find competitive prices. Providers compete to serve requests, while your application keeps one stable integration.
• Make costs and behavior more predictable. Presets define which model a request may use, and cache commitments can preserve a provider and its agreed terms for a continuing conversation.
• Keep prompts private during selection. Providers receive request details needed to quote, but the full request goes only to the selected provider.
• Give providers a practical way to participate. A provider can connect an existing OpenAI-compatible endpoint, publish prices, and decide which work it can serve.

Choose your path

Inference users

Use Canopy from an application, SDK, or AI coding tool. You need an inference API key and a preset model ID from your organization.

Get started as an inference user

Inference providers

Connect inference capacity to the marketplace. You need an approved provider account, a supported model endpoint, and a provider agent that maintains an outbound WebSocket connection.

Get started as an inference provider

How it works

1. An application sends a normal Chat Completions request with an Canopy preset in model.
2. Canopy checks the preset and the providers that can serve it.
3. Eligible providers return a time-limited quote without receiving the prompt.
4. Canopy selects a valid route and sends the complete request to that provider.
5. The provider response is streamed back in the format the application expects.
6. When a conversation repeats a usable prefix, Canopy can reuse an active provider commitment instead of starting from scratch.

See the request flow and provider responsibilities

Current support

The current API works with OpenAI-compatible LLM clients and supports:

• chat requests;
• streamed and non-streamed responses; and
• client cancellation while a request is in progress.

The system is an MVP. Billing, provider settlement, and automatic provider failover are not yet part of the user-facing flow. A provider failure after streaming starts ends that stream rather than silently switching providers.