Start here · 1 of 3
Overview
llm is a set of Rust libraries for calling language models through one neutral turn, with credentials, routes, retries and costs kept explicit.
LLM
llm is a Rust workspace for calling language models. It gives a caller one neutral interface, a turn, and keeps everything that varies between vendors and deployments behind it: the wire protocol, the account, the credential, the route, the retry, the price and the machine a model runs on.
An agent loop, a service or a command-line tool depends on Model::turn. Behind that port an
operator can put a self-hosted vLLM server, a metered API account or a subscription login, and
change it without touching the caller.
Everything here is tested against recorded response bytes, loopback sockets and in-process fakes. The gate makes no paid provider call and starts no GPU. No provider route is qualified yet; read Status and Limitations before you build on it.
What it is not
- Not an agent loop. llm never executes a tool and holds no approval. A tool definition is a name, a description and a JSON Schema; when the model calls a tool, the caller runs it.
- Not a credential store. It resolves credentials through a resolver the embedding injects; it never runs a login flow, and reads a vendor's login file only where the caller points it. The one write it can make, renewing a Codex login, is opt-in.
- Not a gateway service. llm serves nothing and provisions nothing. llm-gateway (GitHub) serves models; llm is the client side.
Where it sits
| Neighbour | Relation |
|---|---|
| Loom (GitHub) | Consumes llm: Loom's crates depend on llm's client crates at a release tag. |
| inference (GitHub) | Consumes llm: its LLM node wraps llm_core::Model from llm at a release tag. |
| Harness (GitHub) | Consumes llm once its cutover lands: Atlas ADR 0097 decides that Harness drops its own model wire and credential crates and pins llm 0.7.0, in Harness's own work. |
| llm-gateway (GitHub) | Consumes llm: serves models to clients through llm's client crates at a release tag, and reads Runpod's provider description from llm-providers. |
| ESS (GitHub) | Specifies llm: every library crate has an ESS domain, run by the conformance suite. |
| Secrets (GitHub) | A credential source: llm-credentials feature secrets resolves references by name in its storage. |
More columns: swipe horizontally, or focus the table and use the arrow keys.
Why it exists
Model clients tend to fail quietly: they guess where they should have refused. llm is built around refusing those guesses.
| A common failure | What llm does instead |
|---|---|
| A missing usage counter is billed as zero | Every usage field is optional on its own. Unknown stays unknown |
| The configured model name is reported as the one that answered | The model the endpoint reported is a separate, optional observation. A route alias never fills the gap |
| A failed request is replayed elsewhere as if it were free | Every failure says whether the request was not-sent, rejected, accepted or unknown, and whether its class may be retried. A possibly billed attempt stays in the record |
| Provider state from one model is replayed against another | Opaque state carries six coordinates (protocol, provider, account, endpoint, model, binding revision). All six must match before it is sent |
| A rejected subscription credential falls back to a billable key | Billing kind is independent of protocol and credential, and an unauthorized failure never falls back |
| An error message leaks a header or the upstream response body | Diagnostics use fixed codes and messages, never headers, request bodies or upstream text |
More columns: swipe horizontally, or focus the table and use the arrow keys.
What you can do with it today
- Run a turn against any Responses, Messages or Chat Completions endpoint you declare, including an anonymous local vLLM server. Call a local endpoint
- Force one tool call on any model and get its JSON arguments back, through a Codex login or
any other
Model. Call a model with one forced tool - Run turns from a synchronous loop, on several threads at once. Use llm from a synchronous loop
- Build the client a catalog declares: one call turns a serving model's binding into a Chat Completions, Responses or Messages client. Call a local endpoint
- Read a provider description: a provider's inference URL template, wires, authentication and pinned control-plane operations, with Runpod's built in. Formats
- Declare routes in TOML and ask which target a request would get and why every other target was rejected; no secret is resolved to answer. Explain a route
- Retry and fall back: a retriable failure that showed nothing is retried on the same target, then the run moves to the next named target. Routing
- Resolve credentials at request time from a resolver you inject, a protected file, a keychain entry, an environment variable, a JSON document, a Codex login or the Secrets library's storage. Resolve a local secret
- Price recorded usage in exact decimals, and enforce a spending limit in a durable SQLite ledger. Price recorded usage
Where to go next
- Getting started: build the workspace and run the offline examples.
- The five concepts: the distinctions the whole design rests on.
- Crates: every package and its features.