Guides · 2 of 12

Call a local endpoint

One anonymous turn to a vLLM-compatible server, with the request llm sent and the answer it observed.

crates/llm-docs/examples/local_endpoint.rs, crates/llm-models, crates/llm-chat (client, fixtures/vllm-text-no-usage.sse), crates/llm-credentials (AnonymousResolver), examples/catalog.toml

Call a local endpoint

This guide sends one real turn to a Chat Completions server on your own machine, such as vLLM. It uses the local-small serving model from examples/catalog.toml, which declares an anonymous, self-hosted account:

[[accounts]]
id = "local"
provider_id = "my-lab"
auth_kind = "anonymous"
billing_kind = "self-hosted"

[[endpoints]]
id = "local-vllm"
account_id = "local"
base_url = "http://127.0.0.1:8000/v1"

[[models]]
id = "small"
upstream_name = "example/Small-Model"

Replace base_url, upstream_name and the declared capabilities with your server's facts. upstream_name is the model name the server answers to.

The program

The program is an example in the repository. Start your server on 127.0.0.1:8000, then run it from the repository root:

cargo run --locked -p llm-docs --example local_endpoint

In your own project it needs llm-core, llm-models, llm-http, llm-credentials, llm-routing, and tokio with the rt and macros features; see Getting started for the Git dependency form.

crates/llm-docs/examples/local_endpoint.rs

//! Send one turn to the anonymous Chat Completions server `examples/catalog.toml` declares.
//!
//! Start a vLLM-compatible server on `127.0.0.1:8000`, then run
//! `cargo run --locked -p llm-docs --example local_endpoint` from the repository root.
use llm_core::{Cancel, Id, Item, Model, TurnRequest, VecSink};
use llm_credentials::SecretResolver;
use llm_http::{HttpClient, Limits};
use llm_routing::Catalog;
use std::{collections::BTreeMap, sync::Arc};

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let catalog = Catalog::parse(&std::fs::read_to_string("examples/catalog.toml")?)?;
    // The local account is anonymous, so the caller supplies no resolver for it.
    let resolvers: BTreeMap<Id, Arc<dyn SecretResolver>> = BTreeMap::new();
    let client = llm_models::port(
        &catalog,
        &Id::new("local-small")?,
        HttpClient::new(Limits::default())?,
        &resolvers,
    )?;

    let request = TurnRequest::new("small", vec![Item::user("Hallo")]);
    let mut sink = VecSink::new(64, 64 * 1024);
    let outcome = client.turn(&request, &mut sink, &Cancel::new()).await?;

    println!("text:           {}", sink.text());
    println!("stop reason:    {:?}", outcome.stop_reason);
    println!(
        "upstream model: {:?}",
        outcome.observation.upstream_model.map(|m| m.to_string())
    );
    println!("usage:          {:?}", outcome.observation.usage);
    println!("final usage:    {}", outcome.observation.final_usage);
    Ok(())
}

llm_models::port takes a serving-model id and builds the client its binding declares: the binding's chat-completions protocol selects the Chat Completions client. The resolver map is keyed by account id; the anonymous account needs none, and a bearer or api-key account with no entry is refused unauthorized before anything is sent. The request names small, the catalog's own model id, which must match the binding's model.

What happens

The client sends one POST to http://127.0.0.1:8000/v1/chat/completions with this body:

{"messages":[{"content":"Hallo","role":"user"}],"model":"example/Small-Model","stream":true,"stream_options":{"include_usage":true}}

• model is the declared upstream_name, not the caller's small.
• stream_options.include_usage is always sent, so the server is asked for counters.
• No Authorization header is sent, because the account is anonymous. The resolver is never called.

Run against a local server that answered with the vLLM response fixture crates/llm-chat/fixtures/vllm-text-no-usage.sse, the program printed:

text:           Guten Tag
stop reason:    EndTurn
upstream model: Some("Qwen/Qwen3-8B")
usage:          None
final usage:    true

Read the last three lines carefully:

• upstream model is what the server said served the request. It differs from the declared example/Small-Model, and LLM reports the difference instead of hiding it.
• usage: None: this fixture's server sent no counters. They are unknown, not zero.
• final usage: true: the stream terminated, so what was reported is all there will be.

Without the catalog

A caller that builds the protocol client itself, with ChatClient::new, ResponsesClient::new or MessagesClient::new, still passes a resolver. For an anonymous binding, pass llm_credentials::AnonymousResolver:

let client = ChatClient::new(binding, http, Arc::new(AnonymousResolver));

The client of an anonymous binding never asks it for a secret. It holds none, so every resolve is refused with SecretError::Missing: a credentialed binding handed it by mistake fails unauthorized before sending anything, and routing does not fall back from that. It is the resolver llm_models::port attaches to an anonymous account's client, and it needs no llm-credentials feature.

Next

• Use bearer or api-key accounts with a real resolver: Resolve a local secret.
• Put several targets behind one alias and fall back between them: Routing.
• The same call builds the Responses or Messages client for a serving model declared over that protocol. llm_models::CatalogModels builds every serving model's port at once and serves them to ordered fallback.