Running sessions and workers.

The first worker up boots the VM with cloud-init, waits for bootstrap, installs the three static helpers from --binaries (default: x86_64-unknown-linux-musl/release under $CARGO_TARGET_DIR, or under ~/.cache/b10x-target/mantle when that is unset, which is where task build-worker puts them), installs the Claude token when [claude] is configured, starts Substrate, installs pinned Codex and prints the worker’s identity and facts.

On AWS, worker up also creates a CloudWatch alarm that stops the instance after 24 five-minute periods (two hours) below 2% average CPU. --no-idle-stop leaves it out. worker down stops the VM; the data volume stays, and retained cloud storage may still incur charges.

On a worker that already runs, worker up starts services but does not restart a running older Substrate daemon, defers changed shared binaries while services are active, and refuses to replace helpers installed from a release bundle. Use offline upgrades for those.

A worker created by 0.2.0 runs Substrate 0.7.10. The 0.3.1 CLI and helpers cannot use it, and neither worker up nor an offline upgrade moves it to Substrate 0.16.1: sessions on 0.3.1 need a new worker. See upgrading from 0.2.0.

mantle stop my-project
mantle restart my-project
mantle destroy my-project --yes

• stop ends and retires the agent and keeps the workspace, the edited files and the private login home. destroy NAME --yes deletes them.
• restart starts a fresh agent execution in the same workspace with the persisted agent, cwd, environment and resource policy; a command session runs its stored argv again. It does not clone the repositories again or overwrite the private home. It does not resume the conversation.
• attach never starts an agent. A missing workspace is refused, never recreated. A retained name stays reserved until destroy.
• exec uses the existing workspace and injects no agent credential.

Interruptions and concurrent commands

For automation, pass --expected-session-id ID with the recorded id, not just the reusable name: the CLI compares it before any remote change. Concurrent commands claim a durable generation and intent before acting. An interrupted restart observes its original admission operation instead of starting another process; run the same command again to reconcile. A missing or unknown observation leaves the command incomplete with a nonzero exit. A confirmed running or ended admission can be stopped or destroyed even when readiness failed. An interrupted first materialization stays incomplete when its termination cannot be proven; it is not reported as a retained workspace.

Records from 0.1.x

A historical STOPPED record means destroyed and cannot be restarted. Migration keeps a historical STOPPING record as a destroy intent, so retrying it may still delete the workspace. Records without a validated launch context cannot be restarted from today’s configuration. Changed worker bindings and unsupported persisted policy versions are refused. Retention does not recover a lost worker or volume.

mantle exec exits with the remote exit code (0–255) and maps the INT, TERM and KILL signals to 130, 143 and 137. A refused, missing, contradictory or indeterminate result exits nonzero. After a valid Substrate response is decoded, stdout and stderr keep their bytes and the status line follows on stderr. A malformed response fails without printing raw output. Networked Cargo builds through exec need the proxy set explicitly; see session environment.

mantle profile add personal --config "$HOME/.config/mantle/personal.toml" \
  --state-dir "$HOME/.local/state/mantle-personal"
mantle profile list
mantle profile show personal
mantle --profile personal list
MANTLE_PROFILE=personal mantle worker status

• A profile stores two absolute paths in a private file under ~/.config/mantle/profiles/. Registering one does not copy credentials, read the configuration, create the state directory or contact a worker. Existing names cannot be overwritten.
• Names: lowercase letters, digits, _ and -, starting with a letter or digit, at most 63 characters.
• --profile overrides MANTLE_PROFILE. A named selection refuses MANTLE_CONFIG and MANTLE_STATE_DIR; unset them first. Without a profile, those two variables (absolute paths) and the defaults ~/.config/mantle/config.toml and ~/.local/state/mantle apply.
• Each state directory holds its own SQLite database, SSH key and known hosts. Use one state directory per profile. Attachments use separate short private socket directories, also with long or Unicode state paths.
• Profile entries and selected paths refuse symlinks and unsafe file types. Configuration and state ancestors must be owned by you or root and not writable by others; root-owned sticky directories are allowed.
• State paths may contain spaces, #, colons and Unicode. Quotes, backslashes, % tokens and ${...} are refused because OpenSSH would interpret them.

mantle --profile personal doctor
mantle --profile personal doctor --json --timeout-secs 5

Checks, in order: selection, configuration, local state, the recorded worker and placement, the provider, strict SSH, the Substrate service and socket, installed Mantle binary versions with live Substrate version and wire compatibility, and the common confinement facts. Each reports healthy, failed or skipped. A failure skips the checks that depend on it and exits nonzero. JSON stays structured even when selection or configuration fails. Reports give fixed advice and omit raw provider, SSH and configuration contents. A healthy worker does not prove agent authentication.

Doctor needs an existing SSH key and pinned known host. It never creates keys, accepts new hosts, runs an agent credential command, initializes or migrates the database, changes records or restarts services. SQLite opens read-only and query-only and sees committed WAL data. Your provider’s normal authentication plugins may run. Bounds: each external probe 10 seconds by default (1–120), with timed-out process groups and tunnel descendants killed; configuration input 1 MiB; database and sidecar files 64 MiB; SQLite busy and query handling 250 ms.

mantle --profile personal list --json
mantle --profile personal status my-project --json --timeout-secs 10
mantle --profile personal status --session-id SESSION_ID --json

Use the JSON forms in scripts; human status may print launcher diagnostics and is not a stable format. list --json includes ended records and does not contact the worker. status --json separates the recorded identity and state from bounded Substrate observations, including unavailable or refused ones; a lookup by session id also finds destroyed records. In both, a command session’s record carries argv, its declared argv; Claude Code and Codex records have no argv key. Neither fetches terminal output. The recorded authentication method is metadata, not proof of login.

To own what you create, start with mantle start MANIFEST --detached --receipt PATH. The receipt is written only after a successful start and holds the immutable session id and the profile binding. PATH must not exist; put it in a private directory (Mantle does not check the directory). Persist it before acting on the session: a name lookup cannot replace a lost receipt. An interrupted start may have created resources without writing a receipt; treat that as unresolved rather than claiming or destroying the name.

mantle-acceptance ships in the GNU bundle. It runs the same cases for Claude Code and Codex against a disposable session it creates, on a worker and profile you already configured. It does not provision workers or adopt existing sessions.

mkdir -m 0700 "$HOME/.cache/mantle-acceptance-run"
mantle-acceptance run --real --mantle "$HOME/.local/mantle/bin/mantle" \
  --profile personal --agent codex --manifest examples/codex.yaml \
  --checkpoint "$HOME/.cache/mantle-acceptance-run/run.json"

For Claude Code use --agent claude-code with a Claude manifest and a profile whose [claude] source is configured. Add --release-manifest /absolute/path/manifest.json to bind the run to a verified bundle: the manifest, both sibling archives and the CLI’s digest must match, otherwise the run is refused. Without it, the source identity is recorded as unknown; a version string or install directory name is not proof of source.

Login, a model and tool turn and the visual check happen in your terminal through the attach command the runner prints. Use it unchanged: it runs the run’s private copy of the CLI with the resolved configuration and state paths pinned. Then submit only what you observed:

mantle-acceptance resume --real \
  --checkpoint "$HOME/.cache/mantle-acceptance-run/run.json" \
  --expected-run-id RUN_ID --expected-exec-id EXEC_ID \
  --attest login --attest model-tool --attest visual
mantle-acceptance report --checkpoint "$HOME/.cache/mantle-acceptance-run/run.json"
mantle-acceptance cleanup --real --checkpoint "$HOME/.cache/mantle-acceptance-run/run.json"

• Exit 0: every required case complete. Exit 1: a case failed. Exit 2: incomplete or waiting for you; resume with missing attestations returns 2 with the same ids.
• report reads the checkpoint without contacting the worker. cleanup removes only the owned disposable workspace and its login state and returns the aggregate status: cleanup after an aborted run can still return 2, and a failed case stays failed.
• The checkpoint binds the CLI’s digest, the resolved profile and the exact session identities. Cleanup needs the creation receipt; a matching name or prefix is not enough. resume checks the bindings again.
• The runner does not read authentication caches, store terminal transcripts or infer login from a running process. Attestations are your statements, kept apart from machine observations. The retained-workspace case shows that the workspace and a marker file survive a fresh agent execution, not that a conversation resumes.
• The runner’s own tests use controlled CLI and PTY fixtures. They do not make a live, authenticated qualification.

mantle worker upgrade replaces the three Mantle helpers and keeps worker data, configuration, Claude, Codex’s generation link and the Substrate daemon. A different Substrate version or revision is refused. It does not drain sessions and never stops, masks, unmasks, restarts or signals a service. --check also works against an older installed helper: it uses bounded read-only SSH observations and uploads nothing.

1. Prepare the worker

Take an exclusive maintenance window. Save wanted work and retire every execution. Keep other administrators and older Mantle clients away for the whole window: they do not take the new installation lock. Through mantle worker ssh:

sudo systemctl stop substrate.service mantle-egress.service
sudo install -d -m 0700 /var/lib/mantle/maintenance/units
sudo mv --no-clobber /etc/systemd/system/substrate.service /var/lib/mantle/maintenance/units/substrate.service
sudo mv --no-clobber /etc/systemd/system/mantle-egress.service /var/lib/mantle/maintenance/units/mantle-egress.service
sudo systemctl mask substrate.service mantle-egress.service
sudo systemctl daemon-reload

Run these only on first entry: inspect saved originals that already exist and never overwrite them. The originals must keep the rendered unit bytes, root:root, mode 0644 and one hard link. A plain mask cannot replace the regular unit files Mantle installs in /etc/systemd/system, and a runtime mask alone does not override them; that is why they are moved. The checker refuses missing originals, unsupported overrides, ineffective masks, queued jobs, nonzero service PIDs, populated cgroups and unknown observations; it inspects the cgroup subtree even when ControlGroup is empty.

2. Check and apply from the laptop

mantle worker upgrade --check --manifest /absolute/path/to/manifest.json
mantle worker upgrade --apply --manifest /absolute/path/to/manifest.json

The check report separates installed versions, the selected source and digests, Substrate compatibility and the maintenance prerequisites. Equal version strings alone never count as current. Apply repeats the observations under /opt/mantle/install.lock, stages a complete sibling of /opt/mantle/bin and swaps the two directories in one atomic exchange; a filesystem without exchange support is refused, and there is no file-by-file fallback. Each transferred file is at most 64 MiB and keeps its verified digest. Codex installation and helper reconciliation take the same host lock first.

3. Read the result

• applied-restart-required: success. Services stay masked and inactive. The previous directory and a synced journal remain under /opt/mantle.
• recovery-required: keep /opt/mantle/upgrade-pending.json and the .upgrade-* directories; do not remove or rename them to force a retry. Running the same apply again inspects the directory identities and completes the transaction or verifies the restoration. Rollback is reported only after the exact previous directory is seen restored.
• recovered-other-bundle: an interrupted earlier journal took precedence; the result names its source and exits nonzero. Review it, then apply the intended bundle again.
• A disconnected transfer can leave a private /var/tmp/mantle-delivery.* stage for inspection; it is not an installed generation. Codex installation is refused while a helper transaction is unfinished. A completed journal is checked without replaying rollback.

4. Leave maintenance

sudo systemctl unmask substrate.service mantle-egress.service
sudo systemctl unmask --runtime substrate.service mantle-egress.service
sudo mv --no-clobber /var/lib/mantle/maintenance/units/substrate.service /etc/systemd/system/substrate.service
sudo mv --no-clobber /var/lib/mantle/maintenance/units/mantle-egress.service /etc/systemd/system/mantle-egress.service
sudo systemctl daemon-reload
sudo systemctl start mantle-egress.service substrate.service

Check the restored unit files before starting, then run mantle doctor from the laptop. Installed versions do not prove the running daemon’s identity or that an agent can authenticate.

The upgrade transaction is verified by package tests and native scenarios on controlled filesystems. No release record shows it applied to a live worker.