Lesson 1 of 6 · 0%Build an evidence-first incident recordNext
Course map

Troubleshooting, upgrades and recovery

0 of 6 complete0 of 6

Lesson 1.1 · 75 minutes

Build an evidence-first incident record

Capture enough signal to choose a safe next action before a restart, edit, or upgrade destroys the useful clues.

Skip course map

Verifiedon 2026.7.1

Action boundary

Before you act

Expected result
A redacted incident record separates observed facts, evidence locations, and hypotheses, and names one safe next check.
Failure mode
A restart, config edit, or broad log dump overwrites the evidence needed to understand the symptom.
Rollback
Stop intervention, preserve the current record locally, and escalate using the stated stop condition.

Why the first record is a technical control

When an OpenClaw gateway appears unhealthy, the tempting move is to restart it. Sometimes that is the right safety action. Often it is merely the fastest way to erase the moment in which the failure was observable. A restart can rotate logs, clear a transient queue view, change the process identity, trigger migrations, or make a flaky channel look healthy for five minutes. The operator who starts with a record preserves the option to act; the operator who starts with a repair may lose the ability to explain what happened.

Treat the first fifteen minutes as an evidence envelope. It is deliberately small: the user-visible impact, the time window and timezone, the installed version and install method, the affected boundary, the most recent relevant change, and the smallest set of read-only checks that can distinguish competing explanations. Do not write a root cause in the first paragraph. Write an observation such as “Telegram delivery stopped at 21:14 UTC after a gateway restart” and keep “configuration migration rejected a key” in a hypothesis field until a source and an observed result support it.

Use the committed 2026.7.1 game-day receipt as the model for a redacted command receipt. It records package integrity, exit-status-bearing mutations, the unsupported restore subcommand, the adapted rollback boundary, and post-restore checks without retaining synthetic secrets or raw logs.

This distinction is not pedantry. It prevents a plausible narrative from becoming an irreversible action. It also makes the handoff reproducible: a second operator can run the same bounded check and disagree with the hypothesis without disagreeing about the facts.

Flow diagram showing impact observation, timestamped capture, a low-confidence hypothesis, one read-only check, and a stop gate before restart or edit.
The evidence envelope narrows the next safe action; it does not manufacture a diagnosis.
  1. Observation records impact, scope, and time without a causal claim.
  2. Capture includes version, command, exit status, and redacted evidence location.
  3. A hypothesis is explicitly low confidence until a check supports it.
  4. Restart, edit, or upgrade waits behind the runbook stop gate.

Classify the boundary before collecting everything

OpenClaw is a gateway, not a single opaque process. A message can fail at the sender or channel account, gateway reachability, a managed service, configuration loading, a plugin or integration, an agent state store, or an external model/provider. Your first classification should be coarse enough to be reliable:

Boundary Question Safe evidence first
Client or channel Is the request reaching OpenClaw, and is one account affected? Channel status or a bounded live probe, with identifiers redacted
Gateway process Is the gateway reachable and answering its control surface? openclaw gateway status, openclaw health --json
Local configuration Can the installed build parse and validate the active shape? openclaw config validate, openclaw doctor --lint --json
Runtime or service Did the managed process start, stay up, and bind the expected surface? Service status, readiness endpoint where documented, recent logs
State or migration Does a database or session store match the running build? Version/schema evidence and the documented doctor or backup verification path

The table is a decision aid, not permission to run every command at once. Pick the smallest check that can falsify the current hypothesis. If the gateway does not answer, a live health RPC is not evidence that a channel account is healthy; if the gateway answers but one channel is dead, a global restart is not the least-change next step.

Worked example: a channel stopped after an upgrade

Assume a synthetic instance running OpenClaw 2026.7.1 reports that a single Slack account stopped delivering messages at 14:12 UTC. The operator has a maintenance record showing an update at 14:05 UTC. The first record should look like this:

Incident: INC-042 (synthetic)
Impact: Slack account A does not deliver replies; WebChat remains reachable.
Window: 2026-08-01 14:12–14:20 UTC; first report is approximate.
Version: openclaw 2026.7.1; install method and executable path recorded separately.
Recent change: gateway update completed at 14:05 UTC; exact result is unverified.
Boundary: likely channel account or post-update compatibility; gateway reachability not yet disputed.
Facts: WebChat health responds; no customer payload copied into this record.
Hypotheses: channel plugin compatibility (low confidence); credential expiry (low confidence).
Next check: run channel status/probe and doctor lint read-only; no restart yet.
Stop condition: any command proposes a write, exposes a token, or changes the evidence window.
Owner: operator A; review at 14:30 UTC.

The important details are the negative facts. WebChat remaining reachable narrows the blast radius, but does not prove the Slack plugin is the cause. “Update completed” is not the same as “post-upgrade checks passed.” The record keeps both unknowns visible.

The current official Doctor reference distinguishes read-only openclaw doctor --lint from repair-oriented --fix, and documents JSON findings and exit codes. The Health reference defines the gateway health snapshot; the Status reference covers diagnostic status and probes. Use those pages for the exact flags supported by the installed binary. Do not paste an older command into a production shell because its name sounds familiar.

Version-scoped, read-only evidence capture
openclaw --version && openclaw doctor --lint --json

Expected output: version: 2026.7.1; JSON findings with ok/checksRun/findings fields

That expected shape is a documentation contract for this course’s supported version, not a promise that every host has zero findings. Record the real output, exit code, timestamp, and redaction decision. If the command is unavailable, record that failure rather than substituting a different tool silently.

Build a redacted bundle, not a log landfill

The official Gateway diagnostics export can create a local ZIP containing sanitized status, health, log summaries, config shape, and stability events. The documentation still says to treat the bundle like a secret until reviewed: credentials and payloads are designed to be redacted, but local runtime facts remain sensitive. A useful evidence bundle has a manifest, source command, UTC timestamp, exit status, and a reviewer who checked the output before sharing it.

Prefer a bounded diagnostic export over copying an entire state directory. Never include API keys, cookies, authorization headers, full chat text, prompts, tool outputs, webhook bodies, customer identifiers, private hostnames, or local usernames. Replace them with markers such as [redacted:token] and record only that redaction occurred. Preserve enough surrounding context to reproduce the classification: subsystem name, status code, duration, queue count, plugin identifier if safe, and the relevant version.

If a command wants to mutate state, asks for a credential, or requires an unapproved production action, stop. A no-change decision with a clean handoff is a successful incident outcome when the evidence does not justify intervention.

Lab: the 15-minute incident envelope

Use a disposable OpenClaw instance or a synthetic paper scenario. Introduce one bounded symptom—for example, an intentionally invalid channel configuration in a sandbox. Do not use real credentials. Complete these steps:

  1. Write the impact, time window, version, recent change, and boundary classification before touching the configuration.
  2. Run the smallest read-only checks available in the current CLI help and official docs. Record exact command, exit status, UTC time, and expected versus actual result.
  3. Create a redacted evidence index. For each artifact, record what it proves and what it cannot prove.
  4. Write one hypothesis and one next check. Add a stop condition that would prevent a broad repair.
  5. Ask a peer to identify the affected boundary and next action using only the record.

Expected output is a one-page incident record plus an evidence index. Failure cases include a missing timestamp, an unredacted token, a command that changed state, or a hypothesis written as fact. Roll back the exercise by discarding the synthetic secret and restoring the sandbox baseline; do not “fix” the record after the fact. The learner artifact is the redacted incident envelope and a short peer-review note.

Local practice

Evidence envelope checkpoint

Every step remains visible without JavaScript. When enabled, this browser stores checks on this device only.

0 of 5 checked

Checkpoint: restart first?

A teammate proposes restarting because it is quick. The correct answer is: preserve the relevant time window and current status first, unless the approved incident runbook requires an immediate safety action to protect people, data, or access. If an emergency action is required, record the reason and timestamp before or immediately after it, then mark which evidence became unavailable. Speed is valuable; unrecorded speed is just uncertainty moved downstream.

Source receipt and version boundary

This lesson is scoped to OpenClaw stable 2026.7.1, verified against upstream revision 2d2ddc43d0dcf71f31283d780f9fe9ff4cc04fe4 in the course record. The live documentation pages were checked on 2026-08-01 UTC and may describe later behavior; the installed CLI and its exact release runbook win when they disagree. Primary receipts: Doctor, Health, Status, Diagnostics export, and v2026.7.1 release notes.

Source provenanceVerification and sources

Review receipt rr_recovery_evidence_first_incident

Outcome
approved
Method
source-review
Reviewer
academy-recovery-gameday-review
Reviewed

Evidence

Limitations

  • Game-day evidence is limited to synthetic, loopback-only state and the represented 2026.7.1 boundaries. The exact pinned CLI rejected `backup restore`; the adapted archive-manifest rollback was exercised only in disposable paths. Independent technical review approved this bounded evidence; restore remains a manual procedure because OpenClaw 2026.7.1 provides backup create/verify but no backup restore subcommand.

Open the public evidence snapshot

Lesson checkpoint

Ready to move on?

Mark this lesson complete when you can apply its outcome without relying on the examples above.