Lesson 4 of 6 · 0%Operate durable jobs with retry and idempotency gatesNext
Course map

Multi-Agent Workflows, Goals and Production Operations

0 of 6 complete0 of 6

Lesson 4.1 · 60 minutes

Operate durable jobs with retry and idempotency gates

Use persistent scheduled work without turning ambiguous failures into duplicate action or an unbounded queue.

Skip course map
Current lessonOperate durable jobs with retry and idempotency gates0% complete · 0/6 lessons

Multi-Agent Workflows, Goals and Production Operations

0% complete · Current: Operate durable jobs with retry and idempotency gates

Verifiedon 2026.7.1-2

Action boundary

Before you act

Expected result
A scheduled workflow has an owner, wall-clock budget, run-history check, idempotency key, retry policy, and an explicit no-retry class.
Failure mode
A transport timeout is treated as proof of non-execution, causing a second run to repeat an action whose first run may still be active or complete.
Rollback
Disable the job, inspect the durable run history and output ledger, reconcile the idempotency key, then resume only after an operator classifies the outcome.

Durable does not mean unattended

OpenClaw cron jobs persist job definitions, runtime state, and run history in shared SQLite state; each execution creates a background task record. That makes inspection possible, but it does not make a job safe to retry blindly.

The CLI documentation explicitly treats a lost Gateway connection as ambiguous: the Gateway may have accepted and may still finish the turn. Check Gateway status and the session transcript before retrying or rerunning, so the same work is not executed twice. Reuse a run ID only when the workflow’s deduplication design supports it.

Lab: create a retry policy

Write a policy table for the release-intelligence pipeline:

Failure class Automatic action Operator action
Official URL returns 503 before content is read bounded retry with backoff escalate after retry budget
Source content changes between read and review stop re-run review using a new fingerprint
Gateway timeout after draft dispatch no automatic retry inspect run/session record and draft ledger
Duplicate source event suppress using source URL + revision key record suppression
Publication step unavailable hold human owner decides next delivery attempt

An idempotency key should represent the effect you mean to produce, for example release-digest:<source-revision>:<week>, not a random task attempt. Store the key beside the source fingerprint and terminal state.

Observable job definition

For every durable job record: schedule, timezone, selected agent/role, wall-clock budget, model/fallback policy, input key, output location, delivery target, retry class, alert owner, and disable command. A job that cannot be disabled and inspected is not ready for production.

Tabletop check

Simulate a task whose worker loses connectivity after it writes a draft but before it reports success. Your evidence pack must answer: Which input key was active? Does a draft already exist? Is the process still running? What side effect is forbidden until a human reviews it? If you cannot answer all four, retrying is unsafe.

Source provenanceVerification and sources

Review receipt rr_multi_agent_durable_jobs_retries

Outcome
approved
Method
source-review
Reviewer
scout-independent-review
Reviewed

Evidence

Limitations

  • Durable jobs require testing against the operator’s delivery boundaries.

Open the public evidence snapshot

Lesson checkpoint

Ready to move on?

Mark this lesson complete when you can apply its outcome without relying on the examples above.