Building a Background Agent Recovery CLI: The Three-Gate Check

The overnight coding agent wrote “Done.” The chat was green. The shipping cost file on disk was still yesterday’s version.

I do not mark that run recovered. Recovered means three things I can show a junior without opening the vendor UI: a file that exists, a job log that names why the process stopped, and one restart from the last good commit that is allowed to fail once.

The question is not whether the agent demos well. The question is whether the next person can restart the job from a known commit when I am offline.

Chat said done is not recovered

Vendors train you to trust the last message. VS Code even stores a checkpoint on every agent response, and Restore Checkpoint rolls the workspace and the chat back together [Source: https://code.visualstudio.com/learn/foundations/reviewing-and-controlling-agent-changes]. That is a useful undo. It is not a recovery record.

Addy Osmani’s long-running-agent write-up names the failure I still see: the model forgets, it declares the task complete when it is not, and a single sitting is the wrong shape for overnight work. State has to live outside the chat [Source: https://addyosmani.com/blog/long-running-agents/]. Anthropic’s harness post is the same idea in files: a progress log on disk, a feature list that starts failing, and the next session reading those files instead of the last boast [Source: https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents].

On this desk the translation is smaller than a product. I do not wait for a new CLI brand. I wait for three artifacts:

If any gate is empty, the job is hung or lying. I do not open a second agent to “just finish it.” That is how you get two half-writes and no owner.

Gate 1 — the file has to be on disk

Juniors get hurt here because the chat lists paths. Listing is cheap. Writing is the work.

I pin the expected path on the ticket before the first prompt. For a shipping bug that is one file, not the whole tree. The map-as-allow-list rule from Your Coding Agent Needs a Map, Not a Bigger Context Window — Part 2 still applies: if the agent grepped a sibling folder, the run already left the ticket.

The check is boring on purpose. A one-year developer can run it.

 1#!/usr/bin/env bash
 2# scripts/assert-recovery-file.sh
 3# Usage: assert-recovery-file.sh PATH JOB_START_EPOCH
 4set -euo pipefail
 5
 6path="${1:?expected file path}"
 7start="${2:?job start unix time}"
 8
 9if [[ ! -f "$path" ]]; then
10  echo "FAIL: recovered file missing: $path" >&2
11  exit 1
12fi
13
14mtime="$(stat -c %Y "$path" 2>/dev/null || stat -f %m "$path")"
15if [[ "$mtime" -lt "$start" ]]; then
16  echo "FAIL: $path mtime $mtime is older than job start $start" >&2
17  exit 1
18fi
19
20echo "PASS: $path exists and is newer than job start"

I store the job start as a number in artifacts/job-start.txt when the wrapper launches. If the file is missing, I do not argue with the chat. The gate failed.

Claude Code keeps file snapshots for the 100 most recent checkpoints in a session, and /rewind (or Esc twice on an empty prompt) opens a restore menu [Source: https://code.claude.com/docs/en/checkpointing]. Use that when you need to undo a bad turn. Do not treat rewind as proof the overnight file landed. Rewind is a chat tool. Gate 1 is stat.

Claude Code also says checkpointing does not track files changed by bash, and it is not a replacement for Git [Source: https://code.claude.com/docs/en/checkpointing]. That is why gate 3 is git reset --hard to a named good commit, not Esc twice on a hung overnight wrapper.

Gate 2 — the job log has to name the stop

A process that exits 0 with an empty log is the same lie as a green cron. I already wrote the empty-output version in Empty Tool Output Is Not Success Until the Harness Says Interrupted. Recovery needs one more line: why it stopped.

Accept only a small set of stop reasons. Everything else is “unknown” and fails the gate.

1STOP_REASON=completed
2STOP_REASON=interrupted
3STOP_REASON=timeout
4STOP_REASON=oom
5STOP_REASON=tests_failed

The wrapper writes that line. The model does not. If the agent wants to claim completed, it still has to leave the file from gate 1. The log is the harness speaking.

 1#!/usr/bin/env python3
 2"""scripts/assert-stop-reason.py — fail unless the job log names a stop."""
 3from __future__ import annotations
 4
 5import sys
 6from pathlib import Path
 7
 8ALLOWED = {
 9    "completed",
10    "interrupted",
11    "timeout",
12    "oom",
13    "tests_failed",
14}
15
16
17def main() -> int:
18    log_path = Path(sys.argv[1] if len(sys.argv) > 1 else "artifacts/job.log")
19    if not log_path.is_file():
20        print(f"FAIL: job log missing: {log_path}", file=sys.stderr)
21        return 1
22    text = log_path.read_text(encoding="utf-8", errors="replace")
23    reasons = [
24        line.split("=", 1)[1].strip()
25        for line in text.splitlines()
26        if line.startswith("STOP_REASON=")
27    ]
28    if not reasons:
29        print("FAIL: job log has no STOP_REASON= line", file=sys.stderr)
30        return 1
31    last = reasons[-1]
32    if last not in ALLOWED:
33        print(f"FAIL: unknown STOP_REASON={last}", file=sys.stderr)
34        return 1
35    print(f"PASS: STOP_REASON={last}")
36    return 0
37
38
39if __name__ == "__main__":
40    raise SystemExit(main())

Addy Osmani’s limitation section is blunt: auditing 24 hours of autonomous activity is a human-time problem, and structured artifacts (PRs, commits, briefings, test runs) are how you make it tractable [Source: https://addyosmani.com/blog/long-running-agents/]. A STOP_REASON= line is that artifact at the smallest size that still names the stop.

Gate 3 — one fail-once restart from the last good commit

Restart is where teams leak. Chat said done, someone hits Run again, the agent writes a second patch on top of a dirty tree, and now nobody knows which commit was good.

I tag the last good commit before the overnight job starts. Recovery is allowed to hard-reset to that tag once. If the second run also fails a gate, the ticket goes to a human. There is no third click.

 1#!/usr/bin/env bash
 2# scripts/fail-once-restart.sh
 3# Usage: fail-once-restart.sh GOOD_REF JOB_ID
 4set -euo pipefail
 5
 6good="${1:?last good commit or tag}"
 7job_id="${2:?job id}"
 8stamp_dir="artifacts/restarts"
 9mkdir -p "$stamp_dir"
10stamp="$stamp_dir/${job_id}.once"
11
12if [[ -f "$stamp" ]]; then
13  echo "FAIL: restart already used for $job_id — human owns the next step" >&2
14  exit 1
15fi
16
17git rev-parse --verify "$good^{commit}" >/dev/null
18git reset --hard "$good"
19date -Iseconds >"$stamp"
20echo "PASS: reset to $good; restart stamp written"
21echo "Re-run the job wrapper now. Do not click a third time."

VS Code’s restore is a conversation rollback. Git reset is the production rollback. Keep them separate. Restore Checkpoint when you are still in the IDE and the turn went wrong [Source: https://code.visualstudio.com/learn/foundations/reviewing-and-controlling-agent-changes]. Use the fail-once script when the overnight wrapper is the process you trust.

Long-running production still needs a sandbox that survives a directory change and a human who owns the budget. That page is Long-Running AI Agents — From Demos to Production. This page is the morning after: the chat is closed, and you still have to prove recovered.

Wire the three gates in one wrapper

You do not need a published binary named agent-recover. You need a wrapper that refuses to print success until the three files exist. Put it next to the Laravel app so the next person on the ticket can run it.

 1#!/usr/bin/env bash
 2# scripts/run-overnight-agent.sh
 3set -euo pipefail
 4
 5job_id="${JOB_ID:-shipping-$(date +%Y%m%d)}"
 6expected="${EXPECTED_FILE:-app/Services/ShippingCost.php}"
 7good="${GOOD_REF:-recovery-good}"
 8root="$(git rev-parse --show-toplevel)"
 9art="$root/artifacts"
10mkdir -p "$art"
11
12date +%s >"$art/job-start.txt"
13start="$(cat "$art/job-start.txt")"
14
15# Replace this block with your real agent command.
16# The wrapper owns the log. The model does not write STOP_REASON=.
17set +e
18your_agent_cmd --task "$job_id"
19agent_ec=$?
20set -e
21
22if [[ "$agent_ec" -eq 0 ]]; then
23  echo "STOP_REASON=completed" >>"$art/job.log"
24else
25  echo "STOP_REASON=tests_failed" >>"$art/job.log"
26fi
27
28set +e
29"$root/scripts/assert-recovery-file.sh" "$root/$expected" "$start"
30file_ec=$?
31"$root/scripts/assert-stop-reason.py" "$art/job.log"
32log_ec=$?
33set -e
34
35if [[ "$file_ec" -eq 0 && "$log_ec" -eq 0 ]]; then
36  echo "RECOVERED: file + stop reason present"
37  exit 0
38fi
39
40echo "NOT RECOVERED: running fail-once restart"
41"$root/scripts/fail-once-restart.sh" "$good" "$job_id"

The wrapper is the CLI. If you later extract a Go binary, the contract stays the same: expected path, stop line, one stamp file. Do not wait for a GitHub release to start using the three files.

What this is not

This is not the five-name editor contract from Five Things I Refused This Week. That list is who pins the model, who merges, who retries. This page is only: is the overnight job recovered.

This is not a second zombie-detection essay. Detection without a file is still a dashboard. Recovery is the file.

This is not “restore the chat and call it a night.” Chat restore is for a bad turn while you are watching. Overnight recovery is for a process you were not watching.

Anthropic’s initializer agent writes claude-progress.txt so the next session is not blank [Source: https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents]. Keep a progress file if you want. I still fail the job if the shipping file is old. Progress text without an mtime check is another chat.

 1task: "Fix shipping cost calculation"
 2expected_file: app/Services/ShippingCost.php
 3stop_reasons:
 4  - completed
 5  - interrupted
 6  - timeout
 7  - oom
 8  - tests_failed
 9restart:
10  good_ref: recovery-good
11  max: 1
12checks:
13  - id: file_newer_than_job
14    type: file_mtime
15    path: app/Services/ShippingCost.php
16  - id: tests
17    type: command
18    command: php artisan test --filter=ShippingCost

Name the owner of the three files

A wrapper without an owner becomes folklore. Put three names on the ticket before the overnight run:

1. Who writes EXPECTED_FILE
2. Who reads artifacts/job.log in the morning
3. Who is allowed to click the fail-once restart

If those names are blank, do not start the agent. Abort is not stop. Abort still needs a retry owner. I ranked that refusal already. Here the artifact is the stamp file under artifacts/restarts/.

The operations hub for this family is /ai-agent-operations/. The map page tells the agent where it may search. The long-running page tells you the sandbox still has to hold after cd. This page tells you when you may say the job came back.

What you should do Monday morning

1. Pick one overnight job this week. Write the expected path on the ticket (app/Services/ShippingCost.php or your equivalent). Do not start the agent until the path is there.
2. Copy assert-recovery-file.sh, assert-stop-reason.py, and fail-once-restart.sh into scripts/. Run them against last night’s run even if chat said done.
3. Tag recovery-good on the last commit you would actually ship. That is the only reset target.
4. Create artifacts/job.log from the wrapper, not from the model. Require a STOP_REASON= line.
5. If the first restart stamp already exists, do not click Run. Assign a human. That person reads the diff.
6. Link this page, the map allow-list, and long-running agents on the ticket so the next person does not invent a fourth gate.

If you are new to this desk, start at /start-here/ and the developer tools hub. Then come back and fail one overnight job on purpose so you see the stamp file.

Further reading

Source Addy Osmani: Long-running Agents

Source Anthropic: Effective harnesses for long-running agents

Source VS Code: Reviewing and controlling agent changes

Source Claude Code docs: Checkpointing