A Green Cron Exit Is Not a Finished Job

Your nightly embedding pipeline exits 0. The logs show INFO Job completed. The monitoring dashboard stays green. Three days later you discover the vector index hasn’t been updated since Tuesday — the agent skipped the upsert because the upstream API returned 429, caught the exception, logged a warning, and returned success. The cron scheduler sees exit 0 and marks the run healthy. This is not a bug. This is the default behavior of every cron implementation since 1979 (Version 7 Unix).

The question is not whether this demos well; it is whether it survives maintenance, handoff, and local constraints. In the Modoo Laravel SaaS projects we maintain across Jakarta and Ho Chi Minh City, silent cron failures have caused more production incidents than any deployment rollback. The pattern repeats: an agent wrapper catches all exceptions to “be resilient,” logs at WARN level, exits cleanly, and the business assumes the work happened. It didn’t.

The Anatomy of a Zombie Task

A zombie task is any scheduled job that reports success while producing no verifiable side effect. The term borrows from Unix: a process that has terminated but still has an entry in the process table. In cron land, the “process table” is your monitoring dashboard, and the “terminated process” is the business logic that never ran.

Bob Renze documented this pattern in production: his agent would stall on LLM calls, hit a timeout, catch the exception, and return exit 0 because the framework treated “handled exception” as success [Source: https://dev.to/bobrenze/how-ai-agents-handle-stalled-tasks-and-timeouts-lessons-from-my-production-failure-1jj9]. The downstream agent waited for a completion signal that never came. The cascade took 4 hours to detect.

Stack Overflow’s Agents TIL captures the core insight: “Don’t just add a Slack/Telegram alert on exit != 0. The original TIL’s point is that exit 0 is the lie. The alert needs to fire when ok=false, which is a semantic check the script itself has to make” [Source: https://agents.stackoverflow.com/tils/ce824637-609e-4ed9-a642-6b2935b77db2].

Three Flavors of Silent Failure

The “precondition skip” row is the only legitimate green. The rest are zombies wearing a success badge.

Proof Artifacts: What Production Actually Needs

UptimeRobot’s 2026 AI monitoring guide states the baseline: “Invest in the right monitoring tools, set up robust logging and tracing, and build a strategy that keeps your agents reliable for the long haul” [Source: https://uptimerobot.com/knowledge-hub/monitoring/ai-agent-monitoring-best-practices-tools-and-metrics/]. But tooling is not strategy. Strategy is defining what proof looks like for each job.

A proof artifact is a durable, queryable record that the expected side effect occurred. Three patterns cover 90% of cases:

1. Heartbeat File with Timestamp + Payload Hash

 1#!/usr/bin/env bash
 2# /opt/jobs/embedding-pipeline.sh
 3set -euo pipefail
 4
 5JOB_NAME="embedding-pipeline"
 6HEARTBEAT_DIR="/var/lib/job-heartbeats"
 7EXPECTED_ITEMS=10000
 8MIN_ITEMS_THRESHOLD=8000  # 80% floor
 9
10mkdir -p "$HEARTBEAT_DIR"
11
12# Run the actual work — capture structured output
13RESULT_JSON=$(php artisan app:embedding-pipeline --json 2>&1)
14EXIT_CODE=$?
15
16# Parse the agent's own success claim
17ITEMS_PROCESSED=$(echo "$RESULT_JSON" | jq -r '.items_processed // 0')
18UPSERT_COUNT=$(echo "$RESULT_JSON" | jq -r '.upsert_count // 0')
19LAST_VECTOR_ID=$(echo "$RESULT_JSON" | jq -r '.last_vector_id // ""')
20
21# Semantic success check — THIS is what matters
22if (( ITEMS_PROCESSED < MIN_ITEMS_THRESHOLD )); then
23    STATUS="degraded"
24    MESSAGE="Only $ITEMS_PROCESSED items processed (threshold: $MIN_ITEMS_THRESHOLD)"
25    EXIT_CODE=1
26elif [[ -z "$LAST_VECTOR_ID" ]]; then
27    STATUS="failed"
28    MESSAGE="No vector ID returned — upsert likely skipped"
29    EXIT_CODE=1
30else
31    STATUS="ok"
32    MESSAGE="Processed $ITEMS_PROCESSED items, upserted $UPSERT_COUNT vectors"
33fi
34
35# Write heartbeat — machine readable + human readable
36cat > "$HEARTBEAT_DIR/$JOB_NAME.json" <<EOF
37{
38  "job": "$JOB_NAME",
39  "run_at": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
40  "status": "$STATUS",
41  "exit_code": $EXIT_CODE,
42  "items_processed": $ITEMS_PROCESSED,
43  "upsert_count": $UPSERT_COUNT,
44  "last_vector_id": "$LAST_VECTOR_ID",
45  "message": "$MESSAGE",
46  "hostname": "$(hostname)",
47  "pid": $$
48}
49EOF
50
51# Also write a simple timestamp for dead-man's-snitch style checks
52date -u +%s > "$HEARTBEAT_DIR/$JOB_NAME.last_success"
53
54echo "$MESSAGE"
55exit $EXIT_CODE

The heartbeat file serves three consumers: your monitoring scraper (Prometheus/Grafana), your on-call dashboard (human readable), and your rollback logic (machine decidable).

2. Side-Effect Verification Query

Digital Thought Disruption’s control plane series emphasizes: “Design a safe AI agent execution runtime with idempotency, isolation, post-condition verification, compensation, rollback, and evidence controls” [Source: https://digitalthoughtdisruption.com/2026/07/25/execute-verify-rollback-agent-actions/]. The post-condition verification is the side-effect check.

 1# /opt/checks/verify_embedding_freshness.py
 2#!/usr/bin/env python3
 3"""
 4Verification probe: runs AFTER the cron job, queries the actual side effect.
 5Deploy as a separate scheduled check or as a Prometheus exporter endpoint.
 6"""
 7import os
 8import json
 9import psycopg2
10from datetime import datetime, timezone, timedelta
11from qdrant_client import QdrantClient
12
13QDRANT_URL = os.getenv("QDRANT_URL", "http://localhost:6333")
14PG_DSN = os.getenv("PG_DSN")
15COLLECTION = "documents"
16MAX_AGE_HOURS = 26  # Cron runs daily; allow 2h buffer
17
18def check_vector_freshness() -> dict:
19    client = QdrantClient(url=QDRANT_URL)
20    
21    # Get the most recent vector timestamp
22    scroll_result = client.scroll(
23        collection_name=COLLECTION,
24        limit=1,
25        with_payload=["updated_at"],
26        order_by="updated_at",
27        direction="desc"
28    )
29    
30    if not scroll_result[0]:
31        return {"ok": False, "reason": "Collection empty", "last_update": None}
32    
33    last_update_str = scroll_result[0][0].payload.get("updated_at")
34    if not last_update_str:
35        return {"ok": False, "reason": "No updated_at field on latest vector", "last_update": None}
36    
37    last_update = datetime.fromisoformat(last_update_str.replace('Z', '+00:00'))
38    age_hours = (datetime.now(timezone.utc) - last_update).total_seconds() / 3600
39    
40    # Also verify row count didn't drop (detects silent truncation)
41    count_result = client.count(collection_name=COLLECTION, exact=True)
42    current_count = count_result.count
43    
44    # Fetch expected count from source of truth
45    with psycopg2.connect(PG_DSN) as conn:
46        with conn.cursor() as cur:
47            cur.execute("SELECT count(*) FROM documents WHERE embedding_ready = true")
48            expected_count = cur.fetchone()[0]
49    
50    count_ratio = current_count / expected_count if expected_count > 0 else 0
51    
52    ok = (age_hours <= MAX_AGE_HOURS) and (count_ratio >= 0.95)
53    
54    return {
55        "ok": ok,
56        "last_update": last_update.isoformat(),
57        "age_hours": round(age_hours, 2),
58        "current_count": current_count,
59        "expected_count": expected_count,
60        "count_ratio": round(count_ratio, 3),
61        "reason": None if ok else f"Age: {age_hours:.1f}h (max {MAX_AGE_HOURS}), Ratio: {count_ratio:.1%}"
62    }
63
64if __name__ == "__main__":
65    result = check_vector_freshness()
66    print(json.dumps(result, indent=2))
67    exit(0 if result["ok"] else 1)

This probe runs independently of the job. It doesn’t trust the job’s self-report. It queries the effect.

3. Last-Success Stamp with Ownership

TianPan’s decision provenance work makes this explicit: “When autonomous agents take consequential actions, having logs is not the same as having accountability. A practical guide to designing decision provenance for production agentic systems — event schemas, ownership handoffs, hallucination attribution, and the compliance requirements that make this non-optional” [Source: https://tianpan.co/blog/2026-04-19-decision-provenance-agentic-systems].

 1# /etc/job-registry/embedding-pipeline.yaml
 2# Source of truth for: who owns this, what proves it ran, what to do when proof is missing
 3job:
 4  name: embedding-pipeline
 5  schedule: "0 3 * * *"  # Daily 03:00 UTC
 6  owner:
 7    name: "Platform Team"
 8    slack: "#platform-oncall"
 9    pagerduty: "PXXXXXX"
10  proof:
11    type: "composite"
12    heartbeat_file: "/var/lib/job-heartbeats/embedding-pipeline.json"
13    verification_probe: "/opt/checks/verify_embedding_freshness.py"
14    max_age_hours: 26
15    min_count_ratio: 0.95
16  escalation:
17    - after_minutes: 30
18      action: "alert_owner_slack"
19    - after_minutes: 60
20      action: "page_oncall"
21    - after_minutes: 120
22      action: "create_incident"
23  rollback:
24    # Compensation action when proof is missing for > 2 runs
25    trigger: "missing_proof_count >= 2"
26    action: "rebuild_index_from_source"
27    command: "php artisan app:embedding-pipeline --force-full-rebuild"

The registry entry is the contract. It says: here is the proof, here is the owner, here is what happens when proof goes missing.

The Monitoring Gap: Why Your Dashboard Lies

UptimeRobot’s AI observability guide notes: “Best practices for observability include establishing baselines, using tracing and logging, adding explainability, monitoring ethics, and ensuring uptime” [Source: https://uptimerobot.com/knowledge-hub/observability/ai-observability-the-complete-guide/]. But “uptime” for an agent cron is not “the process stayed up.” It’s “the side effect happened.”

Traditional Cron Monitoring vs. Agent Cron Monitoring

The critical difference: traditional cron assumes the script is the work. Agent cron assumes the script triggers work that happens elsewhere (vector DB, search index, external API). The proof must live at the destination, not the source.

Dead Man’s Switch Pattern

The simplest production-grade pattern: the job writes a timestamp. A separate checker alerts if the timestamp is too old.

 1# /opt/checks/dead_mans_snitch.sh
 2#!/usr/bin/env bash
 3# Runs every 5 minutes via systemd timer or cron
 4# Alerts if ANY registered job's last_success is stale
 5
 6REGISTRY_DIR="/etc/job-registry"
 7HEARTBEAT_DIR="/var/lib/job-heartbeats"
 8ALERT_WEBHOOK="${ALERT_WEBHOOK:-}"
 9
10for registry in "$REGISTRY_DIR"/*.yaml; do
11    [[ -f "$registry" ]] || continue
12    
13    job_name=$(basename "$registry" .yaml)
14    max_age_hours=$(yq '.job.proof.max_age_hours // 26' "$registry")
15    max_age_seconds=$((max_age_hours * 3600))
16    
17    stamp_file="$HEARTBEAT_DIR/$job_name.last_success"
18    if [[ ! -f "$stamp_file" ]]; then
19        msg="⚠️ $job_name: NO HEARTBEAT FILE (never ran or file deleted)"
20        curl -s -X POST -d "text=$msg" "$ALERT_WEBHOOK" >/dev/null
21        continue
22    fi
23    
24    last_success=$(cat "$stamp_file")
25    now=$(date +%s)
26    age=$((now - last_success))
27    
28    if (( age > max_age_seconds )); then
29        hours=$((age / 3600))
30        msg="🔴 $job_name: STALE by ${hours}h (max ${max_age_hours}h). Last: $(date -d "@$last_success" -u)"
31        curl -s -X POST -d "text=$msg" "$ALERT_WEBHOOK" >/dev/null
32    fi
33done

This 30-line script catches the class of failures that exit codes never will: the job that never started, the job that started but crashed before writing the heartbeat, the job that wrote a heartbeat but the side effect failed.

Ownership: The Missing Field in Every Cron Entry

Every zombie task has one thing in common: no named human owns the proof. The cron entry has a command. The monitoring has a dashboard. But when the proof goes missing at 3 AM, who gets paged? Who decides “rerun” vs “investigate” vs “rollback”?

TianPan’s provenance model includes “ownership handoffs” as a first-class concept [Source: https://tianpan.co/blog/2026-04-19-decision-provenance-agentic-systems]. In cron terms: the job registry must contain an owner with escalation contacts. Not a team alias. A human (or rotation) with a phone number.

The Owner Contract

 1owner:
 2  primary:
 3    name: "Sarah Chen"
 4    slack: "@sarah.chen"
 5    phone: "+62-812-XXXX-XXXX"  # Real number for pages
 6    timezone: "Asia/Jakarta"
 7  secondary:
 8    name: "Budi Santoso"
 9    slack: "@budi.santoso"
10    phone: "+62-813-XXXX-XXXX"
11    timezone: "Asia/Ho_Chi_Minh"
12  escalation_policy: "follow_the_sun"

This is not bureaucracy. This is the difference between “someone will notice” and “Sarah gets paged at 3 AM and knows exactly which verification probe failed and which rollback command to run.”

Rollback and Compensation: When Proof Is Missing

Digital Thought Disruption’s control plane series defines the full loop: “execute, verify, rollback” [Source: https://digitalthoughtdisruption.com/2026/07/25/execute-verify-rollback-agent-actions/]. Most teams stop at verify. Rollback is where the money is.

Compensation Actions by Failure Mode

The key insight: the compensation action belongs in the job registry, not in the job code. The job code does the work. The registry defines what “work not done” looks like and how to fix it.

 1# Extended rollback section in job registry
 2rollback:
 3  triggers:
 4    - condition: "missing_proof_count >= 1"
 5      action: "retry_with_backoff"
 6      max_retries: 3
 7      backoff_minutes: [15, 60, 240]
 8    - condition: "missing_proof_count >= 2"
 9      action: "force_full_rebuild"
10      command: "php artisan app:embedding-pipeline --force-full-rebuild --since=48h"
11    - condition: "verification_probe_failed && count_ratio < 0.5"
12      action: "page_owner_and_block_downstream"
13      # Prevents dependent jobs from running on stale data
14      block_jobs: ["search-index-refresh", "recommendation-training"]
15  notifications:
16    - channel: "slack"
17      template: "rollback_triggered.md"
18    - channel: "pagerduty"
19      severity: "critical"

This is the execute-verify-rollback loop made concrete. The monitoring system doesn’t just alert — it acts.

Implementation Checklist: From Zero to Production

You don’t need all three proof patterns on day one. Start here:

Week 1: Heartbeat Files

• Create /var/lib/job-heartbeats with appropriate permissions
• Modify top 5 critical cron jobs to write job_name.json + job_name.last_success
• Deploy dead man’s switch checker (runs every 5 min)
• Add Slack webhook for alerts

Week 2: Verification Probes

• Write one verification probe for the highest-impact job
• Deploy as Prometheus exporter endpoint (/metrics/job-name)
• Add Grafana dashboard panel: “Last Successful Side Effect”
• Set alert rule: job_verification_ok == 0

Week 3: Job Registry + Ownership

• Create /etc/job-registry with YAML for each job
• Assign named owners with phone numbers
• Define max_age_hours and min_count_ratio per job
• Document rollback commands in registry

Week 4: Automated Compensation

• Implement retry-with-backoff for first failure
• Implement force-rebuild for second failure
• Add downstream blocking for verification failures
• Run a game day: manually corrupt a heartbeat, verify alert + rollback

What You Should Do Monday Morning

1. Audit your top 10 cron jobs — For each, answer: “If this job exited 0 but did nothing, how would I know?” If the answer is “I wouldn’t,” that job needs a heartbeat file this week.

2. Pick one verification probe — Choose the job whose silent failure hurts most (billing sync, embedding pipeline, search index refresh). Write a 50-line script that queries the actual side effect. Deploy it as a separate check.

3. Create the job registry — /etc/job-registry/ with one YAML per critical job. Include: schedule, owner (name + phone), proof requirements, escalation timeline, rollback command. Commit to git.

4. Deploy the dead man’s switch — 30-line bash script, systemd timer every 5 minutes, Slack webhook. This catches the “job never ran” and “job ran but heartbeat missing” cases immediately.

5. Assign owners with phone numbers — Not Slack channels. Not team aliases. A human who carries a phone and knows the rollback command. Put it in the registry.

6. Schedule a game day — Within two weeks, manually break a heartbeat file. Verify the alert fires, the owner gets paged, the rollback runs. Fix whatever breaks.

The goal is not perfect observability. The goal is: when a zombie task appears, a named human knows within minutes and has a documented path to fix it.

Further Reading

• Source Bob Renze: How AI Agents Handle Stalled Tasks and Timeouts

  — Production failure postmortem with concrete timeout and stall patterns

• Source Digital Thought Disruption: Execute, Verify, Rollback Agent Actions

  — Control plane architecture for agent safety

• Source TianPan: Decision Provenance in Agentic Systems

  — Audit trails, ownership handoffs, compliance requirements

Related hubs: AI Agent Operations · Developer Tools · Start Here

See also: [[zombie-task-detection]], [[cron-silent-failure-patterns-infra]], [[parallel-agent-shared-checkout]]