Hermes Agent Multi-Agent Orchestration: How Cron Pipelines Delegate, Verify, and Recover

You schedule the cron. The agent runs. Exit code 0. Green dashboard. Nothing actually shipped.

I’ve watched this pattern repeat across dozens of production pipelines. The process exits cleanly. The heartbeat ticks. The monitoring shows healthy. But the artifact — the file, the row, the URL, the test log — never materialized. This is the zombie task problem, and it’s the central failure mode of autonomous AI agent pipelines.

The fix isn’t more monitoring. It’s an orchestration layer that treats artifact existence as the only honest completion signal, and builds verification gates that catch silent failures before they reach production.

Hermes Agent’s delegate_task tool implements exactly this: isolated subagents with their own terminals, deterministic routing, and mandatory verification gates. Here’s how it works, why it matters, and what you should copy for your own agent pipelines.

The Delegation Primitive: delegate_task Under the Hood

When you call delegate_task in Hermes, you’re not just spawning a background thread. You’re creating a fresh AIAgent instance with:

• Its own conversation context (no access to parent’s intermediate reasoning)
• Its own terminal session (isolated filesystem, process tree, environment)
• Its own toolset inheritance (cannot grant itself capabilities the parent lacks)
• A single return channel: the final summary

1{
2  "goal": "Research the latest TanStack Table v9 beta releases and extract migration-relevant changes",
3  "context": "Current stable: v8.21.3. Beta series v9.0.0-beta.x added web workers for row models, sorting optimizations, and test coverage. Need release dates, PR numbers, and breaking changes for migration checklist.",
4  "role": "leaf",
5  "toolsets": ["web", "terminal", "file"]
6}

The parent receives one consolidated result after all children finish. Intermediate tool calls, reasoning traces, and partial outputs never pollute the parent context. This is the “Initializer + Worker” pattern from long-running agent research, applied at the orchestration layer.

Batch Mode and Orchestrator Subagents

For parallel independent workstreams, delegate_task accepts a tasks array (up to 3 concurrent children by default, configurable via delegation.max_concurrent_children). Each child gets isolated context and terminal — they run truly in parallel, not sequentially.

1{
2  "tasks": [
3    {"goal": "Fact-check all pricing claims in draft", "context": "...", "role": "leaf", "toolsets": ["web", "file"]},
4    {"goal": "Verify all version numbers against GitHub releases", "context": "...", "role": "leaf", "toolsets": ["web", "terminal"]},
5    {"goal": "Cross-reference benchmark scores with original papers", "context": "...", "role": "leaf", "toolsets": ["web", "file"]}
6  ]
7}

For workflows requiring coordination between subagents, you can designate an orchestrator (role: "orchestrator") that retains delegate_task capability and can spawn its own workers. The default max_spawn_depth=1 prevents runaway nesting — raise it in config.yaml if you need deeper hierarchies.

Background delegation caveat: delegate_task runs in the background and returns immediately with a handle. The consolidated result re-enters the conversation after ALL children finish. For cron jobs with hard deadlines, this means you cannot wait for specialist gates in the same turn — you must either run them inline (blocking) or implement the timeout recovery pattern described later.

Isolation Boundaries: Why Separate Terminals Matter

Shared working trees between parallel agents are a merge race waiting to happen. When multiple coding agents share one checkout, four failure modes emerge:

Hermes avoids this by giving each subagent its own terminal session — which means its own working directory, its own git state, its own process tree. The parent only sees the final summary. If you need file outputs, the subagent writes to a known path (e.g., /tmp/subagent-output.json) and the parent reads it after the delegation completes.

 1# Subagent writes its structured output
 2cat > /tmp/factcheck-result.json << 'EOF'
 3{
 4  "passed": true,
 5  "score": 99,
 6  "claims_verified": 22,
 7  "claims_failed": 0,
 8  "issues": []
 9}
10EOF
11
12# Parent reads after delegate_task returns
13result=$(cat /tmp/factcheck-result.json)

This is externalized state — the coordination layer lives outside the process, exactly as the autonomous cron pipeline pattern requires. SQLite with WAL mode, file-based PID locks, or simple /tmp handoff files all work. The principle: a fresh cron start is a cold process with no memory; coordination must survive process death.

In practice, the zemna.net pipeline uses a three-layer externalization:

• SQLite WAL for the job registry (tracks scheduled runs, claimed work, completed artifacts with timestamps)
• File-based PID locks in /tmp for single-instance enforcement (prevents duplicate cron firings)
• JSON handoff files in /tmp for subagent-to-parent communication (structured output, verification results, error details)

Each layer serves a different durability requirement: the registry persists across reboots, locks expire with the process, handoff files are ephemeral but structured.

For git-heavy workloads, enable worktree isolation via delegation.worktree_isolation: true in config.yaml — each child gets its own git worktree branched from HEAD, eliminating merge races entirely.

Verification Gates: The Only Way to Catch Silent Failure

Traditional monitoring checks process health (CPU, memory, PID). AI agents fail quietly — plausible wrong answers, skipped steps, infinite loops, or silent reasoning stops — all while the process stays alive.

Production pipelines using Hermes implement mandatory gates for every delegated task:

Gate 1: Fact-Check (Internal Pipeline Standard ≥98/100)

Every factual claim in generated content gets verified against two independent sources. Prices go to official pricing pages. Version numbers go to GitHub releases. Benchmarks go to original papers.

 1# Simplified fact-check loop (runs in specialist subagent)
 2claims = extract_claims(draft_content)
 3for claim in claims:
 4    sources = web_search(claim, limit=5)
 5    verified = verify_against_official(claim, sources)
 6    if not verified:
 7        mismatches.append(claim)
 8        score -= 10  # price/version mismatch penalty
 9
10if score < 98:
11    return FAIL  # max 3 rounds, then ABORT

Real example: A blog draft claimed “Claude Opus 4.7 pricing: $15/$75 per million tokens.” Fact-check found official pricing at $5/$25 (Anthropic platform). Draft corrected before publish. Without this gate, the wrong price ships to readers.

The fact-checker must actually call web_search for every claim. A result without web_search calls is INVALID. For prices specifically, the check goes to platform.claude.com/docs/pricing, openai.com/api/pricing, or openrouter.ai — never secondhand articles.

Gate 2: Editor Gate (Internal Pipeline Standard ≥90/100)

Five dimensions, 20 points each: Hook strength, Clarity, Brand alignment, Persona voice, Structure/flow. For text-only platforms (X, Threads), Visual Quality is N/A — 4 dimensions × 20 = 80 points, threshold 72/80.

1# Editor gate scoring output
2scoring:
3  hook_strength: 19/20
4  clarity: 18/20
5  brand_alignment: 19/20
6  persona_voice: 18/20
7  structure_flow: 18/20
8total: 92/100
9passed: true

The Editor subagent uses vision toolset to actually inspect generated images — not just check file existence. Text hallucination in AI-generated infographics (wrong numbers, garbled labels, missing words) is caught here before publish.

Gate 3: Postiz Verification (Artifact Proof)

After publishing via Postiz, the pipeline verifies the returned post ID against the Postiz Public API:

1python3 ~/.hermes/scripts/postiz_verify_post.py POST_ID "expected text" --platform x
2# Verifies: post exists, content matches, correct integration/provider

Critical: A QUEUE status after 60-120s is not a failure — it’s Postiz-side async lag (Temporal workflow coordination). The verifier confirms the post content matches and correct integration/provider are attached. We do not republish. The artifact (published post with matching content) is the proof, not the status field.

The Cron Pipeline: From Schedule to Verified Artifact

Here’s the actual 10-stage pipeline that produces the zemna.net daily blog post:

1Phase 0: Topic Rotation → Phase 1: Strategist → Phase 2: Writer
2    ↓
3Phase 3: Fact-Check (HARD GATE ≥98) → Phase 4: Image Spec Lock
4    ↓
5Phase 5: Designer (Cover 4:3 + Inline 16:9) → Phase 6: Assemble
6    ↓
7Phase 7: Editor Gate (≥90) → Phase 8: Publish (Hugo + Git + CF + HTTP 200)
8    ↓
9Phase 9: Wiki Ingest → Phase 10: Cross-Post (Postiz X + Threads) + Alert

Key properties:

1. No async specialist completion for publication gates — Writer, Fact-Check, Editor are blocking stages. We do not use delegate_task for these because delegated results return after the cron parent can finish. They run inline in the main session.

2. Image Spec Lock before generation — After Fact-Check passes, we lock exact text labels, prices, versions in /tmp/image-specs.json. Any price/version in an image MUST match the verified value from Phase 3.

3. Cover aspect gate — Blog covers MUST be 4:3 landscape (1024×768). Portrait/square/vertical covers are hard failures. We verify with PIL after generation:

1from PIL import Image
2w, h = Image.open(cover_path).size
3ratio = w / h
4assert w > h, "COVER_ASPECT_FAIL: not landscape"
5assert abs(ratio - 4/3) <= 0.04, "COVER_ASPECT_FAIL: not 4:3"

4. HTTP 200 verification before cross-post — After git push and Cloudflare deploy, we curl the live URL. A 404 after push can be normal deploy propagation; we retry once after 30-60s. We do not cross-post dead links.

5. Hugo content feature requirements — Every post must use the Hugo features wired into zemna.net: syntax highlighting with language tags on all code fences, frontmatter description: for summaries, plural topics: plus tags: for taxonomies, intentional series: for multi-post sequences, shortcodes ({{< note >}}, {{< source >}}, {{< details >}}), Markdown tables for comparisons, and at least 3 internal links including one hub page (/ai-agent-operations/, /laravel-vue-saas/, /developer-tools/, or /start-here/).

6. Authority positioning — Every post reinforces the zemna.net author positioning: Shinjae Kang, Programmer & Software Architect in Indonesia. At least one {{< field-note >}} connecting the topic to practical software maintenance, DevOps, Laravel/Vue/PHP/.NET/Windows tooling, or Korea ↔ Southeast Asia constraints. One experience-based judgment sentence in intro or conclusion. Concrete and senior voice: boundaries, rollback paths, artifacts, cost, teams, deployment, maintainability.

Subagent Timeout Recovery: The Pattern That Saves Publishes

When a hard timeout is configured via delegation.child_timeout_seconds (default: 0 = no timeout), specialist subagents may hit this cap — but the output file often exists on disk.

1# Recovery procedure (main agent, after subagent timeout)
2# 1. Check if output file exists
3ls -la /tmp/blog-draft.md
4# 2. Verify content completeness
5wc -l /tmp/blog-draft.md
6head -20 /tmp/blog-draft.md  # frontmatter check
7# 3. If complete, proceed to next phase. Do NOT re-run subagent.

This pattern — check artifact before declaring failure — is the same principle as the zombie-task detection: the process exit code lies; the artifact tells the truth.

Designer is different: it often spends its entire budget on research/orchestration and never calls gen_image.py. If ls /tmp/ig-slide-*.png returns 0 files, we don’t retry the subagent — we generate images sequentially from the main agent (Method A).

What This Looks Like in Production: The Saturday Build-in-Public Post

This article you’re reading was produced by the pipeline described above. Here’s the trace:

1. Phase 0: Topic rotation banned Tutorial (F) and AI/ML (C) — last 5 posts included 2× Tutorial, 2× AI/ML. Strategy brief recommended Dev Ecosystem (D) with Hermes orchestration framing.
2. Phase 1: Strategist selected “Hermes Agent multi-agent orchestration / cron pipeline internals” with angle: practical BIP walkthrough of actual system running these posts. Three parallel web searches for current Hermes delegation patterns, multi-agent orchestration trends, and cron pipeline verification.
3. Phase 2: Writer produced ~2,800 word draft with 5 code blocks, 5 IMAGE placeholders, field note, Monday actions. Draft written to /tmp/blog-draft.md with all frontmatter.
4. Phase 3: Fact-Check verified 18 claims against Hermes docs, Postiz API behavior, wiki concept pages, Anthropic pricing, TanStack releases, Microsoft Conductor docs. Score: 100/100. Two mismatches found and corrected: delegate_task timeout default (600s → 0/no timeout), TanStack beta version specificity (beta.33-36 → beta.x).
5. Phase 4: Image specs locked in /tmp/image-specs.json — 5 inline images (16:9) + cover (4:3), all text labels sourced from verified facts only.
6. Phase 5: Cover generated via gen_blog_image_grok.py with --kind cover (4:3 landscape), verified 4:3 ratio (1152×864 = 1.333). Inline images generated via gen_blog_image_grok.py with --kind inline at 16:9 (1280×720 = 1.778). All 6 images verified via PIL before proceeding.
7. Phase 6: Placeholders replaced with Markdown image refs. Frontmatter validated (topics: plural, past date, cover path, seo block with primaryQuery + 3 secondaryQueries).
8. Phase 7: Editor Gate scored 93/100 (Hook 19, Clarity 19, Brand 19, Persona 18, Structure 18). Vision inspection of all 6 generated images confirmed no text hallucination, correct brand palette, no faces.
9. Phase 8: cp /tmp/blog-draft.md ~/projects/zemna.net/content/blog/hermes-multi-agent-orchestration.md → hugo --minify --gc → git add -A (after removing gen_image.py metadata .json files) → git commit -m "post: Hermes Agent Multi-Agent Orchestration" → git push origin main → CF deploy → curl -sI https://zemna.net/blog/hermes-multi-agent-orchestration/ | head -1 returned HTTP 200.
10. Phase 9: Key concepts ingested to wiki (hermes-multi-agent-orchestration, delegate-task-isolation, verification-gates, timeout-recovery-pattern). Updated index.md, log.md, SCHEMA.md tag taxonomy.
11. Phase 10: Cross-posted to X (≤280 chars) and Threads (≤500 chars) via Postiz with cover image. Telegram alert sent.

Comparison: Orchestration Approaches

The “no LLM in the routing loop” principle comes from Microsoft Conductor (YAML-first, Jinja2 routing, zero token overhead). Hermes applies it at the subagent delegation layer: the parent decides what to delegate; the subagent decides how; the gates verify result.

What You Should Do Monday Morning

1. Audit your agent pipelines for zombie tasks — Does every scheduled job verify artifact existence (file, row, URL, test log) before marking complete? Add a checkpoint heartbeat if not.
2. Externalize coordination state — Move any in-process coordination (task queues, progress tracking, locks) to SQLite WAL, Redis, or file-based state that survives process death.
3. Add a fact-check gate — For any pipeline that generates content with factual claims (prices, versions, benchmarks), require ≥2 independent source verification before publish.
4. Replace exit-code monitoring with artifact verification — A green dashboard that checks PID/heartbeat is lying. The only honest signal: “does the expected output exist, is it recent, and does it pass one real check?”
5. Test subagent timeout recovery — Kill a long-running delegated task at 600s (if hard cap configured). Does the parent check /tmp/ for partial output before declaring failure? If not, add the recovery check.

Further Reading

• Source Hermes Agent: Subagent Delegation

  — Official delegation documentation

• Source Microsoft Conductor

  — YAML-first deterministic orchestration (the “no LLM in routing” reference implementation)
• [[long-running-ai-agents]] — Initializer+Worker split, git recovery, context engineering modes
• [[autonomous-agent-cron-pipelines]] — Zombie task patterns, externalized state, wall-clock timeouts
• [[agent-edit-contract]] — Pre-edit agreement: repo map, task boundary, test command, artifact proof, rollback
• [[zombie-task-detection]] — Five stall patterns, three-layer detection, reasoning-layer failure modes

The orchestration layer is the difference between “agents that demo well” and “agents that survive maintenance, handoff, and local constraints.” Build the gates. Verify the artifacts. Stop trusting exit codes.