aios/docs/AIOS_AUTOMATION_QUORUM_CONTROLLER.md

AIOS GPT + one-executor automation quorum

Authority and release boundary

The Gateway is the only authoritative state writer. It must hold the SQLite writer lease before changing task state. A model response, service process, configured credential, or agent presence does not establish health. Release remains paused until all live acceptance rows are backed by logs and exit codes.

Minimum quorum:

central_gpt=HEALTHY AND (openclaw=HEALTHY OR codex=HEALTHY)

UNKNOWN never counts as healthy. If one executor is proven healthy, an unhealthy or unknown standby is P3 telemetry only and cannot block the active gate. If no executor is proven healthy, the controller blocks. Central GPT UNKNOWN or UNHEALTHY always blocks.

Live health receipts

The controller reads three root-owned metadata-only JSON receipts from /run/aios-health. Files must be regular files, must not be symlinks, and neither files nor parent directories may be group/world writable. The controller rejects fields named secret, token, password, api_key, authorization, credential, private_key, environment, env, argv, stdout, or stderr at any depth.

A receipt claiming HEALTHY must contain:

{
  "subject": "codex",
  "health": "HEALTHY",
  "live_probe_succeeded": true,
  "observed_at_epoch": 1784347200
}

The central GPT receipt additionally requires either workspace_agent.validated_live=true or responses_api.validated_live=true. Responses fallback is allowed only after a real Responses decision probe succeeded; credential/configuration presence is insufficient.

Durable behavior

SQLite persists ingress event digests, optimistic task state versions, leases, checkpoints, lifecycle events, effect IDs, outbox messages, blocker counts, observed Git refs, and project summaries. Reusing an event/effect/outbox ID with different content fails closed. An expired lease may be reclaimed; a live lease cannot be stolen.

On active-executor failure, a healthy standby receives the same contract_commit, event_id, state version, and checkpoint. The former executor is recorded as INTERRUPTED_AGENT_FAILOVER. Unique effect IDs prevent replay of completed side effects.

The Gateway persists these events even when OpenClaw is offline:

TASK_DISPATCHED
TASK_STARTED
TASK_ENDED
DECISION_APPLIED
PROJECT_SUMMARY
PROJECT_COMPLETED

OpenClaw mirroring is optional and uses the durable outbox with bounded retry and dead-letter. HTTP ingress failure spools a metadata-only file for polling; the configured local Git refs are also polled and changes missed by webhooks become idempotent events. Repeated blockers beyond the configured threshold enter DEAD_LETTER.

Install, verify, and rollback

Installation is versioned and does not overwrite an existing release:

sudo scripts/install_automation_quorum_controller.sh "$PWD" 20260718.1
sudo systemctl enable --now aios-automation-quorum-controller.timer

Run deterministic verification before enablement:

scripts/verify_automation_quorum_controller.sh "$PWD"

Rollback accepts only an existing non-symlink directory below /opt/aios-automation-quorum/releases. Repeating rollback to the current target is a no-op:

sudo scripts/rollback_automation_quorum_controller.sh \
  /opt/aios-automation-quorum/releases/20260718.0

Live acceptance matrix

Each row requires captured logs, timestamps, exit codes, committed OIDs, Gitleaks output, and where applicable live Gateway DB evidence. Static unit tests are necessary but cannot close the gate.

1. GPT + OpenClaw healthy, Codex stopped: two unattended cycles.
2. GPT + Codex healthy, OpenClaw stopped: two unattended cycles.
3. Both executors healthy: OpenClaw gets repetitive infrastructure; Codex gets complex code/security.
4. Active stops mid-task: standby continues from checkpoint with no duplicate effect.
5. Standby alone stops: active flow continues and no repair gate is created.
6. GPT stops: execution blocks even with two healthy executors.
7. GPT healthy and both executors stop: BLOCKED_NO_HEALTHY_EXECUTOR.
8. UNKNOWN is never counted as healthy.
9. HTTP ingress failure recovers through file polling.
10. Missed webhook recovers through Git ref polling.
11. Gateway restart restores events, leases, and outbox.
12. Repeated event ID does not create a second dispatch; changed payload dead-letters.
13. Repeated blocker beyond threshold dead-letters.
14. Workspace Agent failure falls back only with a live-validated Responses channel.
15. OpenClaw offline: Codex lifecycle and project summary remain complete in Gateway.
16. Codex offline: OpenClaw lifecycle and project summary remain complete in Gateway.

Until all rows pass against the live service, retain:

AUTOMATION_CLOSED=NOT_STARTED
BASELINE=V1R4C6
RELEASE=paused
PRODUCTION_RELEASE=NOT_STARTED