aios/docs/superpowers/specs/2026-08-18-worker-availability-runtime.md

AIOS Worker Availability Runtime

Priority

This runtime exists to make the already verified Online Agent execution path continuously usable without repeated human intervention.

The system already proved that generic, Hermes and OpenClaw workers can claim and complete real Online Tasks. The remaining availability defect was operational: evidence-freeze workflows left the worker services stopped, while registry ACTIVE records remained stale.

The availability contract therefore separates:

AUTO_CLAIM_MECHANISM=VERIFIED
CURRENT_WORKER_AVAILABILITY=<fresh runtime state>

A historical successful claim is not current liveness.

Source of truth for idle liveness

An idle worker may have no Task lease heartbeat. Therefore Task heartbeat alone cannot prove that the worker is continuously polling.

aios_online_agent_worker.py now writes:

<AIOS_WORKER_STATE_ROOT>/worker-liveness.json

on:

• process start: STARTING;
• every successful poll: POLL_OK;
• poll failure: POLL_FAILED;
• every successful Task lease heartbeat while executing: BUSY.

The file contains:

• agent ID/type/version;
• process PID;
• monotonically increasing local sequence;
• last_seen_epoch;
• poll/heartbeat intervals;
• current Task/stage while busy.

The liveness file is worker-owned observability state only. It does not write or mutate Task, Job, Git, WriterLease or registry databases.

Self-healing guard

aios-worker-availability-guard.timer invokes a root-owned oneshot guard every 30 seconds.

The guard manages:

aios-online-agent-worker-20260815.service
aios-hermes-online-agent-worker.service
aios-openclaw-online-agent-worker.service

and the required external runtime dependency:

aios-external-agent-executor.service

For each worker:

1. missing systemd unit → fail closed;
2. disabled → systemctl enable;
3. inactive → systemctl start;
4. active but liveness older than 90 seconds → restart;
5. repeated stale restart within 60 seconds → do not restart again; report degraded.

Long-running Tasks are safe because the Task lease-heartbeat thread refreshes BUSY liveness.

The guard never claims a Task and never touches Task/Job/Git databases.

Maintenance inhibit

Planned maintenance creates:

/run/aios-worker-availability/maintenance.lock

While this file exists the guard performs no start/restart actions and reports:

MAINTENANCE_INHIBITED

Maintenance tooling must remove the file when done. This prevents the self-healer from fighting intentional service changes.

Status

The guard writes:

/run/aios-worker-availability/status.json

with current worker/dependency states, liveness ages and recovery actions.

A worker is currently available only when:

systemd load_state=loaded
systemd active_state=active
unit enabled/static
worker-liveness.json present
last_seen_age <= 90s

Registry ACTIVE without fresh poll/busy liveness is not sufficient.

Boot/restart behavior

The timer is:

OnBootSec=15s
OnUnitActiveSec=30s
Persistent=true

Therefore reboot, manual stop left behind by another workflow, process crash, and hung poll loop are all repaired without requiring a human to notice and intervene.

Normal worker units retain their own Restart=always; the availability guard handles the cases that Restart=always cannot, especially a deliberate systemctl stop left after an evidence workflow.

Acceptance

The live acceptance requires all of the following:

guard timer active/enabled
external executor active/enabled
generic worker active/enabled + fresh liveness
Hermes worker active/enabled + fresh liveness
OpenClaw worker active/enabled + fresh liveness

Then a new read-only Online Task is projected through canonical Bridge 18892.

Success requires:

TASK_CREATED
→ generic worker auto claim
→ TASK_HEARTBEAT
→ RESULT_SUBMITTED
→ official TASK_VERIFIED(SUCCEEDED)

After the Task, the services must remain active and liveness must continue advancing. The acceptance also stops a worker intentionally once and proves the guard starts it again with a new fresh liveness sequence.

Only after these gates may current status be upgraded from:

OVERALL=PARTIAL

to:

PERSISTENT_WORKER_LIVENESS=VERIFIED
NEW_TASK_AUTO_CONSUMPTION=VERIFIED
WORKER_AVAILABILITY_RUNTIME=VERIFIED_CANDIDATE

Protected main and Production remain out of scope.

Forced source-capture projection compatibility

The deployed canonical Bridge currently projects a default source_capture envelope even when a governed binding request asks for binding-only behavior. This must not be allowed to make always-on workers consume engineering Tasks before their deterministic plan is registered.

The worker therefore distinguishes a real read-only source capture from an engineering placeholder:

source_capture
+ capability_scope contains governed.write.branch or governed.write.file
= PENDING_DETERMINISTIC_PLAN

For such a placeholder the worker:

1. re-resolves through the canonical plan registry without replaying the stale source-capture envelope;
2. if no deterministic plan exists, emits TASK_SKIPPED_PENDING_DETERMINISTIC_PLAN and does not claim;
3. if the registered canonical envelope is deterministic and the governed binding still matches, uses that deterministic envelope and claims normally.

A real read-only source-capture Task with only git.read/git.test still replays its exact persisted payload and remains claimable.

This makes persistent workers safe with the currently deployed Bridge while preserving the same Task/Job binding. No Task DB rewrite, cancellation, replacement Task, fake NEEDS_REWORK, or pre-claim metadata mutation is required.

A later Bridge cleanup may implement true binding-only projection, but Worker Availability does not depend on that cleanup.