Compare commits

...
Author SHA1 Message Date
AIOS Governed Continuation ad20eeb5c6 fix(agent-task): recover runtime projection after source capture 2026-08-20 10:17:11 +08:00
AIOS Governed Continuation 0ee456da94 feat(agent-task): add initial execution projection recovery 2026-08-20 10:00:44 +08:00
AIOS Governed Continuation d957a216a6 fix(release): separate immutable V2R2C2 lineage from engineering lineage 2026-08-19 23:00:07 +08:00
AIOS Governed Continuation b4b4416c93 feat(control-plane): add admission invariant repair core 2026-08-19 22:28:38 +08:00
AIOS Governed Continuation 17deb91ca7 fix(task-api): normalize release candidate capability 2026-08-19 18:06:41 +08:00
AIOS Governed Continuation 71e736b05f feat(release): add dedicated release preparation worker 2026-08-19 17:13:20 +08:00
AIOS Governed Continuation 5703ed7ae0 feat(governance): persist external release capability projection outbox 2026-08-19 14:21:05 +08:00
AIOS Governed Continuation 57218959b0 feat(governance): add durable Owner authorization projection 2026-08-19 13:49:45 +08:00
AIOS Governed Continuation 867d4a5173 fix(evidence): make all-modified-files scanner PEM-header precise 2026-08-19 11:18:23 +08:00
AIOS Governed Continuation 3a02a25d79 feat(gpt-edge): add owner-approved activation and rollback controller 2026-08-19 10:34:20 +08:00
AIOS Governed Continuation 69a51d50c7 feat(gpt-edge): add disabled-by-default TLS edge 2026-08-19 09:48:00 +08:00
AIOS Governed Continuation 7f12094e58 feat(online-task): add credential-isolated external GPT read-only ingress 2026-08-19 09:20:52 +08:00
AIOS Governed Continuation 2ae50a2a3d fix(model-governance): persist policy floor source of truth 2026-08-19 01:44:04 +08:00
AIOS Governed Continuation e9426dc0b8 fix(model-governance): reserve OpenClaw injected context 2026-08-18 20:02:52 +08:00
AIOS Governed Continuation d31267e89c fix(external-agent): scope request identity by task attempt 2026-08-18 17:01:38 +08:00
AIOS Governed Continuation f4022d3d1e feat(model-governance): enforce routed budgets and runtime policy 2026-08-18 13:18:20 +08:00
AIOS Governed Continuation 008bea3324 feat(online-agent): persist workers and defer engineering placeholders 2026-08-18 12:24:42 +08:00
AIOS Governed Continuation 52ef41e45f fix(online-agent): persist external runtime completion checkpoint phase 2026-08-18 08:46:05 +08:00
AIOS Governed Continuation 416e5e295d fix(online-agent): replay persisted external runtime envelope through canonical projection 2026-08-18 07:59:36 +08:00
AIOS Governed Continuation 4f81344509 fix(online-agent): normalize external runtime capabilities 2026-08-18 07:31:27 +08:00
AIOS Governed Continuation 8efdaea22f fix(online-agent): add provider-only Hermes/OpenClaw credential projection 2026-08-18 06:42:53 +08:00
AIOS Governed Continuation 3b48740b17 feat(online-agent): add Hermes and OpenClaw runtime workers 2026-08-18 05:13:52 +08:00
AIOS Governed Continuation 0c330afe03 fix(online-agent): preserve source-capture envelope during canonical replay 2026-08-18 04:16:54 +08:00
AIOS Governed Continuation 764bb2f08e feat(online-agent): add owner-controlled same-binding execution rebind 2026-08-18 03:45:27 +08:00
AIOS Governed Continuation 37814da581 feat(online-agent): generalize compact interface-map profiles 2026-08-18 03:27:02 +08:00
AIOS Governed Continuation 6de31e0f3e fix(task-plane): preserve fleet fairness through legacy oracle 2026-08-17 23:49:32 +08:00
AIOS Governed Continuation 51d7a356ca feat(task-plane): wire fleet scheduler into real pull dispatch 2026-08-17 22:04:21 +08:00
AIOS Governed Continuation 19ce458a1f feat(task-plane): add durable multi-agent fleet scheduler 2026-08-17 20:23:25 +08:00
AIOS Governed Continuation 12a0936e65 fix(online-agent): recover deterministic create-new retries after policy denial 2026-08-17 18:19:06 +08:00
AIOS Governed Continuation 68b95fea0d feat(online-agent): add durable checkpoint and retry-safe resume 2026-08-17 14:11:20 +08:00
AIOS Governed Continuation 8a07817910 feat(governance): bind trace tool and credential scopes 2026-08-17 12:36:52 +08:00
AIOS Governed Continuation a642b04577 feat(handoff): add durable scoped message fabric 2026-08-17 01:51:53 +08:00
AIOS Governed Continuation bb80f8cc2f feat(task-plane): enforce first-class scope isolation 2026-08-16 23:34:47 +08:00
AIOS Governed Continuation 0eb6e1a864 fix(gitops): integrate verified continuation lineage onto mainline candidate 2026-08-16 18:31:52 +08:00
148 changed files with 31885 additions and 125 deletions
+14 -7
View File
@@ -13,6 +13,7 @@ from hashlib import sha256
from pathlib import Path
import json
from typing import Any
from message_handoff import stable_event_id
def utc_now() -> str:
@@ -55,13 +56,19 @@ class AgentEventBus:
payload: dict[str, Any],
correlation_id: str | None = None,
) -> dict[str, Any]:
event_id = "evt_" + sha256(
json.dumps(
{"kind": kind, "source": source, "summary": summary, "payload": payload},
sort_keys=True,
ensure_ascii=False,
).encode("utf-8")
).hexdigest()[:16]
event_id = stable_event_id(
source=source, target=target, message_type=kind, payload=payload,
tenant_id=(payload.get('tenant_id') if isinstance(payload, dict) else None),
user_id=(payload.get('user_id') if isinstance(payload, dict) else None),
project_id=(payload.get('project_id') if isinstance(payload, dict) else None),
conversation_id=(payload.get('conversation_id') if isinstance(payload, dict) else None),
task_id=(payload.get('task_id') if isinstance(payload, dict) else None),
run_id=(payload.get('run_id') if isinstance(payload, dict) else None),
correlation_id=(payload.get('correlation_id') if isinstance(payload, dict) else None),
causation_id=(payload.get('causation_id') if isinstance(payload, dict) else None),
sequence=(payload.get('sequence') if isinstance(payload, dict) else None),
idempotency_key=(payload.get('idempotency_key') if isinstance(payload, dict) else None),
)
event = AgentEvent(event_id, kind, source, target, summary, payload, correlation_id=correlation_id)
self._write("pending", event.to_dict())
self._audit("created", event.to_dict())
@@ -7,7 +7,7 @@ class AgentEventObservabilityBridge:
def __init__(self, hooks: AgentObservabilityHooks | None = None):
self.hooks = hooks or AgentObservabilityHooks()
def consume(self, event: dict):
def consume(self, event: dict, *, actor=None, trace=None):
kind = event.get("kind")
payload = event.get("payload", {})
if kind == "agent_started":
@@ -16,6 +16,8 @@ class AgentEventObservabilityBridge:
visible_scope=payload.get("scope", "self"),
entry_agent=payload.get("agent", "unknown"),
execution_id=event.get("id"),
actor=actor,
trace=trace,
)
if kind == "agent_delegated":
return self.hooks.delegate(event["execution_id"], payload.get("agent", "unknown"))
+12
View File
@@ -31,6 +31,18 @@ class AgentExecutionRecord:
status: str = "completed"
started_at: str = field(default_factory=utc_now)
ended_at: str = field(default_factory=utc_now)
trace_id: str | None = None
span_id: str | None = None
parent_span_id: str | None = None
correlation_id: str | None = None
causation_id: str | None = None
tenant_id: str | None = None
project_id: str | None = None
conversation_id: str | None = None
task_id: str | None = None
claim_id: str | None = None
run_id: str | None = None
message_id: str | None = None
def to_dict(self):
return asdict(self)
+30 -6
View File
@@ -9,25 +9,49 @@ try:
except Exception:
APIRouter = None
from .agent_observability import AgentExecutionRecord
from .agent_observability import AgentExecutionRecord, AgentObservability
from .agent_observability_dashboard import AgentObservabilityDashboard
from .agent_observability_store import AgentObservabilityStore
from .trace_tool_credential_governance import GovernanceActor, TraceContext, authorize_scope
store = AgentObservabilityStore()
dashboard = AgentObservabilityDashboard(store.load())
def record_agent_execution(payload: dict[str, Any]):
record = AgentExecutionRecord(**payload)
def record_agent_execution(payload: dict[str, Any], *, actor: GovernanceActor | dict[str, Any] | None = None, trace: TraceContext | None = None):
scoped = any(payload.get(name) is not None for name in (
"trace_id", "span_id", "tenant_id", "project_id", "conversation_id", "task_id", "run_id"
))
values = dict(payload)
if scoped:
if actor is None or trace is None:
raise RuntimeError("TRACE_ACTOR_CONTEXT_REQUIRED")
trusted = GovernanceActor.from_value(actor)
authorize_scope(trusted, trace.scope)
bound = {
"trace_id": trace.trace_id, "span_id": trace.span_id,
"parent_span_id": trace.parent_span_id, "correlation_id": trace.correlation_id,
"causation_id": trace.causation_id, "tenant_id": trace.scope.tenant_id,
"user_id": trace.scope.user_id, "project_id": trace.scope.project_id,
"conversation_id": trace.scope.conversation_id, "task_id": trace.task_id,
"claim_id": trace.claim_id, "run_id": trace.scope.run_id, "message_id": trace.message_id,
}
for name, authoritative in bound.items():
if values.get(name) not in (None, authoritative):
raise RuntimeError("TRACE_SCOPE_MISMATCH_DENY:" + name)
values[name] = authoritative
record = AgentExecutionRecord(**values)
return {"status": "ok", "record": store.append(record)}
def get_agent_observability(user_id: str | None = None, group_id: str | None = None, role: str = "user"):
records = store.query(user_id, group_id, role)
def get_agent_observability(user_id: str | None = None, group_id: str | None = None, role: str = "user", *, actor: GovernanceActor | dict[str, Any] | None = None):
records = store.query(user_id, group_id, role, actor=actor)
visible = AgentObservability()
visible.records = [AgentExecutionRecord(**record) for record in records]
return {
"status": "ok",
"records": records,
"summary": store.load().summary(user_id if role != "owner" else None),
"summary": visible.summary(None),
}
@@ -14,6 +14,11 @@ from uuid import uuid4
from .agent_observability import AgentExecutionRecord
from .agent_observability_store import AgentObservabilityStore
from .trace_tool_credential_governance import (
GovernanceActor,
TraceContext,
authorize_scope,
)
def utc_now() -> str:
@@ -36,6 +41,7 @@ class AgentRunContext:
input_tokens: int = 0
output_tokens: int = 0
status: str = "running"
trace: TraceContext | None = None
def delegate(self, agent_name: str) -> None:
if agent_name not in self.path:
@@ -59,7 +65,19 @@ class AgentObservabilityHooks:
visible_scope: str,
entry_agent: str,
execution_id: str | None = None,
*,
actor: GovernanceActor | dict[str, Any] | None = None,
trace: TraceContext | None = None,
) -> dict[str, Any]:
if execution_id is not None and execution_id in self.active:
raise RuntimeError("OBSERVABILITY_EXECUTION_CONFLICT")
if trace is not None:
if actor is None:
raise RuntimeError("TRACE_ACTOR_CONTEXT_REQUIRED")
trusted_actor = GovernanceActor.from_value(actor)
authorize_scope(trusted_actor, trace.scope)
if trace.scope.user_id is not None and trace.scope.user_id != user_id:
raise RuntimeError("TRACE_SCOPE_MISMATCH_DENY:user_id")
run = AgentRunContext(
execution_id=execution_id or f"exec_{uuid4().hex[:12]}",
user_id=user_id,
@@ -67,6 +85,7 @@ class AgentObservabilityHooks:
entry_agent=entry_agent,
path=[entry_agent],
used_hermes=entry_agent.lower() == "hermes",
trace=trace,
)
self.active[run.execution_id] = run
return {"status": "started", "execution_id": run.execution_id, "run": run.__dict__}
@@ -106,6 +125,18 @@ class AgentObservabilityHooks:
status=status,
started_at=run.started_at,
ended_at=ended_at,
trace_id=run.trace.trace_id if run.trace else None,
span_id=run.trace.span_id if run.trace else None,
parent_span_id=run.trace.parent_span_id if run.trace else None,
correlation_id=run.trace.correlation_id if run.trace else None,
causation_id=run.trace.causation_id if run.trace else None,
tenant_id=run.trace.scope.tenant_id if run.trace else None,
project_id=run.trace.scope.project_id if run.trace else None,
conversation_id=run.trace.scope.conversation_id if run.trace else None,
task_id=run.trace.task_id if run.trace else None,
claim_id=run.trace.claim_id if run.trace else None,
run_id=run.trace.scope.run_id if run.trace else None,
message_id=run.trace.message_id if run.trace else None,
)
return {"status": "stored", "record": self.store.append(record)}
@@ -12,6 +12,12 @@ from pathlib import Path
from typing import Any
from .agent_observability import AgentExecutionRecord, AgentObservability
from .trace_tool_credential_governance import (
GovernanceActor,
GovernanceError,
GovernanceScope,
authorize_scope,
)
class AgentObservabilityStore:
@@ -23,6 +29,11 @@ class AgentObservabilityStore:
def append(self, record: AgentExecutionRecord) -> dict[str, Any]:
path = self.records_dir / f"{record.execution_id}.json"
data = record.to_dict()
if path.exists():
existing = json.loads(path.read_text(encoding="utf-8"))
if existing == data:
return existing
raise RuntimeError("OBSERVABILITY_EXECUTION_CONFLICT")
path.write_text(json.dumps(data, ensure_ascii=False, indent=2), encoding="utf-8")
return data
@@ -33,8 +44,36 @@ class AgentObservabilityStore:
observability.records.append(AgentExecutionRecord(**raw))
return observability
def query(self, user_id: str | None = None, group_id: str | None = None, role: str = "user") -> list[dict[str, Any]]:
def query(self, user_id: str | None = None, group_id: str | None = None, role: str = "user", *, actor: GovernanceActor | dict[str, Any] | None = None) -> list[dict[str, Any]]:
records = [record.to_dict() for record in self.load().records]
scoped = [record for record in records if record.get("tenant_id") is not None]
legacy = [record for record in records if record.get("tenant_id") is None]
if actor is not None:
trusted = GovernanceActor.from_value(actor)
visible: list[dict[str, Any]] = []
for record in scoped:
try:
authorize_scope(
trusted,
GovernanceScope(
"tenant",
tenant_id=record.get("tenant_id"),
user_id=record.get("user_id"),
project_id=record.get("project_id"),
conversation_id=record.get("conversation_id"),
run_id=record.get("run_id"),
),
)
except GovernanceError:
continue
visible.append(record)
if trusted.privileged:
visible.extend(legacy)
else:
visible.extend(record for record in legacy if record.get("user_id") == trusted.scope.user_id)
return visible
# Caller-supplied role remains a legacy compatibility path only.
records = legacy
if role == "owner":
return records
if role == "group_admin" and group_id:
@@ -0,0 +1,722 @@
"""Owner-controlled CAS-bound execution-envelope rebind for Online Agent Tasks.
This service exists for a narrow recovery case: a Task was projected/claimed with a read-only
source_capture envelope before its already-registered deterministic plan became authoritative.
It never creates Tasks/Jobs/branches and never accepts arbitrary metadata patches. It only replaces
metadata.execution + the derived execution_projection for the same governed binding, after an
official NEEDS_REWORK transition has returned the Task to QUEUED with no active lease.
"""
from __future__ import annotations
from dataclasses import dataclass
from datetime import datetime, timezone
import hashlib
import hmac
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
from pathlib import Path
import re
import sqlite3
from typing import Any, Callable, Mapping, Sequence
import urllib.error
import urllib.request
from aios_online_agent_execution import (
GovernedExecutionBinding,
validate_execution_envelope,
)
TASK_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{2,179}$")
KEY_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{7,159}$")
SHA_RE = re.compile(r"^[0-9a-f]{64}$")
MAX_BODY = 64 * 1024
AUTHORIZATION_ID_ENV = "AIOS_AGENT_TASK_EXECUTION_REBIND_AUTHORIZATION_ID"
class ExecutionRebindError(RuntimeError):
def __init__(self, code: str, status: int = 409) -> None:
super().__init__(code)
self.code = code
self.status = status
def _canonical(value: object) -> bytes:
return json.dumps(
value, sort_keys=True, separators=(",", ":"), ensure_ascii=False
).encode("utf-8")
def _sha(value: object) -> str:
return hashlib.sha256(_canonical(value)).hexdigest()
def _now() -> str:
return datetime.now(timezone.utc).isoformat()
def _require_string(
value: object,
code: str,
pattern: re.Pattern[str] | None = None,
) -> str:
if not isinstance(value, str) or not value.strip():
raise ExecutionRebindError(code, 400)
value = value.strip()
if pattern is not None and pattern.fullmatch(value) is None:
raise ExecutionRebindError(code, 400)
return value
def _required_authorization_id() -> str:
value = os.environ.get(AUTHORIZATION_ID_ENV, "").strip()
if not value:
raise RuntimeError("TASK_EXECUTION_REBIND_AUTHORIZATION_ID_MISSING")
if KEY_RE.fullmatch(value) is None:
raise RuntimeError("TASK_EXECUTION_REBIND_AUTHORIZATION_ID_INVALID")
return value
@dataclass(frozen=True)
class ExecutionRebindRequest:
expected_status: str
expected_revision: int
expected_attempt: int
previous_verification_sha256: str
expected_old_execution_digest: str
expected_new_execution_digest: str
authorization_id: str
idempotency_key: str
reason: str
@classmethod
def from_mapping(cls, value: Mapping[str, Any]) -> "ExecutionRebindRequest":
allowed = {
"expected_status",
"expected_revision",
"expected_attempt",
"previous_verification_sha256",
"expected_old_execution_digest",
"expected_new_execution_digest",
"authorization_id",
"idempotency_key",
"reason",
}
if set(value) != allowed:
raise ExecutionRebindError("EXECUTION_REBIND_FIELDS_INVALID", 400)
status = _require_string(value["expected_status"], "EXPECTED_STATUS_INVALID")
if status != "QUEUED":
raise ExecutionRebindError("EXPECTED_STATUS_INVALID", 400)
revision = value["expected_revision"]
attempt = value["expected_attempt"]
if (
not isinstance(revision, int)
or isinstance(revision, bool)
or revision < 0
or not isinstance(attempt, int)
or isinstance(attempt, bool)
or attempt < 1
):
raise ExecutionRebindError("EXECUTION_REBIND_INTEGER_INVALID", 400)
verification = _require_string(
value["previous_verification_sha256"], "VERIFICATION_SHA256_INVALID"
)
old_digest = _require_string(
value["expected_old_execution_digest"], "OLD_EXECUTION_DIGEST_INVALID"
)
new_digest = _require_string(
value["expected_new_execution_digest"], "NEW_EXECUTION_DIGEST_INVALID"
)
for digest, code in (
(verification, "VERIFICATION_SHA256_INVALID"),
(old_digest, "OLD_EXECUTION_DIGEST_INVALID"),
(new_digest, "NEW_EXECUTION_DIGEST_INVALID"),
):
if SHA_RE.fullmatch(digest) is None:
raise ExecutionRebindError(code, 400)
if old_digest == new_digest:
raise ExecutionRebindError("EXECUTION_REBIND_DIGEST_UNCHANGED", 400)
authorization = _require_string(
value["authorization_id"], "AUTHORIZATION_ID_INVALID", KEY_RE
)
key = _require_string(value["idempotency_key"], "IDEMPOTENCY_KEY_INVALID", KEY_RE)
reason = _require_string(value["reason"], "EXECUTION_REBIND_REASON_INVALID")
if len(reason) > 512:
raise ExecutionRebindError("EXECUTION_REBIND_REASON_INVALID", 400)
return cls(
status,
revision,
attempt,
verification,
old_digest,
new_digest,
authorization,
key,
reason,
)
def public(self) -> dict[str, Any]:
return {
"expected_status": self.expected_status,
"expected_revision": self.expected_revision,
"expected_attempt": self.expected_attempt,
"previous_verification_sha256": self.previous_verification_sha256,
"expected_old_execution_digest": self.expected_old_execution_digest,
"expected_new_execution_digest": self.expected_new_execution_digest,
"authorization_id": self.authorization_id,
"idempotency_key": self.idempotency_key,
"reason": self.reason,
}
class CanonicalBridgeExecutionResolver:
"""Resolve the already-registered plan from the canonical governed Bridge."""
def __init__(self, base_url: str, control_token: str) -> None:
self.base_url = base_url.rstrip("/")
self.control_token = control_token
def __call__(
self,
task_id: str,
required_capabilities: Sequence[str],
) -> dict[str, Any]:
body = json.dumps(
{
"task_id": task_id,
"required_capabilities": list(required_capabilities),
},
sort_keys=True,
separators=(",", ":"),
).encode("utf-8")
request = urllib.request.Request(
self.base_url + "/v2/execution-envelope",
method="POST",
data=body,
headers={
"Authorization": "Bearer " + self.control_token,
"Content-Type": "application/json",
"Accept": "application/json",
},
)
try:
with urllib.request.urlopen(request, timeout=15) as response:
raw = response.read()
status = response.status
except urllib.error.HTTPError as exc:
raw = exc.read()
status = exc.code
try:
value = json.loads(raw.decode("utf-8"))
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
raise ExecutionRebindError("CANONICAL_EXECUTION_RESPONSE_INVALID", 502) from exc
if not isinstance(value, Mapping):
raise ExecutionRebindError("CANONICAL_EXECUTION_RESPONSE_INVALID", 502)
if status != 200:
raise ExecutionRebindError(
str(value.get("error") or "CANONICAL_EXECUTION_RESOLUTION_FAILED"),
502 if status >= 500 else 409,
)
return dict(value)
class ExecutionRebindStore:
"""Official owner-controlled mutation with CAS, plan resolution and durable audit."""
def __init__(
self,
db_path: Path,
resolver: Callable[[str, Sequence[str]], Mapping[str, Any]],
) -> None:
self.db_path = Path(db_path)
self.resolver = resolver
def _connect(self) -> sqlite3.Connection:
connection = sqlite3.connect(self.db_path, timeout=30.0)
connection.row_factory = sqlite3.Row
connection.execute("PRAGMA foreign_keys=ON")
connection.execute("PRAGMA busy_timeout=30000")
return connection
@staticmethod
def _columns(
connection: sqlite3.Connection,
table: str,
) -> dict[str, sqlite3.Row]:
return {
str(row["name"]): row
for row in connection.execute(f'PRAGMA table_info("{table}")')
}
@staticmethod
def _json_value(raw: object, code: str) -> dict[str, Any]:
if isinstance(raw, Mapping):
return dict(raw)
if not isinstance(raw, str):
raise ExecutionRebindError(code, 409)
try:
value = json.loads(raw)
except json.JSONDecodeError as exc:
raise ExecutionRebindError(code, 409) from exc
if not isinstance(value, Mapping):
raise ExecutionRebindError(code, 409)
return dict(value)
@staticmethod
def _verification_hash(raw: object) -> str:
if raw in (None, ""):
value: object = {}
elif isinstance(raw, str):
try:
value = json.loads(raw)
except json.JSONDecodeError as exc:
raise ExecutionRebindError("TASK_VERIFICATION_INVALID", 409) from exc
else:
value = raw
return _sha(value)
@staticmethod
def _metadata_column(columns: Mapping[str, sqlite3.Row]) -> str:
for name in ("metadata_json", "metadata"):
if name in columns:
return name
raise ExecutionRebindError("TASK_METADATA_SCHEMA_UNSUPPORTED", 503)
@staticmethod
def _required_capabilities(
task: sqlite3.Row,
columns: Mapping[str, sqlite3.Row],
old_execution: Mapping[str, Any],
) -> list[str]:
for field in ("required_capabilities_json", "required_capabilities"):
if field not in columns:
continue
raw = task[field]
if raw in (None, ""):
continue
if isinstance(raw, str):
try:
value = json.loads(raw)
except json.JSONDecodeError:
value = [item for item in raw.split(",") if item.strip()]
else:
value = raw
if isinstance(value, list) and all(isinstance(item, str) for item in value):
return [item.strip() for item in value if item.strip()]
value = old_execution.get("capability_scope")
if isinstance(value, list) and all(isinstance(item, str) for item in value):
return [item for item in value if item]
raise ExecutionRebindError("TASK_CAPABILITIES_UNAVAILABLE", 503)
@staticmethod
def _event_table(
connection: sqlite3.Connection,
) -> tuple[str, dict[str, sqlite3.Row]]:
tables = {
row["name"]
for row in connection.execute(
"SELECT name FROM sqlite_master WHERE type='table'"
)
}
for name in ("task_events", "events"):
if name not in tables:
continue
columns = ExecutionRebindStore._columns(connection, name)
if "task_id" in columns and ("event_type" in columns or "type" in columns):
return name, columns
raise ExecutionRebindError("TASK_EVENT_TABLE_UNSUPPORTED", 503)
@staticmethod
def _insert_event(
connection: sqlite3.Connection,
task_id: str,
payload: Mapping[str, Any],
) -> None:
table, columns = ExecutionRebindStore._event_table(connection)
values: dict[str, Any] = {"task_id": task_id}
if "event_type" in columns:
values["event_type"] = "TASK_EXECUTION_REBIND_AUTHORIZED"
else:
values["type"] = "TASK_EXECUTION_REBIND_AUTHORIZED"
payload_text = json.dumps(payload, sort_keys=True, separators=(",", ":"))
if "payload_json" in columns:
values["payload_json"] = payload_text
elif "payload" in columns:
values["payload"] = payload_text
else:
raise ExecutionRebindError("TASK_EVENT_PAYLOAD_UNSUPPORTED", 503)
if "created_at" in columns:
values["created_at"] = _now()
if "agent_id" in columns:
values["agent_id"] = "owner-control"
if "actor_id" in columns:
values["actor_id"] = "owner-control"
names = list(values)
quoted = ",".join('"' + name + '"' for name in names)
placeholders = ",".join("?" for _ in names)
connection.execute(
f'INSERT INTO "{table}" ({quoted}) VALUES ({placeholders})',
tuple(values[name] for name in names),
)
@staticmethod
def _ensure_rebind_table(connection: sqlite3.Connection) -> None:
connection.execute(
"""
CREATE TABLE IF NOT EXISTS task_execution_rebinds (
task_id TEXT NOT NULL,
idempotency_key TEXT NOT NULL,
request_sha256 TEXT NOT NULL,
authorization_id TEXT NOT NULL,
previous_verification_sha256 TEXT NOT NULL,
old_execution_digest TEXT NOT NULL,
new_execution_digest TEXT NOT NULL,
expected_revision INTEGER NOT NULL,
applied_revision INTEGER NOT NULL,
attempt INTEGER NOT NULL,
reason TEXT NOT NULL,
created_at TEXT NOT NULL,
PRIMARY KEY(task_id,idempotency_key)
)
"""
)
def _load_and_validate(
self,
connection: sqlite3.Connection,
task_id: str,
request: ExecutionRebindRequest,
) -> tuple[sqlite3.Row, dict[str, sqlite3.Row], str, dict[str, Any], dict[str, Any]]:
columns = self._columns(connection, "tasks")
required = {
"task_id",
"status",
"attempt",
"revision",
"claimed_by",
"lease_token_hash",
"lease_expires_at",
"verification_json",
}
if not required.issubset(columns):
raise ExecutionRebindError("TASK_SCHEMA_EXECUTION_REBIND_UNSUPPORTED", 503)
metadata_column = self._metadata_column(columns)
task = connection.execute(
"SELECT * FROM tasks WHERE task_id=?",
(task_id,),
).fetchone()
if task is None:
raise ExecutionRebindError("TASK_NOT_FOUND", 404)
if (
str(task["status"]) != request.expected_status
or int(task["revision"]) != request.expected_revision
or int(task["attempt"]) != request.expected_attempt
):
raise ExecutionRebindError("EXECUTION_REBIND_CAS_MISMATCH", 409)
if task["lease_token_hash"] not in (None, "") or task["lease_expires_at"] not in (None, ""):
raise ExecutionRebindError("EXECUTION_REBIND_ACTIVE_LEASE_DENY", 409)
verification_sha = self._verification_hash(task["verification_json"])
if verification_sha != request.previous_verification_sha256:
raise ExecutionRebindError("EXECUTION_REBIND_VERIFICATION_DRIFT", 409)
metadata = self._json_value(
task[metadata_column],
"TASK_METADATA_INVALID",
)
execution = metadata.get("execution")
if not isinstance(execution, Mapping):
raise ExecutionRebindError("TASK_EXECUTION_METADATA_MISSING", 409)
old_execution = validate_execution_envelope(dict(execution))
if old_execution["execution_digest"] != request.expected_old_execution_digest:
raise ExecutionRebindError("EXECUTION_REBIND_OLD_DIGEST_DRIFT", 409)
if old_execution.get("action") != "source_capture":
raise ExecutionRebindError("EXECUTION_REBIND_SOURCE_CAPTURE_REQUIRED", 409)
projection = metadata.get("execution_projection")
if isinstance(projection, Mapping):
projected_digest = str(projection.get("execution_digest") or "")
if projected_digest and projected_digest != request.expected_old_execution_digest:
raise ExecutionRebindError("EXECUTION_REBIND_PROJECTION_DRIFT", 409)
return task, columns, metadata_column, metadata, old_execution
def rebind(
self,
task_id: str,
request: ExecutionRebindRequest,
) -> dict[str, Any]:
task_id = _require_string(task_id, "TASK_ID_INVALID", TASK_RE)
required_authorization = _required_authorization_id()
if request.authorization_id != required_authorization:
raise ExecutionRebindError("EXECUTION_REBIND_AUTHORIZATION_ID_MISMATCH", 403)
request_sha = _sha(request.public())
# Idempotent replay must win before CAS validation because a successful
# rebind necessarily increments the Task revision and changes the persisted envelope.
with self._connect() as connection:
self._ensure_rebind_table(connection)
replay = connection.execute(
"SELECT * FROM task_execution_rebinds WHERE task_id=? AND idempotency_key=?",
(task_id, request.idempotency_key),
).fetchone()
if replay is not None:
if replay["request_sha256"] != request_sha:
raise ExecutionRebindError("EXECUTION_REBIND_IDEMPOTENCY_CONFLICT", 409)
connection.commit()
return {
"task_id": task_id,
"status": "QUEUED",
"attempt": int(replay["attempt"]),
"revision": int(replay["applied_revision"]),
"old_execution_digest": replay["old_execution_digest"],
"new_execution_digest": replay["new_execution_digest"],
"authorization_id": replay["authorization_id"],
"idempotent": True,
}
# Read/CAS snapshot first. Canonical Bridge resolution happens outside the DB write lock.
with self._connect() as connection:
task, columns, _, _, old_execution = self._load_and_validate(
connection, task_id, request
)
capabilities = self._required_capabilities(task, columns, old_execution)
binding = GovernedExecutionBinding.from_mapping(old_execution)
resolved = dict(self.resolver(task_id, capabilities))
new_execution = validate_execution_envelope(resolved, binding=binding)
if new_execution.get("action") != "deterministic":
raise ExecutionRebindError("EXECUTION_REBIND_DETERMINISTIC_PLAN_REQUIRED", 409)
if new_execution["execution_digest"] != request.expected_new_execution_digest:
raise ExecutionRebindError("EXECUTION_REBIND_NEW_DIGEST_DRIFT", 409)
if new_execution["execution_digest"] == old_execution["execution_digest"]:
raise ExecutionRebindError("EXECUTION_REBIND_DIGEST_UNCHANGED", 409)
# Re-read under BEGIN IMMEDIATE and apply only if the exact snapshot is unchanged.
with self._connect() as connection:
connection.execute("BEGIN IMMEDIATE")
self._ensure_rebind_table(connection)
replay = connection.execute(
"SELECT * FROM task_execution_rebinds WHERE task_id=? AND idempotency_key=?",
(task_id, request.idempotency_key),
).fetchone()
if replay is not None:
if replay["request_sha256"] != request_sha:
raise ExecutionRebindError("EXECUTION_REBIND_IDEMPOTENCY_CONFLICT", 409)
connection.commit()
return {
"task_id": task_id,
"status": "QUEUED",
"attempt": int(replay["attempt"]),
"revision": int(replay["applied_revision"]),
"old_execution_digest": replay["old_execution_digest"],
"new_execution_digest": replay["new_execution_digest"],
"authorization_id": replay["authorization_id"],
"idempotent": True,
}
task, columns, metadata_column, metadata, current_old = self._load_and_validate(
connection, task_id, request
)
if current_old["execution_digest"] != old_execution["execution_digest"]:
raise ExecutionRebindError("EXECUTION_REBIND_OLD_DIGEST_DRIFT", 409)
new_metadata = dict(metadata)
new_metadata["execution"] = new_execution
new_metadata["execution_projection"] = {
"schema": "aios.online-agent.task-projection.v1",
"task_id": binding.task_id,
"job_id": binding.job_id,
"execution_digest": new_execution["execution_digest"],
}
applied_revision = request.expected_revision + 1
previous_claimed_by = task["claimed_by"]
cursor = connection.execute(
f"""
UPDATE tasks
SET "{metadata_column}"=?, revision=?, claimed_by=NULL,
lease_token_hash=NULL, lease_expires_at=NULL
WHERE task_id=? AND status=? AND revision=? AND attempt=?
AND claimed_by IS ? AND lease_token_hash IS NULL AND lease_expires_at IS NULL
""",
(
json.dumps(new_metadata, sort_keys=True, separators=(",", ":")),
applied_revision,
task_id,
request.expected_status,
request.expected_revision,
request.expected_attempt,
previous_claimed_by,
),
)
if cursor.rowcount != 1:
raise ExecutionRebindError("EXECUTION_REBIND_CAS_MISMATCH", 409)
event_payload = {
"schema": "aios.task-execution-rebind.event.v1",
"task_id": task_id,
"job_id": binding.job_id,
"authorization_id": request.authorization_id,
"idempotency_key": request.idempotency_key,
"reason": request.reason,
"previous_verification_sha256": request.previous_verification_sha256,
"old_execution_digest": old_execution["execution_digest"],
"new_execution_digest": new_execution["execution_digest"],
"old_action": old_execution.get("action"),
"new_action": new_execution.get("action"),
"previous_revision": request.expected_revision,
"applied_revision": applied_revision,
"attempt": request.expected_attempt,
"request_sha256": request_sha,
"previous_claimed_by": previous_claimed_by,
"stale_claim_normalized": previous_claimed_by not in (None, ""),
}
self._insert_event(connection, task_id, event_payload)
connection.execute(
"""
INSERT INTO task_execution_rebinds(
task_id,idempotency_key,request_sha256,authorization_id,
previous_verification_sha256,old_execution_digest,new_execution_digest,
expected_revision,applied_revision,attempt,reason,created_at
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?)
""",
(
task_id,
request.idempotency_key,
request_sha,
request.authorization_id,
request.previous_verification_sha256,
old_execution["execution_digest"],
new_execution["execution_digest"],
request.expected_revision,
applied_revision,
request.expected_attempt,
request.reason,
_now(),
),
)
connection.commit()
return {
"task_id": task_id,
"status": "QUEUED",
"attempt": request.expected_attempt,
"revision": applied_revision,
"old_execution_digest": old_execution["execution_digest"],
"new_execution_digest": new_execution["execution_digest"],
"authorization_id": request.authorization_id,
"idempotent": False,
}
class ExecutionRebindHandler(BaseHTTPRequestHandler):
store: ExecutionRebindStore | None = None
control_token: str = ""
def log_message(self, *_args: Any) -> None:
return
def _json(self, value: Mapping[str, Any], status: int) -> None:
data = json.dumps(
dict(value), sort_keys=True, separators=(",", ":")
).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def _authorized(self) -> bool:
header = self.headers.get("Authorization", "")
return (
header.startswith("Bearer ")
and bool(self.control_token)
and hmac.compare_digest(header[7:], self.control_token)
)
def do_GET(self) -> None:
if self.path == "/health":
self._json(
{"status": "ok", "service": "aios-agent-task-execution-rebind"},
200,
)
else:
self._json({"error": "NOT_FOUND"}, 404)
def do_POST(self) -> None:
match = re.fullmatch(r"/v1/tasks/([^/]+)/execution-rebind", self.path)
if match is None:
self._json({"error": "NOT_FOUND"}, 404)
return
if not self._authorized():
self._json({"error": "UNAUTHORIZED"}, 401)
return
try:
length = int(self.headers.get("Content-Length", "0"))
except ValueError:
self._json({"error": "CONTENT_LENGTH_INVALID"}, 400)
return
if length < 1 or length > MAX_BODY:
self._json({"error": "REQUEST_SIZE_INVALID"}, 413)
return
try:
value = json.loads(self.rfile.read(length).decode("utf-8"))
if not isinstance(value, Mapping):
raise ValueError
request = ExecutionRebindRequest.from_mapping(value)
if self.store is None:
raise ExecutionRebindError("RUNTIME_UNAVAILABLE", 503)
result = self.store.rebind(match.group(1), request)
self._json(result, 200 if result["idempotent"] else 201)
except ExecutionRebindError as exc:
self._json({"error": exc.code}, exc.status)
except (UnicodeDecodeError, json.JSONDecodeError, ValueError):
self._json({"error": "REQUEST_INVALID"}, 400)
def load_control_token(path: Path) -> str:
if path.is_symlink() or not path.is_file():
raise RuntimeError("TASK_EXECUTION_REBIND_CONTROL_CREDENTIAL_UNAVAILABLE")
value = path.read_text(encoding="utf-8").strip()
if len(value) < 32:
raise RuntimeError("TASK_EXECUTION_REBIND_CONTROL_CREDENTIAL_INVALID")
return value
def serve() -> None:
db = Path(os.environ.get("AIOS_AGENT_TASK_DB", "/var/lib/aios-agent-task/state.sqlite3"))
_required_authorization_id()
token_path = Path(
os.environ.get(
"AIOS_AGENT_TASK_CONTROL_TOKEN_FILE",
str(
Path(os.environ.get("CREDENTIALS_DIRECTORY", "/run/credentials"))
/ "agent_task_control_token"
),
)
)
token = load_control_token(token_path)
bridge = os.environ.get(
"AIOS_GOVERNED_WRITE_API_URL",
"http://127.0.0.1:18892",
)
host = os.environ.get("AIOS_TASK_EXECUTION_REBIND_HOST", "127.0.0.1")
port = int(os.environ.get("AIOS_TASK_EXECUTION_REBIND_PORT", "18942"))
ExecutionRebindHandler.store = ExecutionRebindStore(
db,
CanonicalBridgeExecutionResolver(bridge, token),
)
ExecutionRebindHandler.control_token = token
ThreadingHTTPServer((host, port), ExecutionRebindHandler).serve_forever()
if __name__ == "__main__":
serve()
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,427 @@
"""Owner-controlled, CAS-bound retry extension for exhausted Online Agent Tasks."""
from __future__ import annotations
from dataclasses import dataclass
from datetime import datetime, timezone
import hashlib
import hmac
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
from pathlib import Path
import re
import sqlite3
from typing import Any, Mapping
TASK_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{2,179}$")
KEY_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{7,159}$")
MAX_BODY = 64 * 1024
AUTHORIZE_EXTENSION_ID_ENV = "AIOS_AGENT_TASK_AUTHORIZATION_ID"
class RetryExtensionError(RuntimeError):
def __init__(self, code: str, status: int = 409) -> None:
super().__init__(code)
self.code = code
self.status = status
def _canonical(value: object) -> bytes:
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
def _sha(value: object) -> str:
return hashlib.sha256(_canonical(value)).hexdigest()
def _required_authorization_id() -> str:
value = os.environ.get(AUTHORIZE_EXTENSION_ID_ENV, "").strip()
if not value:
raise RuntimeError("TASK_RETRY_AUTHORIZATION_ID_MISSING")
if KEY_RE.fullmatch(value) is None:
raise RuntimeError("TASK_RETRY_AUTHORIZATION_ID_INVALID")
return value
def _now() -> str:
return datetime.now(timezone.utc).isoformat()
def _require_string(value: object, code: str, pattern: re.Pattern[str] | None = None) -> str:
if not isinstance(value, str) or not value.strip():
raise RetryExtensionError(code, 400)
value = value.strip()
if pattern is not None and pattern.fullmatch(value) is None:
raise RetryExtensionError(code, 400)
return value
@dataclass(frozen=True)
class RetryExtensionRequest:
expected_status: str
expected_revision: int
expected_attempt: int
expected_max_attempts: int
next_max_attempts: int
previous_verification_sha256: str
authorization_id: str
idempotency_key: str
@classmethod
def from_mapping(cls, value: Mapping[str, Any]) -> "RetryExtensionRequest":
allowed = {
"expected_status", "expected_revision", "expected_attempt", "expected_max_attempts",
"next_max_attempts", "previous_verification_sha256", "authorization_id", "idempotency_key",
}
if set(value) != allowed:
raise RetryExtensionError("RETRY_EXTENSION_FIELDS_INVALID", 400)
status = _require_string(value["expected_status"], "EXPECTED_STATUS_INVALID")
if status != "FAILED":
raise RetryExtensionError("EXPECTED_STATUS_INVALID", 400)
ints: dict[str, int] = {}
for field in ("expected_revision", "expected_attempt", "expected_max_attempts", "next_max_attempts"):
item = value[field]
if not isinstance(item, int) or isinstance(item, bool) or item < 0:
raise RetryExtensionError("RETRY_EXTENSION_INTEGER_INVALID", 400)
ints[field] = item
if ints["expected_attempt"] != ints["expected_max_attempts"]:
raise RetryExtensionError("RETRY_EXTENSION_NOT_EXHAUSTED", 409)
if ints["next_max_attempts"] != ints["expected_max_attempts"] + 1:
raise RetryExtensionError("RETRY_EXTENSION_MUST_BE_BOUNDED_ONE_STEP", 400)
if ints["next_max_attempts"] > 50:
raise RetryExtensionError("RETRY_EXTENSION_LIMIT_EXCEEDED", 400)
verification = _require_string(value["previous_verification_sha256"], "VERIFICATION_SHA256_INVALID")
if re.fullmatch(r"[0-9a-f]{64}", verification) is None:
raise RetryExtensionError("VERIFICATION_SHA256_INVALID", 400)
authorization = _require_string(value["authorization_id"], "AUTHORIZATION_ID_INVALID", KEY_RE)
key = _require_string(value["idempotency_key"], "IDEMPOTENCY_KEY_INVALID", KEY_RE)
return cls(
status,
ints["expected_revision"],
ints["expected_attempt"],
ints["expected_max_attempts"],
ints["next_max_attempts"],
verification,
authorization,
key,
)
def public(self) -> dict[str, Any]:
return {
"expected_status": self.expected_status,
"expected_revision": self.expected_revision,
"expected_attempt": self.expected_attempt,
"expected_max_attempts": self.expected_max_attempts,
"next_max_attempts": self.next_max_attempts,
"previous_verification_sha256": self.previous_verification_sha256,
"authorization_id": self.authorization_id,
"idempotency_key": self.idempotency_key,
}
class RetryExtensionStore:
"""Owns the official mutation. Direct/manual DB edits remain forbidden."""
def __init__(self, db_path: Path) -> None:
self.db_path = Path(db_path)
def _connect(self) -> sqlite3.Connection:
connection = sqlite3.connect(self.db_path, timeout=30.0)
connection.row_factory = sqlite3.Row
connection.execute("PRAGMA foreign_keys=ON")
connection.execute("PRAGMA busy_timeout=30000")
return connection
@staticmethod
def _columns(connection: sqlite3.Connection, table: str) -> dict[str, sqlite3.Row]:
return {str(row["name"]): row for row in connection.execute(f'PRAGMA table_info("{table}")')}
@staticmethod
def _event_table(connection: sqlite3.Connection) -> tuple[str, dict[str, sqlite3.Row]]:
tables = {
row["name"]
for row in connection.execute("SELECT name FROM sqlite_master WHERE type='table'")
}
for name in ("task_events", "events"):
if name not in tables:
continue
columns = RetryExtensionStore._columns(connection, name)
if "task_id" in columns and ("event_type" in columns or "type" in columns):
return name, columns
raise RetryExtensionError("TASK_EVENT_TABLE_UNSUPPORTED", 503)
@staticmethod
def _verification_hash(raw: object) -> str:
if raw in (None, ""):
value: object = {}
elif isinstance(raw, str):
try:
value = json.loads(raw)
except json.JSONDecodeError as exc:
raise RetryExtensionError("TASK_VERIFICATION_INVALID", 409) from exc
else:
value = raw
return _sha(value)
@staticmethod
def _ensure_extension_table(connection: sqlite3.Connection) -> None:
connection.execute(
"""
CREATE TABLE IF NOT EXISTS task_retry_extensions (
task_id TEXT NOT NULL,
idempotency_key TEXT NOT NULL,
request_sha256 TEXT NOT NULL,
authorization_id TEXT NOT NULL,
previous_verification_sha256 TEXT NOT NULL,
previous_max_attempts INTEGER NOT NULL,
next_max_attempts INTEGER NOT NULL,
expected_revision INTEGER NOT NULL,
applied_revision INTEGER NOT NULL,
created_at TEXT NOT NULL,
PRIMARY KEY(task_id, idempotency_key)
)
"""
)
@staticmethod
def _insert_event(
connection: sqlite3.Connection,
task_id: str,
payload: Mapping[str, Any],
) -> None:
table, columns = RetryExtensionStore._event_table(connection)
values: dict[str, Any] = {"task_id": task_id}
if "event_type" in columns:
values["event_type"] = "TASK_RETRY_EXTENSION_AUTHORIZED"
else:
values["type"] = "TASK_RETRY_EXTENSION_AUTHORIZED"
payload_text = json.dumps(payload, sort_keys=True, separators=(",", ":"))
if "payload_json" in columns:
values["payload_json"] = payload_text
elif "payload" in columns:
values["payload"] = payload_text
else:
raise RetryExtensionError("TASK_EVENT_PAYLOAD_UNSUPPORTED", 503)
if "created_at" in columns:
values["created_at"] = _now()
if "agent_id" in columns:
values["agent_id"] = "owner-control"
if "actor_id" in columns:
values["actor_id"] = "owner-control"
names = list(values)
sql = f'INSERT INTO "{table}" ({",".join(f"""\"{name}\"""" for name in names)}) VALUES ({",".join("?" for _ in names)})'
connection.execute(sql, tuple(values[name] for name in names))
def extend(self, task_id: str, request: RetryExtensionRequest) -> dict[str, Any]:
task_id = _require_string(task_id, "TASK_ID_INVALID", TASK_RE)
required_authorization_id = _required_authorization_id()
if request.authorization_id != required_authorization_id:
raise RetryExtensionError("RETRY_EXTENSION_AUTHORIZATION_ID_MISMATCH", 403)
request_payload = request.public()
request_sha = _sha(request_payload)
with self._connect() as connection:
connection.execute("BEGIN IMMEDIATE")
self._ensure_extension_table(connection)
replay = connection.execute(
"SELECT * FROM task_retry_extensions WHERE task_id=? AND idempotency_key=?",
(task_id, request.idempotency_key),
).fetchone()
if replay is not None:
if replay["request_sha256"] != request_sha:
raise RetryExtensionError("RETRY_EXTENSION_IDEMPOTENCY_CONFLICT", 409)
connection.commit()
return {
"task_id": task_id,
"status": "QUEUED",
"attempt": request.expected_attempt,
"max_attempts": int(replay["next_max_attempts"]),
"revision": int(replay["applied_revision"]),
"idempotent": True,
"authorization_id": replay["authorization_id"],
"previous_verification_sha256": replay["previous_verification_sha256"],
}
columns = self._columns(connection, "tasks")
required = {
"task_id", "status", "attempt", "max_attempts", "revision", "claimed_by",
"lease_token_hash", "lease_expires_at", "verification_json",
}
if not required.issubset(columns):
raise RetryExtensionError("TASK_SCHEMA_RETRY_EXTENSION_UNSUPPORTED", 503)
task = connection.execute("SELECT * FROM tasks WHERE task_id=?", (task_id,)).fetchone()
if task is None:
raise RetryExtensionError("TASK_NOT_FOUND", 404)
actual_verification_sha = self._verification_hash(task["verification_json"])
if actual_verification_sha != request.previous_verification_sha256:
raise RetryExtensionError("RETRY_EXTENSION_VERIFICATION_DRIFT", 409)
if (
task["status"] != request.expected_status
or int(task["revision"]) != request.expected_revision
or int(task["attempt"]) != request.expected_attempt
or int(task["max_attempts"]) != request.expected_max_attempts
):
raise RetryExtensionError("RETRY_EXTENSION_CAS_MISMATCH", 409)
for lease_field in ("lease_token_hash", "lease_expires_at"):
if lease_field in columns and task[lease_field] not in (None, ""):
raise RetryExtensionError("RETRY_EXTENSION_ACTIVE_LEASE_DENY", 409)
previous_claimed_by = task["claimed_by"]
applied_revision = request.expected_revision + 1
cursor = connection.execute(
"""
UPDATE tasks
SET status='QUEUED', max_attempts=?, revision=?, claimed_by=NULL,
lease_token_hash=NULL, lease_expires_at=NULL
WHERE task_id=? AND status=? AND revision=? AND attempt=? AND max_attempts=?
AND claimed_by IS ?
AND lease_token_hash IS NULL AND lease_expires_at IS NULL
""",
(
request.next_max_attempts,
applied_revision,
task_id,
request.expected_status,
request.expected_revision,
request.expected_attempt,
request.expected_max_attempts,
previous_claimed_by,
),
)
if cursor.rowcount != 1:
raise RetryExtensionError("RETRY_EXTENSION_CAS_MISMATCH", 409)
event_payload = {
"schema": "aios.task-retry-extension.event.v1",
"task_id": task_id,
"authorization_id": request.authorization_id,
"idempotency_key": request.idempotency_key,
"previous_verification_sha256": request.previous_verification_sha256,
"previous_max_attempts": request.expected_max_attempts,
"next_max_attempts": request.next_max_attempts,
"previous_revision": request.expected_revision,
"applied_revision": applied_revision,
"request_sha256": request_sha,
"previous_claimed_by": previous_claimed_by,
"stale_terminal_claim_normalized": previous_claimed_by not in (None, ""),
}
self._insert_event(connection, task_id, event_payload)
connection.execute(
"""
INSERT INTO task_retry_extensions(
task_id,idempotency_key,request_sha256,authorization_id,previous_verification_sha256,
previous_max_attempts,next_max_attempts,expected_revision,applied_revision,created_at
) VALUES (?,?,?,?,?,?,?,?,?,?)
""",
(
task_id,
request.idempotency_key,
request_sha,
request.authorization_id,
request.previous_verification_sha256,
request.expected_max_attempts,
request.next_max_attempts,
request.expected_revision,
applied_revision,
_now(),
),
)
connection.commit()
return {
"task_id": task_id,
"status": "QUEUED",
"attempt": request.expected_attempt,
"max_attempts": request.next_max_attempts,
"revision": applied_revision,
"idempotent": False,
"authorization_id": request.authorization_id,
"previous_verification_sha256": request.previous_verification_sha256,
}
class RetryExtensionHandler(BaseHTTPRequestHandler):
store: RetryExtensionStore | None = None
control_token: str = ""
def log_message(self, *_args: Any) -> None:
return
def _json(self, value: Mapping[str, Any], status: int) -> None:
data = json.dumps(dict(value), sort_keys=True, separators=(",", ":")).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def _authorized(self) -> bool:
header = self.headers.get("Authorization", "")
return (
header.startswith("Bearer ")
and bool(self.control_token)
and hmac.compare_digest(header[7:], self.control_token)
)
def do_GET(self) -> None:
if self.path == "/health":
self._json({"status": "ok", "service": "aios-agent-task-retry-extension"}, 200)
else:
self._json({"error": "NOT_FOUND"}, 404)
def do_POST(self) -> None:
match = re.fullmatch(r"/v1/tasks/([^/]+)/retry-extension", self.path)
if match is None:
self._json({"error": "NOT_FOUND"}, 404)
return
if not self._authorized():
self._json({"error": "UNAUTHORIZED"}, 401)
return
try:
length = int(self.headers.get("Content-Length", "0"))
except ValueError:
self._json({"error": "CONTENT_LENGTH_INVALID"}, 400)
return
if length < 1 or length > MAX_BODY:
self._json({"error": "REQUEST_SIZE_INVALID"}, 413)
return
try:
value = json.loads(self.rfile.read(length).decode("utf-8"))
if not isinstance(value, Mapping):
raise ValueError
request = RetryExtensionRequest.from_mapping(value)
if self.store is None:
raise RetryExtensionError("RUNTIME_UNAVAILABLE", 503)
result = self.store.extend(match.group(1), request)
self._json(result, 200 if result["idempotent"] else 201)
except RetryExtensionError as exc:
self._json({"error": exc.code}, exc.status)
except (UnicodeDecodeError, json.JSONDecodeError, ValueError):
self._json({"error": "REQUEST_INVALID"}, 400)
def load_control_token(path: Path) -> str:
if path.is_symlink() or not path.is_file():
raise RuntimeError("TASK_RETRY_CONTROL_CREDENTIAL_UNAVAILABLE")
value = path.read_text(encoding="utf-8").strip()
if len(value) < 32:
raise RuntimeError("TASK_RETRY_CONTROL_CREDENTIAL_INVALID")
return value
def serve() -> None:
db = Path(os.environ.get("AIOS_AGENT_TASK_DB", "/var/lib/aios-agent-task/state.sqlite3"))
_required_authorization_id()
token_path = Path(
os.environ.get(
"AIOS_AGENT_TASK_CONTROL_TOKEN_FILE",
str(Path(os.environ.get("CREDENTIALS_DIRECTORY", "/run/credentials")) / "agent_task_control_token"),
)
)
host = os.environ.get("AIOS_TASK_RETRY_HOST", "127.0.0.1")
port = int(os.environ.get("AIOS_TASK_RETRY_PORT", "18941"))
RetryExtensionHandler.store = RetryExtensionStore(db)
RetryExtensionHandler.control_token = load_control_token(token_path)
ThreadingHTTPServer((host, port), RetryExtensionHandler).serve_forever()
if __name__ == "__main__":
serve()
@@ -0,0 +1,470 @@
#!/usr/bin/env python3
"""Canonical AIOS all-modified-files evidence packager.
The bundle contract is intentionally compatible with the historical
aios_package_all_modified_files_v1.sh output while keeping secret scanning in
version-controlled, tested code.
"""
from __future__ import annotations
from datetime import datetime, timezone
import argparse
import hashlib
import json
import os
from pathlib import Path
import re
import subprocess
import tempfile
import zipfile
from typing import Iterable
OID_RE = re.compile(r"^[0-9a-f]{40}$")
TASK_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{2,179}$")
# A PEM private-key finding requires the complete PEM header. Source code,
# tests, docs, or error text containing only "BEGIN PRIVATE KEY" are not
# credentials and must not be rejected.
PEM_PRIVATE_KEY_RE = re.compile(
rb"-----BEGIN (?:RSA |EC |DSA |OPENSSH )?PRIVATE KEY-----",
re.IGNORECASE,
)
BEARER_RE = re.compile(
rb"authorization\s*:\s*bearer\s+[A-Za-z0-9._~+/=-]{20,}",
re.IGNORECASE,
)
KEY_VALUE_SECRET_RE = re.compile(
rb"(?:password|passwd|secret|api[_-]?key)\s*[:=]\s*[^\s,;]{12,}",
re.IGNORECASE,
)
DOCUMENT_SUFFIXES = {".md", ".rst", ".adoc", ".txt"}
DOCUMENT_PREFIXES = (
"README",
"CHANGELOG",
"RUNBOOK",
"ADR",
"ARCHITECTURE",
"DEPLOYMENT",
"TROUBLESHOOTING",
)
class PackagerError(RuntimeError):
pass
def sha256(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def secret_finding(data: bytes) -> str | None:
if PEM_PRIVATE_KEY_RE.search(data):
return "PEM_PRIVATE_KEY"
if BEARER_RE.search(data):
return "AUTHORIZATION_BEARER"
if KEY_VALUE_SECRET_RE.search(data):
return "KEY_VALUE_SECRET"
return None
def scan_secret(data: bytes, label: str) -> None:
finding = secret_finding(data)
if finding is not None:
raise PackagerError(
f"MODIFIED_FILES_SECRET_SCAN_MATCH:{finding}:{label}"
)
def git_bytes(git_dir: Path, oid: str, path: str) -> bytes:
try:
return subprocess.check_output(
[
"/usr/bin/git",
f"--git-dir={git_dir}",
"show",
f"{oid}:{path}",
],
timeout=30,
)
except subprocess.CalledProcessError as exc:
raise PackagerError(
"MODIFIED_FILES_GIT_SHOW_FAILED:" + path
) from exc
def git_text(git_dir: Path, *args: str) -> str:
try:
return subprocess.check_output(
["/usr/bin/git", f"--git-dir={git_dir}", *args],
text=True,
timeout=30,
)
except subprocess.CalledProcessError as exc:
raise PackagerError(
"MODIFIED_FILES_GIT_COMMAND_FAILED:" + " ".join(args)
) from exc
def validate_lineage(git_dir: Path, base: str, candidate: str) -> None:
if OID_RE.fullmatch(base) is None or OID_RE.fullmatch(candidate) is None:
raise PackagerError("MODIFIED_FILES_OID_INVALID")
for oid in (base, candidate):
try:
subprocess.run(
[
"/usr/bin/git",
f"--git-dir={git_dir}",
"cat-file",
"-e",
f"{oid}^{{commit}}",
],
check=True,
timeout=30,
)
except subprocess.CalledProcessError as exc:
raise PackagerError("MODIFIED_FILES_COMMIT_MISSING:" + oid) from exc
if subprocess.run(
[
"/usr/bin/git",
f"--git-dir={git_dir}",
"merge-base",
"--is-ancestor",
base,
candidate,
],
check=False,
timeout=30,
).returncode != 0:
raise PackagerError("MODIFIED_FILES_LINEAGE_INVALID")
def safe_destination(root: Path, relative: str) -> Path:
path = Path(relative)
if path.is_absolute() or ".." in path.parts:
raise PackagerError("MODIFIED_FILE_PATH_INVALID:" + relative)
destination = root / path
destination.parent.mkdir(parents=True, exist_ok=True)
return destination
def is_document_path(path: str) -> bool:
value = Path(path)
if path.startswith("docs/"):
return True
if value.suffix.lower() in DOCUMENT_SUFFIXES:
return True
upper = value.name.upper()
return any(upper.startswith(prefix) for prefix in DOCUMENT_PREFIXES)
def diff_entries(
git_dir: Path,
base: str,
candidate: str,
before_root: Path,
after_root: Path,
) -> list[dict[str, object]]:
raw = git_text(
git_dir,
"diff",
"--name-status",
"-M",
"-C",
base,
candidate,
)
entries: list[dict[str, object]] = []
for line in raw.splitlines():
if not line.strip():
continue
parts = line.split("\t")
status = parts[0]
code = status[0]
old_path: str | None = None
new_path: str | None = None
if code in {"R", "C"}:
if len(parts) != 3:
raise PackagerError(
"MODIFIED_FILES_DIFF_PARSE_INVALID:" + line
)
old_path, new_path = parts[1], parts[2]
else:
if len(parts) != 2:
raise PackagerError(
"MODIFIED_FILES_DIFF_PARSE_INVALID:" + line
)
new_path = parts[1] if code != "D" else None
old_path = parts[1] if code != "A" else None
entry: dict[str, object] = {"status": status}
if old_path is not None:
entry["previous_path"] = old_path
before = git_bytes(git_dir, base, old_path)
scan_secret(before, "before:" + old_path)
safe_destination(before_root, old_path).write_bytes(before)
entry["before_sha256"] = sha256(before)
entry["before_bytes"] = len(before)
if new_path is not None:
entry["path"] = new_path
after = git_bytes(git_dir, candidate, new_path)
scan_secret(after, "after:" + new_path)
safe_destination(after_root, new_path).write_bytes(after)
entry["after_sha256"] = sha256(after)
entry["after_bytes"] = len(after)
entry["deleted"] = code == "D"
entry["added"] = code == "A"
entry["renamed"] = code == "R"
entry["copied"] = code == "C"
entries.append(entry)
return entries
def execution_entries(
extra_root: Path | None,
execution_root: Path,
) -> list[dict[str, object]]:
if extra_root is None or not extra_root.exists():
return []
if extra_root.is_symlink() or not extra_root.is_dir():
raise PackagerError("EXTRA_FILES_DIR_INVALID")
entries: list[dict[str, object]] = []
for path in sorted(extra_root.rglob("*")):
if path.is_symlink():
raise PackagerError(
"EXTRA_FILE_SYMLINK_REJECTED:" + str(path)
)
if not path.is_file():
continue
relative = path.relative_to(extra_root).as_posix()
data = path.read_bytes()
scan_secret(data, "execution:" + relative)
safe_destination(execution_root, relative).write_bytes(data)
entries.append(
{
"path": relative,
"sha256": sha256(data),
"bytes": len(data),
}
)
return entries
def write_checksums(bundle: Path) -> None:
checks: list[tuple[str, str]] = []
for path in sorted(bundle.rglob("*")):
if path.is_file() and path.name != "checksums.sha256":
checks.append(
(
sha256(path.read_bytes()),
path.relative_to(bundle).as_posix(),
)
)
(bundle / "checksums.sha256").write_text(
"".join(f"{digest} {name}\n" for digest, name in checks),
encoding="utf-8",
)
def verify_checksums(bundle: Path) -> None:
checksum_path = bundle / "checksums.sha256"
for line in checksum_path.read_text(encoding="utf-8").splitlines():
digest, relative = line.split(" ", 1)
path = bundle / relative
if not path.is_file() or sha256(path.read_bytes()) != digest:
raise PackagerError(
"MODIFIED_FILES_CHECKSUM_CLOSURE_INVALID:" + relative
)
def build_bundle(
*,
git_dir: Path,
base: str,
candidate: str,
task_id: str,
extra_dir: Path | None = None,
output_root: Path | None = None,
now: datetime | None = None,
) -> dict[str, object]:
if TASK_RE.fullmatch(task_id) is None:
raise PackagerError("TASK_ID_INVALID")
validate_lineage(git_dir, base, candidate)
timestamp = (now or datetime.now(timezone.utc)).strftime(
"%Y%m%dT%H%M%SZ"
)
safe_task = re.sub(r"[^A-Za-z0-9._-]+", "-", task_id).strip("-")
if output_root is None:
output_root = Path(
tempfile.mkdtemp(
prefix=f"aios-modified-files-bundle-{safe_task}-{timestamp}-",
dir="/var/tmp",
)
)
else:
output_root.mkdir(parents=True, exist_ok=True)
bundle = output_root / "bundle"
before_root = bundle / "repo" / "before"
after_root = bundle / "repo" / "after"
execution_root = bundle / "execution"
before_root.mkdir(parents=True)
after_root.mkdir(parents=True)
execution_root.mkdir(parents=True)
changes = diff_entries(
git_dir,
base,
candidate,
before_root,
after_root,
)
execution_files = execution_entries(extra_dir, execution_root)
document_changes = [
entry
for entry in changes
if is_document_path(
str(entry.get("path") or entry.get("previous_path") or "")
)
]
execution_documents = [
entry
for entry in execution_files
if is_document_path(str(entry["path"]))
]
documents_index = {
"schema": "aios.modified-documents.index.v1",
"task_id": task_id,
"base_oid": base,
"candidate_oid": candidate,
"repository_documents": document_changes,
"execution_documents": execution_documents,
"repository_document_count": len(document_changes),
"execution_document_count": len(execution_documents),
}
(bundle / "documents-index.json").write_text(
json.dumps(documents_index, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
manifest = {
"schema": "aios.all-modified-files.bundle.v1",
"task_id": task_id,
"base_oid": base,
"candidate_oid": candidate,
"generated_at": (
now or datetime.now(timezone.utc)
).isoformat(),
"changed_file_count": len(changes),
"execution_file_count": len(execution_files),
"repository_document_count": len(document_changes),
"execution_document_count": len(execution_documents),
"changes": changes,
"execution_files": execution_files,
}
(bundle / "all-modified-files-manifest.json").write_text(
json.dumps(manifest, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
write_checksums(bundle)
verify_checksums(bundle)
zip_name = (
f"AIOS_{task_id}_ALL_MODIFIED_FILES_{candidate[:12]}.zip"
.replace("/", "-")
)
zip_path = output_root / zip_name
with zipfile.ZipFile(
zip_path,
"w",
compression=zipfile.ZIP_DEFLATED,
) as archive:
for path in sorted(bundle.rglob("*")):
if path.is_symlink():
raise PackagerError(
"BUNDLE_SYMLINK_REJECTED:" + str(path)
)
if path.is_file():
archive.write(
path,
arcname=path.relative_to(bundle).as_posix(),
)
zip_sha = sha256(zip_path.read_bytes())
summary = {
"schema": "aios.all-modified-files.bundle-summary.v1",
"task_id": task_id,
"base_oid": base,
"candidate_oid": candidate,
"bundle_path": str(zip_path),
"bundle_sha256": zip_sha,
"changed_file_count": len(changes),
"execution_file_count": len(execution_files),
"repository_document_count": len(document_changes),
"execution_document_count": len(execution_documents),
}
(output_root / "ALL_MODIFIED_FILES_BUNDLE.json").write_text(
json.dumps(summary, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
return summary
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("base")
parser.add_argument("candidate")
parser.add_argument("task_id")
parser.add_argument("extra_dir", nargs="?")
parser.add_argument(
"--authority-git-dir",
default=os.environ.get(
"AIOS_AUTHORITY_GIT_DIR",
"/data/aios-gitops/repo.git",
),
)
parser.add_argument("--output-root")
args = parser.parse_args()
try:
summary = build_bundle(
git_dir=Path(args.authority_git_dir),
base=args.base,
candidate=args.candidate,
task_id=args.task_id,
extra_dir=Path(args.extra_dir) if args.extra_dir else None,
output_root=(
Path(args.output_root) if args.output_root else None
),
)
except PackagerError as exc:
print("FATAL:" + str(exc))
return 20
print("ALL_MODIFIED_FILES_BUNDLE=PASS")
print("ALL_MODIFIED_FILES_BUNDLE_PATH=" + str(summary["bundle_path"]))
print("ALL_MODIFIED_FILES_BUNDLE_SHA256=" + str(summary["bundle_sha256"]))
print("CHANGED_FILE_COUNT=" + str(summary["changed_file_count"]))
print("EXECUTION_FILE_COUNT=" + str(summary["execution_file_count"]))
print(
"REPOSITORY_DOCUMENT_COUNT="
+ str(summary["repository_document_count"])
)
print(
"EXECUTION_DOCUMENT_COUNT="
+ str(summary["execution_document_count"])
)
print(
"ALL_MODIFIED_FILES_EVIDENCE="
+ str(Path(str(summary["bundle_path"])).parent)
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -94,6 +94,91 @@ class AuthorityGitExecutor:
self._git(["worktree", "add", "--detach", str(path), base_oid])
return path
def recover_failed_detached_worktree_exact(
self,
*,
path: Path,
expected_oid: str,
stale_oid: str,
target_ref: str,
) -> str:
"""Recreate one exact clean, detached, single-attempt worktree at its durable head."""
_oid(expected_oid, "expected_oid")
_oid(stale_oid, "stale_oid")
if not TASK_BRANCH_RE.fullmatch(target_ref):
raise AuthorityGitError("TASK_BRANCH_SCOPE_DENIED")
if self.read_ref(target_ref) is not None:
raise AuthorityGitError("AUTHORITY_TARGET_BRANCH_ALREADY_EXISTS")
if (
not path.is_absolute()
or path.parent != self.worktree_root
or path.name in {"", ".", ".."}
or path.is_symlink()
or not path.is_dir()
):
raise AuthorityGitError("WORKTREE_RECOVERY_PATH_INVALID")
head = subprocess.run(
["git", "-C", str(path), "rev-parse", "HEAD"],
text=True, capture_output=True, check=False,
)
if head.returncode != 0 or head.stdout.strip() != stale_oid or stale_oid == expected_oid:
raise AuthorityGitError("WORKTREE_RECOVERY_HEAD_INVALID")
symbolic = subprocess.run(
["git", "-C", str(path), "symbolic-ref", "-q", "HEAD"],
text=True, capture_output=True, check=False,
)
if symbolic.returncode != 1:
raise AuthorityGitError("WORKTREE_RECOVERY_NOT_DETACHED")
status = subprocess.run(
["git", "-C", str(path), "status", "--porcelain=v1", "--untracked-files=all"],
text=True, capture_output=True, check=False,
)
if status.returncode != 0 or status.stdout:
raise AuthorityGitError("WORKTREE_RECOVERY_NOT_CLEAN")
parents = self._git(["rev-list", "--parents", "-n", "1", stale_oid]).split()
if parents != [stale_oid, expected_oid]:
raise AuthorityGitError("WORKTREE_RECOVERY_LINEAGE_INVALID")
registered = [
line.removeprefix("worktree ")
for line in self._git(["worktree", "list", "--porcelain"]).splitlines()
if line.startswith("worktree ")
]
if registered.count(str(path)) != 1:
raise AuthorityGitError("WORKTREE_RECOVERY_REGISTRATION_INVALID")
self._git(["worktree", "remove", "--force", str(path)])
if path.exists() or path.is_symlink():
raise AuthorityGitError("WORKTREE_RECOVERY_REMOVE_FAILED")
recreated = self.create_detached_worktree(expected_oid, path.name, path.parent)
readback = subprocess.run(
["git", "-C", str(recreated), "rev-parse", "HEAD"],
text=True, capture_output=True, check=False,
)
detached = subprocess.run(
["git", "-C", str(recreated), "symbolic-ref", "-q", "HEAD"],
text=True, capture_output=True, check=False,
)
if readback.returncode != 0 or readback.stdout.strip() != expected_oid or detached.returncode != 1:
raise AuthorityGitError("WORKTREE_RECOVERY_READBACK_MISMATCH")
return stale_oid
def create_task_branch_CAS(self, branch: str, expected_base_oid: str, new_oid: str) -> str:
if not TASK_BRANCH_RE.fullmatch(branch):
raise AuthorityGitError("TASK_BRANCH_SCOPE_DENIED")
_oid(expected_base_oid, "expected_base_oid")
_oid(new_oid, "new_oid")
current = self.read_ref(branch)
if current is not None:
if current == new_oid:
return new_oid
raise AuthorityGitError("AUTHORITY_TARGET_BRANCH_ALREADY_EXISTS")
if not self.ancestry_check(expected_base_oid, new_oid):
raise AuthorityGitError("NEW_OID_NOT_DESCENDANT")
self._git(["update-ref", branch, new_oid, "0" * 40])
readback = self.read_ref(branch)
if readback != new_oid:
raise AuthorityGitError("AUTHORITY_CREATE_READBACK_MISMATCH")
return readback
def advance_task_branch_CAS(self, branch: str, expected_oid: str, new_oid: str) -> str:
if not TASK_BRANCH_RE.fullmatch(branch):
raise AuthorityGitError("TASK_BRANCH_SCOPE_DENIED")
@@ -0,0 +1,288 @@
"""Unified autonomous control-plane primitives.
This is deliberately a library, not another daemon. Task API, workers and host
recovery code consume the same Admission -> Invariant -> Repair semantics.
"""
from __future__ import annotations
from dataclasses import asdict, dataclass
from enum import Enum
import re
from typing import Callable, Iterable, Mapping, Sequence
from aios_capability_contract import (
capability_contract,
has_capabilities,
normalize_required_capabilities,
normalize_worker_capabilities,
registered_capability,
validate_capability_contract_registry,
)
class FailureClass(str, Enum):
CAPABILITY_CONTRACT_ERROR = "CAPABILITY_CONTRACT_ERROR"
CREDENTIAL_PROJECTION_ERROR = "CREDENTIAL_PROJECTION_ERROR"
DEPENDENCY_UNAVAILABLE = "DEPENDENCY_UNAVAILABLE"
POLICY_DENIED = "POLICY_DENIED"
RESOURCE_UNAVAILABLE = "RESOURCE_UNAVAILABLE"
WORKER_CRASH = "WORKER_CRASH"
LEASE_LOST = "LEASE_LOST"
BUSINESS_PERMANENT = "BUSINESS_PERMANENT"
UNKNOWN = "UNKNOWN"
class RepairLevel(int, Enum):
AUTO_LOCAL = 0
AUTO_REPAIR_TASK = 1
OWNER_APPROVAL = 2
MANUAL = 3
@dataclass(frozen=True)
class InvariantResult:
name: str
passed: bool
detail: str = ""
def as_dict(self) -> dict[str, object]:
return asdict(self)
@dataclass(frozen=True)
class AdmissionDecision:
admitted: bool
state: str
invariants: tuple[InvariantResult, ...]
failure_class: FailureClass | None = None
repair_level: RepairLevel | None = None
repair_action: str | None = None
def as_dict(self) -> dict[str, object]:
return {
"schema": "aios.runtime-admission.decision.v1",
"admitted": self.admitted,
"state": self.state,
"invariants": [item.as_dict() for item in self.invariants],
"failure_class": None if self.failure_class is None else self.failure_class.value,
"repair_level": None if self.repair_level is None else int(self.repair_level),
"repair_action": self.repair_action,
}
@dataclass(frozen=True)
class RepairDecision:
failure_class: FailureClass
level: RepairLevel
action: str
consumes_business_attempt: bool
def as_dict(self) -> dict[str, object]:
return {
"failure_class": self.failure_class.value,
"level": int(self.level),
"action": self.action,
"consumes_business_attempt": self.consumes_business_attempt,
}
def classify_failure(error: object, *, phase: str = "") -> FailureClass:
text = (str(error) + " " + str(phase)).lower()
if any(token in text for token in (
"credential", "permission denied", "token registry", "secret", "unauthorized"
)):
return FailureClass.CREDENTIAL_PROJECTION_ERROR
if any(token in text for token in (
"transport", "unavailable", "timeout", "connection", "bad gateway", "http_5"
)):
return FailureClass.DEPENDENCY_UNAVAILABLE
if any(token in text for token in (
"capability", "eligible", "matcher", "contract"
)):
return FailureClass.CAPABILITY_CONTRACT_ERROR
if any(token in text for token in (
"policy", "forbidden", "approval required", "not authorized"
)):
return FailureClass.POLICY_DENIED
if any(token in text for token in ("lease lost", "lease_expired", "heartbeat_failed")):
return FailureClass.LEASE_LOST
if any(token in text for token in ("out of memory", "disk full", "resource")):
return FailureClass.RESOURCE_UNAVAILABLE
if any(token in text for token in ("worker crash", "process exited", "signal")):
return FailureClass.WORKER_CRASH
if any(token in text for token in ("invalid release", "business permanent")):
return FailureClass.BUSINESS_PERMANENT
return FailureClass.UNKNOWN
def repair_for_failure(failure: FailureClass) -> RepairDecision:
table = {
FailureClass.CAPABILITY_CONTRACT_ERROR: RepairDecision(
failure, RepairLevel.AUTO_REPAIR_TASK,
"create_same_lineage_capability_contract_repair", False,
),
FailureClass.CREDENTIAL_PROJECTION_ERROR: RepairDecision(
failure, RepairLevel.AUTO_REPAIR_TASK,
"restore_service_private_credential_projection", False,
),
FailureClass.DEPENDENCY_UNAVAILABLE: RepairDecision(
failure, RepairLevel.AUTO_LOCAL,
"retry_dependency_health_and_resume", False,
),
FailureClass.RESOURCE_UNAVAILABLE: RepairDecision(
failure, RepairLevel.AUTO_LOCAL,
"heal_runtime_resource_and_resume", False,
),
FailureClass.WORKER_CRASH: RepairDecision(
failure, RepairLevel.AUTO_LOCAL,
"restart_worker_and_resume_same_lineage", False,
),
FailureClass.LEASE_LOST: RepairDecision(
failure, RepairLevel.AUTO_LOCAL,
"recover_lease_and_resume_same_lineage", False,
),
FailureClass.POLICY_DENIED: RepairDecision(
failure, RepairLevel.OWNER_APPROVAL,
"request_bound_owner_authorization", False,
),
FailureClass.BUSINESS_PERMANENT: RepairDecision(
failure, RepairLevel.MANUAL,
"terminal_business_failure", True,
),
FailureClass.UNKNOWN: RepairDecision(
failure, RepairLevel.AUTO_REPAIR_TASK,
"create_diagnostic_repair_task", False,
),
}
return table[failure]
def _probe(
name: str,
probes: Mapping[str, Callable[[], bool]],
) -> InvariantResult:
fn = probes.get(name)
if fn is None:
return InvariantResult(name, False, "PROBE_MISSING")
try:
passed = bool(fn())
except Exception as exc:
return InvariantResult(name, False, "PROBE_ERROR:" + type(exc).__name__)
return InvariantResult(name, passed, "PASS" if passed else "NOT_READY")
def runtime_admission(
*,
worker_capabilities: Iterable[object],
required_capabilities: Iterable[object],
requested_operations: Sequence[str] = (),
probes: Mapping[str, Callable[[], bool]] | None = None,
) -> AdmissionDecision:
"""Admission is evaluated before claim; failures do not consume attempts."""
worker_values = tuple(worker_capabilities)
required_values = tuple(required_capabilities)
probe_map = probes or {}
invariants: list[InvariantResult] = []
registry = validate_capability_contract_registry()
invariants.append(InvariantResult(
"capability_contract_registry",
bool(registry.get("valid")),
"PASS" if registry.get("valid") else "REGISTRY_INVALID",
))
task_known = all(registered_capability(item) for item in required_values)
worker_known = all(registered_capability(item, worker=True) for item in worker_values)
invariants.extend((
InvariantResult("task_capabilities_registered", task_known, "PASS" if task_known else "UNKNOWN_TASK_CAPABILITY"),
InvariantResult("worker_capabilities_registered", worker_known, "PASS" if worker_known else "UNKNOWN_WORKER_CAPABILITY"),
))
capability_match = (
task_known
and worker_known
and has_capabilities(worker_values, required_values)
)
invariants.append(InvariantResult(
"capability_match",
capability_match,
"PASS" if capability_match else "WORKER_NOT_ELIGIBLE",
))
normalized = normalize_required_capabilities(required_values)
contracts = [
capability_contract(item)
for item in normalized
if not str(item).startswith("exact:")
]
contracts = [item for item in contracts if item is not None]
forbidden: set[str] = set()
dependencies: set[str] = set()
health: set[str] = set()
for contract in contracts:
forbidden.update(contract.forbidden)
dependencies.update(contract.dependencies)
health.update(contract.health_requirements)
requested = {str(item).strip().lower() for item in requested_operations if str(item).strip()}
forbidden_hit = sorted(requested & {item.lower() for item in forbidden})
invariants.append(InvariantResult(
"forbidden_operations_absent",
not forbidden_hit,
"PASS" if not forbidden_hit else "FORBIDDEN:" + ",".join(forbidden_hit),
))
for name in sorted(dependencies | health):
invariants.append(_probe(name, probe_map))
failed = [item for item in invariants if not item.passed]
if not failed:
return AdmissionDecision(
True,
"EXECUTION_READY",
tuple(invariants),
)
details = " ".join(item.name + ":" + item.detail for item in failed)
failure = classify_failure(details, phase="admission")
if any(item.name == "forbidden_operations_absent" and not item.passed for item in failed):
failure = FailureClass.POLICY_DENIED
elif any("credential" in item.name for item in failed):
failure = FailureClass.CREDENTIAL_PROJECTION_ERROR
elif any(item.name in {"capability_contract_registry", "task_capabilities_registered", "worker_capabilities_registered", "capability_match"} for item in failed):
failure = FailureClass.CAPABILITY_CONTRACT_ERROR
elif any(item.name in dependencies or item.name in health for item in failed):
failure = FailureClass.DEPENDENCY_UNAVAILABLE
repair = repair_for_failure(failure)
return AdmissionDecision(
False,
"EXECUTION_READY_FAILED",
tuple(invariants),
failure,
repair.level,
repair.action,
)
def system_invariant_snapshot(
*,
worker_capabilities: Iterable[object],
required_capabilities: Iterable[object],
requested_operations: Sequence[str] = (),
probes: Mapping[str, Callable[[], bool]] | None = None,
) -> dict[str, object]:
decision = runtime_admission(
worker_capabilities=worker_capabilities,
required_capabilities=required_capabilities,
requested_operations=requested_operations,
probes=probes,
)
return {
"schema": "aios.system-invariants.v1",
"status": "HEALTHY" if decision.admitted else "DEGRADED",
"execution_ready": decision.admitted,
"admission": decision.as_dict(),
}
@@ -0,0 +1,334 @@
"""Canonical fail-closed capability semantics for task eligibility."""
from __future__ import annotations
from collections.abc import Iterable
from dataclasses import asdict, dataclass
from typing import Any
_TASK_ALIASES: dict[str, frozenset[str]] = {
"repository_read": frozenset({"repo.read"}),
"git.read": frozenset({"repo.read"}),
"repo.read": frozenset({"repo.read"}),
"test_execution": frozenset({"test.execute"}),
"git.test": frozenset({"test.execute"}),
"test.execute": frozenset({"test.execute"}),
"governed_git_write": frozenset({"governed.git.write.branch", "governed.git.write.file"}),
"governed.git.write": frozenset({"governed.git.write.branch", "governed.git.write.file"}),
"governed.write.branch": frozenset({"governed.git.write.branch"}),
"governed.write.file": frozenset({"governed.git.write.file"}),
"shell": frozenset({"runtime.shell"}),
"git": frozenset({"runtime.git"}),
"python": frozenset({"runtime.python"}),
"code.read": frozenset({"legacy.code.read"}),
"code.review": frozenset({"legacy.code.review"}),
"code.write": frozenset({"legacy.code.write"}),
"code.write.patch": frozenset({"legacy.code.write.patch"}),
"test.python": frozenset({"legacy.test.python"}),
"test.shell.allowlisted": frozenset({"legacy.test.shell.allowlisted"}),
"git.commit": frozenset({"legacy.git.commit"}),
"git.diff.read": frozenset({"legacy.git.diff.read"}),
"git.force_push": frozenset({"legacy.git.force_push"}),
"deploy.production": frozenset({"legacy.deploy.production"}),
"secret.transform": frozenset({"legacy.secret.transform"}),
"secure_file_transformer": frozenset({"legacy.secure_file_transformer"}),
"runtime.shadow": frozenset({"legacy.runtime.shadow"}),
"review.security": frozenset({"legacy.review.security"}),
"review.api-contract": frozenset({"legacy.review.api-contract"}),
"gpu.train": frozenset({"legacy.gpu.train"}),
"agent.runtime.hermes": frozenset({"agent.runtime.hermes"}),
"agent.runtime.openclaw": frozenset({"agent.runtime.openclaw"}),
"release:candidate": frozenset({"release:candidate"}),
}
_WORKER_ALIASES: dict[str, frozenset[str]] = {
"git.read": frozenset({"repo.read"}),
"repo.read": frozenset({"repo.read"}),
"repository_read": frozenset({"repo.read"}),
"git.test": frozenset({"test.execute"}),
"test.execute": frozenset({"test.execute"}),
"test_execution": frozenset({"test.execute"}),
"governed.write.branch": frozenset({"governed.git.write.branch"}),
"governed.git.write.branch": frozenset({"governed.git.write.branch"}),
"governed.write.file": frozenset({"governed.git.write.file"}),
"governed.git.write.file": frozenset({"governed.git.write.file"}),
"shell": frozenset({"runtime.shell"}),
"git": frozenset({"runtime.git"}),
"python": frozenset({"runtime.python"}),
"code.read": frozenset({"legacy.code.read"}),
"code.review": frozenset({"legacy.code.review"}),
"code.write": frozenset({"legacy.code.write"}),
"code.write.patch": frozenset({"legacy.code.write.patch"}),
"test.python": frozenset({"legacy.test.python"}),
"test.shell.allowlisted": frozenset({"legacy.test.shell.allowlisted"}),
"git.commit": frozenset({"legacy.git.commit"}),
"git.diff.read": frozenset({"legacy.git.diff.read"}),
"git.force_push": frozenset({"legacy.git.force_push"}),
"deploy.production": frozenset({"legacy.deploy.production"}),
"secret.transform": frozenset({"legacy.secret.transform"}),
"secure_file_transformer": frozenset({"legacy.secure_file_transformer"}),
"runtime.shadow": frozenset({"legacy.runtime.shadow"}),
"review.security": frozenset({"legacy.review.security"}),
"review.api-contract": frozenset({"legacy.review.api-contract"}),
"gpu.train": frozenset({"legacy.gpu.train"}),
"agent.runtime.hermes": frozenset({"agent.runtime.hermes"}),
"agent.runtime.openclaw": frozenset({"agent.runtime.openclaw"}),
"release:candidate": frozenset({"release:candidate"}),
}
@dataclass(frozen=True)
class CapabilityContract:
"""Versioned runtime contract attached to one canonical semantic capability."""
name: str
min_version: int = 1
max_version: int = 1
permissions: tuple[str, ...] = ()
forbidden: tuple[str, ...] = ()
dependencies: tuple[str, ...] = ()
health_requirements: tuple[str, ...] = ()
def as_dict(self) -> dict[str, Any]:
return asdict(self)
# This registry is the semantic source of truth. Alias tables below are retained
# as compatibility adapters; startup/tests validate every alias target against
# this registry so a name cannot be accepted in one layer and be unknown in
# another.
_CAPABILITY_CONTRACTS: dict[str, CapabilityContract] = {
"repo.read": CapabilityContract("repo.read", permissions=("repository_read",)),
"test.execute": CapabilityContract("test.execute", permissions=("test_execution",)),
"governed.git.write.branch": CapabilityContract(
"governed.git.write.branch",
permissions=("governed_branch_write",),
dependencies=("writer_lease", "governed_gitops"),
),
"governed.git.write.file": CapabilityContract(
"governed.git.write.file",
permissions=("governed_file_write",),
dependencies=("writer_lease", "governed_gitops"),
),
"governed.git.write": CapabilityContract(
"governed.git.write",
permissions=("governed_git_write",),
dependencies=("writer_lease", "governed_gitops"),
),
"runtime.shell": CapabilityContract("runtime.shell", permissions=("runtime_shell",)),
"runtime.git": CapabilityContract("runtime.git", permissions=("runtime_git",)),
"runtime.python": CapabilityContract("runtime.python", permissions=("runtime_python",)),
"legacy.code.read": CapabilityContract("legacy.code.read"),
"legacy.code.review": CapabilityContract("legacy.code.review"),
"legacy.code.write": CapabilityContract("legacy.code.write"),
"legacy.code.write.patch": CapabilityContract("legacy.code.write.patch"),
"legacy.test.python": CapabilityContract("legacy.test.python"),
"legacy.test.shell.allowlisted": CapabilityContract("legacy.test.shell.allowlisted"),
"legacy.git.commit": CapabilityContract("legacy.git.commit"),
"legacy.git.diff.read": CapabilityContract("legacy.git.diff.read"),
"legacy.git.force_push": CapabilityContract(
"legacy.git.force_push",
forbidden=("protected_ref_bypass",),
),
"legacy.deploy.production": CapabilityContract(
"legacy.deploy.production",
permissions=("production_deploy",),
dependencies=("owner_approval", "deployment_controller"),
health_requirements=("approval_envelope_verified", "runtime_acceptance_available"),
),
"legacy.secret.transform": CapabilityContract("legacy.secret.transform"),
"legacy.secure_file_transformer": CapabilityContract("legacy.secure_file_transformer"),
"legacy.runtime.shadow": CapabilityContract("legacy.runtime.shadow"),
"legacy.review.security": CapabilityContract("legacy.review.security"),
"legacy.review.api-contract": CapabilityContract("legacy.review.api-contract"),
"legacy.gpu.train": CapabilityContract("legacy.gpu.train"),
"agent.runtime.hermes": CapabilityContract(
"agent.runtime.hermes",
dependencies=("hermes_runtime",),
health_requirements=("worker_liveness",),
),
"agent.runtime.openclaw": CapabilityContract(
"agent.runtime.openclaw",
dependencies=("openclaw_runtime",),
health_requirements=("worker_liveness",),
),
"release:candidate": CapabilityContract(
"release:candidate",
min_version=1,
max_version=1,
permissions=("prepare_release",),
forbidden=(
"production_deploy",
"submit_owner_approval",
"tag_publish",
"baseline_registry_finalize",
"runtime_switch",
),
dependencies=(
"credential_projection",
"trusted_builder",
),
health_requirements=(
"authenticated_probe",
"release_api_health",
"worker_liveness",
),
),
}
def capability_contract(name: object) -> CapabilityContract | None:
"""Return the contract for one canonical or registered capability name."""
text = _clean(name)
direct = _CAPABILITY_CONTRACTS.get(text)
if direct is not None:
return direct
mapped = _TASK_ALIASES.get(text) or _WORKER_ALIASES.get(text)
if mapped is None or len(mapped) != 1:
return None
return _CAPABILITY_CONTRACTS.get(next(iter(mapped)))
def registered_capability(name: object, *, worker: bool = False) -> bool:
"""Fail-closed validation used by registration/task admission."""
text = _clean(name)
if "*" in text:
return False
aliases = _WORKER_ALIASES if worker else _TASK_ALIASES
mapped = aliases.get(text)
if mapped is None:
return text in _CAPABILITY_CONTRACTS
return all(item in _CAPABILITY_CONTRACTS for item in mapped)
def validate_capability_contract_registry() -> dict[str, object]:
"""Prove alias/contract closure and contract safety invariants."""
alias_targets: set[str] = set()
for mapping in (_TASK_ALIASES, _WORKER_ALIASES):
for values in mapping.values():
alias_targets.update(values)
missing = sorted(alias_targets - set(_CAPABILITY_CONTRACTS))
invalid_versions = sorted(
name
for name, contract in _CAPABILITY_CONTRACTS.items()
if contract.min_version < 1 or contract.max_version < contract.min_version
)
permission_forbidden_overlap = sorted(
name
for name, contract in _CAPABILITY_CONTRACTS.items()
if set(contract.permissions) & set(contract.forbidden)
)
return {
"schema": "aios.capability-contract-registry.v1",
"contract_count": len(_CAPABILITY_CONTRACTS),
"missing_alias_targets": missing,
"invalid_versions": invalid_versions,
"permission_forbidden_overlap": permission_forbidden_overlap,
"valid": not missing and not invalid_versions and not permission_forbidden_overlap,
}
def capability_contract_snapshot() -> dict[str, dict[str, Any]]:
return {
name: contract.as_dict()
for name, contract in sorted(_CAPABILITY_CONTRACTS.items())
}
_UNKNOWN_PREFIX = "exact:"
def _clean(value: object) -> str:
if not isinstance(value, str):
raise ValueError("capability must be a string")
text = value.strip().lower()
if not text:
raise ValueError("capability must not be empty")
return text
def normalize_required_capabilities(values: Iterable[object]) -> frozenset[str]:
"""Map task-facing names to one semantic vocabulary.
Unknown names remain explicit exact requirements. They are never treated
as wildcards or granted by a failed normalization.
"""
result: set[str] = set()
for value in values:
text = _clean(value)
if "*" in text:
result.add(_UNKNOWN_PREFIX + text)
else:
mapped = _TASK_ALIASES.get(text)
if mapped is None or any(item not in _CAPABILITY_CONTRACTS for item in mapped):
result.add(_UNKNOWN_PREFIX + text)
else:
result.update(mapped)
return frozenset(result)
def normalize_worker_capabilities(values: Iterable[object]) -> frozenset[str]:
"""Map registered worker names to effective semantic capabilities."""
result: set[str] = set()
for value in values:
text = _clean(value)
if "*" in text:
continue
mapped = _WORKER_ALIASES.get(text)
if mapped is not None and all(item in _CAPABILITY_CONTRACTS for item in mapped):
result.update(mapped)
if {
"governed.git.write.branch",
"governed.git.write.file",
}.issubset(result):
result.add("governed.git.write")
return frozenset(result)
def has_capabilities(
worker_capabilities: Iterable[object],
required_capabilities: Iterable[object],
) -> bool:
"""Return the single eligibility decision used by poll and claim."""
required = normalize_required_capabilities(required_capabilities)
effective = normalize_worker_capabilities(worker_capabilities)
if any(value.startswith(_UNKNOWN_PREFIX) for value in required):
return False
return required.issubset(effective)
def required_capabilities_subset(
granted_requirements: Iterable[object],
required_requirements: Iterable[object],
) -> bool:
"""Check semantic capability containment for delegation constraints."""
granted = normalize_required_capabilities(granted_requirements)
required = normalize_required_capabilities(required_requirements)
if any(value.startswith(_UNKNOWN_PREFIX) for value in granted | required):
return False
return required.issubset(granted)
def normalized_capability_matrix(
worker_capabilities: Iterable[object],
required_capabilities: Iterable[object],
) -> dict[str, object]:
"""Expose non-secret decision evidence for tests and diagnostics."""
worker_values = tuple(worker_capabilities)
required_values = tuple(required_capabilities)
worker = normalize_worker_capabilities(worker_values)
required = normalize_required_capabilities(required_values)
return {
"worker_effective": sorted(worker),
"task_required": sorted(required),
"eligible": has_capabilities(worker_values, required_values),
}
@@ -13,6 +13,8 @@ BRIDGE_OPERATIONS = (
"getGovernedWriteCapabilities",
"createGovernedTask",
"acquireWriterLease",
"bootstrapWriterLease",
"renewWriterLease",
"commitFileGoverned",
"commitFilesGoverned",
"getGitOpsJob",
@@ -31,6 +33,8 @@ BRIDGE_OPERATIONS = (
OPERATION_REQUIRED_FIELDS = {
"createGovernedTask": ("task_id", "repository", "base_oid", "target_branch", "risk", "description", "idempotency_key"),
"acquireWriterLease": ("schema", "task_id", "repository", "branch_mode", "base_oid", "expected_branch_oid", "target_branch", "generation", "lease_seconds"),
"bootstrapWriterLease": ("schema", "task_id", "repository", "branch_mode", "base_oid", "expected_branch_oid", "target_branch", "generation", "lease_seconds"),
"renewWriterLease": ("schema", "task_id", "repository", "branch_mode", "base_oid", "expected_branch_oid", "target_branch", "generation", "lease_seconds"),
"commitFileGoverned": ("task_id", "repository", "expected_branch_oid", "target_branch", "generation", "idempotency_key", "path", "content_base64", "message"),
"commitFilesGoverned": ("task_id", "repository", "expected_branch_oid", "target_branch", "generation", "idempotency_key", "files", "message"),
"getGitOpsJob": ("job_id",),
@@ -46,6 +50,49 @@ OPERATION_REQUIRED_FIELDS = {
"getControlPlaneSnapshot": (),
}
BRIDGE_OPERATION_CONTRACTS = {
"acquireWriterLease": {
"method": "POST",
"path": "/v2/writer-lease",
"required_fields": OPERATION_REQUIRED_FIELDS["acquireWriterLease"],
"response_fields": (
"task_id", "job_id", "generation", "lease_seconds", "lease_expires_at",
"create_target_branch", "worktree_mode", "cas_ref_update_required",
),
},
"bootstrapWriterLease": {
"method": "POST",
"path": "/v2/writer-lease/bootstrap",
"required_fields": OPERATION_REQUIRED_FIELDS["bootstrapWriterLease"],
"response_fields": (
"task_id", "job_id", "generation", "lease_seconds", "lease_expires_at",
"create_target_branch", "worktree_mode", "cas_ref_update_required",
"recovered", "bootstrapped", "runtime_rehydrated",
"durable_branch_lease_present", "idempotent",
),
},
"renewWriterLease": {
"method": "POST",
"path": "/v2/writer-lease/renew",
"required_fields": OPERATION_REQUIRED_FIELDS["renewWriterLease"],
"response_fields": (
"task_id", "job_id", "generation", "lease_seconds", "lease_expires_at",
"recovered", "bootstrapped", "runtime_rehydrated",
"durable_branch_lease_present", "idempotent",
),
},
"commitFileGoverned": {
"method": "POST", "path": "/v2/commit-file",
"required_fields": OPERATION_REQUIRED_FIELDS["commitFileGoverned"],
"response_fields": ("job_id", "commit_oid", "commit_visible", "branch_mode", "target_branch", "generation"),
},
"commitFilesGoverned": {
"method": "POST", "path": "/v2/commit-files",
"required_fields": OPERATION_REQUIRED_FIELDS["commitFilesGoverned"],
"response_fields": ("job_id", "commit_oid", "commit_visible", "branch_mode", "target_branch", "generation"),
},
}
class RuntimeDriftError(RuntimeError):
pass
@@ -122,6 +169,14 @@ class BridgeV2Adapter:
"legacy_writes_enabled": False,
"governed_writes_enabled": True,
"operations": list(BRIDGE_OPERATIONS),
"operation_contracts": {
operation: {
**contract,
"required_fields": list(contract["required_fields"]),
"response_fields": list(contract["response_fields"]),
}
for operation, contract in BRIDGE_OPERATION_CONTRACTS.items()
},
"force_push": False,
"main_write": False,
"tag_write": False,
@@ -151,6 +206,8 @@ class BridgeV2Adapter:
def createGovernedTask(self, request: Mapping[str, Any]) -> Any: return self._call("createGovernedTask", request)
def acquireWriterLease(self, request: Mapping[str, Any]) -> Any: return self._call("acquireWriterLease", request)
def bootstrapWriterLease(self, request: Mapping[str, Any]) -> Any: return self._call("bootstrapWriterLease", request)
def renewWriterLease(self, request: Mapping[str, Any]) -> Any: return self._call("renewWriterLease", request)
def commitFileGoverned(self, request: Mapping[str, Any]) -> Any: return self._call("commitFileGoverned", request)
def commitFilesGoverned(self, request: Mapping[str, Any]) -> Any: return self._call("commitFilesGoverned", request)
def getGitOpsJob(self, request: Mapping[str, Any]) -> Any: return self._call("getGitOpsJob", request)
File diff suppressed because it is too large Load Diff
+131 -33
View File
@@ -51,11 +51,13 @@ class AuthorityPort(Protocol):
def branch_exists(self, branch: str) -> bool: ...
def branch_head(self, branch: str) -> str: ...
def is_ancestor(self, older: str, newer: str) -> bool: ...
def create_task_branch_cas(self, branch: str, expected_base: str, new: str) -> str: ...
def compare_and_swap_branch(self, branch: str, expected: str, new: str) -> str: ...
class RemotePort(Protocol):
def repository_identity(self) -> str: ...
def push_new_exact(self, branch: str, new: str) -> None: ...
def push_exact(self, branch: str, expected: str, new: str) -> None: ...
def read_git_ref(self, branch: str) -> str | None: ...
def read_api_branch(self, branch: str) -> str | None: ...
@@ -103,6 +105,9 @@ class AuthorityGitAdapter:
def is_ancestor(self, older: str, newer: str) -> bool:
return self.executor.ancestry_check(older, newer)
def create_task_branch_cas(self, branch: str, expected_base: str, new: str) -> str:
return self.executor.create_task_branch_CAS(self._ref(branch), expected_base, new)
def compare_and_swap_branch(self, branch: str, expected: str, new: str) -> str:
return self.executor.advance_task_branch_CAS(self._ref(branch), expected, new)
@@ -258,6 +263,15 @@ class GiteaContinuationRemote:
raise ContinuationExecutionError("GITEA_GIT_READBACK_INVALID")
return oid
def push_new_exact(self, branch: str, new: str) -> None:
self._require_projected()
self.repository_identity()
if self.read_git_ref(branch) is not None:
raise ContinuationExecutionError("GITEA_TARGET_BRANCH_ALREADY_EXISTS")
_run(["git", "push", self.remote_name, f"{new}:refs/heads/{branch}"], cwd=self.workspace)
if self.read_git_ref(branch) != new:
raise ContinuationExecutionError("GITEA_CREATE_READBACK_MISMATCH")
def push_exact(self, branch: str, expected: str, new: str) -> None:
self._require_projected()
self.repository_identity()
@@ -329,6 +343,7 @@ class ContinuationPermissions:
@dataclass(frozen=True, slots=True)
class ContinuationRuntimeBinding:
request: WriterLeaseRequestV2
job_id: str
workspace_path: Path
lease_not_before: datetime
lease_expires_at: datetime
@@ -389,13 +404,20 @@ class ContinuationExecutor:
def observe_generation(self, branch: str, generation: int) -> None:
self._observed_generations[branch] = max(generation, self._observed_generations.get(branch, 0))
def _max_generation(self, branch: str) -> int:
def _max_generation(self, branch: str, *, exclude_job_id: str | None = None) -> int:
observed = self._observed_generations.get(branch, 0)
with self.controller.store._connect() as connection:
row = connection.execute(
"SELECT MAX(generation) AS generation FROM branch_leases WHERE branch=?",
(branch,),
).fetchone()
if exclude_job_id is None:
row = connection.execute(
"SELECT MAX(generation) AS generation FROM branch_leases WHERE branch=?",
(branch,),
).fetchone()
else:
row = connection.execute(
"SELECT MAX(generation) AS generation FROM branch_leases "
"WHERE branch=? AND job_id!=?",
(branch, exclude_job_id),
).fetchone()
durable = int(row["generation"] or 0)
return max(observed, durable)
@@ -404,6 +426,7 @@ class ContinuationExecutor:
return {
"schema": WriterLeaseRequestV2.schema,
"task_id": request.task_id,
"job_id": request.job_id,
"repository": request.repository,
"branch_mode": request.branch_mode,
"base_oid": request.base_oid,
@@ -413,10 +436,21 @@ class ContinuationExecutor:
"lease_seconds": request.lease_seconds,
}
def _validate_preflight(self, binding: ContinuationRuntimeBinding) -> None:
def _validate_preflight(self, binding: ContinuationRuntimeBinding, *, job_id: str | None = None) -> None:
request = binding.request
normalized = validate_writer_lease_request_v2(self._request_mapping(request), self.authority)
if normalized != {**self._request_mapping(request), "create_target_branch": False, "worktree_mode": "detached_at_expected_head", "cas_ref_update_required": True}:
request_mapping = self._request_mapping(request)
normalized = validate_writer_lease_request_v2(request_mapping, self.authority)
expected_normalized = {
**request_mapping,
"create_target_branch": request.branch_mode == "CREATE_NEW",
"worktree_mode": (
"detached_at_base_oid"
if request.branch_mode == "CREATE_NEW"
else "detached_at_expected_head"
),
"cas_ref_update_required": True,
}
if normalized != expected_normalized:
raise ContinuationExecutionError("CONTINUATION_NORMALIZATION_INVALID")
if not binding.original_receipt_valid or not binding.original_grant_consumed:
raise ContinuationExecutionError("ORIGINAL_GRANT_LINEAGE_INVALID")
@@ -426,7 +460,7 @@ class ContinuationExecutor:
raise ContinuationExecutionError("LEASE_NOT_ACTIVE")
if now >= binding.lease_expires_at:
raise ContinuationExecutionError("LEASE_EXPIRED")
if request.generation <= self._max_generation(request.target_branch):
if request.generation <= self._max_generation(request.target_branch, exclude_job_id=job_id):
raise ContinuationExecutionError("GENERATION_NOT_MONOTONIC")
if not self.workspace.is_detached():
raise ContinuationExecutionError("WORKTREE_NOT_DETACHED")
@@ -452,57 +486,102 @@ class ContinuationExecutor:
"generation": request.generation,
}
def execute(self, binding: ContinuationRuntimeBinding, commit: ContinuationCommit) -> dict[str, object]:
payload = commit.payload()
submitted = self.controller.submit(self._job_request(binding, commit, payload), payload=payload)
current = self.controller.store.read(submitted.job_id)
if JobState(current["state"]) == JobState.COMPLETED:
def execute(
self,
binding: ContinuationRuntimeBinding,
commit: ContinuationCommit,
*,
before_complete: Callable[[str], None] | None = None,
) -> dict[str, object]:
commit.payload()
if not binding.job_id or binding.request.job_id != binding.job_id:
raise ContinuationExecutionError("RUNTIME_LINEAGE_MISSING_JOB_ID")
current = self.controller.store.read(binding.job_id)
current_state = JobState(current["state"])
if current_state == JobState.COMPLETED:
return self.controller.store._receipt(current, duplicate=True).as_dict()
self._validate_preflight(binding)
if current_state == JobState.REMOTE_VERIFIED:
receipt = json.loads(str(current["receipt_json"] or "{}"))
commit_oid = str(receipt.get("authority_oid") or "")
if not OID_RE.fullmatch(commit_oid):
raise ContinuationExecutionError("GOVERNED_COMMIT_OID_INVALID")
if before_complete is not None:
before_complete(commit_oid)
return self.controller.advance(
binding.job_id,
JobState.COMPLETED,
side_effect_state="REMOTE_VERIFIED",
receipt=receipt,
).as_dict()
self._validate_preflight(binding, job_id=binding.job_id)
request = binding.request
projected = False
try:
self.credential.project()
projected = True
row = self.controller.store.read(submitted.job_id)
if JobState(row["state"]) == JobState.REQUESTED:
row = self.controller.store.read(binding.job_id)
state = JobState(row["state"])
if state == JobState.REQUESTED:
self.controller.advance(
submitted.job_id,
binding.job_id,
JobState.AUTHORIZED,
side_effect_state="AUTHORIZED",
receipt={"task_id": request.task_id, "original_grant_consumed": True},
expected_revision=int(row["revision"]),
)
row = self.controller.store.read(submitted.job_id)
if JobState(row["state"]) == JobState.AUTHORIZED:
row = self.controller.store.read(binding.job_id)
state = JobState(row["state"])
if state == JobState.AUTHORIZED:
self.controller.lease_job(
submitted.job_id,
lease_id=f"lease-{submitted.job_id}",
binding.job_id,
lease_id=f"lease-{binding.job_id}",
lineage_anchor_oid=request.base_oid,
current_head_oid=request.expected_branch_oid,
generation=request.generation,
lease_expires_at=binding.lease_expires_at.astimezone(timezone.utc).isoformat(),
expected_revision=int(row["revision"]),
)
self.controller.advance(submitted.job_id, JobState.WORKTREE_READY, side_effect_state="WORKTREE_READY")
self.controller.advance(submitted.job_id, JobState.EXECUTING, side_effect_state="IN_PROGRESS")
row = self.controller.store.read(binding.job_id)
state = JobState(row["state"])
if state == JobState.LEASED:
self.controller.advance(binding.job_id, JobState.WORKTREE_READY,
side_effect_state="WORKTREE_READY", expected_revision=int(row["revision"]))
row = self.controller.store.read(binding.job_id)
state = JobState(row["state"])
if state == JobState.WORKTREE_READY:
self.controller.advance(binding.job_id, JobState.EXECUTING,
side_effect_state="IN_PROGRESS", expected_revision=int(row["revision"]))
elif state != JobState.EXECUTING:
raise ContinuationExecutionError("JOB_EXECUTION_STATE_INVALID")
new_oid = self.workspace.commit_files(commit.files, commit.message)
if not OID_RE.fullmatch(new_oid) or not self.authority.is_ancestor(request.expected_branch_oid, new_oid):
raise ContinuationExecutionError("NON_FAST_FORWARD_COMMIT")
authority_oid = self.authority.compare_and_swap_branch(request.target_branch, request.expected_branch_oid, new_oid)
if request.branch_mode == "CREATE_NEW":
authority_oid = self.authority.create_task_branch_cas(request.target_branch, request.base_oid, new_oid)
else:
authority_oid = self.authority.compare_and_swap_branch(request.target_branch, request.expected_branch_oid, new_oid)
if authority_oid != new_oid:
raise ContinuationExecutionError("AUTHORITY_CAS_READBACK_MISMATCH")
self.controller.advance(
submitted.job_id,
binding.job_id,
JobState.LOCAL_VERIFIED,
side_effect_state="LOCAL_VERIFIED",
receipt={"authority_oid": authority_oid, "force_push": False},
)
try:
self.controller.advance(submitted.job_id, JobState.REMOTE_SYNCING, side_effect_state="IN_PROGRESS")
self.remote.push_exact(request.target_branch, request.expected_branch_oid, new_oid)
self.controller.advance(binding.job_id, JobState.REMOTE_SYNCING, side_effect_state="IN_PROGRESS")
if request.branch_mode == "CREATE_NEW":
self.remote.push_new_exact(request.target_branch, new_oid)
else:
self.remote.push_exact(request.target_branch, request.expected_branch_oid, new_oid)
except Exception:
return self._remote_pending(submitted.job_id, authority_oid, new_oid)
return self._finish_remote_verification(submitted.job_id, authority_oid, new_oid)
return self._remote_pending(binding.job_id, authority_oid, new_oid)
return self._finish_remote_verification(
binding.job_id,
authority_oid,
new_oid,
before_complete=before_complete,
)
finally:
if projected:
self.credential.revoke()
@@ -525,7 +604,14 @@ class ContinuationExecutor:
)
return result.as_dict()
def _finish_remote_verification(self, job_id: str, authority_oid: str, new_oid: str) -> dict[str, object]:
def _finish_remote_verification(
self,
job_id: str,
authority_oid: str,
new_oid: str,
*,
before_complete: Callable[[str], None] | None = None,
) -> dict[str, object]:
row = self.controller.store.read(job_id)
branch = row["branch"]
request = json.loads(row["request_json"])["binding"]
@@ -542,8 +628,20 @@ class ContinuationExecutor:
"reconciliation_decision": decision.action.value,
"force_push": False,
}
self.controller.advance(job_id, JobState.REMOTE_VERIFIED, side_effect_state="REMOTE_VERIFIED", receipt=receipt)
return self.controller.advance(job_id, JobState.COMPLETED, side_effect_state="REMOTE_VERIFIED", receipt=receipt).as_dict()
self.controller.advance(
job_id,
JobState.REMOTE_VERIFIED,
side_effect_state="REMOTE_VERIFIED",
receipt=receipt,
)
if before_complete is not None:
before_complete(new_oid)
return self.controller.advance(
job_id,
JobState.COMPLETED,
side_effect_state="REMOTE_VERIFIED",
receipt=receipt,
).as_dict()
def reconcile(self, job_id: str) -> dict[str, object]:
row = self.controller.store.read(job_id)
@@ -49,6 +49,7 @@ class WriterLeaseRequestV2:
target_branch: str
generation: int
lease_seconds: int = 900
job_id: str = ""
schema = "aios.task-writer-lease.v2"
accepted_schemas = frozenset({"aios.task-writer-lease.v2", "aios.task-writer-lease-request.v2"})
@@ -56,6 +57,8 @@ class WriterLeaseRequestV2:
def validate(self, probe: BranchProbe) -> None:
if not self.task_id or self.repository != "aios/aios":
raise ContinuationError("request_identity_invalid")
if not self.job_id or not self.job_id.startswith("job-"):
raise ContinuationError("job_id_invalid")
if self.branch_mode not in {"CREATE_NEW", "CONTINUE_EXISTING"}:
raise ContinuationError("branch_mode_invalid")
_oid(self.base_oid, "base_oid")
+50 -4
View File
@@ -6,6 +6,7 @@ from datetime import datetime, timedelta, timezone
import os
from pathlib import Path
import secrets
import re
from typing import Any, Callable, Mapping
@@ -21,9 +22,14 @@ class CredentialLease:
scopes: tuple[str, ...]
provider: str
state: str = "ACTIVE"
scope_type: str = "system"
tenant_id: str | None = None
user_id: str | None = None
project_id: str | None = None
run_id: str | None = None
def as_dict(self) -> dict[str, Any]:
return {"lease_id": self.lease_id, "task_id": self.task_id, "repository": self.repository, "target_branch": self.target_branch, "generation": self.generation, "issued_at": self.issued_at, "expires_at": self.expires_at, "scopes": list(self.scopes), "provider": self.provider, "state": self.state}
return {"lease_id": self.lease_id, "task_id": self.task_id, "repository": self.repository, "target_branch": self.target_branch, "generation": self.generation, "issued_at": self.issued_at, "expires_at": self.expires_at, "scopes": list(self.scopes), "provider": self.provider, "state": self.state, "scope_type": self.scope_type, "tenant_id": self.tenant_id, "user_id": self.user_id, "project_id": self.project_id, "run_id": self.run_id}
class FileCredentialProvider:
@@ -43,11 +49,19 @@ class CredentialBroker:
self.now = now or (lambda: datetime.now(timezone.utc))
self._leases: dict[str, CredentialLease] = {}
def issue(self, *, task_id: str, repository: str, target_branch: str, generation: int, scopes: tuple[str, ...] = ("git:push", "gitea:read"), lease_seconds: int = 900) -> dict[str, Any]:
def issue(self, *, task_id: str, repository: str, target_branch: str, generation: int, scopes: tuple[str, ...] = ("git:push", "gitea:read"), lease_seconds: int = 900, scope_type: str = "system", tenant_id: str | None = None, user_id: str | None = None, project_id: str | None = None, run_id: str | None = None) -> dict[str, Any]:
if repository != "aios/aios" or not target_branch.startswith("agent/") or generation < 1 or not 1 <= lease_seconds <= 86400:
raise ValueError("CREDENTIAL_LEASE_BINDING_INVALID")
if scope_type == "tenant":
if not tenant_id:
raise ValueError("CREDENTIAL_SCOPE_DENY")
elif scope_type in {"system", "global"}:
if any((tenant_id, user_id, project_id, run_id)):
raise ValueError("CREDENTIAL_SCOPE_DENY")
else:
raise ValueError("CREDENTIAL_SCOPE_DENY")
now = self.now().astimezone(timezone.utc)
lease = CredentialLease("lease-" + secrets.token_hex(16), task_id, repository, target_branch, generation, now.isoformat(), (now + timedelta(seconds=lease_seconds)).isoformat(), tuple(scopes), type(self.provider).__name__)
lease = CredentialLease("lease-" + secrets.token_hex(16), task_id, repository, target_branch, generation, now.isoformat(), (now + timedelta(seconds=lease_seconds)).isoformat(), tuple(scopes), type(self.provider).__name__, "ACTIVE", scope_type, tenant_id, user_id, project_id, run_id)
self._leases[lease.lease_id] = lease
return lease.as_dict()
@@ -64,7 +78,39 @@ class CredentialBroker:
lease = self._get(lease_id)
if lease.state != "ACTIVE":
raise RuntimeError("CREDENTIAL_LEASE_NOT_ACTIVE")
return self.provider.environment(lease)
environment = self.provider.environment(lease)
for key, value in environment.items():
upper = str(key).upper()
if not isinstance(value, str):
raise RuntimeError("CREDENTIAL_SECRET_EXPOSURE_DENY")
sensitive = any(marker in upper for marker in ("CREDENTIAL", "TOKEN", "SECRET", "PASSWORD", "PRIVATE_KEY", "API_KEY"))
if re.search(r"(?i)bearer\s+\S+|(?:token|secret|password|api[_-]?key)\s*[:=]", value):
raise RuntimeError("CREDENTIAL_SECRET_EXPOSURE_DENY")
if sensitive and upper.endswith("_FILE"):
candidate = Path(value)
credential_root = Path("/run/credentials")
try:
resolved = candidate.resolve(strict=False)
except OSError as exc:
raise RuntimeError("CREDENTIAL_SECRET_EXPOSURE_DENY") from exc
if not candidate.is_absolute() or not (resolved == credential_root or credential_root in resolved.parents):
raise RuntimeError("CREDENTIAL_SECRET_EXPOSURE_DENY")
elif sensitive and upper.endswith("_REFERENCE"):
if re.fullmatch(r"(?:systemd-credential|vault|secret-manager):[A-Za-z0-9][A-Za-z0-9._:/#-]{0,255}", value) is None:
raise RuntimeError("CREDENTIAL_SECRET_EXPOSURE_DENY")
elif sensitive:
raise RuntimeError("CREDENTIAL_SECRET_EXPOSURE_DENY")
return environment
def authorize(self, lease_id: str, *, task_id: str, operation: str, scope_type: str = "system", tenant_id: str | None = None, user_id: str | None = None, project_id: str | None = None, run_id: str | None = None) -> Mapping[str, str]:
lease = self._get(lease_id)
if lease.state != "ACTIVE" or lease.task_id != task_id or operation not in lease.scopes:
raise RuntimeError("CREDENTIAL_SCOPE_DENY")
requested = (scope_type, tenant_id, user_id, project_id, run_id)
actual = (lease.scope_type, lease.tenant_id, lease.user_id, lease.project_id, lease.run_id)
if requested != actual:
raise RuntimeError("CREDENTIAL_SCOPE_DENY")
return self.environment(lease_id)
def status(self, lease_id: str) -> dict[str, Any]:
return self._get(lease_id).as_dict()
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
@@ -27,6 +27,19 @@ class ServiceUnavailable(JobError):
pass
@dataclass(frozen=True, slots=True)
class LeaseRenewalResult:
"""Durable outcome of one writer-lease recovery attempt."""
renewed: bool
idempotent: bool
durable: bool
lease_expires_at: str | None
def __bool__(self) -> bool:
return self.renewed or self.idempotent
class BindingConflict(JobError):
pass
@@ -66,6 +79,14 @@ def utc_now() -> str:
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
def parse_utc(value: str) -> datetime:
normalized = value[:-1] + "+00:00" if value.endswith("Z") else value
parsed = datetime.fromisoformat(normalized)
if parsed.tzinfo is None:
raise ValueError("LEASE_EXPIRY_INVALID")
return parsed.astimezone(timezone.utc)
def canonical_json(value: Any) -> bytes:
return (
json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False) + "\n"
@@ -404,13 +425,16 @@ class GitOpsJobController:
self.store._mirror(result)
return self.store._receipt(result, duplicate=True)
def update_lease(self, lease_id: str, *, expected_revision: int, current_head_oid: str, generation: int, state: str = "ACTIVE") -> None:
def update_lease(self, lease_id: str, *, expected_revision: int, current_head_oid: str, generation: int, state: str = "ACTIVE", lease_expires_at: str | None = None) -> None:
if len(current_head_oid) != 40 or generation < 1:
raise JobError("LEASE_BINDING_INVALID")
try:
with self.store._connect() as connection:
connection.execute("BEGIN IMMEDIATE")
updated = connection.execute("UPDATE branch_leases SET current_head_oid=?,generation=?,state=?,revision=revision+1 WHERE lease_id=? AND revision=?", (current_head_oid, generation, state, lease_id, expected_revision))
if lease_expires_at is None:
updated = connection.execute("UPDATE branch_leases SET current_head_oid=?,generation=?,state=?,revision=revision+1 WHERE lease_id=? AND revision=?", (current_head_oid, generation, state, lease_id, expected_revision))
else:
updated = connection.execute("UPDATE branch_leases SET current_head_oid=?,generation=?,lease_expires_at=?,state=?,revision=revision+1 WHERE lease_id=? AND revision=?", (current_head_oid, generation, lease_expires_at, state, lease_id, expected_revision))
if updated.rowcount != 1:
raise JobError("LEASE_REVISION_CAS_CONFLICT")
connection.commit()
@@ -419,6 +443,94 @@ class GitOpsJobController:
except sqlite3.Error as exc:
raise ServiceUnavailable("JOB_DB_UNAVAILABLE") from exc
def renew_lease(
self,
*,
branch: str,
lineage_anchor_oid: str,
current_head_oid: str,
generation: int,
lease_expires_at: str,
) -> LeaseRenewalResult:
if (
not branch
or len(lineage_anchor_oid) != 40
or len(current_head_oid) != 40
or generation < 1
or not lease_expires_at.strip()
):
raise JobError("LEASE_BINDING_INVALID")
try:
with self.store._connect() as connection:
connection.execute("BEGIN IMMEDIATE")
rows = connection.execute(
"SELECT lease_id, revision, generation, state, lease_expires_at "
"FROM branch_leases "
"WHERE branch=? AND lineage_anchor_oid=? AND current_head_oid=? "
"AND generation=? AND state='ACTIVE' ORDER BY revision DESC",
(branch, lineage_anchor_oid, current_head_oid, generation),
).fetchall()
if not rows:
connection.commit()
return LeaseRenewalResult(False, False, False, None)
if len(rows) != 1:
raise JobError("LEASE_BINDING_CONFLICT")
row = rows[0]
try:
current_expires_at = parse_utc(str(row["lease_expires_at"]))
requested_expires_at = parse_utc(lease_expires_at)
except (TypeError, ValueError) as exc:
raise JobError("LEASE_EXPIRY_INVALID") from exc
now = datetime.now(timezone.utc)
if requested_expires_at <= now:
raise JobError("LEASE_EXPIRY_INVALID")
if current_expires_at > now:
connection.commit()
return LeaseRenewalResult(
False,
True,
True,
str(row["lease_expires_at"]),
)
updated = connection.execute(
"UPDATE branch_leases SET lease_expires_at=?,revision=revision+1 "
"WHERE lease_id=? AND revision=? AND generation=? "
"AND state='ACTIVE' AND lease_expires_at=?",
(
lease_expires_at,
row["lease_id"],
int(row["revision"]),
generation,
row["lease_expires_at"],
),
)
if updated.rowcount != 1:
current = connection.execute(
"SELECT lease_expires_at, generation, state FROM branch_leases "
"WHERE lease_id=?",
(row["lease_id"],),
).fetchone()
if (
current is not None
and int(current["generation"]) == generation
and current["state"] == "ACTIVE"
and parse_utc(str(current["lease_expires_at"])) > now
):
connection.commit()
return LeaseRenewalResult(
False,
True,
True,
str(current["lease_expires_at"]),
)
raise JobError("LEASE_REVISION_CAS_CONFLICT")
connection.commit()
return LeaseRenewalResult(True, False, True, lease_expires_at)
except JobError:
raise
except sqlite3.Error as exc:
raise ServiceUnavailable("JOB_DB_UNAVAILABLE") from exc
def advance(self, job_id: str, state: JobState, *, side_effect_state: str, receipt: Mapping[str, Any] | None = None, expected_revision: int | None = None) -> JobReceipt:
return self._advance(
job_id,
+456
View File
@@ -0,0 +1,456 @@
#!/usr/bin/env python3
"""Disabled-by-default TLS edge for the read-only GPT Online Task ingress.
The internal AIOS ingress remains loopback-only. This service is a separate edge
that may bind non-loopback only when the owner explicitly approves it. It
forwards only the tiny read-only ingress API and never proxies the canonical
Bridge or Task API.
"""
from __future__ import annotations
from collections import defaultdict, deque
from dataclasses import dataclass
import hashlib
import hmac
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
from pathlib import Path
import re
import secrets
import ssl
import sys
import threading
import time
import urllib.error
import urllib.request
from typing import Any, Mapping
from urllib.parse import urlsplit
UPSTREAM = "http://127.0.0.1:18945"
MAX_BODY = 64 * 1024
MAX_RESPONSE = 256 * 1024
MAX_CONCURRENT_REQUESTS = 32
REQUEST_TIMEOUT_SECONDS = 20
RATE_WINDOW_SECONDS = 60
RATE_LIMIT_REQUESTS = 60
RATE_BURST_REQUESTS = 15
RATE_MAX_CLIENTS = 4096
_AUDIT_SALT = secrets.token_bytes(32)
PUBLIC_APPROVAL_ENV = "AIOS_GPT_EDGE_PUBLIC_BIND_APPROVED"
PUBLIC_DOMAIN_ENV = "AIOS_GPT_EDGE_PUBLIC_DOMAIN"
TASK_PATH_RE = re.compile(r"^/v1/read-only-tasks(?:/[A-Za-z0-9][A-Za-z0-9._:-]{7,127})?$")
def _client_pseudonym(value: str) -> str:
return hashlib.sha256(_AUDIT_SALT + value.encode("utf-8", "replace")).hexdigest()[:16]
def audit(event: str, **fields: object) -> None:
safe = {
"event": event,
"ts_unix_ms": int(time.time() * 1000),
}
for key, value in fields.items():
if key in {"authorization", "body", "token", "cookie", "client_ip"}:
continue
safe[key] = value
print(
json.dumps(safe, sort_keys=True, separators=(",", ":")),
file=sys.stderr,
flush=True,
)
class EdgeError(RuntimeError):
def __init__(self, code: str, status: int = 400) -> None:
super().__init__(code)
self.code = code
self.status = status
def canonical(value: object) -> bytes:
return json.dumps(
value,
ensure_ascii=False,
sort_keys=True,
separators=(",", ":"),
).encode("utf-8")
def public_bind_allowed(host: str, approval: str) -> str:
value = host.strip()
if value in {"127.0.0.1", "::1", "localhost"}:
return value
if approval != "1":
raise RuntimeError("GPT_EDGE_PUBLIC_BIND_OWNER_APPROVAL_REQUIRED")
if value not in {"0.0.0.0", "::"}:
raise RuntimeError("GPT_EDGE_BIND_ADDRESS_INVALID")
return value
def validated_public_domain(value: str) -> str:
domain = value.strip().lower()
if not domain:
return ""
if len(domain) > 253 or re.fullmatch(
r"(?=.{1,253}\Z)(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?",
domain,
) is None:
raise RuntimeError("GPT_EDGE_PUBLIC_DOMAIN_INVALID")
return domain
def validated_edge_identity(host: str, approval: str, domain: str) -> tuple[str, str]:
bind_host = public_bind_allowed(host, approval)
public_domain = validated_public_domain(domain)
if bind_host not in {"127.0.0.1", "::1", "localhost"} and not public_domain:
raise RuntimeError("GPT_EDGE_PUBLIC_DOMAIN_REQUIRED")
return bind_host, public_domain
def validate_tls_file(path: Path, *, key: bool) -> Path:
if path.is_symlink() or not path.is_file():
raise RuntimeError("GPT_EDGE_TLS_FILE_INVALID")
size = path.stat().st_size
if size < 16 or size > 256 * 1024:
raise RuntimeError("GPT_EDGE_TLS_FILE_INVALID")
if key and (path.stat().st_mode & 0o077):
raise RuntimeError("GPT_EDGE_TLS_KEY_MODE_INVALID")
return path
def build_tls_context(cert_path: Path, key_path: Path) -> ssl.SSLContext:
cert = validate_tls_file(cert_path, key=False)
key = validate_tls_file(key_path, key=True)
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.minimum_version = ssl.TLSVersion.TLSv1_2
context.options |= ssl.OP_NO_COMPRESSION
context.load_cert_chain(certfile=str(cert), keyfile=str(key))
return context
def allowed_route(method: str, raw_path: str) -> str:
parsed = urlsplit(raw_path)
if parsed.query or parsed.fragment:
raise EdgeError("QUERY_OR_FRAGMENT_DENY", 400)
path = parsed.path
if method == "GET" and path in {"/health", "/openapi.json"}:
return path
if method == "POST" and path == "/v1/read-only-tasks":
return path
if method == "GET" and TASK_PATH_RE.fullmatch(path):
return path
raise EdgeError("ROUTE_DENY", 404)
def route_label(method: str, path: str) -> str:
if method == "GET" and path == "/health":
return "health"
if method == "GET" and path == "/openapi.json":
return "openapi"
if method == "POST" and path == "/v1/read-only-tasks":
return "task_create"
if method == "GET" and path.startswith("/v1/read-only-tasks/"):
return "task_status"
return "unknown"
def forwarded_headers(headers: Mapping[str, str], *, body: bool) -> dict[str, str]:
result = {"Accept": "application/json"}
authorization = headers.get("Authorization")
if not isinstance(authorization, str) or not authorization.startswith("Bearer "):
raise EdgeError("AUTHORIZATION_REQUIRED", 401)
token = authorization[7:]
if len(token) < 32 or len(token) > 4096 or any(ch.isspace() for ch in token):
raise EdgeError("AUTHORIZATION_INVALID", 401)
result["Authorization"] = authorization
if body:
content_type = headers.get("Content-Type", "")
if content_type.split(";", 1)[0].strip().lower() != "application/json":
raise EdgeError("CONTENT_TYPE_INVALID", 415)
result["Content-Type"] = "application/json"
return result
class RateLimiter:
def __init__(
self,
*,
window_seconds: int = RATE_WINDOW_SECONDS,
limit: int = RATE_LIMIT_REQUESTS,
burst: int = RATE_BURST_REQUESTS,
max_clients: int = RATE_MAX_CLIENTS,
now_fn=time.monotonic,
) -> None:
self.window_seconds = window_seconds
self.limit = limit
self.burst = burst
self.max_clients = max_clients
self.now_fn = now_fn
self._lock = threading.Lock()
self._events: dict[str, deque[float]] = defaultdict(deque)
def _prune_stale(self, cutoff: float) -> None:
stale = [
key
for key, events in self._events.items()
if not events or events[-1] <= cutoff
]
for key in stale:
self._events.pop(key, None)
def allowed(self, key: str) -> bool:
now = self.now_fn()
cutoff = now - self.window_seconds
with self._lock:
if key not in self._events and len(self._events) >= self.max_clients:
self._prune_stale(cutoff)
if len(self._events) >= self.max_clients:
return False
events = self._events[key]
while events and events[0] <= cutoff:
events.popleft()
if len(events) >= self.limit + self.burst:
return False
events.append(now)
return True
@dataclass(frozen=True)
class Proxy:
upstream: str = UPSTREAM
timeout: int = REQUEST_TIMEOUT_SECONDS
def request(
self,
method: str,
path: str,
headers: Mapping[str, str],
body: bytes | None,
) -> tuple[int, bytes, str]:
if self.upstream != UPSTREAM:
raise EdgeError("UPSTREAM_DRIFT_DENY", 503)
req = urllib.request.Request(
self.upstream + path,
method=method,
data=body,
headers=dict(headers),
)
try:
with urllib.request.urlopen(req, timeout=self.timeout) as response:
raw = response.read(MAX_RESPONSE + 1)
status = response.status
content_type = response.headers.get("Content-Type", "application/json")
except urllib.error.HTTPError as exc:
raw = exc.read(MAX_RESPONSE + 1)
status = exc.code
content_type = exc.headers.get("Content-Type", "application/json")
except Exception as exc:
raise EdgeError("UPSTREAM_UNAVAILABLE", 502) from exc
if len(raw) > MAX_RESPONSE:
raise EdgeError("UPSTREAM_RESPONSE_TOO_LARGE", 502)
if content_type.split(";", 1)[0].strip().lower() != "application/json":
raise EdgeError("UPSTREAM_CONTENT_TYPE_INVALID", 502)
return status, raw, "application/json"
class EdgeServer(ThreadingHTTPServer):
daemon_threads = True
request_queue_size = 64
allow_reuse_address = True
class Handler(BaseHTTPRequestHandler):
proxy = Proxy()
limiter = RateLimiter()
concurrency = threading.BoundedSemaphore(MAX_CONCURRENT_REQUESTS)
expected_domain = ""
server_version = "AIOSGPTIngressEdge/1.0"
sys_version = ""
def log_message(self, *_args: Any) -> None:
return
def respond_json(self, value: Mapping[str, Any], status: int) -> None:
raw = canonical(value)
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Cache-Control", "no-store")
self.send_header("X-Content-Type-Options", "nosniff")
self.send_header("Referrer-Policy", "no-referrer")
self.send_header("Content-Security-Policy", "default-src 'none'")
self.send_header("Strict-Transport-Security", "max-age=31536000")
self.send_header("Content-Length", str(len(raw)))
self.end_headers()
self.wfile.write(raw)
def send_proxy_response(self, status: int, raw: bytes, content_type: str) -> None:
self.send_response(status)
self.send_header("Content-Type", content_type)
self.send_header("Cache-Control", "no-store")
self.send_header("X-Content-Type-Options", "nosniff")
self.send_header("Referrer-Policy", "no-referrer")
self.send_header("Content-Security-Policy", "default-src 'none'")
self.send_header("Strict-Transport-Security", "max-age=31536000")
self.send_header("Content-Length", str(len(raw)))
self.end_headers()
self.wfile.write(raw)
def client_key(self) -> str:
return str(self.client_address[0])
def validate_host(self) -> None:
if not self.expected_domain:
return
host = self.headers.get("Host", "").split(":", 1)[0].lower()
if not hmac.compare_digest(host, self.expected_domain):
raise EdgeError("HOST_HEADER_DENY", 421)
def body(self) -> bytes:
try:
length = int(self.headers.get("Content-Length", "-1"))
except ValueError as exc:
raise EdgeError("CONTENT_LENGTH_INVALID") from exc
if length < 1 or length > MAX_BODY:
raise EdgeError("REQUEST_SIZE_INVALID", 413)
raw = self.rfile.read(length)
if len(raw) != length:
raise EdgeError("REQUEST_BODY_TRUNCATED", 400)
# Validate JSON at the edge so malformed bodies do not consume upstream work.
try:
value = json.loads(raw.decode("utf-8"))
except Exception as exc:
raise EdgeError("REQUEST_JSON_INVALID", 400) from exc
if not isinstance(value, Mapping):
raise EdgeError("REQUEST_JSON_OBJECT_REQUIRED", 400)
return canonical(value)
def handle_proxy(self, method: str) -> None:
client_id = _client_pseudonym(self.client_key())
started = time.monotonic()
acquired = self.concurrency.acquire(blocking=False)
if not acquired:
audit("EDGE_BUSY", client_id=client_id, method=method)
self.respond_json({"error": "EDGE_BUSY"}, 503)
return
label = "unknown"
try:
self.validate_host()
path = allowed_route(method, self.path)
label = route_label(method, path)
if path == "/health" and method == "GET":
self.respond_json(
{
"status": "ok",
"service": "aios-gpt-https-edge",
"upstream": "loopback-readonly-ingress",
},
200,
)
audit(
"EDGE_REQUEST",
client_id=client_id,
method=method,
route=label,
status=200,
duration_ms=int((time.monotonic() - started) * 1000),
)
return
if not self.limiter.allowed(self.client_key()):
audit(
"EDGE_RATE_LIMIT",
client_id=client_id,
method=method,
route=label,
)
self.respond_json({"error": "RATE_LIMITED"}, 429)
return
has_body = method == "POST"
headers = forwarded_headers(self.headers, body=has_body)
body = self.body() if has_body else None
status, raw, content_type = self.proxy.request(
method,
path,
headers,
body,
)
self.send_proxy_response(status, raw, content_type)
audit(
"EDGE_REQUEST",
client_id=client_id,
method=method,
route=label,
status=status,
duration_ms=int((time.monotonic() - started) * 1000),
)
except EdgeError as exc:
audit(
"EDGE_REJECT",
client_id=client_id,
method=method,
route=label,
status=exc.status,
error=exc.code,
)
self.respond_json({"error": exc.code}, exc.status)
finally:
self.concurrency.release()
def do_GET(self) -> None:
self.handle_proxy("GET")
def do_POST(self) -> None:
self.handle_proxy("POST")
def do_PUT(self) -> None:
self.respond_json({"error": "METHOD_DENY"}, 405)
def do_PATCH(self) -> None:
self.respond_json({"error": "METHOD_DENY"}, 405)
def do_DELETE(self) -> None:
self.respond_json({"error": "METHOD_DENY"}, 405)
def do_CONNECT(self) -> None:
self.respond_json({"error": "METHOD_DENY"}, 405)
def do_OPTIONS(self) -> None:
self.respond_json({"error": "METHOD_DENY"}, 405)
def serve() -> None:
host, public_domain = validated_edge_identity(
os.environ.get("AIOS_GPT_EDGE_HOST", "127.0.0.1"),
os.environ.get(PUBLIC_APPROVAL_ENV, "0"),
os.environ.get(PUBLIC_DOMAIN_ENV, ""),
)
port = int(os.environ.get("AIOS_GPT_EDGE_PORT", "18443"))
if not 1024 <= port <= 65535:
raise RuntimeError("GPT_EDGE_PORT_INVALID")
Handler.expected_domain = public_domain
credentials = Path(os.environ.get("CREDENTIALS_DIRECTORY", "/run/credentials"))
cert_path = Path(
os.environ.get(
"AIOS_GPT_EDGE_TLS_CERT_FILE",
str(credentials / "tls_cert"),
)
)
key_path = Path(
os.environ.get(
"AIOS_GPT_EDGE_TLS_KEY_FILE",
str(credentials / "tls_key"),
)
)
context = build_tls_context(cert_path, key_path)
server = EdgeServer((host, port), Handler)
server.socket = context.wrap_socket(server.socket, server_side=True)
server.serve_forever()
if __name__ == "__main__":
serve()
@@ -0,0 +1,561 @@
#!/usr/bin/env python3
"""Owner-approved activation/rollback controller for the AIOS GPT HTTPS edge.
This controller never creates approval, TLS, or ingress credentials. It consumes
owner-managed inputs, applies one exact public-bind overlay, verifies the live
edge, and provides an idempotent rollback path.
"""
from __future__ import annotations
from dataclasses import dataclass
from datetime import datetime, timezone
import argparse
import hashlib
import json
import os
from pathlib import Path
import re
import socket
import ssl
import stat
import subprocess
import tempfile
import time
import urllib.request
from typing import Any, Mapping
APPROVAL_SCHEMA = "aios.gpt-https-edge.public-approval.v1"
STATE_SCHEMA = "aios.gpt-https-edge.activation-state.v1"
OID_RE = re.compile(r"^[0-9a-f]{40}$")
SHA_RE = re.compile(r"^[0-9a-f]{64}$")
APPROVAL_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$")
DOMAIN_RE = re.compile(
r"(?=.{1,253}\Z)"
r"(?:[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?\.)+"
r"[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?"
)
EDGE_UNIT = "aios-gpt-https-edge.service"
INGRESS_UNIT = "aios-gpt-online-task-ingress.service"
PUBLIC_HOST = "0.0.0.0"
PUBLIC_PORT = 18443
INGRESS_HEALTH = "http://127.0.0.1:18945/health"
DROPIN_NAME = "zzzzzzzzzz-owner-approved-public.conf"
class ActivationError(RuntimeError):
pass
def canonical(value: object) -> bytes:
return json.dumps(
value,
ensure_ascii=False,
sort_keys=True,
separators=(",", ":"),
).encode("utf-8")
def sha256_bytes(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def sha256_file(path: Path) -> str:
return sha256_bytes(path.read_bytes())
def parse_time(value: object, *, field: str) -> datetime:
if not isinstance(value, str) or not value.strip():
raise ActivationError("APPROVAL_" + field.upper() + "_INVALID")
text = value.strip()
if text.endswith("Z"):
text = text[:-1] + "+00:00"
try:
parsed = datetime.fromisoformat(text)
except ValueError as exc:
raise ActivationError("APPROVAL_" + field.upper() + "_INVALID") from exc
if parsed.tzinfo is None:
raise ActivationError("APPROVAL_" + field.upper() + "_INVALID")
return parsed.astimezone(timezone.utc)
def validate_domain(value: object) -> str:
if not isinstance(value, str):
raise ActivationError("APPROVAL_DOMAIN_INVALID")
domain = value.strip().lower()
if DOMAIN_RE.fullmatch(domain) is None:
raise ActivationError("APPROVAL_DOMAIN_INVALID")
return domain
def validate_root_file(
path: Path,
*,
exact_mode: int | None = None,
max_size: int = 256 * 1024,
) -> Path:
if path.is_symlink() or not path.is_file():
raise ActivationError("OWNER_FILE_INVALID")
info = path.stat()
if info.st_uid != 0 or info.st_gid != 0:
raise ActivationError("OWNER_FILE_OWNERSHIP_INVALID")
mode = stat.S_IMODE(info.st_mode)
if exact_mode is not None and mode != exact_mode:
raise ActivationError("OWNER_FILE_MODE_INVALID")
if info.st_size < 16 or info.st_size > max_size:
raise ActivationError("OWNER_FILE_SIZE_INVALID")
return path
def certificate_dns_names(cert_path: Path) -> tuple[str, ...]:
try:
decoded = ssl._ssl._test_decode_cert(str(cert_path))
except Exception as exc:
raise ActivationError("TLS_CERT_DECODE_INVALID") from exc
names = [
value
for kind, value in decoded.get("subjectAltName", ())
if kind == "DNS"
]
if not names:
for rdn in decoded.get("subject", ()):
for key, value in rdn:
if key == "commonName":
names.append(value)
return tuple(names)
def cert_matches_domain(cert_path: Path, domain: str) -> bool:
names = certificate_dns_names(cert_path)
try:
return bool(names) and any(ssl._dnsname_match(name, domain) for name in names)
except ssl.CertificateError:
return False
def validate_tls_pair(
cert_path: Path,
key_path: Path,
domain: str,
*,
now_fn=time.time,
) -> dict[str, Any]:
validate_root_file(cert_path)
validate_root_file(key_path, exact_mode=0o600)
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.minimum_version = ssl.TLSVersion.TLSv1_2
context.options |= ssl.OP_NO_COMPRESSION
try:
context.load_cert_chain(str(cert_path), str(key_path))
except Exception as exc:
raise ActivationError("TLS_CERT_KEY_PAIR_INVALID") from exc
if not cert_matches_domain(cert_path, domain):
raise ActivationError("TLS_CERT_DOMAIN_MISMATCH")
decoded = ssl._ssl._test_decode_cert(str(cert_path))
not_after = decoded.get("notAfter")
if not not_after:
raise ActivationError("TLS_CERT_EXPIRY_MISSING")
remaining = ssl.cert_time_to_seconds(not_after) - now_fn()
if remaining < 7 * 86400:
raise ActivationError("TLS_CERT_EXPIRY_TOO_CLOSE")
return {
"cert_sha256": sha256_file(cert_path),
"remaining_days": int(remaining // 86400),
}
@dataclass(frozen=True)
class Approval:
approval_id: str
candidate_oid: str
domain: str
bind_host: str
port: int
tls_cert_sha256: str
owner: str
issued_at: datetime
not_before: datetime
expires_at: datetime
raw_sha256: str
@classmethod
def load(
cls,
path: Path,
*,
candidate_oid: str,
cert_path: Path,
now: datetime | None = None,
) -> "Approval":
validate_root_file(path, exact_mode=0o600, max_size=64 * 1024)
try:
value = json.loads(path.read_text(encoding="utf-8"))
except Exception as exc:
raise ActivationError("APPROVAL_JSON_INVALID") from exc
if not isinstance(value, Mapping):
raise ActivationError("APPROVAL_JSON_INVALID")
allowed = {
"schema",
"approval_id",
"candidate_oid",
"domain",
"public_bind",
"bind_host",
"port",
"tls_cert_sha256",
"issued_at",
"not_before",
"expires_at",
"owner",
}
if set(value) != allowed or value.get("schema") != APPROVAL_SCHEMA:
raise ActivationError("APPROVAL_FIELDS_INVALID")
approval_id = value.get("approval_id")
if not isinstance(approval_id, str) or APPROVAL_ID_RE.fullmatch(approval_id) is None:
raise ActivationError("APPROVAL_ID_INVALID")
if not OID_RE.fullmatch(str(value.get("candidate_oid") or "")):
raise ActivationError("APPROVAL_CANDIDATE_INVALID")
if value["candidate_oid"] != candidate_oid:
raise ActivationError("APPROVAL_CANDIDATE_MISMATCH")
if value.get("public_bind") is not True:
raise ActivationError("APPROVAL_PUBLIC_BIND_REQUIRED")
if value.get("bind_host") != PUBLIC_HOST or value.get("port") != PUBLIC_PORT:
raise ActivationError("APPROVAL_BINDING_INVALID")
cert_sha = str(value.get("tls_cert_sha256") or "")
if SHA_RE.fullmatch(cert_sha) is None:
raise ActivationError("APPROVAL_CERT_SHA_INVALID")
if not cert_path.is_file() or sha256_file(cert_path) != cert_sha:
raise ActivationError("APPROVAL_CERT_SHA_MISMATCH")
domain = validate_domain(value.get("domain"))
owner = value.get("owner")
if not isinstance(owner, str) or not owner.strip() or len(owner) > 255:
raise ActivationError("APPROVAL_OWNER_INVALID")
issued_at = parse_time(value.get("issued_at"), field="issued_at")
not_before = parse_time(value.get("not_before"), field="not_before")
expires_at = parse_time(value.get("expires_at"), field="expires_at")
current = now or datetime.now(timezone.utc)
if not (issued_at <= not_before <= expires_at):
raise ActivationError("APPROVAL_TIME_ORDER_INVALID")
if current < not_before:
raise ActivationError("APPROVAL_NOT_YET_VALID")
if current >= expires_at:
raise ActivationError("APPROVAL_EXPIRED")
if expires_at - issued_at > __import__("datetime").timedelta(days=7):
raise ActivationError("APPROVAL_VALIDITY_TOO_LONG")
return cls(
approval_id=approval_id,
candidate_oid=candidate_oid,
domain=domain,
bind_host=PUBLIC_HOST,
port=PUBLIC_PORT,
tls_cert_sha256=cert_sha,
owner=owner.strip(),
issued_at=issued_at,
not_before=not_before,
expires_at=expires_at,
raw_sha256=sha256_file(path),
)
def validate_release(release_root: Path) -> dict[str, Path]:
if release_root.is_symlink() or not release_root.is_dir():
raise ActivationError("IMMUTABLE_RELEASE_INVALID")
edge = release_root / "active-runtime/app/aios_gpt_https_edge.py"
controller = release_root / "active-runtime/app/aios_gpt_https_edge_activation.py"
for path in (edge, controller):
if path.is_symlink() or not path.is_file():
raise ActivationError("IMMUTABLE_RELEASE_INVALID")
return {"edge": edge, "controller": controller}
def render_dropin(approval: Approval, release_root: Path) -> str:
app = release_root / "active-runtime/app/aios_gpt_https_edge.py"
return (
"[Service]\n"
f"Environment=AIOS_GPT_EDGE_HOST={approval.bind_host}\n"
f"Environment=AIOS_GPT_EDGE_PORT={approval.port}\n"
"Environment=AIOS_GPT_EDGE_PUBLIC_BIND_APPROVED=1\n"
f"Environment=AIOS_GPT_EDGE_PUBLIC_DOMAIN={approval.domain}\n"
"ExecStart=\n"
f"ExecStart=/usr/bin/python3 -B {app}\n"
f"ReadOnlyPaths={release_root}\n"
)
def atomic_write(path: Path, data: str, mode: int = 0o600) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
fd, name = tempfile.mkstemp(prefix="." + path.name + ".", dir=path.parent)
try:
os.fchmod(fd, mode)
with os.fdopen(fd, "w", encoding="utf-8") as handle:
handle.write(data)
handle.flush()
os.fsync(handle.fileno())
os.replace(name, path)
finally:
if os.path.exists(name):
os.unlink(name)
class Systemd:
def run(self, *args: str, check: bool = True) -> subprocess.CompletedProcess[str]:
return subprocess.run(
["/usr/bin/systemctl", *args],
check=check,
text=True,
capture_output=True,
timeout=30,
)
def active(self, unit: str) -> bool:
return self.run("is-active", "--quiet", unit, check=False).returncode == 0
def enabled(self, unit: str) -> bool:
return self.run("is-enabled", "--quiet", unit, check=False).returncode == 0
def show(self, unit: str, prop: str) -> str:
return self.run("show", unit, f"--property={prop}", "--value").stdout.strip()
def require_ingress_ready(systemd: Systemd) -> None:
if not systemd.active(INGRESS_UNIT):
raise ActivationError("INGRESS_SERVICE_NOT_ACTIVE")
try:
with urllib.request.urlopen(INGRESS_HEALTH, timeout=5) as response:
if response.status != 200:
raise ActivationError("INGRESS_HEALTH_FAILED")
value = json.loads(response.read().decode())
except ActivationError:
raise
except Exception as exc:
raise ActivationError("INGRESS_HEALTH_FAILED") from exc
if value.get("status") != "ok":
raise ActivationError("INGRESS_HEALTH_FAILED")
def tls_probe(domain: str, cert_path: Path, *, port: int = PUBLIC_PORT) -> None:
context = ssl.create_default_context(cafile=str(cert_path))
context.check_hostname = True
context.verify_mode = ssl.CERT_REQUIRED
try:
with socket.create_connection(("127.0.0.1", port), timeout=5) as sock:
with context.wrap_socket(sock, server_hostname=domain) as tls_sock:
if tls_sock.version() not in {"TLSv1.2", "TLSv1.3"}:
raise ActivationError("EDGE_TLS_VERSION_INVALID")
except ActivationError:
raise
except Exception as exc:
raise ActivationError("EDGE_TLS_PROBE_FAILED") from exc
def receipt_for(
approval: Approval,
release_root: Path,
*,
state: str,
) -> dict[str, Any]:
return {
"schema": STATE_SCHEMA,
"state": state,
"approval_id": approval.approval_id,
"approval_sha256": approval.raw_sha256,
"candidate_oid": approval.candidate_oid,
"domain": approval.domain,
"bind_host": approval.bind_host,
"port": approval.port,
"tls_cert_sha256": approval.tls_cert_sha256,
"release_root": str(release_root),
"updated_at": datetime.now(timezone.utc).isoformat(),
}
def activate(
*,
approval: Approval,
release_root: Path,
cert_path: Path,
key_path: Path,
dropin_path: Path,
state_path: Path,
systemd: Systemd,
) -> dict[str, Any]:
if os.geteuid() != 0:
raise ActivationError("ROOT_REQUIRED")
release = validate_release(release_root)
app = release["edge"]
require_ingress_ready(systemd)
validate_tls_pair(cert_path, key_path, approval.domain)
if state_path.is_file():
try:
existing = json.loads(state_path.read_text(encoding="utf-8"))
except Exception:
existing = {}
if (
existing.get("schema") == STATE_SCHEMA
and existing.get("state") == "ACTIVE"
and existing.get("approval_sha256") == approval.raw_sha256
and existing.get("candidate_oid") == approval.candidate_oid
and systemd.active(EDGE_UNIT)
):
return existing
dropin = render_dropin(approval, release_root)
modified = False
try:
atomic_write(dropin_path, dropin, mode=0o644)
modified = True
systemd.run("daemon-reload")
systemd.run("enable", EDGE_UNIT)
systemd.run("restart", EDGE_UNIT)
if not systemd.active(EDGE_UNIT):
raise ActivationError("EDGE_SERVICE_NOT_ACTIVE")
exec_start = systemd.show(EDGE_UNIT, "ExecStart")
environment = systemd.show(EDGE_UNIT, "Environment")
if str(app) not in exec_start:
raise ActivationError("EDGE_EFFECTIVE_EXECSTART_INVALID")
for expected in (
f"AIOS_GPT_EDGE_HOST={approval.bind_host}",
f"AIOS_GPT_EDGE_PORT={approval.port}",
"AIOS_GPT_EDGE_PUBLIC_BIND_APPROVED=1",
f"AIOS_GPT_EDGE_PUBLIC_DOMAIN={approval.domain}",
):
if expected not in environment:
raise ActivationError("EDGE_EFFECTIVE_ENVIRONMENT_INVALID")
tls_probe(approval.domain, cert_path, port=approval.port)
receipt = receipt_for(approval, release_root, state="ACTIVE")
atomic_write(state_path, json.dumps(receipt, indent=2, sort_keys=True) + "\n")
return receipt
except Exception:
if modified:
rollback(
dropin_path=dropin_path,
state_path=state_path,
systemd=systemd,
reason="activation_failed",
)
raise
def rollback(
*,
dropin_path: Path,
state_path: Path,
systemd: Systemd,
reason: str,
) -> dict[str, Any]:
if os.geteuid() != 0:
raise ActivationError("ROOT_REQUIRED")
systemd.run("stop", EDGE_UNIT, check=False)
systemd.run("disable", EDGE_UNIT, check=False)
dropin_path.unlink(missing_ok=True)
systemd.run("daemon-reload")
if systemd.active(EDGE_UNIT):
raise ActivationError("EDGE_ROLLBACK_SERVICE_STILL_ACTIVE")
previous: dict[str, Any] = {}
if state_path.is_file():
try:
previous = json.loads(state_path.read_text(encoding="utf-8"))
except Exception:
previous = {}
receipt = {
"schema": STATE_SCHEMA,
"state": "ROLLED_BACK",
"reason": reason[:255],
"candidate_oid": previous.get("candidate_oid"),
"approval_id": previous.get("approval_id"),
"domain": previous.get("domain"),
"updated_at": datetime.now(timezone.utc).isoformat(),
}
atomic_write(state_path, json.dumps(receipt, indent=2, sort_keys=True) + "\n")
return receipt
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("command", choices=("check", "activate", "rollback"))
parser.add_argument("--candidate")
parser.add_argument("--approval", default="/etc/aios-gitops/gpt-https-edge.approved")
parser.add_argument("--tls-cert", default="/etc/aios-gitops/gpt-https-edge-tls.crt")
parser.add_argument("--tls-key", default="/etc/aios-gitops/gpt-https-edge-tls.key")
parser.add_argument("--release-root")
parser.add_argument(
"--dropin",
default=f"/etc/systemd/system/{EDGE_UNIT}.d/{DROPIN_NAME}",
)
parser.add_argument(
"--state",
default="/var/lib/aios-gpt-https-edge-activation/state.json",
)
parser.add_argument("--reason", default="owner_requested")
args = parser.parse_args()
systemd = Systemd()
if args.command == "rollback":
value = rollback(
dropin_path=Path(args.dropin),
state_path=Path(args.state),
systemd=systemd,
reason=args.reason,
)
print(json.dumps(value, indent=2, sort_keys=True))
return 0
candidate = str(args.candidate or "")
if OID_RE.fullmatch(candidate) is None:
raise SystemExit("CANDIDATE_REQUIRED")
release_root = (
Path(args.release_root)
if args.release_root
else Path(f"/opt/aios/releases/{candidate}-gpt-https-edge-public")
)
cert_path = Path(args.tls_cert)
key_path = Path(args.tls_key)
approval = Approval.load(
Path(args.approval),
candidate_oid=candidate,
cert_path=cert_path,
)
tls = validate_tls_pair(cert_path, key_path, approval.domain)
release = validate_release(release_root)
require_ingress_ready(systemd)
if args.command == "check":
value = {
"status": "READY",
"candidate_oid": candidate,
"approval_id": approval.approval_id,
"approval_sha256": approval.raw_sha256,
"domain": approval.domain,
"bind_host": approval.bind_host,
"port": approval.port,
"edge_sha256": sha256_file(release["edge"]),
"controller_sha256": sha256_file(release["controller"]),
**tls,
"public_network_mutation": False,
}
print(json.dumps(value, indent=2, sort_keys=True))
return 0
value = activate(
approval=approval,
release_root=release_root,
cert_path=cert_path,
key_path=key_path,
dropin_path=Path(args.dropin),
state_path=Path(args.state),
systemd=systemd,
)
print(json.dumps(value, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,816 @@
#!/usr/bin/env python3
"""Credential-isolated host-side ingress for externally scheduled read-only Online Tasks.
This service is intentionally not a public AIOS control plane. It accepts a tiny,
authenticated external intent and translates it to the canonical loopback Bridge.
Callers cannot select repository coordinates, capabilities, execution action,
worker, branch policy, credentials, verification decisions, or write permissions.
"""
from __future__ import annotations
from dataclasses import dataclass
from datetime import datetime, timezone
import hashlib
import hmac
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import json
import os
from pathlib import Path
import re
import sqlite3
import threading
import urllib.error
import urllib.request
from typing import Any, Mapping
SCHEMA = "aios.gpt-online-task-ingress.request.v1"
RESPONSE_SCHEMA = "aios.gpt-online-task-ingress.response.v1"
OPENAPI_VERSION = "3.1.0"
REQUEST_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{7,127}$")
TASK_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{2,179}$")
JOB_ID_RE = re.compile(r"^job-[0-9a-f]{32}$")
OID_RE = re.compile(r"^[0-9a-f]{40}$")
MAX_BODY = 64 * 1024
MAX_GOAL = 12_000
MAX_INSTRUCTIONS = 24_000
MAX_CRITERIA = 16
MAX_CRITERION_TEXT = 1_000
READONLY_CAPABILITIES = ("git.read", "git.test")
REPOSITORY = "aios/aios"
class IngressError(RuntimeError):
def __init__(self, code: str, status: int = 400) -> None:
super().__init__(code)
self.code = code
self.status = status
def canonical(value: object) -> bytes:
return json.dumps(
value,
ensure_ascii=False,
sort_keys=True,
separators=(",", ":"),
).encode("utf-8")
def digest(value: object) -> str:
return hashlib.sha256(canonical(value)).hexdigest()
def now() -> str:
return datetime.now(timezone.utc).isoformat()
def load_secret(path: Path, *, code: str) -> str:
if path.is_symlink() or not path.is_file():
raise RuntimeError(code + "_UNAVAILABLE")
value = path.read_text(encoding="utf-8").strip()
if len(value) < 32 or "\n" in value or "\r" in value:
raise RuntimeError(code + "_INVALID")
return value
def normalized_request(value: Mapping[str, Any]) -> dict[str, Any]:
allowed = {
"schema",
"request_id",
"goal",
"instructions",
"success_criteria",
"constraints",
}
if set(value) - allowed:
raise IngressError("UNKNOWN_FIELDS")
if value.get("schema") != SCHEMA:
raise IngressError("SCHEMA_INVALID")
request_id = value.get("request_id")
if not isinstance(request_id, str) or REQUEST_ID_RE.fullmatch(request_id) is None:
raise IngressError("REQUEST_ID_INVALID")
goal = value.get("goal")
if not isinstance(goal, str) or not goal.strip() or len(goal) > MAX_GOAL:
raise IngressError("GOAL_INVALID")
instructions = value.get("instructions", "")
if not isinstance(instructions, str) or len(instructions) > MAX_INSTRUCTIONS:
raise IngressError("INSTRUCTIONS_INVALID")
criteria = value.get("success_criteria") or []
if not isinstance(criteria, list) or len(criteria) > MAX_CRITERIA:
raise IngressError("SUCCESS_CRITERIA_INVALID")
normalized_criteria = []
for item in criteria:
if not isinstance(item, Mapping) or set(item) != {"id", "description"}:
raise IngressError("SUCCESS_CRITERIA_INVALID")
criterion_id = item.get("id")
description = item.get("description")
if (
not isinstance(criterion_id, str)
or not criterion_id.strip()
or len(criterion_id) > 120
or not isinstance(description, str)
or not description.strip()
or len(description) > MAX_CRITERION_TEXT
):
raise IngressError("SUCCESS_CRITERIA_INVALID")
normalized_criteria.append(
{"id": criterion_id.strip(), "description": description.strip()}
)
constraints = value.get("constraints") or {}
if not isinstance(constraints, Mapping):
raise IngressError("CONSTRAINTS_INVALID")
allowed_constraints = {
"max_files",
"focus_paths",
"notes",
}
if set(constraints) - allowed_constraints:
raise IngressError("CONSTRAINTS_FIELDS_INVALID")
normalized_constraints: dict[str, Any] = {}
if "max_files" in constraints:
max_files = constraints["max_files"]
if (
not isinstance(max_files, int)
or isinstance(max_files, bool)
or not 1 <= max_files <= 500
):
raise IngressError("MAX_FILES_INVALID")
normalized_constraints["max_files"] = max_files
if "focus_paths" in constraints:
paths = constraints["focus_paths"]
if (
not isinstance(paths, list)
or len(paths) > 32
or not all(
isinstance(path, str)
and path
and len(path) <= 512
and not path.startswith("/")
and ".." not in Path(path).parts
for path in paths
)
):
raise IngressError("FOCUS_PATHS_INVALID")
normalized_constraints["focus_paths"] = paths
if "notes" in constraints:
notes = constraints["notes"]
if not isinstance(notes, str) or len(notes) > 4_000:
raise IngressError("CONSTRAINT_NOTES_INVALID")
normalized_constraints["notes"] = notes
return {
"schema": SCHEMA,
"request_id": request_id,
"goal": goal.strip(),
"instructions": instructions,
"success_criteria": normalized_criteria,
"constraints": normalized_constraints,
}
def validated_bind_host(value: str) -> str:
host = value.strip()
if host not in {"127.0.0.1", "::1", "localhost"}:
raise RuntimeError("GPT_INGRESS_NON_LOOPBACK_BIND_DENY")
return host
def task_id_for(request_id: str) -> str:
suffix = hashlib.sha256(request_id.encode("utf-8")).hexdigest()[:24]
return "gpt-readonly-" + suffix
def branch_for(task_id: str) -> str:
return "agent/external-gpt-readonly/" + task_id[-24:]
@dataclass(frozen=True)
class Upstream:
bridge_base: str
task_base: str
control_token: str
timeout: int = 20
def _request(
self,
base: str,
method: str,
path: str,
body: Mapping[str, Any] | None = None,
) -> tuple[int, dict[str, Any]]:
raw_body = None if body is None else canonical(body)
headers = {
"Authorization": "Bearer " + self.control_token,
"Accept": "application/json",
}
if raw_body is not None:
headers["Content-Type"] = "application/json"
req = urllib.request.Request(
base.rstrip("/") + path,
method=method,
data=raw_body,
headers=headers,
)
try:
with urllib.request.urlopen(req, timeout=self.timeout) as response:
raw = response.read()
status = response.status
except urllib.error.HTTPError as exc:
raw = exc.read()
status = exc.code
except Exception as exc:
raise IngressError("UPSTREAM_TRANSPORT_FAILED", 503) from exc
try:
value = json.loads(raw.decode("utf-8"))
except Exception as exc:
raise IngressError("UPSTREAM_RESPONSE_INVALID", 502) from exc
if not isinstance(value, Mapping):
raise IngressError("UPSTREAM_RESPONSE_INVALID", 502)
return status, dict(value)
def control_snapshot(self) -> dict[str, Any]:
status, value = self._request(
self.bridge_base,
"GET",
"/v2/control-plane-snapshot",
)
if status != 200:
raise IngressError("CONTROL_SNAPSHOT_UNAVAILABLE", 503)
return value
def create_governed_task(
self,
body: Mapping[str, Any],
) -> tuple[int, dict[str, Any]]:
return self._request(
self.bridge_base,
"POST",
"/v2/governed-task",
body,
)
def execution_envelope(
self,
body: Mapping[str, Any],
) -> tuple[int, dict[str, Any]]:
return self._request(
self.bridge_base,
"POST",
"/v2/execution-envelope",
body,
)
def get_task(self, task_id: str) -> tuple[int, dict[str, Any]]:
return self._request(
self.task_base,
"GET",
"/v1/tasks/" + task_id,
)
class Store:
def __init__(self, path: Path) -> None:
self.path = Path(path)
self._lock = threading.RLock()
self.path.parent.mkdir(parents=True, exist_ok=True)
with self._connect() as con:
con.execute(
"""
CREATE TABLE IF NOT EXISTS external_task_requests(
request_id TEXT PRIMARY KEY,
request_sha256 TEXT NOT NULL,
task_id TEXT NOT NULL UNIQUE,
job_id TEXT,
base_oid TEXT,
state TEXT NOT NULL,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
response_json TEXT
)
"""
)
con.commit()
def _connect(self) -> sqlite3.Connection:
con = sqlite3.connect(self.path, timeout=30)
con.row_factory = sqlite3.Row
con.execute("PRAGMA busy_timeout=30000")
return con
def get(self, request_id: str) -> sqlite3.Row | None:
with self._connect() as con:
return con.execute(
"SELECT * FROM external_task_requests WHERE request_id=?",
(request_id,),
).fetchone()
def reserve(self, request: Mapping[str, Any], task_id: str) -> sqlite3.Row:
request_sha = digest(request)
timestamp = now()
with self._lock, self._connect() as con:
con.execute("BEGIN IMMEDIATE")
row = con.execute(
"SELECT * FROM external_task_requests WHERE request_id=?",
(request["request_id"],),
).fetchone()
if row is not None:
if row["request_sha256"] != request_sha:
raise IngressError("IDEMPOTENCY_CONFLICT", 409)
con.commit()
return row
con.execute(
"""
INSERT INTO external_task_requests(
request_id,request_sha256,task_id,state,created_at,updated_at
) VALUES(?,?,?,?,?,?)
""",
(
request["request_id"],
request_sha,
task_id,
"PENDING",
timestamp,
timestamp,
),
)
con.commit()
row = self.get(request["request_id"])
assert row is not None
return row
def bound(
self,
request_id: str,
*,
job_id: str,
base_oid: str,
response: Mapping[str, Any],
) -> None:
with self._lock, self._connect() as con:
con.execute(
"""
UPDATE external_task_requests
SET job_id=?,base_oid=?,state='BOUND',response_json=?,updated_at=?
WHERE request_id=?
""",
(
job_id,
base_oid,
json.dumps(response, sort_keys=True, separators=(",", ":")),
now(),
request_id,
),
)
con.commit()
class IngressService:
def __init__(self, upstream: Upstream, store: Store) -> None:
self.upstream = upstream
self.store = store
@staticmethod
def _main_oid(snapshot: Mapping[str, Any]) -> str:
candidates = [
snapshot.get("authoritative_head_oid"),
snapshot.get("head"),
(snapshot.get("aggregate") or {}).get("authoritative_head_oid")
if isinstance(snapshot.get("aggregate"), Mapping)
else None,
]
for candidate in candidates:
if isinstance(candidate, str) and OID_RE.fullmatch(candidate):
return candidate
raise IngressError("AUTHORITATIVE_HEAD_UNAVAILABLE", 503)
@staticmethod
def _public_response(
request_id: str,
task_id: str,
job_id: str,
state: str,
) -> dict[str, Any]:
return {
"schema": RESPONSE_SCHEMA,
"request_id": request_id,
"task_id": task_id,
"job_id": job_id,
"state": state,
"execution_mode": "source_capture",
"capabilities": list(READONLY_CAPABILITIES),
"write_access": False,
"official_verification_required": True,
}
def _recover_existing(
self,
request: Mapping[str, Any],
row: sqlite3.Row,
) -> dict[str, Any] | None:
task_id = str(row["task_id"])
status, task = self.upstream.get_task(task_id)
if status == 404:
return None
if status != 200:
raise IngressError("TASK_RECOVERY_FAILED", 503)
metadata = task.get("metadata")
execution = (
metadata.get("execution")
if isinstance(metadata, Mapping)
and isinstance(metadata.get("execution"), Mapping)
else {}
)
job_id = str(execution.get("job_id") or row["job_id"] or "")
base_oid = str(execution.get("base_oid") or row["base_oid"] or "")
if (
task.get("task_id") != task_id
or not JOB_ID_RE.fullmatch(job_id)
or not OID_RE.fullmatch(base_oid)
or execution.get("action") != "source_capture"
or sorted(execution.get("capability_scope") or [])
!= sorted(READONLY_CAPABILITIES)
or (execution.get("governance") or {}).get("writer_lease") is not False
):
raise IngressError("TASK_RECOVERY_BINDING_INVALID", 409)
response = self._public_response(
request["request_id"],
task_id,
job_id,
str(task.get("status") or "BOUND"),
)
self.store.bound(
request["request_id"],
job_id=job_id,
base_oid=base_oid,
response=response,
)
return response
def create(self, raw: Mapping[str, Any]) -> dict[str, Any]:
request = normalized_request(raw)
task_id = task_id_for(request["request_id"])
row = self.store.reserve(request, task_id)
if row["state"] == "BOUND" and row["response_json"]:
return json.loads(row["response_json"])
recovered = self._recover_existing(request, row)
if recovered is not None:
return recovered
snapshot = self.upstream.control_snapshot()
base_oid = self._main_oid(snapshot)
target_branch = branch_for(task_id)
binding_key = "gpt-readonly-binding-" + digest(
{"request_id": request["request_id"], "base_oid": base_oid}
)[:24]
constraints = {
**request["constraints"],
"repository_mutation": "forbidden",
"control_plane_side_effects": "forbidden",
"credential_access": "forbidden",
}
body = {
"task_id": task_id,
"repository": REPOSITORY,
"base_oid": base_oid,
"expected_branch_oid": base_oid,
"target_branch": target_branch,
"branch_mode": "CREATE_NEW",
"generation": 1,
"risk": "low",
"description": "Externally requested read-only governed Online Task.",
"idempotency_key": binding_key,
"priority": 0,
"max_attempts": 3,
"require_online_task_projection": True,
"online_task": {
"goal": request["goal"],
"instructions": request["instructions"],
"success_criteria": request["success_criteria"],
"constraints": constraints,
"required_capabilities": list(READONLY_CAPABILITIES),
"priority": 0,
"max_attempts": 3,
},
"execution": {
"action": "source_capture",
"adapter": "source_capture",
"capability_scope": list(READONLY_CAPABILITIES),
"source_signals": [
"external read-only source analysis request",
"no repository mutation no credential access no control-plane side effects",
],
},
}
status, value = self.upstream.create_governed_task(body)
if status not in {200, 201, 202}:
if status == 409:
recovered = self._recover_existing(request, row)
if recovered is not None:
return recovered
raise IngressError(
"GOVERNED_TASK_CREATE_FAILED",
409 if status == 409 else 503,
)
job_id = str(value.get("job_id") or "")
if value.get("task_id") != task_id or JOB_ID_RE.fullmatch(job_id) is None:
raise IngressError("GOVERNED_TASK_RESPONSE_INVALID", 502)
envelope_status, envelope = self.upstream.execution_envelope(
{
"task_id": task_id,
"job_id": job_id,
"required_capabilities": list(READONLY_CAPABILITIES),
"execution": body["execution"],
}
)
if envelope_status != 200:
raise IngressError("EXECUTION_PROJECTION_FAILED", 503)
if (
envelope.get("task_id") != task_id
or envelope.get("job_id") != job_id
or envelope.get("base_oid") != base_oid
or envelope.get("action") != "source_capture"
or sorted(envelope.get("capability_scope") or [])
!= sorted(READONLY_CAPABILITIES)
or (envelope.get("governance") or {}).get("writer_lease") is not False
):
raise IngressError("EXECUTION_PROJECTION_INVALID", 502)
response = self._public_response(
request["request_id"],
task_id,
job_id,
"BOUND",
)
self.store.bound(
request["request_id"],
job_id=job_id,
base_oid=base_oid,
response=response,
)
return response
def status(self, request_id: str) -> dict[str, Any]:
if REQUEST_ID_RE.fullmatch(request_id) is None:
raise IngressError("REQUEST_ID_INVALID")
row = self.store.get(request_id)
if row is None:
raise IngressError("REQUEST_NOT_FOUND", 404)
task_id = str(row["task_id"])
status, task = self.upstream.get_task(task_id)
if status == 200 and not row["job_id"]:
metadata = task.get("metadata")
execution = (
metadata.get("execution")
if isinstance(metadata, Mapping)
and isinstance(metadata.get("execution"), Mapping)
else {}
)
job_id = str(execution.get("job_id") or "")
base_oid = str(execution.get("base_oid") or "")
if (
JOB_ID_RE.fullmatch(job_id)
and OID_RE.fullmatch(base_oid)
and execution.get("action") == "source_capture"
and sorted(execution.get("capability_scope") or [])
== sorted(READONLY_CAPABILITIES)
and (execution.get("governance") or {}).get("writer_lease") is False
):
recovered = self._public_response(
request_id,
task_id,
job_id,
str(task.get("status") or "BOUND"),
)
self.store.bound(
request_id,
job_id=job_id,
base_oid=base_oid,
response=recovered,
)
row = self.store.get(request_id) or row
if status == 404:
return {
"schema": RESPONSE_SCHEMA,
"request_id": request_id,
"task_id": task_id,
"state": "PENDING",
"official_verification_required": True,
}
if status != 200:
raise IngressError("TASK_STATUS_UNAVAILABLE", 503)
result = task.get("result") if isinstance(task.get("result"), Mapping) else {}
verification = (
task.get("verification")
if isinstance(task.get("verification"), Mapping)
else {}
)
safe = {
"schema": RESPONSE_SCHEMA,
"request_id": request_id,
"task_id": task_id,
"job_id": str(row["job_id"] or ""),
"state": str(task.get("status") or "UNKNOWN"),
"attempt": int(task.get("attempt") or 0),
"claimed": bool(task.get("claimed_by")),
"result_status": result.get("status"),
"result_classification": result.get("classification"),
"official_verification": verification.get("decision"),
"official_classification": verification.get("classification"),
"write_access": False,
}
if result.get("execution_mode") == "source_capture":
snapshot = result.get("source_snapshot")
if isinstance(snapshot, Mapping):
safe["source_snapshot_sha256"] = snapshot.get("payload_sha256")
safe["source_snapshot_file_count"] = snapshot.get("file_count")
return safe
def openapi_document() -> dict[str, Any]:
return {
"openapi": OPENAPI_VERSION,
"info": {
"title": "AIOS Read-Only Online Task Ingress",
"version": "1.0.0",
},
"paths": {
"/v1/read-only-tasks": {
"post": {
"operationId": "createReadOnlyOnlineTask",
"security": [{"bearerAuth": []}],
"requestBody": {"required": True},
"responses": {"200": {"description": "Task accepted"}},
}
},
"/v1/read-only-tasks/{request_id}": {
"get": {
"operationId": "getReadOnlyOnlineTaskStatus",
"security": [{"bearerAuth": []}],
"responses": {"200": {"description": "Sanitized task status"}},
}
},
},
"components": {
"securitySchemes": {
"bearerAuth": {
"type": "http",
"scheme": "bearer",
}
}
},
}
class Handler(BaseHTTPRequestHandler):
service: IngressService | None = None
ingress_token: str = ""
def log_message(self, *_args: Any) -> None:
return
def respond(self, value: Mapping[str, Any], status: int) -> None:
raw = canonical(value)
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Cache-Control", "no-store")
self.send_header("Content-Length", str(len(raw)))
self.end_headers()
self.wfile.write(raw)
def authorized(self) -> bool:
header = self.headers.get("Authorization", "")
return (
header.startswith("Bearer ")
and bool(self.ingress_token)
and hmac.compare_digest(header[7:], self.ingress_token)
)
def body(self) -> dict[str, Any]:
try:
length = int(self.headers.get("Content-Length", "-1"))
except ValueError as exc:
raise IngressError("CONTENT_LENGTH_INVALID") from exc
if length < 1 or length > MAX_BODY:
raise IngressError("REQUEST_SIZE_INVALID", 413)
try:
value = json.loads(self.rfile.read(length).decode("utf-8"))
except Exception as exc:
raise IngressError("REQUEST_JSON_INVALID") from exc
if not isinstance(value, Mapping):
raise IngressError("REQUEST_OBJECT_REQUIRED")
return dict(value)
def do_GET(self) -> None:
if self.path == "/health":
self.respond(
{
"status": "ok",
"service": "aios-gpt-online-task-ingress",
"mode": "read-only",
},
200,
)
return
if self.path == "/openapi.json":
self.respond(openapi_document(), 200)
return
if not self.authorized():
self.respond({"error": "UNAUTHORIZED"}, 401)
return
match = re.fullmatch(r"/v1/read-only-tasks/([^/?#]+)", self.path)
if match is None:
self.respond({"error": "NOT_FOUND"}, 404)
return
try:
if self.service is None:
raise IngressError("RUNTIME_UNAVAILABLE", 503)
self.respond(self.service.status(match.group(1)), 200)
except IngressError as exc:
self.respond({"error": exc.code}, exc.status)
def do_POST(self) -> None:
if self.path != "/v1/read-only-tasks":
self.respond({"error": "NOT_FOUND"}, 404)
return
if not self.authorized():
self.respond({"error": "UNAUTHORIZED"}, 401)
return
try:
if self.service is None:
raise IngressError("RUNTIME_UNAVAILABLE", 503)
value = self.service.create(self.body())
self.respond(value, 200)
except IngressError as exc:
self.respond({"error": exc.code}, exc.status)
def serve() -> None:
ingress_token_path = Path(
os.environ.get(
"AIOS_GPT_INGRESS_TOKEN_FILE",
str(
Path(os.environ.get("CREDENTIALS_DIRECTORY", "/run/credentials"))
/ "gpt_ingress_token"
),
)
)
control_token_path = Path(
os.environ.get(
"AIOS_AGENT_TASK_CONTROL_TOKEN_FILE",
str(
Path(os.environ.get("CREDENTIALS_DIRECTORY", "/run/credentials"))
/ "agent_task_control_token"
),
)
)
ingress_token = load_secret(
ingress_token_path,
code="GPT_INGRESS_CREDENTIAL",
)
control_token = load_secret(
control_token_path,
code="CONTROL_PLANE_CREDENTIAL",
)
state_root = Path(
os.environ.get(
"AIOS_GPT_INGRESS_STATE_ROOT",
"/var/lib/aios-gpt-online-task-ingress",
)
)
host = validated_bind_host(
os.environ.get("AIOS_GPT_INGRESS_HOST", "127.0.0.1")
)
port = int(os.environ.get("AIOS_GPT_INGRESS_PORT", "18945"))
if not 1024 <= port <= 65535:
raise RuntimeError("GPT_INGRESS_PORT_INVALID")
Handler.service = IngressService(
Upstream(
bridge_base=os.environ.get(
"AIOS_GOVERNED_WRITE_API_URL",
"http://127.0.0.1:18892",
),
task_base=os.environ.get(
"AIOS_AGENT_TASK_API_URL",
"http://127.0.0.1:18940",
),
control_token=control_token,
),
Store(state_root / "state.sqlite3"),
)
Handler.ingress_token = ingress_token
ThreadingHTTPServer((host, port), Handler).serve_forever()
if __name__ == "__main__":
serve()
@@ -0,0 +1,306 @@
"""Uncredentialed deterministic executor for governed Online Agent changes.
This process never receives Agent/Owner/Bridge credentials and never writes Git refs.
It operates only on an exchange bundle and returns hash-bound changed files + test evidence.
"""
from __future__ import annotations
import base64
from dataclasses import dataclass
import errno
import hashlib
import io
import json
import os
from pathlib import Path
import re
import shutil
import socket
import subprocess
import tarfile
import tempfile
import time
from typing import Any, Mapping, Sequence
ALLOWED_PROGRAMS = {"/usr/bin/python3", "/usr/bin/git", "/bin/true", "/usr/bin/true"}
MAX_ARCHIVE_BYTES = 64 * 1024 * 1024
MAX_FILE_BYTES = 4 * 1024 * 1024
SECRET_ENV_TOKENS = ("TOKEN", "SECRET", "PASSWORD", "CREDENTIAL", "AUTHORIZATION", "PRIVATE_KEY")
REQUEST_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{15,254}$")
SENSITIVE_VALUE_RE = re.compile(
r"(?i)(authorization\s*[:=]\s*bearer\s+\S+|authorization\s+bearer\s+\S+|"
r"(?:password|passwd|secret|token|api[_-]?key)\s*[:=]\s*\S+)"
)
class IsolatedExecutorError(RuntimeError):
pass
def sha256(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def canonical(value: object) -> bytes:
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
def safe_relative(value: object) -> str:
if not isinstance(value, str) or not value or value.startswith("/"):
raise IsolatedExecutorError("PATH_INVALID")
path = Path(value)
if ".." in path.parts or "." in path.parts or any(part in {"", ".git"} for part in path.parts):
raise IsolatedExecutorError("PATH_INVALID")
return path.as_posix()
def resolve_under(root: Path, relative: str) -> Path:
root = root.resolve()
path = (root / safe_relative(relative)).resolve(strict=False)
if path == root or root not in path.parents:
raise IsolatedExecutorError("PATH_ESCAPE")
return path
def _credential_exposure(directory: Path) -> str | None:
"""Return an exposure code; EACCES is the intended credential isolation proof."""
try:
if not directory.exists():
return None
if directory.is_symlink():
return "CREDENTIAL_MOUNT_SYMLINK"
if os.access(directory, os.R_OK):
return "CREDENTIAL_MOUNT_ACCESSIBLE"
if directory.is_dir():
for child in directory.iterdir():
if child.is_file() and os.access(child, os.R_OK):
return "CREDENTIAL_FILE_ACCESSIBLE"
except PermissionError:
return None
return None
def assert_uncredentialed() -> None:
credentials = Path("/run/credentials")
candidates = {credentials}
credentials_directory = Path(os.environ.get("CREDENTIALS_DIRECTORY", "/run/credentials"))
candidates.add(credentials_directory)
leaked = sorted(
key for key, value in os.environ.items()
if value
and (any(token in key.upper() for token in SECRET_ENV_TOKENS) or SENSITIVE_VALUE_RE.search(str(value)))
)
if leaked:
raise IsolatedExecutorError("CREDENTIAL_ENV_EXPOSED:" + ",".join(leaked))
for directory in candidates:
exposure = _credential_exposure(directory)
if exposure is not None:
raise IsolatedExecutorError(exposure)
# In the intended PrivateNetwork namespace host loopback services are unreachable.
try:
probe = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except OSError as exc:
if exc.errno in {errno.EACCES, errno.EPERM, errno.EAFNOSUPPORT}:
return
raise IsolatedExecutorError("NETWORK_ISOLATION_PROBE_FAILED") from exc
probe.settimeout(0.15)
try:
if probe.connect_ex(("127.0.0.1", 18892)) == 0:
raise IsolatedExecutorError("GOVERNED_BRIDGE_REACHABLE")
finally:
probe.close()
def extract_source(archive: Path, root: Path) -> None:
if archive.is_symlink() or not archive.is_file() or archive.stat().st_size > MAX_ARCHIVE_BYTES:
raise IsolatedExecutorError("SOURCE_ARCHIVE_INVALID")
with tarfile.open(archive, "r:gz") as tf:
for member in tf.getmembers():
path = Path(member.name)
if path.is_absolute() or ".." in path.parts or ".git" in path.parts:
raise IsolatedExecutorError("SOURCE_ARCHIVE_PATH_INVALID")
if member.issym() or member.islnk():
raise IsolatedExecutorError("SOURCE_ARCHIVE_LINK_DENY")
tf.extractall(root, filter="data")
def run_command(item: Mapping[str, Any], root: Path) -> dict[str, Any]:
argv = item.get("argv")
if not isinstance(argv, Sequence) or isinstance(argv, (str, bytes)) or not argv:
raise IsolatedExecutorError("COMMAND_INVALID")
argv = [str(part) for part in argv]
if argv[0] not in ALLOWED_PROGRAMS:
raise IsolatedExecutorError("COMMAND_PROGRAM_DENY")
if argv[0] == "/usr/bin/python3" and "-m" in argv:
index = argv.index("-m")
if index + 1 >= len(argv) or argv[index + 1] not in {"pytest", "py_compile"}:
raise IsolatedExecutorError("PYTHON_MODULE_DENY")
if argv[0] == "/usr/bin/git" and any(
token in {"push", "update-ref", "commit", "merge", "rebase", "reset", "clean", "branch"}
for token in argv[1:]
):
raise IsolatedExecutorError("GIT_MUTATION_DENY")
timeout = int(item.get("timeout_seconds") or 300)
if not 1 <= timeout <= 1800:
raise IsolatedExecutorError("COMMAND_TIMEOUT_INVALID")
cwd = root if item.get("cwd", ".") == "." else resolve_under(root, str(item.get("cwd")))
started = time.monotonic()
completed = subprocess.run(
argv,
cwd=cwd,
capture_output=True,
text=True,
check=False,
timeout=timeout,
env={"PATH": "/usr/bin:/bin", "LC_ALL": "C.UTF-8"},
)
result = {
"label": str(item.get("label") or "command"),
"exit_code": completed.returncode,
"stdout_sha256": sha256(completed.stdout.encode("utf-8", "replace")),
"stderr_sha256": sha256(completed.stderr.encode("utf-8", "replace")),
"duration_ms": int((time.monotonic() - started) * 1000),
}
if completed.returncode != 0:
raise IsolatedExecutorError(
"COMMAND_FAILED:" + result["label"] + ":" + str(completed.returncode)
)
return result
def apply_changes(request: Mapping[str, Any], root: Path) -> list[str]:
changes = request.get("change_set")
if not isinstance(changes, list) or not changes:
raise IsolatedExecutorError("CHANGE_SET_INVALID")
changed: list[str] = []
for change in changes:
if not isinstance(change, Mapping):
raise IsolatedExecutorError("CHANGE_INVALID")
relative = safe_relative(change.get("path"))
path = resolve_under(root, relative)
current = path.read_bytes() if path.is_file() else None
expected = change.get("expected_sha256")
if expected is not None and (current is None or sha256(current) != expected):
raise IsolatedExecutorError("SOURCE_DRIFT:" + relative)
operation = change.get("operation")
if operation == "write_file":
content = change.get("content")
if not isinstance(content, str):
raise IsolatedExecutorError("WRITE_CONTENT_INVALID")
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(content, encoding="utf-8")
elif operation == "replace_text":
if current is None:
raise IsolatedExecutorError("REPLACE_SOURCE_MISSING")
old = str(change.get("old"))
new = str(change.get("new"))
count = int(change.get("count", 1))
text = current.decode("utf-8")
if text.count(old) != count:
raise IsolatedExecutorError("REPLACE_COUNT_INVALID:" + relative)
path.write_text(text.replace(old, new), encoding="utf-8")
elif operation == "delete_file":
if path.exists():
path.unlink()
else:
raise IsolatedExecutorError("CHANGE_OPERATION_INVALID")
changed.append(relative)
return sorted(set(changed))
def changed_bundle(root: Path, changed: list[str]) -> list[dict[str, Any]]:
files = []
for relative in changed:
path = resolve_under(root, relative)
if path.exists():
data = path.read_bytes()
if len(data) > MAX_FILE_BYTES:
raise IsolatedExecutorError("CHANGED_FILE_TOO_LARGE:" + relative)
files.append({
"path": relative,
"deleted": False,
"sha256": sha256(data),
"content_b64": base64.b64encode(data).decode("ascii"),
})
else:
files.append({"path": relative, "deleted": True, "sha256": None, "content_b64": None})
return files
def execute_request(request_path: Path, result_path: Path) -> None:
assert_uncredentialed()
request = json.loads(request_path.read_text(encoding="utf-8"))
if request.get("schema") != "aios.isolated-execution.request.v1":
raise IsolatedExecutorError("REQUEST_SCHEMA_INVALID")
request_id = request.get("request_id")
if not isinstance(request_id, str) or not REQUEST_ID_RE.fullmatch(request_id):
raise IsolatedExecutorError("REQUEST_ID_INVALID")
source_archive = Path(str(request.get("source_archive") or ""))
if not source_archive.is_absolute():
raise IsolatedExecutorError("SOURCE_ARCHIVE_PATH_INVALID")
expected_archive_sha = str(request.get("source_archive_sha256") or "")
if sha256(source_archive.read_bytes()) != expected_archive_sha:
raise IsolatedExecutorError("SOURCE_ARCHIVE_SHA_MISMATCH")
with tempfile.TemporaryDirectory(prefix="aios-isolated-executor-") as temporary:
root = Path(temporary) / "workspace"
root.mkdir()
extract_source(source_archive, root)
changed = apply_changes(request, root)
commands = [
run_command(item, root)
for item in request.get("verification", [])
if isinstance(item, Mapping)
]
result = {
"schema": "aios.isolated-execution.result.v1",
"request_id": request_id,
"request_sha256": sha256(canonical(request)),
"changed_files": changed_bundle(root, changed),
"verification": commands,
"credential_isolation": {
"credential_env_absent": True,
"credential_mount_inaccessible": True,
"governed_bridge_unreachable": True,
},
}
result["result_sha256"] = sha256(canonical(result))
temporary_result = result_path.with_suffix(".tmp")
temporary_result.write_text(json.dumps(result, sort_keys=True, separators=(",", ":")), encoding="utf-8")
os.replace(temporary_result, result_path)
def serve(root: Path) -> None:
assert_uncredentialed()
requests = root / "requests"
results = root / "results"
failed = root / "failed"
for path in (requests, results, failed):
path.mkdir(parents=True, exist_ok=True)
while True:
for request in sorted(requests.glob("*.json")):
if request.is_symlink():
continue
result = results / request.name
try:
execute_request(request, result)
request.unlink()
except Exception as exc:
(failed / request.name).write_text(
json.dumps({"error": str(exc) or type(exc).__name__}, sort_keys=True),
encoding="utf-8",
)
request.unlink(missing_ok=True)
time.sleep(0.5)
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--root", default="/var/lib/aios-online-agent-executor")
parser.add_argument("--request")
parser.add_argument("--result")
args = parser.parse_args()
if args.request and args.result:
execute_request(Path(args.request), Path(args.result))
else:
serve(Path(args.root))
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,232 @@
#!/usr/bin/env python3
"""Root-owned Model Governance configuration reconciler.
The config contains provider/model/policy metadata only. Raw credentials are rejected by
aios_model_governance and never stored in the governance database.
"""
from __future__ import annotations
import argparse
import hashlib
import json
from pathlib import Path
from typing import Any, Mapping
from aios_model_governance import (
ModelGovernanceError,
ModelGovernanceStore,
ModelRecord,
ProviderRecord,
ScopePolicy,
)
SCHEMA = "aios.model-governance.bootstrap.v1"
MAX_CONFIG_BYTES = 512 * 1024
def canonical(value: object) -> bytes:
return json.dumps(
value, ensure_ascii=False, sort_keys=True, separators=(",", ":")
).encode("utf-8")
def fail(code: str) -> None:
raise SystemExit(code)
def _load(path: Path) -> dict[str, Any]:
if path.is_symlink() or not path.is_file() or path.stat().st_size > MAX_CONFIG_BYTES:
fail("MODEL_GOVERNANCE_CONFIG_INVALID")
try:
value = json.loads(path.read_text(encoding="utf-8"))
except json.JSONDecodeError:
fail("MODEL_GOVERNANCE_CONFIG_INVALID")
if not isinstance(value, Mapping) or value.get("schema") != SCHEMA:
fail("MODEL_GOVERNANCE_CONFIG_SCHEMA_INVALID")
return dict(value)
def _provider_desired(value: Mapping[str, Any]) -> ProviderRecord:
return ProviderRecord(
provider_id=str(value.get("provider_id") or ""),
credential_ref=value.get("credential_ref"),
enabled=bool(value.get("enabled", True)),
failure_domain=str(value.get("failure_domain") or "default"),
max_failures=int(value.get("max_failures") or 3),
cooldown_seconds=int(value.get("cooldown_seconds") or 60),
)
def _model_desired(value: Mapping[str, Any]) -> ModelRecord:
return ModelRecord(
model_id=str(value.get("model_id") or ""),
provider_id=str(value.get("provider_id") or ""),
version=str(value.get("version") or ""),
capabilities=tuple(value.get("capabilities") or ()),
input_cost_micros_per_1k=int(value.get("input_cost_micros_per_1k") or 0),
output_cost_micros_per_1k=int(value.get("output_cost_micros_per_1k") or 0),
max_input_tokens=int(value.get("max_input_tokens") or 0),
max_output_tokens=int(value.get("max_output_tokens") or 0),
evaluation_state=str(value.get("evaluation_state") or "APPROVED"),
evaluation_sha256=str(value.get("evaluation_sha256") or ""),
enabled=bool(value.get("enabled", True)),
priority=int(value.get("priority") or 100),
)
def _policy_desired(value: Mapping[str, Any]) -> ScopePolicy:
return ScopePolicy(
scope_type=str(value.get("scope_type") or ""),
tenant_id=value.get("tenant_id"),
project_id=value.get("project_id"),
allowed_models=tuple(value.get("allowed_models") or ()),
denied_providers=tuple(value.get("denied_providers") or ()),
required_capabilities=tuple(value.get("required_capabilities") or ()),
pinned_model_id=value.get("pinned_model_id"),
pinned_model_version=value.get("pinned_model_version"),
max_input_tokens_per_request=int(value.get("max_input_tokens_per_request") or 0),
max_output_tokens_per_request=int(value.get("max_output_tokens_per_request") or 0),
daily_token_budget=int(value.get("daily_token_budget") or 0),
daily_cost_budget_micros=int(value.get("daily_cost_budget_micros") or 0),
fallback_limit=int(value.get("fallback_limit") or 0),
)
def _row_dict(row: object) -> dict[str, Any] | None:
return dict(row) if row is not None else None
def _provider_matches(row: Mapping[str, Any], desired: ProviderRecord) -> bool:
return (
row["credential_ref"] == desired.credential_ref
and bool(row["enabled"]) == desired.enabled
and row["failure_domain"] == desired.failure_domain
and int(row["max_failures"]) == desired.max_failures
and int(row["cooldown_seconds"]) == desired.cooldown_seconds
)
def _model_matches(row: Mapping[str, Any], desired: ModelRecord) -> bool:
return (
row["provider_id"] == desired.provider_id
and row["version"] == desired.version
and tuple(json.loads(row["capabilities_json"])) == tuple(sorted(set(desired.capabilities)))
and int(row["input_cost_micros_per_1k"]) == desired.input_cost_micros_per_1k
and int(row["output_cost_micros_per_1k"]) == desired.output_cost_micros_per_1k
and int(row["max_input_tokens"]) == desired.max_input_tokens
and int(row["max_output_tokens"]) == desired.max_output_tokens
and row["evaluation_state"] == desired.evaluation_state
and row["evaluation_sha256"] == desired.evaluation_sha256
and bool(row["enabled"]) == desired.enabled
and int(row["priority"]) == desired.priority
)
def _policy_matches(row: Mapping[str, Any], desired: ScopePolicy) -> bool:
return (
row["scope_type"] == desired.scope_type
and row["tenant_id"] == desired.tenant_id
and row["project_id"] == desired.project_id
and tuple(json.loads(row["allowed_models_json"])) == tuple(sorted(set(desired.allowed_models)))
and tuple(json.loads(row["denied_providers_json"])) == tuple(sorted(set(desired.denied_providers)))
and tuple(json.loads(row["required_capabilities_json"])) == tuple(sorted(set(desired.required_capabilities)))
and row["pinned_model_id"] == desired.pinned_model_id
and row["pinned_model_version"] == desired.pinned_model_version
and int(row["max_input_tokens_per_request"]) == desired.max_input_tokens_per_request
and int(row["max_output_tokens_per_request"]) == desired.max_output_tokens_per_request
and int(row["daily_token_budget"]) == desired.daily_token_budget
and int(row["daily_cost_budget_micros"]) == desired.daily_cost_budget_micros
and int(row["fallback_limit"]) == desired.fallback_limit
)
def apply_config(db_path: Path, config_path: Path) -> dict[str, Any]:
config = _load(config_path)
config_sha = hashlib.sha256(canonical(config)).hexdigest()
providers = config.get("providers") or []
models = config.get("models") or []
policies = config.get("policies") or []
if not all(isinstance(value, list) for value in (providers, models, policies)):
fail("MODEL_GOVERNANCE_CONFIG_COLLECTION_INVALID")
store = ModelGovernanceStore(db_path)
changes = {"providers": 0, "models": 0, "policies": 0, "unchanged": 0}
try:
for item in providers:
if not isinstance(item, Mapping):
fail("MODEL_GOVERNANCE_PROVIDER_CONFIG_INVALID")
desired = _provider_desired(item)
row = store.conn.execute(
"SELECT * FROM model_governance_providers WHERE provider_id=?",
(desired.provider_id,),
).fetchone()
if row is not None and _provider_matches(row, desired):
changes["unchanged"] += 1
continue
expected = int(row["revision"]) if row is not None else 0
store.register_provider(desired, expected_revision=expected)
changes["providers"] += 1
for item in models:
if not isinstance(item, Mapping):
fail("MODEL_GOVERNANCE_MODEL_CONFIG_INVALID")
desired = _model_desired(item)
row = store.conn.execute(
"SELECT * FROM model_governance_models WHERE model_id=?",
(desired.model_id,),
).fetchone()
if row is not None and _model_matches(row, desired):
changes["unchanged"] += 1
continue
expected = int(row["revision"]) if row is not None else 0
store.register_model(desired, expected_revision=expected)
changes["models"] += 1
for item in policies:
if not isinstance(item, Mapping):
fail("MODEL_GOVERNANCE_POLICY_CONFIG_INVALID")
desired = _policy_desired(item)
row = store.conn.execute(
"SELECT * FROM model_governance_scope_policies WHERE scope_key=?",
(desired.scope_key,),
).fetchone()
if row is not None and _policy_matches(row, desired):
changes["unchanged"] += 1
continue
expected = int(row["revision"]) if row is not None else 0
store.set_scope_policy(desired, expected_revision=expected)
changes["policies"] += 1
if not store.verify_audit_chain():
fail("MODEL_GOVERNANCE_AUDIT_CHAIN_INVALID")
return {
"schema": "aios.model-governance.bootstrap-result.v1",
"database": str(db_path),
"config_sha256": config_sha,
"changes": changes,
"audit_chain_valid": True,
}
finally:
store.close()
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--db", required=True)
parser.add_argument("--config", required=True)
parser.add_argument("--output")
args = parser.parse_args()
try:
result = apply_config(Path(args.db), Path(args.config))
except ModelGovernanceError as exc:
print(json.dumps({"status": "FAILED", "error": exc.code}, sort_keys=True))
return 20
data = json.dumps(result, indent=2, sort_keys=True) + "\n"
if args.output:
Path(args.output).write_text(data, encoding="utf-8")
print(data, end="")
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,302 @@
"""Durable checkpoint journal for the generic Online Agent worker.
The JSON task state remains a fast cache. This SQLite journal is the durable,
append-only evidence used for restart/resume/replay decisions.
"""
from __future__ import annotations
from dataclasses import dataclass
from datetime import datetime, timezone
import hashlib
import json
from pathlib import Path
import sqlite3
from typing import Any, Mapping
PHASE_ORDER = {
"claimed": 10,
"workspace_prepared": 20,
"verification_passed": 30,
"agent_runtime_done": 40,
"source_capture_done": 40,
"commit_done": 40,
"reconciliation_required": 45,
"result_pending": 50,
"result_submitted": 60,
}
class CheckpointError(RuntimeError):
pass
def canonical(value: object) -> bytes:
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False, default=str).encode("utf-8")
def sha256_json(value: object) -> str:
return hashlib.sha256(canonical(value)).hexdigest()
def now() -> str:
return datetime.now(timezone.utc).isoformat()
@dataclass(frozen=True)
class CheckpointBinding:
task_id: str
attempt: int
execution_digest: str
job_id: str
repository: str
base_oid: str
expected_branch_oid: str
target_branch: str
generation: int
def material(self) -> dict[str, Any]:
return {
"task_id": self.task_id,
"attempt": self.attempt,
"execution_digest": self.execution_digest,
"job_id": self.job_id,
"repository": self.repository,
"base_oid": self.base_oid,
"expected_branch_oid": self.expected_branch_oid,
"target_branch": self.target_branch,
"generation": self.generation,
}
@property
def binding_sha256(self) -> str:
return sha256_json(self.material())
class CheckpointStore:
def __init__(self, path: Path) -> None:
self.path = Path(path)
self.path.parent.mkdir(parents=True, exist_ok=True)
with self._connect() as connection:
connection.executescript(
"""
CREATE TABLE IF NOT EXISTS checkpoints(
task_id TEXT NOT NULL,
attempt INTEGER NOT NULL,
sequence INTEGER NOT NULL,
phase TEXT NOT NULL,
phase_order INTEGER NOT NULL,
execution_digest TEXT NOT NULL,
job_id TEXT NOT NULL,
repository TEXT NOT NULL,
base_oid TEXT NOT NULL,
expected_branch_oid TEXT NOT NULL,
target_branch TEXT NOT NULL,
generation INTEGER NOT NULL,
binding_sha256 TEXT NOT NULL,
payload_json TEXT NOT NULL,
payload_sha256 TEXT NOT NULL,
created_at TEXT NOT NULL,
PRIMARY KEY(task_id, attempt, sequence)
);
CREATE INDEX IF NOT EXISTS idx_checkpoints_task_latest
ON checkpoints(task_id, attempt DESC, sequence DESC);
CREATE UNIQUE INDEX IF NOT EXISTS idx_checkpoints_exact_replay
ON checkpoints(task_id, attempt, phase, payload_sha256);
CREATE TABLE IF NOT EXISTS checkpoint_heads(
task_id TEXT PRIMARY KEY,
attempt INTEGER NOT NULL,
sequence INTEGER NOT NULL,
phase TEXT NOT NULL,
phase_order INTEGER NOT NULL,
execution_digest TEXT NOT NULL,
binding_sha256 TEXT NOT NULL,
payload_sha256 TEXT NOT NULL,
updated_at TEXT NOT NULL
);
"""
)
def _connect(self) -> sqlite3.Connection:
connection = sqlite3.connect(self.path, timeout=30.0)
connection.row_factory = sqlite3.Row
connection.execute("PRAGMA foreign_keys=ON")
connection.execute("PRAGMA journal_mode=WAL")
connection.execute("PRAGMA synchronous=FULL")
connection.execute("PRAGMA busy_timeout=30000")
return connection
@staticmethod
def _phase_order(phase: str) -> int:
try:
return PHASE_ORDER[phase]
except KeyError as exc:
raise CheckpointError("CHECKPOINT_PHASE_INVALID") from exc
def append(
self,
binding: CheckpointBinding,
phase: str,
payload: Mapping[str, Any],
) -> dict[str, Any]:
order = self._phase_order(phase)
payload_value = dict(payload)
payload_sha = sha256_json(payload_value)
with self._connect() as connection:
connection.execute("BEGIN IMMEDIATE")
replay = connection.execute(
"""
SELECT * FROM checkpoints
WHERE task_id=? AND attempt=? AND phase=? AND payload_sha256=?
""",
(binding.task_id, binding.attempt, phase, payload_sha),
).fetchone()
if replay is not None:
connection.commit()
return dict(replay)
head = connection.execute(
"SELECT * FROM checkpoint_heads WHERE task_id=?",
(binding.task_id,),
).fetchone()
if head is not None:
head_attempt = int(head["attempt"])
head_order = int(head["phase_order"])
if binding.attempt < head_attempt:
raise CheckpointError("CHECKPOINT_ATTEMPT_REGRESSION")
if binding.attempt == head_attempt:
if head["execution_digest"] != binding.execution_digest:
raise CheckpointError("CHECKPOINT_EXECUTION_DIGEST_CONFLICT")
if head["binding_sha256"] != binding.binding_sha256:
raise CheckpointError("CHECKPOINT_SCOPE_MISMATCH_DENY")
if order < head_order and phase != "reconciliation_required":
raise CheckpointError("CHECKPOINT_PHASE_REGRESSION")
sequence = int(head["sequence"]) + 1
else:
sequence = 1
else:
sequence = 1
created_at = now()
connection.execute(
"""
INSERT INTO checkpoints(
task_id,attempt,sequence,phase,phase_order,execution_digest,job_id,
repository,base_oid,expected_branch_oid,target_branch,generation,
binding_sha256,payload_json,payload_sha256,created_at
) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
""",
(
binding.task_id,
binding.attempt,
sequence,
phase,
order,
binding.execution_digest,
binding.job_id,
binding.repository,
binding.base_oid,
binding.expected_branch_oid,
binding.target_branch,
binding.generation,
binding.binding_sha256,
canonical(payload_value).decode("utf-8"),
payload_sha,
created_at,
),
)
connection.execute(
"""
INSERT INTO checkpoint_heads(
task_id,attempt,sequence,phase,phase_order,execution_digest,
binding_sha256,payload_sha256,updated_at
) VALUES (?,?,?,?,?,?,?,?,?)
ON CONFLICT(task_id) DO UPDATE SET
attempt=excluded.attempt,
sequence=excluded.sequence,
phase=excluded.phase,
phase_order=excluded.phase_order,
execution_digest=excluded.execution_digest,
binding_sha256=excluded.binding_sha256,
payload_sha256=excluded.payload_sha256,
updated_at=excluded.updated_at
""",
(
binding.task_id,
binding.attempt,
sequence,
phase,
order,
binding.execution_digest,
binding.binding_sha256,
payload_sha,
created_at,
),
)
connection.commit()
row = connection.execute(
"SELECT * FROM checkpoints WHERE task_id=? AND attempt=? AND sequence=?",
(binding.task_id, binding.attempt, sequence),
).fetchone()
return dict(row)
def latest(self, task_id: str) -> dict[str, Any] | None:
with self._connect() as connection:
row = connection.execute(
"""
SELECT * FROM checkpoints
WHERE task_id=?
ORDER BY attempt DESC, sequence DESC
LIMIT 1
""",
(task_id,),
).fetchone()
if row is None:
return None
value = dict(row)
value["payload"] = json.loads(value.pop("payload_json"))
return value
def latest_for_attempt(self, task_id: str, attempt: int) -> dict[str, Any] | None:
with self._connect() as connection:
row = connection.execute(
"""
SELECT * FROM checkpoints
WHERE task_id=? AND attempt=?
ORDER BY sequence DESC
LIMIT 1
""",
(task_id, attempt),
).fetchone()
if row is None:
return None
value = dict(row)
value["payload"] = json.loads(value.pop("payload_json"))
return value
def verify_latest_binding(self, binding: CheckpointBinding) -> dict[str, Any] | None:
value = self.latest(binding.task_id)
if value is None:
return None
if int(value["attempt"]) > binding.attempt:
raise CheckpointError("CHECKPOINT_FUTURE_ATTEMPT")
# A previous attempt may be reused only when the immutable execution/scope binding matches.
if value["execution_digest"] != binding.execution_digest:
return None
expected = CheckpointBinding(
task_id=value["task_id"],
attempt=int(value["attempt"]),
execution_digest=value["execution_digest"],
job_id=value["job_id"],
repository=value["repository"],
base_oid=value["base_oid"],
expected_branch_oid=value["expected_branch_oid"],
target_branch=value["target_branch"],
generation=int(value["generation"]),
)
current_scope = binding.material()
historical_scope = expected.material()
current_scope.pop("attempt", None)
historical_scope.pop("attempt", None)
if sha256_json(current_scope) != sha256_json(historical_scope):
raise CheckpointError("CHECKPOINT_SCOPE_MISMATCH_DENY")
return value
@@ -0,0 +1,463 @@
"""Bounded read-only repository interface map for governed Online Agent analysis.
The compact map intentionally returns symbols and short source excerpts rather than a
source archive. It is designed for control-plane consumers that cannot safely ingest a
multi-megabyte repository snapshot.
"""
from __future__ import annotations
import ast
import hashlib
import json
from pathlib import Path
import re
from typing import Any, Iterable, Mapping, Sequence
SCHEMA = "aios.engineering.interface-map.v1"
MAX_RESULT_BYTES = 120 * 1024
TARGET_RESULT_BYTES = 108 * 1024
MAX_MODULES = 40
MAX_SYMBOLS = 80
MAX_TESTS = 40
MAX_EXCERPTS = 25
MAX_EXCERPT_CHARS = 1200
MAX_FILE_BYTES = 2 * 1024 * 1024
ROOTS = (
"active-runtime/app",
"runtime/agent-os",
"tests",
"db/migrations",
"migrations",
"deploy/systemd",
"docs/task-plane",
"docs/superpowers/specs",
)
SUFFIXES = {".py", ".sql", ".service", ".md", ".json", ".yaml", ".yml"}
DEFAULT_TERMS = {
"agent", "worker", "fleet", "scheduler", "scheduling", "task", "claim", "lease",
"heartbeat", "capacity", "concurrency", "priority", "fairness", "queue", "retry",
"backoff", "circuit", "health", "liveness", "scope", "tenant", "project", "trace",
"message", "checkpoint", "recovery", "idempotency", "dispatch", "assignment",
}
FLEET_CONTRACT_TERMS: dict[str, tuple[str, ...]] = {
"FLEET_AGENT_REGISTRATION": ("agent", "register", "worker", "capability", "version"),
"FLEET_CAPACITY_SCHEDULING": ("capacity", "slot", "concurrency", "scheduler", "queue"),
"FLEET_FAIRNESS": ("fair", "priority", "queue", "tenant", "project", "starvation"),
"FLEET_FAULT_ISOLATION": ("failure", "health", "liveness", "timeout", "circuit", "poison"),
"FLEET_WORKER_REASSIGNMENT": ("reassign", "assignment", "claim", "lease", "retry", "worker"),
"FLEET_SCOPE_ISOLATION": ("tenant", "project", "scope", "conversation", "task", "run"),
"FLEET_IDEMPOTENT_DISPATCH": ("idempot", "dispatch", "claim", "lease", "result", "attempt"),
"FLEET_FAILURE_RECOVERY": ("checkpoint", "resume", "recover", "retry", "cleanup", "receipt"),
}
MODEL_GOVERNANCE_CONTRACT_TERMS: dict[str, tuple[str, ...]] = {
"MODEL_PROVIDER_REGISTRY": ("model", "provider", "registry", "endpoint", "capability"),
"MODEL_SELECTION_POLICY": ("model", "select", "route", "policy", "capability"),
"MODEL_VERSION_PINNING": ("model", "version", "pin", "revision", "snapshot"),
"MODEL_BUDGET_ACCOUNTING": ("budget", "token", "cost", "quota", "usage"),
"MODEL_FALLBACK_POLICY": ("fallback", "retry", "route", "provider", "model"),
"MODEL_SCOPE_ISOLATION": ("tenant", "project", "scope", "policy", "model"),
"MODEL_CREDENTIAL_BOUNDARY": ("credential", "secret", "provider", "tool", "token"),
"MODEL_TRACE_AUDIT": ("trace", "audit", "model", "provider", "request"),
"MODEL_EVALUATION_GATE": ("eval", "evaluation", "approval", "quality", "gate"),
"MODEL_ROUTING_RESILIENCE": ("route", "retry", "fallback", "timeout", "health"),
}
PROFILE_CLASSIFICATION = {
"fleet": "P2_INTERFACE_MAP_CAPTURED",
"model_governance": "MODEL_GOVERNANCE_INTERFACE_MAP_CAPTURED",
}
MODEL_PROFILE_MARKERS = {
"model", "provider", "llm", "inference", "routing", "router", "budget",
"token", "cost", "quota", "fallback", "evaluation", "eval", "approval",
"version", "pinning",
}
SENSITIVE_RE = re.compile(
r"(?i)(authorization\s*:\s*bearer\s+\S+|"
r"(?:password|passwd|secret|token|api[_-]?key)\s*[:=]\s*['\"]?[^'\"\s,;]+)"
)
SQL_TABLE_RE = re.compile(
r"(?is)\bcreate\s+table\s+(?:if\s+not\s+exists\s+)?[\"`[]?([A-Za-z_][A-Za-z0-9_.-]*)"
)
class CompactInterfaceMapError(RuntimeError):
pass
def _sha(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def _bounded_excerpt(text: str, start_line: int, end_line: int) -> str:
lines = text.splitlines()
start = max(0, start_line - 2)
end = min(len(lines), max(end_line + 1, start + 1))
excerpt = "\n".join(lines[start:end])
excerpt = SENSITIVE_RE.sub("<redacted-sensitive-source-pattern>", excerpt)
if len(excerpt) > MAX_EXCERPT_CHARS:
excerpt = excerpt[: MAX_EXCERPT_CHARS - 1] + "…"
return excerpt
def _signal_terms(signals: Sequence[str]) -> set[str]:
terms = set(DEFAULT_TERMS)
for signal in signals:
if not isinstance(signal, str):
continue
terms.update(
token.lower()
for token in re.findall(r"[A-Za-z][A-Za-z0-9_.-]{2,63}", signal)
)
return set(sorted(terms)[:96])
def _profile(source_signals: Sequence[str]) -> str:
tokens: set[str] = set()
for signal in source_signals:
if not isinstance(signal, str):
continue
tokens.update(
token.lower()
for token in re.findall(r"[A-Za-z][A-Za-z0-9_.-]{2,63}", signal)
)
model_score = len(tokens & MODEL_PROFILE_MARKERS)
return "model_governance" if model_score >= 2 else "fleet"
def _contract_terms(profile: str) -> Mapping[str, tuple[str, ...]]:
if profile == "model_governance":
return MODEL_GOVERNANCE_CONTRACT_TERMS
if profile == "fleet":
return FLEET_CONTRACT_TERMS
raise CompactInterfaceMapError("INTERFACE_MAP_PROFILE_INVALID")
def _candidate_files(root: Path, terms: set[str]) -> list[dict[str, Any]]:
root = root.resolve()
candidates: list[dict[str, Any]] = []
for relative_root in ROOTS:
directory = (root / relative_root).resolve(strict=False)
if directory == root or root not in directory.parents:
raise CompactInterfaceMapError("INTERFACE_MAP_SCOPE_INVALID")
if not directory.is_dir() or directory.is_symlink():
continue
for path in sorted(directory.rglob("*")):
if path.is_symlink() or not path.is_file() or path.suffix.lower() not in SUFFIXES:
continue
resolved = path.resolve(strict=True)
if root not in resolved.parents:
raise CompactInterfaceMapError("INTERFACE_MAP_PATH_ESCAPE")
data = path.read_bytes()
if len(data) > MAX_FILE_BYTES:
continue
relative = path.relative_to(root).as_posix()
text = data.decode("utf-8", "ignore")
searchable = (relative + "\n" + text[:384000]).lower()
matched = sorted(term for term in terms if term in searchable)
if not matched:
continue
path_bonus = sum(3 for term in terms if term in relative.lower())
test_bonus = 3 if relative.startswith("tests/") else 0
score = min(250, len(matched) + path_bonus + test_bonus)
candidates.append(
{
"path": relative,
"sha256": _sha(data),
"bytes": len(data),
"score": score,
"text": text,
"suffix": path.suffix.lower(),
}
)
candidates.sort(key=lambda item: (-item["score"], item["path"]))
return candidates
def _python_symbols(item: Mapping[str, Any], terms: set[str]) -> list[dict[str, Any]]:
try:
tree = ast.parse(str(item["text"]), filename=str(item["path"]))
except SyntaxError:
return []
result: list[dict[str, Any]] = []
class Visitor(ast.NodeVisitor):
def __init__(self) -> None:
self.stack: list[str] = []
def _record(self, node: ast.AST, kind: str, name: str) -> None:
qualified = ".".join([*self.stack, name]) if self.stack else name
haystack = f"{item['path']} {qualified}".lower()
score = sum(3 if term in haystack else 0 for term in terms)
if name.startswith("test_"):
score += 12
result.append(
{
"path": item["path"],
"symbol": qualified,
"kind": kind,
"line": int(getattr(node, "lineno", 1)),
"end_line": int(getattr(node, "end_lineno", getattr(node, "lineno", 1))),
"score": score + int(item["score"]),
}
)
def visit_ClassDef(self, node: ast.ClassDef) -> Any:
self._record(node, "class", node.name)
self.stack.append(node.name)
self.generic_visit(node)
self.stack.pop()
def visit_FunctionDef(self, node: ast.FunctionDef) -> Any:
self._record(node, "function", node.name)
self.stack.append(node.name)
self.generic_visit(node)
self.stack.pop()
def visit_AsyncFunctionDef(self, node: ast.AsyncFunctionDef) -> Any:
self._record(node, "async_function", node.name)
self.stack.append(node.name)
self.generic_visit(node)
self.stack.pop()
Visitor().visit(tree)
return result
def _surface_ref(symbol: Mapping[str, Any]) -> str:
return f"{symbol['path']}::{symbol['symbol']}"
def _contract_map(
symbols: Sequence[Mapping[str, Any]],
modules: Sequence[Mapping[str, Any]],
contract_terms: Mapping[str, tuple[str, ...]],
) -> dict[str, Any]:
result: dict[str, Any] = {}
module_paths = [str(item["path"]) for item in modules]
for contract, terms in contract_terms.items():
scored: list[tuple[int, str]] = []
for symbol in symbols:
haystack = _surface_ref(symbol).lower()
score = sum(1 for term in terms if term in haystack)
if score:
scored.append((score, _surface_ref(symbol)))
scored.sort(key=lambda item: (-item[0], item[1]))
surfaces = [value for _, value in scored[:6]]
if not surfaces:
for path in module_paths:
score = sum(1 for term in terms if term in path.lower())
if score:
surfaces.append(path)
if len(surfaces) == 6:
break
result[contract] = {
"status": "PARTIAL" if surfaces else "GAP",
"surfaces": surfaces,
}
return result
def _gap_evidence(
symbols: Sequence[Mapping[str, Any]],
profile: str,
) -> tuple[list[str], list[str], list[str]]:
haystack = "\n".join(_surface_ref(item).lower() for item in symbols)
if profile == "model_governance":
invariants = [
"Online Task claim/heartbeat remains the authoritative worker ownership boundary.",
"Provider/model credentials remain outside uncredentialed planner/executor processes.",
"P0-A tenant/project scope and P0-C trace/audit lineage must constrain model routing.",
"P1 retry/checkpoint semantics remain authoritative for model-call execution recovery.",
]
failure_windows: list[str] = []
if "provider" not in haystack and "model" not in haystack:
failure_windows.append("No first-class model/provider registry surface detected.")
if "budget" not in haystack and "cost" not in haystack and "quota" not in haystack:
failure_windows.append("No explicit durable model budget/cost/quota accounting surface detected.")
if "fallback" not in haystack and "route" not in haystack:
failure_windows.append("No explicit model fallback/routing policy surface detected.")
if "eval" not in haystack and "approval" not in haystack:
failure_windows.append("No explicit evaluation/approval gate surface detected for model changes.")
if "version" not in haystack and "pin" not in haystack:
failure_windows.append("No explicit model-version pinning surface detected.")
desired = (
("trace", "P0-C trace/audit lineage"),
("scope", "P0-A tenant/project scope isolation"),
("credential", "P0-C credential governance"),
("checkpoint", "P1 durable execution recovery"),
("message", "P0-B MessageEnvelope/handoff"),
("retry", "P1 retry/idempotency"),
)
else:
invariants = [
"Online Task claim/heartbeat remains the authoritative worker ownership boundary.",
"Governed mutation remains separated from read-only analysis and requires WriterLease.",
"P1 attempt-aware result idempotency and durable checkpoint recovery are reusable fleet primitives.",
"Scope/trace/message lineage must remain attached when work is scheduled or reassigned.",
]
failure_windows = []
if "scheduler" not in haystack:
failure_windows.append("No first-class durable fleet scheduler symbol detected; worker-local polling may still be the scheduling mechanism.")
if "capacity" not in haystack and "slot" not in haystack:
failure_windows.append("No explicit durable worker-capacity/slot primitive detected.")
if "fair" not in haystack and "starvation" not in haystack:
failure_windows.append("No explicit fairness/starvation policy symbol detected.")
if "circuit" not in haystack:
failure_windows.append("No explicit circuit-breaker primitive detected for repeated worker/task failures.")
if "reassign" not in haystack:
failure_windows.append("No explicit worker-reassignment primitive detected beyond lease expiry/retry semantics.")
desired = (
("checkpoint", "P1 durable checkpoint/resume"),
("message", "P0-B MessageEnvelope/handoff"),
("trace", "P0-C trace correlation"),
("scope", "P0-A scope isolation"),
("lease", "claim/WriterLease ownership"),
("heartbeat", "worker liveness"),
("idempot", "retry-safe dispatch/result"),
)
reuse_points: list[str] = []
for token, label in desired:
match = next((item for item in symbols if token in _surface_ref(item).lower()), None)
if match is not None:
reuse_points.append(f"{label}: {_surface_ref(match)}")
return invariants, failure_windows, reuse_points
def _trim_to_budget(value: dict[str, Any]) -> dict[str, Any]:
def size() -> int:
return len(json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":")).encode("utf-8"))
while size() > TARGET_RESULT_BYTES and value["bounded_excerpts"]:
value["bounded_excerpts"].pop()
while size() > TARGET_RESULT_BYTES and len(value["implementation_map"]["functions"]) > 20:
value["implementation_map"]["functions"].pop()
while size() > TARGET_RESULT_BYTES and len(value["implementation_map"]["classes"]) > 15:
value["implementation_map"]["classes"].pop()
while size() > TARGET_RESULT_BYTES and len(value["implementation_map"]["tests"]) > 15:
value["implementation_map"]["tests"].pop()
while size() > TARGET_RESULT_BYTES and len(value["implementation_map"]["modules"]) > 20:
value["implementation_map"]["modules"].pop()
final_size = size()
if final_size > MAX_RESULT_BYTES:
raise CompactInterfaceMapError("INTERFACE_MAP_RESULT_TOO_LARGE")
value["result_bytes"] = final_size
return value
def build_compact_interface_map(
worktree: Path,
source_signals: Sequence[str],
*,
source_commit: str,
) -> dict[str, Any]:
root = Path(worktree)
if not root.is_dir() or root.is_symlink():
raise CompactInterfaceMapError("INTERFACE_MAP_WORKTREE_INVALID")
if re.fullmatch(r"[0-9a-f]{40}", source_commit) is None:
raise CompactInterfaceMapError("INTERFACE_MAP_SOURCE_COMMIT_INVALID")
profile = _profile(source_signals)
terms = _signal_terms(source_signals)
candidates = _candidate_files(root, terms)
if not candidates:
raise CompactInterfaceMapError("INTERFACE_MAP_EMPTY")
module_candidates = candidates[:MAX_MODULES]
modules = [
{
"path": item["path"],
"sha256": item["sha256"],
"bytes": item["bytes"],
"score": item["score"],
}
for item in module_candidates
]
symbols: list[dict[str, Any]] = []
tables: list[dict[str, Any]] = []
systemd_units: list[dict[str, Any]] = []
for item in candidates:
if item["suffix"] == ".py":
symbols.extend(_python_symbols(item, terms))
elif item["suffix"] == ".sql":
for table in SQL_TABLE_RE.findall(str(item["text"])):
tables.append({"path": item["path"], "table": table, "sha256": item["sha256"]})
elif item["suffix"] == ".service":
systemd_units.append({"path": item["path"], "unit": Path(str(item["path"])).name, "sha256": item["sha256"]})
symbols.sort(key=lambda item: (-int(item["score"]), str(item["path"]), str(item["symbol"])))
all_symbols = list(symbols)
test_symbols = [
item for item in all_symbols
if str(item["path"]).startswith("tests/") or str(item["symbol"]).split(".")[-1].startswith("test_")
][: min(MAX_TESTS, 24)]
non_test_symbols = [
item for item in all_symbols
if item not in test_symbols
][: max(0, MAX_SYMBOLS - len(test_symbols))]
symbols = [*test_symbols, *non_test_symbols]
symbols.sort(key=lambda item: (-int(item["score"]), str(item["path"]), str(item["symbol"])))
classes = [
{k: item[k] for k in ("path", "symbol", "line", "end_line")}
for item in symbols
if item["kind"] == "class"
]
functions = [
{k: item[k] for k in ("path", "symbol", "line", "end_line")}
for item in symbols
if item["kind"] in {"function", "async_function"} and not str(item["symbol"]).split(".")[-1].startswith("test_")
]
tests = [
{k: item[k] for k in ("path", "symbol", "line", "end_line")}
for item in symbols
if str(item["path"]).startswith("tests/") or str(item["symbol"]).split(".")[-1].startswith("test_")
][:MAX_TESTS]
excerpts: list[dict[str, Any]] = []
by_path = {str(item["path"]): item for item in candidates}
for symbol in symbols:
if len(excerpts) == MAX_EXCERPTS:
break
item = by_path.get(str(symbol["path"]))
if item is None:
continue
excerpt = _bounded_excerpt(str(item["text"]), int(symbol["line"]), int(symbol["end_line"]))
if not excerpt.strip():
continue
excerpts.append(
{
"path": symbol["path"],
"symbol": symbol["symbol"],
"sha256": item["sha256"],
"excerpt": excerpt,
}
)
invariants, failure_windows, reuse_points = _gap_evidence(all_symbols, profile)
result = {
"schema": SCHEMA,
"profile": profile,
"classification": PROFILE_CLASSIFICATION[profile],
"source_commit": source_commit,
"implementation_map": {
"modules": modules,
"classes": classes[:MAX_SYMBOLS],
"functions": functions[:MAX_SYMBOLS],
"tables": tables[:40],
"systemd_units": systemd_units[:20],
"tests": tests,
"current_invariants": invariants,
"failure_windows": failure_windows,
"reuse_points": reuse_points,
},
"bounded_excerpts": excerpts,
"contracts": _contract_map(all_symbols, modules, _contract_terms(profile)),
}
return _trim_to_budget(result)
@@ -0,0 +1,528 @@
"""Canonical governed execution projection for Online Agent Tasks.
The durable governed binding is authoritative. Natural-language task fields and
worker configuration are never allowed to invent repository, ref, generation, or
idempotency coordinates.
"""
from __future__ import annotations
from dataclasses import asdict, dataclass
import hashlib
import json
import re
from pathlib import Path
from typing import Any, Mapping, Sequence
import urllib.error
import urllib.request
EXECUTION_SCHEMA = "aios.online-agent.execution.v2"
PROJECTION_SCHEMA = "aios.online-agent.task-projection.v1"
OID_RE = re.compile(r"^[0-9a-f]{40}$")
TASK_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{2,179}$")
BRANCH_RE = re.compile(r"^agent/[A-Za-z0-9][A-Za-z0-9._/-]{2,179}$")
KEY_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{7,159}$")
PROTECTED_PREFIXES = ("main", "master", "refs/", "tags/", "tag/", "release/", "baseline/", "production/")
MUTATION_ACTIONS = frozenset({"deterministic"})
READ_ONLY_ACTIONS = frozenset({"source_capture", "agent_runtime"})
SUPPORTED_ACTIONS = frozenset({"deterministic", "source_capture", "agent_runtime"})
AGENT_RUNTIME_ADAPTERS = frozenset({"hermes", "openclaw"})
MAX_AGENT_RUNTIME_INPUT_BYTES = 64 * 1024
class ExecutionProjectionError(RuntimeError):
"""A canonical fail-closed projection or validation error."""
class OnlineTaskProjectionClient:
"""Official HTTP projection client using only a control credential reference."""
def __init__(self, api_base: str, control_token_path: Path) -> None:
self.api_base = api_base.rstrip("/")
self.control_token_path = Path(control_token_path)
def _token(self) -> str:
path = self.control_token_path
if path.is_symlink() or not path.is_file():
_error("ONLINE_TASK_PROJECTION_CONTROL_UNAVAILABLE")
value = path.read_text(encoding="utf-8").strip()
if len(value) < 32:
_error("ONLINE_TASK_PROJECTION_CONTROL_INVALID")
return value
def create(
self,
projected_task: Mapping[str, Any],
*,
binding: "GovernedExecutionBinding",
) -> dict[str, Any]:
body = dict(projected_task)
payload = _canonical(body).encode("utf-8")
request = urllib.request.Request(
self.api_base + "/v1/tasks",
method="POST",
data=payload,
headers={
"Authorization": "Bearer " + self._token(),
"Content-Type": "application/json",
"Accept": "application/json",
},
)
try:
with urllib.request.urlopen(request, timeout=15) as response:
result = json.loads(response.read().decode("utf-8"))
status = response.status
except urllib.error.HTTPError as exc:
raw = exc.read(8192).decode("utf-8", "replace")
try:
error = json.loads(raw).get("error")
except (AttributeError, json.JSONDecodeError):
error = "HTTP_ERROR"
raise ExecutionProjectionError(f"ONLINE_TASK_PROJECTION_FAILED:{exc.code}:{error}") from exc
except (OSError, urllib.error.URLError, TimeoutError) as exc:
raise ExecutionProjectionError("ONLINE_TASK_PROJECTION_UNAVAILABLE") from exc
if status not in {200, 201} or not isinstance(result, dict):
_error("ONLINE_TASK_PROJECTION_RESPONSE_INVALID")
if result.get("task_id") != binding.task_id or result.get("status") not in {"QUEUED", "VERIFYING", "SUCCEEDED"}:
_error("ONLINE_TASK_PROJECTION_RESPONSE_INVALID")
metadata = result.get("metadata")
if not isinstance(metadata, Mapping):
_error("ONLINE_TASK_PROJECTION_RESPONSE_INVALID")
returned_execution = validate_execution_envelope(
_object(metadata.get("execution"), "ONLINE_TASK_PROJECTION_RESPONSE_INVALID"),
binding=binding,
)
submitted_metadata = _object(body.get("metadata"), "ONLINE_TASK_PROJECTION_REQUEST_INVALID")
submitted_execution = validate_execution_envelope(
_object(submitted_metadata.get("execution"), "ONLINE_TASK_PROJECTION_REQUEST_INVALID"),
binding=binding,
)
if _canonical(returned_execution) != _canonical(submitted_execution):
_error("ONLINE_TASK_PROJECTION_RESPONSE_BINDING_INVALID")
return result
def _error(code: str) -> None:
raise ExecutionProjectionError(code)
def _object(value: object, code: str) -> dict[str, Any]:
if not isinstance(value, Mapping):
_error(code)
return dict(value)
def _string(value: object, code: str) -> str:
if not isinstance(value, str) or not value.strip():
_error(code)
return value.strip()
def _task_id(value: object) -> str:
value = _string(value, "TASK_ID_INVALID")
if TASK_RE.fullmatch(value) is None:
_error("TASK_ID_INVALID")
return value
def _oid(value: object, code: str) -> str:
value = _string(value, code)
if OID_RE.fullmatch(value) is None:
_error(code)
return value
def _branch(value: object) -> str:
value = _string(value, "TARGET_BRANCH_INVALID")
if BRANCH_RE.fullmatch(value) is None or value.startswith(PROTECTED_PREFIXES):
_error("TARGET_BRANCH_INVALID")
return value
def _generation(value: object) -> int:
if not isinstance(value, int) or isinstance(value, bool) or value < 1:
_error("GENERATION_INVALID")
return value
def _canonical(value: object) -> str:
return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))
def _digest(value: object) -> str:
return hashlib.sha256(_canonical(value).encode("utf-8")).hexdigest()
def _source_signals(value: object) -> tuple[str, ...]:
"""Normalize untrusted natural-language hints before they enter a signed envelope."""
if value is None:
return ()
if not isinstance(value, Sequence) or isinstance(value, (str, bytes)):
_error("SOURCE_CAPTURE_SIGNALS_INVALID")
result: list[str] = []
for item in value:
if not isinstance(item, str):
_error("SOURCE_CAPTURE_SIGNALS_INVALID")
text = " ".join(item.split()).strip()
if not text:
continue
# Projection may accept long task prose, but execution envelopes stay bounded.
text = text[:512].rstrip()
if text not in result:
result.append(text)
if len(result) == 16:
break
return tuple(result)
def _agent_runtime_input(value: object) -> dict[str, Any]:
if not isinstance(value, Mapping):
_error("AGENT_RUNTIME_INPUT_REQUIRED")
raw = dict(value)
allowed = {
"goal", "instructions", "success_criteria", "constraints",
"scope", "trace_id",
}
if set(raw) - allowed:
_error("AGENT_RUNTIME_INPUT_FIELDS_INVALID")
goal = _string(raw.get("goal"), "AGENT_RUNTIME_GOAL_REQUIRED")
instructions = raw.get("instructions", "")
if not isinstance(instructions, str):
_error("AGENT_RUNTIME_INSTRUCTIONS_INVALID")
criteria = raw.get("success_criteria", [])
if not isinstance(criteria, list) or len(criteria) > 32:
_error("AGENT_RUNTIME_SUCCESS_CRITERIA_INVALID")
normalized_criteria: list[dict[str, str]] = []
for item in criteria:
if not isinstance(item, Mapping):
_error("AGENT_RUNTIME_SUCCESS_CRITERIA_INVALID")
criterion_id = _string(item.get("id"), "AGENT_RUNTIME_SUCCESS_CRITERIA_INVALID")
description = _string(
item.get("description"), "AGENT_RUNTIME_SUCCESS_CRITERIA_INVALID"
)
normalized_criteria.append(
{"id": criterion_id[:120], "description": description[:1000]}
)
constraints = raw.get("constraints", {})
if not isinstance(constraints, Mapping):
_error("AGENT_RUNTIME_CONSTRAINTS_INVALID")
sensitive_markers = ("token", "secret", "password", "credential", "authorization", "api_key", "apikey")
for key in constraints:
lowered = str(key).lower()
if any(marker in lowered for marker in sensitive_markers):
_error("AGENT_RUNTIME_SECRET_FIELD_DENY")
scope = raw.get("scope", {})
if not isinstance(scope, Mapping):
_error("AGENT_RUNTIME_SCOPE_INVALID")
scope = {
"scope_type": str(scope.get("scope_type") or "legacy"),
"tenant_id": scope.get("tenant_id"),
"project_id": scope.get("project_id"),
}
if scope["scope_type"] not in {"legacy", "system", "tenant"}:
_error("AGENT_RUNTIME_SCOPE_INVALID")
if scope["scope_type"] == "tenant":
if not isinstance(scope["tenant_id"], str) or not scope["tenant_id"].strip():
_error("AGENT_RUNTIME_SCOPE_INVALID")
elif scope["tenant_id"] not in (None, "") or scope["project_id"] not in (None, ""):
_error("AGENT_RUNTIME_SCOPE_INVALID")
trace_id = str(raw.get("trace_id") or "").strip()
if not trace_id or len(trace_id) > 255:
_error("AGENT_RUNTIME_TRACE_ID_INVALID")
normalized = {
"goal": goal[:12000],
"instructions": instructions[:32000],
"success_criteria": normalized_criteria,
"constraints": dict(constraints),
"scope": scope,
"trace_id": trace_id,
}
if len(_canonical(normalized).encode("utf-8")) > MAX_AGENT_RUNTIME_INPUT_BYTES:
_error("AGENT_RUNTIME_INPUT_TOO_LARGE")
return normalized
def _capabilities(value: object) -> tuple[str, ...]:
if value is None:
return ()
if not isinstance(value, Sequence) or isinstance(value, (str, bytes)):
_error("CAPABILITY_SCOPE_INVALID")
result = tuple(sorted({_string(item, "CAPABILITY_SCOPE_INVALID") for item in value}))
if any(len(item) > 120 for item in result):
_error("CAPABILITY_SCOPE_INVALID")
return result
@dataclass(frozen=True, slots=True)
class GovernedExecutionBinding:
task_id: str
job_id: str
repository: str
base_oid: str
expected_branch_oid: str
target_branch: str
branch_mode: str
generation: int
idempotency_key: str
@classmethod
def from_mapping(cls, value: Mapping[str, Any]) -> "GovernedExecutionBinding":
data = dict(value)
task_id = _task_id(data.get("task_id"))
job_id = _string(data.get("job_id"), "JOB_ID_REQUIRED")
if TASK_RE.fullmatch(job_id) is None:
_error("JOB_ID_INVALID")
repository = _string(data.get("repository"), "REPOSITORY_INVALID")
if repository != "aios/aios":
_error("WRONG_REPOSITORY_ENVELOPE_DENY")
base_oid = _oid(data.get("base_oid"), "BASE_OID_INVALID")
expected = _oid(data.get("expected_branch_oid"), "EXPECTED_BRANCH_OID_INVALID")
target = _branch(data.get("target_branch"))
mode = _string(data.get("branch_mode"), "BRANCH_MODE_MISSING")
if mode not in {"CREATE_NEW", "CONTINUE_EXISTING"}:
_error("BRANCH_MODE_INVALID")
generation = _generation(data.get("generation"))
if mode == "CREATE_NEW" and expected != base_oid:
_error("EXPECTED_BASE_CONFLICT")
key = _string(data.get("idempotency_key"), "IDEMPOTENCY_KEY_INVALID")
if KEY_RE.fullmatch(key) is None:
_error("IDEMPOTENCY_KEY_INVALID")
return cls(task_id, job_id, repository, base_oid, expected, target, mode, generation, key)
def public(self) -> dict[str, Any]:
return asdict(self)
def binding_from_bootstrap(value: Mapping[str, Any]) -> GovernedExecutionBinding:
return GovernedExecutionBinding.from_mapping(value)
def validate_execution_envelope(
value: Mapping[str, Any],
*,
binding: GovernedExecutionBinding | None = None,
worker_capabilities: Sequence[str] | None = None,
) -> dict[str, Any]:
envelope = dict(value)
if envelope.get("schema") != EXECUTION_SCHEMA:
_error("EXECUTION_ENVELOPE_SCHEMA_VALIDATION")
candidate = GovernedExecutionBinding.from_mapping(envelope)
if binding is not None and candidate != binding:
comparisons = (
(candidate.task_id, binding.task_id, "TASK_BINDING_CONFLICT"),
(candidate.job_id, binding.job_id, "JOB_BINDING_CONFLICT"),
(candidate.repository, binding.repository, "WRONG_REPOSITORY_ENVELOPE_DENY"),
(candidate.base_oid, binding.base_oid, "WRONG_BASE_ENVELOPE_DENY"),
(candidate.expected_branch_oid, binding.expected_branch_oid, "WRONG_EXPECTED_HEAD_ENVELOPE_DENY"),
(candidate.target_branch, binding.target_branch, "WRONG_BRANCH_ENVELOPE_DENY"),
(candidate.generation, binding.generation, "WRONG_GENERATION_ENVELOPE_DENY"),
(candidate.branch_mode, binding.branch_mode, "BRANCH_MODE_BINDING_CONFLICT"),
(candidate.idempotency_key, binding.idempotency_key, "IDEMPOTENCY_BINDING_CONFLICT"),
)
for actual, expected, code in comparisons:
if actual != expected:
_error(code)
action = _string(envelope.get("action"), "EXECUTION_ACTION_MISSING")
if action not in SUPPORTED_ACTIONS:
_error("EXECUTION_ACTION_UNSUPPORTED")
capability_scope = _capabilities(envelope.get("capability_scope"))
if worker_capabilities is not None and not set(capability_scope).issubset(set(worker_capabilities)):
_error("CAPABILITY_MISMATCH_DENY")
workspace = _object(envelope.get("workspace"), "WORKSPACE_POLICY_REQUIRED")
if workspace.get("authority_git_dir") != "/data/aios-gitops/repo.git" or workspace.get("read_only_repository") is not True:
_error("WORKSPACE_POLICY_DENY")
governance = _object(envelope.get("governance"), "GOVERNANCE_POLICY_REQUIRED")
governed_coordinates = (
(governance.get("task_id"), candidate.task_id, "TASK_BINDING_CONFLICT"),
(governance.get("job_id"), candidate.job_id, "JOB_BINDING_CONFLICT"),
(governance.get("idempotency_key"), candidate.idempotency_key, "IDEMPOTENCY_BINDING_CONFLICT"),
)
for actual_coordinate, expected_coordinate, code in governed_coordinates:
if actual_coordinate != expected_coordinate:
_error(code)
commit_key = _string(governance.get("commit_idempotency_key"), "COMMIT_IDEMPOTENCY_KEY_INVALID")
if KEY_RE.fullmatch(commit_key) is None:
_error("COMMIT_IDEMPOTENCY_KEY_INVALID")
if governance.get("writer_lease") is not (action in MUTATION_ACTIONS):
_error("WRITER_LEASE_POLICY_DENY")
for field in ("main_write", "production_write", "tag_write", "baseline_write", "force_push", "protected_ref_bypass"):
if governance.get(field, False) is not False:
_error("POLICY_DENY")
changes = envelope.get("change_set")
verification = envelope.get("verification")
if not isinstance(changes, list) or not isinstance(verification, list):
_error("EXECUTION_OPERATIONS_INVALID")
if action == "source_capture":
if changes or verification:
_error("SOURCE_CAPTURE_MUTATION_DENY")
signals = list(_source_signals(envelope.get("source_signals") or []))
if envelope.get("source_signals") != signals:
_error("SOURCE_CAPTURE_SIGNALS_INVALID")
if envelope.get("runtime_input") not in (None, {}):
_error("SOURCE_CAPTURE_RUNTIME_INPUT_DENY")
elif action == "agent_runtime":
if changes or verification:
_error("AGENT_RUNTIME_MUTATION_DENY")
adapter = str(envelope.get("adapter") or "")
if adapter not in AGENT_RUNTIME_ADAPTERS:
_error("AGENT_RUNTIME_ADAPTER_INVALID")
runtime_input = _agent_runtime_input(envelope.get("runtime_input"))
if envelope.get("runtime_input") != runtime_input:
_error("AGENT_RUNTIME_INPUT_NOT_CANONICAL")
required_runtime_capability = f"agent.runtime.{adapter}"
if required_runtime_capability not in capability_scope:
_error("AGENT_RUNTIME_CAPABILITY_REQUIRED")
if envelope.get("source_signals") not in (None, []):
_error("AGENT_RUNTIME_SOURCE_SIGNALS_DENY")
elif not changes:
_error("CHANGE_SET_INVALID")
declared = str(envelope.get("execution_digest") or "")
material = {key: item for key, item in envelope.items() if key != "execution_digest"}
actual = _digest(material)
if declared != actual:
_error("EXECUTION_DIGEST_INVALID")
return {**material, "execution_digest": actual}
def project_execution_envelope(
binding: GovernedExecutionBinding,
execution_request: Mapping[str, Any] | None = None,
*,
required_capabilities: Sequence[str] = (),
) -> dict[str, Any]:
request = dict(execution_request or {})
action = str(request.get("action") or "source_capture")
if action not in SUPPORTED_ACTIONS:
_error("EXECUTION_ACTION_UNSUPPORTED")
capabilities = _capabilities(request.get("capability_scope", required_capabilities))
governance = {
"task_id": binding.task_id,
"job_id": binding.job_id,
"idempotency_key": binding.idempotency_key,
"commit_idempotency_key": str(request.get("commit_idempotency_key") or f"{binding.task_id}-commit-g{binding.generation}"),
"writer_lease": action in MUTATION_ACTIONS,
"main_write": False,
"production_write": False,
"tag_write": False,
"baseline_write": False,
"force_push": False,
"protected_ref_bypass": False,
}
supplied_governance = request.get("governance")
if supplied_governance is not None:
governance.update(_object(supplied_governance, "GOVERNANCE_POLICY_REQUIRED"))
material: dict[str, Any] = {
"schema": EXECUTION_SCHEMA,
**binding.public(),
"action": action,
"adapter": str(
request.get("adapter")
or ("deterministic" if action == "deterministic" else "source_capture")
),
"capability_scope": list(capabilities),
"change_set": list(request.get("change_set") or []),
"verification": list(request.get("verification") or []),
"source_signals": list(_source_signals(request.get("source_signals") or [])) if action == "source_capture" else [],
"runtime_input": _agent_runtime_input(request.get("runtime_input")) if action == "agent_runtime" else {},
"governance": governance,
"workspace": {
"authority_git_dir": "/data/aios-gitops/repo.git",
"read_only_repository": True,
"cleanup_policy": "always",
},
}
envelope = {**material, "execution_digest": _digest(material)}
return validate_execution_envelope(envelope, binding=binding)
def project_online_task(
binding: GovernedExecutionBinding,
task_request: Mapping[str, Any],
execution_request: Mapping[str, Any] | None = None,
) -> dict[str, Any]:
request = dict(task_request)
required = _capabilities(request.get("required_capabilities"))
execution_value = dict(execution_request or {})
if str(execution_value.get("action") or "source_capture") == "agent_runtime":
adapter = str(execution_value.get("adapter") or "")
if adapter not in AGENT_RUNTIME_ADAPTERS:
_error("AGENT_RUNTIME_ADAPTER_INVALID")
runtime_input = execution_value.get("runtime_input")
tenant_id = request.get("tenant_id")
project_id = request.get("project_id")
declared_scope = str(
request.get("scope_type")
or ("tenant" if isinstance(tenant_id, str) and tenant_id.strip() else "legacy")
)
if runtime_input is None:
runtime_input = {
"goal": request.get("goal"),
"instructions": request.get("instructions") or "",
"success_criteria": request.get("success_criteria") or [],
"constraints": request.get("constraints") or {},
}
elif not isinstance(runtime_input, Mapping):
_error("AGENT_RUNTIME_INPUT_REQUIRED")
runtime_input = dict(runtime_input)
# Scope/trace are authoritative Task identity, never caller-selected runtime hints.
runtime_input["scope"] = {
"scope_type": declared_scope,
"tenant_id": tenant_id,
"project_id": project_id,
}
runtime_input["trace_id"] = str(
request.get("trace_id")
or request.get("run_id")
or binding.task_id
)
execution_value["runtime_input"] = runtime_input
execution = project_execution_envelope(
binding,
execution_value,
required_capabilities=required,
)
metadata = dict(request.get("metadata") or {})
if "execution" in metadata:
validate_execution_envelope(_object(metadata["execution"], "EXECUTION_ENVELOPE_INVALID"), binding=binding)
if metadata["execution"] != execution:
_error("EXECUTION_PROJECTION_CONFLICT")
metadata["execution"] = execution
metadata["execution_projection"] = {
"schema": PROJECTION_SCHEMA,
"task_id": binding.task_id,
"job_id": binding.job_id,
"execution_digest": execution["execution_digest"],
}
return {
**request,
"task_id": binding.task_id,
"metadata": metadata,
"required_capabilities": list(required),
"idempotency_key": str(request.get("idempotency_key") or f"online-task:{binding.idempotency_key}"),
}
def resolve_task_execution(
task: Mapping[str, Any],
binding: GovernedExecutionBinding,
*,
worker_capabilities: Sequence[str] | None = None,
) -> tuple[dict[str, Any], str]:
metadata = task.get("metadata")
if isinstance(metadata, Mapping) and "execution" in metadata:
value = validate_execution_envelope(
_object(metadata["execution"], "EXECUTION_ENVELOPE_INVALID"),
binding=binding,
worker_capabilities=worker_capabilities,
)
return value, "persisted"
projected = project_execution_envelope(
binding,
{"action": "source_capture", "capability_scope": task.get("required_capabilities") or []},
required_capabilities=task.get("required_capabilities") or [],
)
validate_execution_envelope(projected, binding=binding, worker_capabilities=worker_capabilities)
return projected, "recovered_projection"
@@ -0,0 +1,187 @@
"""Durable, authority-bound deterministic execution plans for Online Agent Tasks."""
from __future__ import annotations
from contextlib import contextmanager
from dataclasses import dataclass
import fcntl
import hashlib
import json
from pathlib import Path
from typing import Any, Mapping
from aios_online_agent_execution import (
GovernedExecutionBinding,
project_execution_envelope,
)
class ExecutionPlanError(RuntimeError):
pass
def _canonical(value: object) -> bytes:
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
def _digest(value: object) -> str:
return hashlib.sha256(_canonical(value)).hexdigest()
def _task_key(task_id: str) -> str:
readable = "".join(ch if ch.isalnum() or ch in "._-" else "-" for ch in task_id).strip("-")[:72] or "task"
return readable + "-" + hashlib.sha256(task_id.encode("utf-8")).hexdigest()[:24]
@dataclass(frozen=True)
class ExecutionPlanRecord:
task_id: str
job_id: str
source_head_oid: str
generation: int
idempotency_key: str
plan_digest: str
execution: dict[str, Any]
def public(self) -> dict[str, Any]:
return {
"schema": "aios.online-agent.plan-record.v1",
"task_id": self.task_id,
"job_id": self.job_id,
"source_head_oid": self.source_head_oid,
"generation": self.generation,
"idempotency_key": self.idempotency_key,
"plan_digest": self.plan_digest,
"execution": dict(self.execution),
}
class ExecutionPlanStore:
def __init__(self, root: Path) -> None:
self.root = Path(root)
self.root.mkdir(parents=True, exist_ok=True)
def _path(self, task_id: str) -> Path:
return self.root / (_task_key(task_id) + ".json")
@contextmanager
def _lock(self, task_id: str):
path = self.root / (_task_key(task_id) + ".lock")
with path.open("a+b") as handle:
fcntl.flock(handle.fileno(), fcntl.LOCK_EX)
try:
yield
finally:
fcntl.flock(handle.fileno(), fcntl.LOCK_UN)
@staticmethod
def _source_head(binding: GovernedExecutionBinding) -> str:
return binding.expected_branch_oid if binding.branch_mode == "CONTINUE_EXISTING" else binding.base_oid
def _read(self, task_id: str) -> ExecutionPlanRecord | None:
path = self._path(task_id)
if not path.is_file() or path.is_symlink():
return None
value = json.loads(path.read_text(encoding="utf-8"))
if not isinstance(value, Mapping) or value.get("schema") != "aios.online-agent.plan-record.v1":
raise ExecutionPlanError("EXECUTION_PLAN_RECORD_INVALID")
execution = value.get("execution")
if not isinstance(execution, Mapping):
raise ExecutionPlanError("EXECUTION_PLAN_RECORD_INVALID")
return ExecutionPlanRecord(
task_id=str(value.get("task_id") or ""),
job_id=str(value.get("job_id") or ""),
source_head_oid=str(value.get("source_head_oid") or ""),
generation=int(value.get("generation") or 0),
idempotency_key=str(value.get("idempotency_key") or ""),
plan_digest=str(value.get("plan_digest") or ""),
execution=dict(execution),
)
def register(
self,
binding: GovernedExecutionBinding,
request: Mapping[str, Any],
) -> dict[str, Any]:
with self._lock(binding.task_id):
return self._register_locked(binding, request)
def _register_locked(
self,
binding: GovernedExecutionBinding,
request: Mapping[str, Any],
) -> dict[str, Any]:
if request.get("schema") != "aios.online-agent.plan.v1":
raise ExecutionPlanError("EXECUTION_PLAN_SCHEMA_INVALID")
if request.get("task_id") != binding.task_id or request.get("job_id") != binding.job_id:
raise ExecutionPlanError("EXECUTION_PLAN_BINDING_CONFLICT")
if int(request.get("generation") or 0) != binding.generation:
raise ExecutionPlanError("EXECUTION_PLAN_GENERATION_CONFLICT")
source_head = str(request.get("source_head_oid") or "")
if source_head != self._source_head(binding):
raise ExecutionPlanError("EXECUTION_PLAN_SOURCE_HEAD_CONFLICT")
key = str(request.get("idempotency_key") or "")
if len(key) < 8:
raise ExecutionPlanError("EXECUTION_PLAN_IDEMPOTENCY_INVALID")
execution_request = request.get("execution")
if not isinstance(execution_request, Mapping) or execution_request.get("action") != "deterministic":
raise ExecutionPlanError("EXECUTION_PLAN_DETERMINISTIC_REQUIRED")
envelope = project_execution_envelope(binding, execution_request)
material = {
"task_id": binding.task_id,
"job_id": binding.job_id,
"source_head_oid": source_head,
"generation": binding.generation,
"idempotency_key": key,
"execution": dict(execution_request),
}
plan_digest = _digest(material)
existing = self._read(binding.task_id)
expected_previous = request.get("expected_previous_plan_digest")
if existing is not None:
if existing.idempotency_key == key:
if existing.plan_digest != plan_digest:
raise ExecutionPlanError("EXECUTION_PLAN_IDEMPOTENCY_CONFLICT")
return {**existing.public(), "idempotent": True, "execution_digest": envelope["execution_digest"]}
if expected_previous != existing.plan_digest:
raise ExecutionPlanError("EXECUTION_PLAN_REPLACEMENT_CAS_MISMATCH")
if existing.source_head_oid != source_head:
raise ExecutionPlanError("EXECUTION_PLAN_STALE_REPLACEMENT_DENY")
elif expected_previous not in (None, ""):
raise ExecutionPlanError("EXECUTION_PLAN_REPLACEMENT_CAS_MISMATCH")
record = ExecutionPlanRecord(
task_id=binding.task_id,
job_id=binding.job_id,
source_head_oid=source_head,
generation=binding.generation,
idempotency_key=key,
plan_digest=plan_digest,
execution=dict(execution_request),
)
path = self._path(binding.task_id)
temporary = path.with_suffix(".tmp")
temporary.write_text(
json.dumps(record.public(), sort_keys=True, separators=(",", ":")),
encoding="utf-8",
)
temporary.replace(path)
return {**record.public(), "idempotent": False, "execution_digest": envelope["execution_digest"]}
def resolve(self, binding: GovernedExecutionBinding) -> dict[str, Any] | None:
record = self._read(binding.task_id)
if record is None:
return None
if (
record.task_id != binding.task_id
or record.job_id != binding.job_id
or record.generation != binding.generation
):
raise ExecutionPlanError("EXECUTION_PLAN_BINDING_CONFLICT")
if record.source_head_oid != self._source_head(binding):
raise ExecutionPlanError("EXECUTION_PLAN_SOURCE_HEAD_DRIFT")
envelope = project_execution_envelope(binding, record.execution)
return {
"execution": dict(record.execution),
"plan_digest": record.plan_digest,
"execution_digest": envelope["execution_digest"],
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,82 @@
"""Immutable release-lineage contracts used by activation and recovery."""
from __future__ import annotations
from dataclasses import asdict, dataclass
from typing import Any, Mapping
@dataclass(frozen=True)
class ReleaseLineage:
task_id: str
candidate_id: str
source_commit: str
version: str
required_capabilities: tuple[str, ...]
def as_dict(self) -> dict[str, Any]:
return asdict(self)
FIXED_V2R2C2 = ReleaseLineage(
task_id="r42-release-candidate-v2r2c2-live-20260819",
candidate_id="r42-release-candidate-v2r2c2-20260819",
source_commit="71e736b05fab13efc06af7a4fe2dc022c1cf9a85",
version="V2R2C2",
required_capabilities=("release:candidate",),
)
class ReleaseLineageError(RuntimeError):
pass
def validate_fixed_v2r2c2_task(task: Mapping[str, Any]) -> dict[str, Any]:
"""Validate identity/source/version without mutating Task state.
This contract intentionally does not bind status/attempt/revision, because
those are lifecycle state and may change through governed retry/recovery.
"""
if str(task.get("task_id") or "") != FIXED_V2R2C2.task_id:
raise ReleaseLineageError("FIXED_TASK_ID_DRIFT")
capabilities = task.get("required_capabilities")
if (
not isinstance(capabilities, list)
or tuple(capabilities) != FIXED_V2R2C2.required_capabilities
):
raise ReleaseLineageError("FIXED_TASK_CAPABILITY_DRIFT")
constraints = task.get("constraints")
if not isinstance(constraints, Mapping):
raise ReleaseLineageError("FIXED_TASK_CONSTRAINTS_MISSING")
release = constraints.get("release_candidate")
if not isinstance(release, Mapping):
raise ReleaseLineageError("FIXED_RELEASE_BINDING_MISSING")
actual = {
"candidate_id": str(release.get("candidate_id") or ""),
"source_commit": str(release.get("source_commit") or ""),
"version": str(release.get("version") or ""),
}
expected = {
"candidate_id": FIXED_V2R2C2.candidate_id,
"source_commit": FIXED_V2R2C2.source_commit,
"version": FIXED_V2R2C2.version,
}
if actual != expected:
raise ReleaseLineageError(
"FIXED_RELEASE_LINEAGE_DRIFT:"
+ repr({"expected": expected, "actual": actual})
)
return {
"schema": "aios.release-lineage.validation.v1",
"status": "VERIFIED",
"lineage": FIXED_V2R2C2.as_dict(),
"task_status": task.get("status"),
"attempt": task.get("attempt"),
"max_attempts": task.get("max_attempts"),
"revision": task.get("revision"),
}
@@ -0,0 +1,714 @@
#!/usr/bin/env python3
"""Persistent release-preparation Online Agent worker.
The worker is intentionally narrower than the generic engineering worker:
it can only consume tasks whose required capability set is exactly
{"release:candidate"} and whose payload asks for immutable release-candidate
creation. It never receives Owner signing, deployment, tag, baseline or Git
write credentials.
"""
from __future__ import annotations
from dataclasses import dataclass
from datetime import datetime, timezone
import hashlib
import json
import os
from pathlib import Path
import re
import socket
import threading
import time
import urllib.error
import urllib.request
from typing import Any, Mapping
from aios_autonomous_control_plane import (
classify_failure,
repair_for_failure,
runtime_admission,
)
AGENT_ID = os.environ.get(
"AIOS_RELEASE_WORKER_AGENT_ID",
"online-release-worker-cn-network-mkt-20260819",
).strip()
AGENT_VERSION = os.environ.get(
"AIOS_RELEASE_WORKER_AGENT_VERSION",
"2026-08-19.1-release-candidate",
).strip()
TASK_API = os.environ.get(
"AIOS_AGENT_TASK_API_URL",
"http://127.0.0.1:18940",
).rstrip("/")
RELEASE_API = os.environ.get("AIOS_RELEASE_API_URL", "").rstrip("/")
POLL_SECONDS = int(os.environ.get("AIOS_RELEASE_WORKER_POLL_SECONDS", "3"))
LEASE_SECONDS = int(os.environ.get("AIOS_RELEASE_WORKER_LEASE_SECONDS", "60"))
HEARTBEAT_SECONDS = int(
os.environ.get("AIOS_RELEASE_WORKER_HEARTBEAT_SECONDS", "15")
)
STATE_ROOT = Path(
os.environ.get(
"AIOS_RELEASE_WORKER_STATE_ROOT",
"/var/lib/aios-online-agent-release",
)
)
EXPECTED_CAPABILITIES = frozenset({"release:candidate"})
LIVENESS_PATH = Path(
os.environ.get(
"AIOS_WORKER_LIVENESS_PATH",
str(STATE_ROOT / "worker-liveness.json"),
)
)
CANDIDATE_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{7,255}$")
OID_RE = re.compile(r"^[0-9a-f]{40}$")
VERSION_RE = re.compile(r"^V[0-9]+R[0-9]+C[0-9]+$")
TASK_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{2,179}$")
SECRET_MARKERS = (
"authorization",
"token",
"secret",
"credential",
"password",
"private_key",
"signature",
)
class ReleaseWorkerError(RuntimeError):
def __init__(self, code: str, *, phase: str) -> None:
super().__init__(code)
self.code = code
self.phase = phase
def now() -> str:
return datetime.now(timezone.utc).isoformat()
def canonical(value: object) -> bytes:
return json.dumps(
value,
ensure_ascii=False,
sort_keys=True,
separators=(",", ":"),
).encode("utf-8")
def sha256(value: bytes) -> str:
return hashlib.sha256(value).hexdigest()
def sanitized(value: Any) -> Any:
if isinstance(value, Mapping):
result: dict[str, Any] = {}
for key, inner in value.items():
lowered = str(key).lower()
if any(marker in lowered for marker in SECRET_MARKERS):
result[str(key)] = "[REDACTED]"
else:
result[str(key)] = sanitized(inner)
return result
if isinstance(value, list):
return [sanitized(item) for item in value]
return value
def audit(event: str, **fields: Any) -> None:
STATE_ROOT.mkdir(parents=True, exist_ok=True, mode=0o750)
path = STATE_ROOT / "events.jsonl"
record = {
"schema": "aios.release-worker.event.v1",
"event": event,
"created_at": now(),
"agent_id": AGENT_ID,
**sanitized(fields),
}
with path.open("a", encoding="utf-8") as handle:
handle.write(json.dumps(record, sort_keys=True) + "\n")
os.chmod(path, 0o640)
def sd_notify(message: str) -> None:
target = os.environ.get("NOTIFY_SOCKET", "")
if not target:
return
if target.startswith("@"):
address = "\0" + target[1:]
else:
address = target
sock = socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM)
try:
sock.connect(address)
sock.sendall(message.encode("utf-8"))
except OSError:
# Liveness JSON remains the diagnostic source; systemd will enforce
# WatchdogSec if notify transport is persistently broken.
pass
finally:
sock.close()
def write_liveness(
state: str,
*,
task_id: str | None = None,
stage: str | None = None,
error: str | None = None,
visible_count: int | None = None,
) -> None:
STATE_ROOT.mkdir(parents=True, exist_ok=True, mode=0o750)
value = {
"schema": "aios.online-agent.worker-liveness.v1",
"agent_id": AGENT_ID,
"agent_type": "release-preparation-worker",
"agent_version": AGENT_VERSION,
"pid": os.getpid(),
"state": state,
"last_seen_epoch": time.time(),
"last_seen_at": now(),
"poll_seconds": POLL_SECONDS,
"heartbeat_seconds": HEARTBEAT_SECONDS,
"api_base": TASK_API,
"capabilities": sorted(EXPECTED_CAPABILITIES),
"task_id": task_id,
"stage": stage,
"error": error,
"visible_count": visible_count,
}
LIVENESS_PATH.parent.mkdir(parents=True, exist_ok=True, mode=0o750)
temporary = LIVENESS_PATH.with_name(
"." + LIVENESS_PATH.name + f".{os.getpid()}.tmp"
)
temporary.write_text(
json.dumps(value, ensure_ascii=False, sort_keys=True) + "\n",
encoding="utf-8",
)
os.chmod(temporary, 0o640)
os.replace(temporary, LIVENESS_PATH)
def load_credential(name: str) -> str:
directory = Path(os.environ.get("CREDENTIALS_DIRECTORY", ""))
path = directory / name
if path.is_symlink() or not path.is_file():
raise ReleaseWorkerError(
"CREDENTIAL_UNAVAILABLE:" + name,
phase="startup",
)
value = path.read_text(encoding="utf-8").strip()
if len(value) < 24 or any(ch.isspace() for ch in value):
raise ReleaseWorkerError(
"CREDENTIAL_INVALID:" + name,
phase="startup",
)
return value
def http_json(
base: str,
method: str,
route: str,
token: str,
body: Mapping[str, Any] | None = None,
*,
timeout: int = 30,
) -> tuple[int, dict[str, Any]]:
data = None if body is None else canonical(body)
headers = {"Accept": "application/json"}
if token:
headers["Authorization"] = "Bearer " + token
if data is not None:
headers["Content-Type"] = "application/json"
request = urllib.request.Request(
base + route,
method=method,
data=data,
headers=headers,
)
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
raw, status = response.read(), response.status
except urllib.error.HTTPError as exc:
raw, status = exc.read(), exc.code
except (urllib.error.URLError, TimeoutError) as exc:
raise ReleaseWorkerError(
"HTTP_TRANSPORT_UNAVAILABLE",
phase="network",
) from exc
try:
value = json.loads(raw.decode())
except Exception as exc:
raise ReleaseWorkerError(
"HTTP_RESPONSE_INVALID",
phase="network",
) from exc
return status, value if isinstance(value, dict) else {}
@dataclass(frozen=True)
class ReleaseRequest:
candidate_id: str
source_commit: str
version: str
projection_id: str | None
@classmethod
def from_task(cls, task: Mapping[str, Any]) -> "ReleaseRequest":
if not TASK_RE.fullmatch(str(task.get("task_id") or "")):
raise ReleaseWorkerError("TASK_ID_INVALID", phase="validation")
caps = task.get("required_capabilities")
if (
not isinstance(caps, list)
or frozenset(caps) != EXPECTED_CAPABILITIES
or len(caps) != 1
):
raise ReleaseWorkerError(
"TASK_CAPABILITY_SCOPE_INVALID",
phase="validation",
)
metadata = task.get("metadata")
if not isinstance(metadata, Mapping):
metadata = {}
constraints = task.get("constraints")
if not isinstance(constraints, Mapping):
constraints = {}
release = constraints.get("release_candidate")
if not isinstance(release, Mapping):
release = metadata.get("release_candidate")
if not isinstance(release, Mapping):
release = task.get("release_candidate")
if not isinstance(release, Mapping):
raise ReleaseWorkerError(
"RELEASE_CANDIDATE_PAYLOAD_MISSING",
phase="validation",
)
allowed = {
"candidate_id",
"source_commit",
"version",
"projection_id",
}
if set(release) - allowed:
raise ReleaseWorkerError(
"RELEASE_CANDIDATE_PAYLOAD_FIELDS_INVALID",
phase="validation",
)
candidate_id = str(release.get("candidate_id") or "")
source_commit = str(release.get("source_commit") or "")
version = str(release.get("version") or "")
projection_id = release.get("projection_id")
if CANDIDATE_RE.fullmatch(candidate_id) is None:
raise ReleaseWorkerError(
"RELEASE_CANDIDATE_ID_INVALID",
phase="validation",
)
if OID_RE.fullmatch(source_commit) is None:
raise ReleaseWorkerError(
"RELEASE_SOURCE_COMMIT_INVALID",
phase="validation",
)
if VERSION_RE.fullmatch(version) is None:
raise ReleaseWorkerError(
"RELEASE_VERSION_INVALID",
phase="validation",
)
if projection_id is not None:
projection_id = str(projection_id)
if not re.fullmatch(r"projection-[0-9a-f]{32}", projection_id):
raise ReleaseWorkerError(
"RELEASE_PROJECTION_ID_INVALID",
phase="validation",
)
return cls(
candidate_id=candidate_id,
source_commit=source_commit,
version=version,
projection_id=projection_id,
)
def body(self) -> dict[str, str]:
return {
"candidate_id": self.candidate_id,
"source_commit": self.source_commit,
"version": self.version,
}
class Heartbeat:
def __init__(
self,
agent_token: str,
task_id: str,
lease_token: str,
) -> None:
self.agent_token = agent_token
self.task_id = task_id
self.lease_token = lease_token
self.stage = "claimed"
self.failure: str | None = None
self.stop = threading.Event()
self.thread = threading.Thread(
target=self._run,
name="release-worker-heartbeat",
daemon=True,
)
def first(self) -> None:
status, value = http_json(
TASK_API,
"POST",
f"/v1/tasks/{self.task_id}/heartbeat",
self.agent_token,
{
"lease_token": self.lease_token,
"lease_seconds": LEASE_SECONDS,
"progress": {
"stage": self.stage,
"worker_version": AGENT_VERSION,
},
},
)
if status != 200:
raise ReleaseWorkerError(
"TASK_HEARTBEAT_FAILED",
phase="heartbeat",
)
audit(
"TASK_HEARTBEAT",
task_id=self.task_id,
stage=self.stage,
api_status=status,
)
write_liveness(
"BUSY",
task_id=self.task_id,
stage=self.stage,
)
sd_notify("WATCHDOG=1")
def _run(self) -> None:
while not self.stop.wait(HEARTBEAT_SECONDS):
try:
status, value = http_json(
TASK_API,
"POST",
f"/v1/tasks/{self.task_id}/heartbeat",
self.agent_token,
{
"lease_token": self.lease_token,
"lease_seconds": LEASE_SECONDS,
"progress": {
"stage": self.stage,
"worker_version": AGENT_VERSION,
},
},
)
if status != 200:
self.failure = str(value.get("error") or status)
return
audit(
"TASK_HEARTBEAT",
task_id=self.task_id,
stage=self.stage,
api_status=status,
)
write_liveness(
"BUSY",
task_id=self.task_id,
stage=self.stage,
)
sd_notify("WATCHDOG=1")
except Exception as exc:
self.failure = type(exc).__name__
return
def ensure(self) -> None:
if self.failure:
raise ReleaseWorkerError(
"TASK_LEASE_LOST",
phase="heartbeat",
)
def close(self) -> None:
self.stop.set()
if self.thread.ident is not None:
self.thread.join(timeout=2)
def release_admission(
task: Mapping[str, Any],
*,
release_token: str,
):
required = task.get("required_capabilities")
if not isinstance(required, list):
required = []
def release_api_health() -> bool:
status, value = http_json(
RELEASE_API,
"GET",
"/health",
"",
timeout=10,
)
return status == 200 and str(value.get("status") or "").lower() in {"ok", "healthy"}
def authenticated_probe() -> bool:
status, _ = http_json(
RELEASE_API,
"POST",
"/createReleaseCandidate",
release_token,
{},
timeout=10,
)
# Authentication + credential projection succeeded; the empty release
# request is intentionally rejected by request validation.
return status == 400
probes = {
"credential_projection": authenticated_probe,
"trusted_builder": release_api_health,
"authenticated_probe": authenticated_probe,
"release_api_health": release_api_health,
"worker_liveness": lambda: True,
}
return runtime_admission(
worker_capabilities=EXPECTED_CAPABILITIES,
required_capabilities=required,
requested_operations=("prepare_release",),
probes=probes,
)
def result_payload(
task_id: str,
release: ReleaseRequest,
builder: Mapping[str, Any],
) -> dict[str, Any]:
builder_candidate = builder.get("candidate_id")
if builder_candidate not in {None, release.candidate_id}:
raise ReleaseWorkerError(
"BUILDER_CANDIDATE_BINDING_INVALID",
phase="release",
)
immutable = {
"candidate_id": release.candidate_id,
"source_commit": release.source_commit,
"version": release.version,
}
return {
"status": "SUCCEEDED",
"classification": "RELEASE_CANDIDATE_CREATED",
"execution_mode": "release_candidate",
"adapter": "release_candidate",
"task_id": task_id,
"agent_id": AGENT_ID,
"release_candidate": immutable,
"release_candidate_request_sha256": sha256(canonical(immutable)),
"builder_response": sanitized(dict(builder)),
"projection_id": release.projection_id,
"writer_lease": "NONE",
"changed_files": [],
"direct_main_write": False,
"production_write": False,
"tag_write": False,
"baseline_write": False,
"runtime_switch": False,
"owner_approval_submitted": False,
"credential_material_exposed": False,
}
def execute_task(
task: Mapping[str, Any],
*,
agent_token: str,
release_token: str,
) -> bool:
task_id = str(task.get("task_id") or "")
release = ReleaseRequest.from_task(task)
admission = release_admission(
task,
release_token=release_token,
)
if not admission.admitted:
audit(
"TASK_ADMISSION_BLOCKED",
task_id=task_id,
admission=admission.as_dict(),
)
write_liveness(
"BLOCKED",
task_id=task_id,
stage="execution_ready",
error=admission.failure_class.value if admission.failure_class else "ADMISSION_FAILED",
)
return False
audit(
"TASK_ADMISSION_PASSED",
task_id=task_id,
admission=admission.as_dict(),
)
status, claimed = http_json(
TASK_API,
"POST",
f"/v1/tasks/{task_id}/claim",
agent_token,
{"lease_seconds": LEASE_SECONDS},
)
if status in {409, 423}:
return False
if status != 200:
raise ReleaseWorkerError(
"TASK_CLAIM_FAILED",
phase="claim",
)
lease_token = str(claimed.get("lease_token") or "")
if len(lease_token) < 16:
raise ReleaseWorkerError(
"TASK_LEASE_TOKEN_INVALID",
phase="claim",
)
attempt = int(claimed.get("attempt") or 0)
audit(
"TASK_CLAIMED",
task_id=task_id,
attempt=attempt,
)
heartbeat = Heartbeat(agent_token, task_id, lease_token)
heartbeat.first()
heartbeat.thread.start()
try:
heartbeat.stage = "release_candidate"
heartbeat.ensure()
status, builder = http_json(
RELEASE_API,
"POST",
"/createReleaseCandidate",
release_token,
release.body(),
timeout=120,
)
if status not in {200, 201, 202}:
raise ReleaseWorkerError(
"RELEASE_CANDIDATE_CREATE_FAILED:"
+ str(builder.get("error") or status),
phase="release",
)
heartbeat.ensure()
result = result_payload(task_id, release, builder)
idempotency_key = (
f"{task_id}-attempt-{attempt}-release-result-v1"
)
status, submitted = http_json(
TASK_API,
"POST",
f"/v1/tasks/{task_id}/result",
agent_token,
{
"lease_token": lease_token,
"idempotency_key": idempotency_key,
"result": result,
},
)
if status not in {200, 201, 202}:
raise ReleaseWorkerError(
"TASK_RESULT_SUBMIT_FAILED",
phase="result",
)
audit(
"RESULT_SUBMITTED",
task_id=task_id,
attempt=attempt,
release_candidate_id=release.candidate_id,
)
return True
finally:
heartbeat.close()
def main() -> int:
if not RELEASE_API:
raise SystemExit("FATAL:AIOS_RELEASE_API_URL_REQUIRED")
agent_token = load_credential("agent_token")
release_token = load_credential("release_candidate_token")
STATE_ROOT.mkdir(parents=True, exist_ok=True, mode=0o750)
audit(
"WORKER_STARTED",
capabilities=sorted(EXPECTED_CAPABILITIES),
release_api=RELEASE_API,
)
write_liveness("STARTING", stage="startup")
sd_notify("READY=1\nSTATUS=release worker ready")
while True:
try:
status, response = http_json(
TASK_API,
"POST",
"/v1/tasks/poll",
agent_token,
{"limit": 100},
)
if status != 200:
raise ReleaseWorkerError(
"TASK_POLL_FAILED",
phase="poll",
)
tasks = response.get("tasks")
if not isinstance(tasks, list):
tasks = []
write_liveness(
"IDLE",
stage="poll",
visible_count=len(tasks),
)
sd_notify("WATCHDOG=1\nSTATUS=release worker poll ok")
handled = False
for raw in tasks:
if not isinstance(raw, Mapping):
continue
caps = raw.get("required_capabilities")
if (
not isinstance(caps, list)
or frozenset(caps) != EXPECTED_CAPABILITIES
):
continue
try:
handled = execute_task(
raw,
agent_token=agent_token,
release_token=release_token,
) or handled
except ReleaseWorkerError as exc:
failure = classify_failure(exc.code, phase=exc.phase)
repair = repair_for_failure(failure)
audit(
"TASK_EXECUTION_FAILED",
task_id=raw.get("task_id"),
phase=exc.phase,
error=exc.code,
failure_class=failure.value,
repair=repair.as_dict(),
)
time.sleep(0 if handled else POLL_SECONDS)
except KeyboardInterrupt:
return 0
except Exception as exc:
audit(
"WORKER_LOOP_ERROR",
error=type(exc).__name__,
)
write_liveness(
"DEGRADED",
stage="loop",
error=type(exc).__name__,
)
time.sleep(POLL_SECONDS)
if __name__ == "__main__":
raise SystemExit(main())
@@ -0,0 +1,462 @@
"""Self-healing availability guard for persistent Online Agent workers.
The guard never claims Tasks and never writes Task/Job/Git databases. It only:
- ensures required worker/executor systemd units are enabled + active;
- checks worker-owned liveness JSON written by the real poll/heartbeat loop;
- restarts an active worker only when its liveness is stale;
- honors an explicit maintenance inhibit file;
- writes a bounded status JSON under /run for observability.
Idle liveness is the successful poll loop itself. Busy liveness is refreshed by the
Task lease-heartbeat thread, so long-running Tasks are not mistaken for hung workers.
"""
from __future__ import annotations
from dataclasses import asdict, dataclass
import argparse
import json
import os
from pathlib import Path
import shlex
import subprocess
import tempfile
import time
from typing import Any, Callable, Mapping, Sequence
SCHEMA = "aios.worker-availability.status.v1"
DEFAULT_WORKER_UNITS = (
"aios-online-agent-worker-20260815.service",
"aios-hermes-online-agent-worker.service",
"aios-openclaw-online-agent-worker.service",
)
DEFAULT_DEPENDENCY_UNITS = (
"aios-external-agent-executor.service",
)
DEFAULT_RUN_ROOT = Path("/run/aios-worker-availability")
DEFAULT_STATUS = DEFAULT_RUN_ROOT / "status.json"
DEFAULT_RECOVERY = DEFAULT_RUN_ROOT / "recovery.json"
DEFAULT_INHIBIT = DEFAULT_RUN_ROOT / "maintenance.lock"
DEFAULT_MAX_STALE_SECONDS = 90
DEFAULT_RESTART_COOLDOWN_SECONDS = 60
DEFAULT_STARTUP_WAIT_SECONDS = 10
ACCEPTED_ENABLED_STATES = frozenset({"enabled", "enabled-runtime", "static", "indirect", "generated"})
class AvailabilityGuardError(RuntimeError):
pass
@dataclass(frozen=True)
class UnitSnapshot:
unit: str
role: str
load_state: str
active_state: str
unit_file_state: str
enabled_ok: bool
liveness_path: str | None = None
liveness_state: str | None = None
last_seen_epoch: float | None = None
liveness_age_seconds: float | None = None
fresh: bool | None = None
action: str = "NONE"
detail: str | None = None
class Systemd:
def __init__(
self,
runner: Callable[..., subprocess.CompletedProcess[str]] | None = None,
) -> None:
self.runner = runner or subprocess.run
def _run(self, args: Sequence[str], *, check: bool = False) -> subprocess.CompletedProcess[str]:
result = self.runner(
["/usr/bin/systemctl", *args],
text=True,
capture_output=True,
check=False,
timeout=20,
)
if check and result.returncode != 0:
raise AvailabilityGuardError(
"SYSTEMD_COMMAND_FAILED:"
+ ":".join(args)
+ ":"
+ str(result.returncode)
)
return result
def property(self, unit: str, name: str) -> str:
result = self._run(["show", unit, f"--property={name}", "--value"])
if result.returncode != 0:
return ""
return result.stdout.strip()
def environment(self, unit: str) -> dict[str, str]:
text = self.property(unit, "Environment")
result: dict[str, str] = {}
for item in shlex.split(text):
if "=" not in item:
continue
key, value = item.split("=", 1)
result[key] = value
return result
def enable(self, unit: str) -> None:
self._run(["enable", unit], check=True)
def start(self, unit: str) -> None:
self._run(["start", unit], check=True)
def restart(self, unit: str) -> None:
self._run(["restart", unit], check=True)
def _atomic_json(path: Path, value: Mapping[str, Any]) -> None:
path.parent.mkdir(parents=True, exist_ok=True, mode=0o755)
if path.parent.is_symlink() or path.is_symlink():
raise AvailabilityGuardError("STATUS_PATH_SYMLINK_REJECTED")
fd, temporary = tempfile.mkstemp(prefix=f".{path.name}.", dir=path.parent)
try:
os.fchmod(fd, 0o644)
with os.fdopen(fd, "w", encoding="utf-8") as handle:
json.dump(value, handle, ensure_ascii=False, sort_keys=True, indent=2)
handle.write("\n")
handle.flush()
os.fsync(handle.fileno())
os.replace(temporary, path)
os.chmod(path, 0o644)
finally:
if os.path.exists(temporary):
os.unlink(temporary)
def _load_json(path: Path) -> dict[str, Any] | None:
if path.is_symlink() or not path.is_file():
return None
try:
value = json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError):
return None
return dict(value) if isinstance(value, Mapping) else None
def _state_root(environment: Mapping[str, str], unit: str) -> Path:
explicit = environment.get("AIOS_WORKER_STATE_ROOT")
if explicit:
return Path(explicit)
fallbacks = {
"aios-online-agent-worker-20260815.service": Path(
"/var/lib/aios-online-agent-worker-20260815"
),
"aios-hermes-online-agent-worker.service": Path(
"/var/lib/aios-online-agent-hermes"
),
"aios-openclaw-online-agent-worker.service": Path(
"/var/lib/aios-online-agent-openclaw"
),
}
if unit not in fallbacks:
raise AvailabilityGuardError("WORKER_STATE_ROOT_UNAVAILABLE:" + unit)
return fallbacks[unit]
def _liveness_path(environment: Mapping[str, str], unit: str) -> Path:
explicit = environment.get("AIOS_WORKER_LIVENESS_PATH")
if explicit:
return Path(explicit)
return _state_root(environment, unit) / "worker-liveness.json"
class AvailabilityGuard:
def __init__(
self,
*,
systemd: Systemd | None = None,
worker_units: Sequence[str] = DEFAULT_WORKER_UNITS,
dependency_units: Sequence[str] = DEFAULT_DEPENDENCY_UNITS,
status_path: Path = DEFAULT_STATUS,
recovery_path: Path = DEFAULT_RECOVERY,
inhibit_path: Path = DEFAULT_INHIBIT,
max_stale_seconds: int = DEFAULT_MAX_STALE_SECONDS,
restart_cooldown_seconds: int = DEFAULT_RESTART_COOLDOWN_SECONDS,
startup_wait_seconds: int = DEFAULT_STARTUP_WAIT_SECONDS,
now_fn: Callable[[], float] = time.time,
sleep_fn: Callable[[float], None] = time.sleep,
) -> None:
self.systemd = systemd or Systemd()
self.worker_units = tuple(worker_units)
self.dependency_units = tuple(dependency_units)
self.status_path = Path(status_path)
self.recovery_path = Path(recovery_path)
self.inhibit_path = Path(inhibit_path)
self.max_stale_seconds = int(max_stale_seconds)
self.restart_cooldown_seconds = int(restart_cooldown_seconds)
self.startup_wait_seconds = int(startup_wait_seconds)
self.now_fn = now_fn
self.sleep_fn = sleep_fn
if self.max_stale_seconds < 15:
raise AvailabilityGuardError("MAX_STALE_SECONDS_TOO_SMALL")
if self.restart_cooldown_seconds < 15:
raise AvailabilityGuardError("RESTART_COOLDOWN_TOO_SMALL")
if self.startup_wait_seconds < 0 or self.startup_wait_seconds > 60:
raise AvailabilityGuardError("STARTUP_WAIT_INVALID")
def _recovery_state(self) -> dict[str, float]:
value = _load_json(self.recovery_path) or {}
raw = value.get("last_restart_epoch")
if not isinstance(raw, Mapping):
return {}
result: dict[str, float] = {}
for unit, epoch in raw.items():
try:
result[str(unit)] = float(epoch)
except (TypeError, ValueError):
continue
return result
def _save_recovery_state(self, state: Mapping[str, float]) -> None:
_atomic_json(
self.recovery_path,
{
"schema": "aios.worker-availability.recovery.v1",
"last_restart_epoch": dict(state),
},
)
def _basic_snapshot(self, unit: str, role: str) -> tuple[str, str, str, bool]:
load = self.systemd.property(unit, "LoadState")
active = self.systemd.property(unit, "ActiveState")
enabled = self.systemd.property(unit, "UnitFileState")
return load, active, enabled, enabled in ACCEPTED_ENABLED_STATES
def _ensure_enabled_active(self, unit: str, role: str) -> tuple[str, bool]:
load, active, enabled, enabled_ok = self._basic_snapshot(unit, role)
if load != "loaded":
raise AvailabilityGuardError("REQUIRED_UNIT_NOT_LOADED:" + unit)
action = "NONE"
if not enabled_ok:
self.systemd.enable(unit)
action = "ENABLE"
if active != "active":
self.systemd.start(unit)
action = "START" if action == "NONE" else "ENABLE_START"
return action, active == "active"
def _wait_for_liveness(self, path: Path) -> dict[str, Any] | None:
deadline = self.now_fn() + self.startup_wait_seconds
while True:
value = _load_json(path)
if value is not None:
try:
epoch = float(value.get("last_seen_epoch"))
except (TypeError, ValueError):
epoch = 0.0
if epoch > 0 and self.now_fn() - epoch <= self.max_stale_seconds:
return value
if self.now_fn() >= deadline:
return value
self.sleep_fn(0.5)
def run(self, *, heal: bool) -> dict[str, Any]:
now = self.now_fn()
if self.inhibit_path.exists():
result = {
"schema": SCHEMA,
"status": "MAINTENANCE_INHIBITED",
"generated_at_epoch": now,
"maintenance_inhibit": str(self.inhibit_path),
"workers": [],
"dependencies": [],
"all_fresh": False,
"actions": [],
}
_atomic_json(self.status_path, result)
return result
actions: list[dict[str, str]] = []
dependencies: list[UnitSnapshot] = []
for unit in self.dependency_units:
load, active, enabled, enabled_ok = self._basic_snapshot(unit, "dependency")
action = "NONE"
if heal:
if load != "loaded":
raise AvailabilityGuardError("REQUIRED_UNIT_NOT_LOADED:" + unit)
if not enabled_ok:
self.systemd.enable(unit)
action = "ENABLE"
if active != "active":
self.systemd.start(unit)
action = "START" if action == "NONE" else "ENABLE_START"
if action != "NONE":
actions.append({"unit": unit, "action": action})
load, active, enabled, enabled_ok = self._basic_snapshot(unit, "dependency")
dependencies.append(
UnitSnapshot(
unit=unit,
role="dependency",
load_state=load,
active_state=active,
unit_file_state=enabled,
enabled_ok=enabled_ok,
fresh=(active == "active"),
action=action,
)
)
recovery = self._recovery_state()
workers: list[UnitSnapshot] = []
for unit in self.worker_units:
load, active, enabled, enabled_ok = self._basic_snapshot(unit, "worker")
environment = self.systemd.environment(unit) if load == "loaded" else {}
path = _liveness_path(environment, unit) if load == "loaded" else None
action = "NONE"
if heal:
if load != "loaded":
raise AvailabilityGuardError("REQUIRED_UNIT_NOT_LOADED:" + unit)
if not enabled_ok:
self.systemd.enable(unit)
action = "ENABLE"
if active != "active":
self.systemd.start(unit)
action = "START" if action == "NONE" else "ENABLE_START"
actions.append({"unit": unit, "action": action})
if path is not None:
self._wait_for_liveness(path)
load, active, enabled, enabled_ok = self._basic_snapshot(unit, "worker")
live = _load_json(path) if path is not None else None
last_seen: float | None = None
state: str | None = None
age: float | None = None
fresh = False
detail: str | None = None
if live is not None:
state = str(live.get("state") or "")
try:
last_seen = float(live.get("last_seen_epoch"))
except (TypeError, ValueError):
last_seen = None
if last_seen is not None:
age = max(0.0, now - last_seen)
fresh = age <= self.max_stale_seconds
if live.get("agent_id") in (None, ""):
fresh = False
detail = "LIVENESS_AGENT_ID_MISSING"
else:
detail = "LIVENESS_MISSING"
if heal and active == "active" and not fresh:
previous = float(recovery.get(unit, 0.0))
if now - previous >= self.restart_cooldown_seconds:
self.systemd.restart(unit)
recovery[unit] = now
action = "RESTART_STALE"
actions.append({"unit": unit, "action": action})
if path is not None:
live = self._wait_for_liveness(path)
load, active, enabled, enabled_ok = self._basic_snapshot(unit, "worker")
if live is not None:
state = str(live.get("state") or "")
try:
last_seen = float(live.get("last_seen_epoch"))
except (TypeError, ValueError):
last_seen = None
if last_seen is not None:
age = max(0.0, self.now_fn() - last_seen)
fresh = age <= self.max_stale_seconds
detail = None if fresh else "LIVENESS_STALE_AFTER_RESTART"
else:
detail = "RESTART_COOLDOWN_ACTIVE"
workers.append(
UnitSnapshot(
unit=unit,
role="worker",
load_state=load,
active_state=active,
unit_file_state=enabled,
enabled_ok=enabled_ok,
liveness_path=str(path) if path is not None else None,
liveness_state=state,
last_seen_epoch=last_seen,
liveness_age_seconds=age,
fresh=bool(active == "active" and enabled_ok and fresh),
action=action,
detail=detail,
)
)
if heal:
self._save_recovery_state(recovery)
all_fresh = (
all(item.active_state == "active" and item.enabled_ok and item.fresh for item in workers)
and all(item.active_state == "active" for item in dependencies)
)
result = {
"schema": SCHEMA,
"status": "HEALTHY" if all_fresh else "DEGRADED",
"generated_at_epoch": self.now_fn(),
"max_stale_seconds": self.max_stale_seconds,
"maintenance_inhibit": str(self.inhibit_path),
"workers": [asdict(item) for item in workers],
"dependencies": [asdict(item) for item in dependencies],
"all_fresh": all_fresh,
"actions": actions,
}
_atomic_json(self.status_path, result)
return result
def main() -> int:
parser = argparse.ArgumentParser()
mode = parser.add_mutually_exclusive_group(required=True)
mode.add_argument("--heal", action="store_true")
mode.add_argument("--check", action="store_true")
parser.add_argument("--status-file", default=str(DEFAULT_STATUS))
parser.add_argument("--recovery-file", default=str(DEFAULT_RECOVERY))
parser.add_argument("--inhibit-file", default=str(DEFAULT_INHIBIT))
parser.add_argument(
"--max-stale-seconds",
type=int,
default=int(
os.environ.get(
"AIOS_WORKER_AVAILABILITY_MAX_STALE_SECONDS",
str(DEFAULT_MAX_STALE_SECONDS),
)
),
)
parser.add_argument(
"--restart-cooldown-seconds",
type=int,
default=int(
os.environ.get(
"AIOS_WORKER_AVAILABILITY_RESTART_COOLDOWN_SECONDS",
str(DEFAULT_RESTART_COOLDOWN_SECONDS),
)
),
)
args = parser.parse_args()
guard = AvailabilityGuard(
status_path=Path(args.status_file),
recovery_path=Path(args.recovery_file),
inhibit_path=Path(args.inhibit_file),
max_stale_seconds=args.max_stale_seconds,
restart_cooldown_seconds=args.restart_cooldown_seconds,
)
try:
result = guard.run(heal=args.heal)
except AvailabilityGuardError as exc:
print(json.dumps({"status": "FAILED", "error": str(exc)}, sort_keys=True))
return 20
print(json.dumps(result, ensure_ascii=False, sort_keys=True))
if args.check and not result.get("all_fresh"):
return 10
return 0
if __name__ == "__main__":
raise SystemExit(main())
@@ -964,6 +964,7 @@ def validate_writer_lease_request_v2(raw: Mapping[str, Any], probe: Any) -> dict
required = {
"schema",
"task_id",
"job_id",
"repository",
"branch_mode",
"base_oid",
@@ -976,6 +977,7 @@ def validate_writer_lease_request_v2(raw: Mapping[str, Any], probe: Any) -> dict
raise ProvisioningError("LEASE_REQUEST_V2_FIELDS_REJECTED", 409)
request = WriterLeaseRequestV2(
task_id=str(raw["task_id"]),
job_id=str(raw["job_id"]),
repository=str(raw["repository"]),
branch_mode=str(raw["branch_mode"]),
base_oid=str(raw["base_oid"]),
@@ -988,6 +990,7 @@ def validate_writer_lease_request_v2(raw: Mapping[str, Any], probe: Any) -> dict
return {
"schema": WriterLeaseRequestV2.schema,
"task_id": request.task_id,
"job_id": request.job_id,
"repository": request.repository,
"branch_mode": request.branch_mode,
"base_oid": request.base_oid,
+456
View File
@@ -0,0 +1,456 @@
from __future__ import annotations
import dataclasses
import datetime as dt
import hashlib
import json
import sqlite3
import re
from pathlib import Path
from typing import Any, Iterable, Mapping
UTC = dt.timezone.utc
_ALLOWED_CONFIDENTIALITY = frozenset({"public", "tenant", "project", "private", "run"})
_PRIVILEGED = frozenset({"*", "admin", "system.admin", "message.admin", "task.admin"})
class HandoffError(RuntimeError):
code = "HANDOFF_ERROR"
class MessageScopeMismatch(HandoffError):
code = "MESSAGE_SCOPE_MISMATCH_DENY"
class ConfidentialityScopeDenied(HandoffError):
code = "MESSAGE_CONFIDENTIALITY_SCOPE_ENFORCE"
class MessageExpired(HandoffError):
code = "MESSAGE_EXPIRED"
class SequenceGap(HandoffError):
code = "MESSAGE_SEQUENCE_GAP_DETECT"
class SequenceConflict(HandoffError):
code = "MESSAGE_SEQUENCE_CONFLICT"
class IdempotencyConflict(HandoffError):
code = "MESSAGE_IDEMPOTENCY_CONFLICT"
def _utcnow() -> dt.datetime:
return dt.datetime.now(UTC)
def _iso(value: dt.datetime | str | None) -> str | None:
if value is None:
return None
parsed = dt.datetime.fromisoformat(value.replace("Z", "+00:00")) if isinstance(value, str) else value
if parsed.tzinfo is None:
parsed = parsed.replace(tzinfo=UTC)
return parsed.astimezone(UTC).isoformat()
def _canonical(value: Any) -> str:
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False, default=str)
def _sha(value: Any) -> str:
return hashlib.sha256(_canonical(value).encode("utf-8")).hexdigest()
def _clean(value: Any) -> str | None:
if value is None:
return None
text = str(value).strip()
return text or None
def _tuple_scope(value: Iterable[str] | str | None) -> tuple[str, ...]:
if value is None:
return ()
if isinstance(value, str):
value = value.split(",")
return tuple(sorted({str(item).strip() for item in value if str(item).strip()}))
_SECRET_KEY = re.compile(r"(?i)(?:token|secret|password|private[_-]?key|api[_-]?key)")
_SECRET_VALUE = re.compile(r"(?i)bearer\s+[A-Za-z0-9._~+/-]{8,}|(?:token|secret|password|api[_-]?key)\s*[:=]\s*\S+")
_SAFE_REFERENCE = re.compile(r"^(?:systemd-credential|vault|secret-manager):[A-Za-z0-9][A-Za-z0-9._:/#-]{0,255}$")
def assert_payload_secret_safe(value: Any, *, key: str = "") -> None:
if isinstance(value, Mapping):
for child_key, child in value.items():
assert_payload_secret_safe(child, key=str(child_key))
return
if isinstance(value, (list, tuple)):
for child in value:
assert_payload_secret_safe(child, key=key)
return
if isinstance(value, (bytes, bytearray)):
raise ValueError("CREDENTIAL_SECRET_EXPOSURE_DENY")
if isinstance(value, str):
if _SECRET_VALUE.search(value):
raise ValueError("CREDENTIAL_SECRET_EXPOSURE_DENY")
if _SECRET_KEY.search(key) and value and _SAFE_REFERENCE.fullmatch(value) is None:
raise ValueError("CREDENTIAL_SECRET_EXPOSURE_DENY")
@dataclasses.dataclass(frozen=True)
class DeliveryContext:
tenant_id: str | None = None
user_id: str | None = None
project_id: str | None = None
conversation_id: str | None = None
run_id: str | None = None
capabilities: tuple[str, ...] = ()
@classmethod
def from_actor(cls, actor: Any) -> "DeliveryContext":
def get(name: str) -> Any:
if isinstance(actor, Mapping):
return actor.get(name)
return getattr(actor, name, None)
return cls(
tenant_id=_clean(get("tenant_id")),
user_id=_clean(get("user_id")),
project_id=_clean(get("project_id")),
conversation_id=_clean(get("conversation_id")),
run_id=_clean(get("run_id")),
capabilities=_tuple_scope(get("capabilities") or get("capability_scope")),
)
@property
def privileged(self) -> bool:
return bool(set(self.capabilities) & _PRIVILEGED)
@dataclasses.dataclass(frozen=True)
class MessageEnvelope:
source: str
target: str
message_type: str
payload: Mapping[str, Any]
sequence: int
idempotency_key: str
tenant_id: str | None = None
user_id: str | None = None
project_id: str | None = None
conversation_id: str | None = None
task_id: str | None = None
run_id: str | None = None
correlation_id: str | None = None
causation_id: str | None = None
capability_scope: tuple[str, ...] = ()
confidentiality_scope: str = "private"
expires_at: str | None = None
created_at: str = dataclasses.field(default_factory=lambda: _iso(_utcnow()) or "")
envelope_id: str = ""
def __post_init__(self) -> None:
assert_payload_secret_safe(self.payload)
for field in ("source", "target", "message_type", "idempotency_key", "tenant_id", "user_id",
"project_id", "conversation_id", "task_id", "run_id", "correlation_id", "causation_id"):
object.__setattr__(self, field, _clean(getattr(self, field)) if field not in {
"source", "target", "message_type", "idempotency_key"
} else (_clean(getattr(self, field)) or ""))
object.__setattr__(self, "capability_scope", _tuple_scope(self.capability_scope))
object.__setattr__(self, "confidentiality_scope", (_clean(self.confidentiality_scope) or "private").lower())
object.__setattr__(self, "expires_at", _iso(self.expires_at))
object.__setattr__(self, "created_at", _iso(self.created_at) or "")
if not self.source or not self.target or not self.message_type or not self.idempotency_key:
raise ValueError("MESSAGE_ENVELOPE_IDENTITY_REQUIRED")
if not isinstance(self.sequence, int) or self.sequence < 1:
raise ValueError("MESSAGE_SEQUENCE_MUST_BE_POSITIVE")
if self.confidentiality_scope not in _ALLOWED_CONFIDENTIALITY:
raise ValueError("MESSAGE_CONFIDENTIALITY_SCOPE_INVALID")
if self.confidentiality_scope != "public" and self.tenant_id is None:
raise ValueError("MESSAGE_TENANT_REQUIRED")
if self.confidentiality_scope in {"project", "private", "run"} and self.project_id is None:
raise ValueError("MESSAGE_PROJECT_REQUIRED")
if self.confidentiality_scope in {"private", "run"} and self.user_id is None:
raise ValueError("MESSAGE_USER_REQUIRED")
if self.confidentiality_scope == "run" and self.run_id is None:
raise ValueError("MESSAGE_RUN_REQUIRED")
expected = stable_event_id(
source=self.source, target=self.target, message_type=self.message_type, payload=self.payload,
tenant_id=self.tenant_id, user_id=self.user_id, project_id=self.project_id,
conversation_id=self.conversation_id, task_id=self.task_id, run_id=self.run_id,
correlation_id=self.correlation_id, causation_id=self.causation_id,
sequence=self.sequence, idempotency_key=self.idempotency_key,
)
if self.envelope_id and self.envelope_id != expected:
raise ValueError("MESSAGE_ENVELOPE_ID_MISMATCH")
object.__setattr__(self, "envelope_id", expected)
@property
def payload_sha256(self) -> str:
return _sha(self.payload)
@property
def stream_key(self) -> str:
return _sha({
"tenant_id": self.tenant_id, "source": self.source, "target": self.target,
"conversation_id": self.conversation_id, "task_id": self.task_id, "run_id": self.run_id,
})
def is_expired(self, *, now: dt.datetime | None = None) -> bool:
if self.expires_at is None:
return False
return dt.datetime.fromisoformat(self.expires_at) <= (now or _utcnow()).astimezone(UTC)
def as_record(self) -> dict[str, Any]:
return {
"envelope_id": self.envelope_id, "stream_key": self.stream_key, "source": self.source,
"target": self.target, "message_type": self.message_type, "payload_json": _canonical(self.payload),
"payload_sha256": self.payload_sha256, "tenant_id": self.tenant_id, "user_id": self.user_id,
"project_id": self.project_id, "conversation_id": self.conversation_id, "task_id": self.task_id,
"run_id": self.run_id, "correlation_id": self.correlation_id,
"causation_id": self.causation_id, "sequence": self.sequence,
"capability_scope_json": _canonical(self.capability_scope),
"confidentiality_scope": self.confidentiality_scope, "idempotency_key": self.idempotency_key,
"expires_at": self.expires_at, "created_at": self.created_at,
}
def stable_event_id(*, source: Any, target: Any, message_type: Any, payload: Any,
tenant_id: Any = None, user_id: Any = None, project_id: Any = None,
conversation_id: Any = None, task_id: Any = None, run_id: Any = None,
correlation_id: Any = None, causation_id: Any = None,
sequence: Any = None, idempotency_key: Any = None) -> str:
assert_payload_secret_safe(payload)
material = {
"source": _clean(source), "target": _clean(target), "message_type": _clean(message_type),
"payload_sha256": _sha(payload), "tenant_id": _clean(tenant_id), "user_id": _clean(user_id),
"project_id": _clean(project_id), "conversation_id": _clean(conversation_id),
"task_id": _clean(task_id), "run_id": _clean(run_id), "causation_id": _clean(causation_id),
"sequence": sequence, "idempotency_key": _clean(idempotency_key),
}
# Preserve historical envelope IDs for legacy rows that predate correlation_id.
if _clean(correlation_id) is not None:
material["correlation_id"] = _clean(correlation_id)
return _sha(material)
def bind_envelope_scope(envelope: MessageEnvelope, actor: Any) -> MessageEnvelope:
ctx = DeliveryContext.from_actor(actor)
if ctx.privileged:
return envelope
if ctx.tenant_id is None:
if envelope.tenant_id is not None:
raise MessageScopeMismatch(MessageScopeMismatch.code)
return envelope
updates: dict[str, Any] = {}
for field in ("tenant_id", "user_id", "project_id", "conversation_id", "run_id"):
trusted = getattr(ctx, field)
requested = getattr(envelope, field)
if requested is not None and requested != trusted:
raise MessageScopeMismatch(f"{MessageScopeMismatch.code}:{field}")
if requested is None and trusted is not None:
updates[field] = trusted
if not updates:
return envelope
data = dataclasses.asdict(envelope)
data.update(updates)
data["envelope_id"] = ""
return MessageEnvelope(**data)
def authorize_delivery(actor: Any, envelope: MessageEnvelope) -> None:
ctx = DeliveryContext.from_actor(actor)
if ctx.privileged:
return
if not set(envelope.capability_scope).issubset(set(ctx.capabilities)):
raise ConfidentialityScopeDenied(f"{ConfidentialityScopeDenied.code}:capability")
if envelope.confidentiality_scope == "public":
return
if ctx.tenant_id != envelope.tenant_id:
raise MessageScopeMismatch(f"{MessageScopeMismatch.code}:tenant_id")
if envelope.confidentiality_scope in {"project", "private", "run"} and ctx.project_id != envelope.project_id:
raise ConfidentialityScopeDenied(f"{ConfidentialityScopeDenied.code}:project_id")
if envelope.confidentiality_scope in {"private", "run"} and ctx.user_id != envelope.user_id:
raise ConfidentialityScopeDenied(f"{ConfidentialityScopeDenied.code}:user_id")
if envelope.confidentiality_scope == "run" and ctx.run_id != envelope.run_id:
raise ConfidentialityScopeDenied(f"{ConfidentialityScopeDenied.code}:run_id")
_SCHEMA = """
CREATE TABLE IF NOT EXISTS message_handoff (
envelope_id TEXT PRIMARY KEY,
stream_key TEXT NOT NULL,
source TEXT NOT NULL,
target TEXT NOT NULL,
message_type TEXT NOT NULL,
payload_json TEXT NOT NULL,
payload_sha256 TEXT NOT NULL,
tenant_id TEXT,
user_id TEXT,
project_id TEXT,
conversation_id TEXT,
task_id TEXT,
run_id TEXT,
correlation_id TEXT,
causation_id TEXT,
sequence INTEGER NOT NULL CHECK(sequence >= 1),
capability_scope_json TEXT NOT NULL,
confidentiality_scope TEXT NOT NULL,
idempotency_key TEXT NOT NULL,
expires_at TEXT,
created_at TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'PENDING',
delivered_at TEXT,
acked_at TEXT,
UNIQUE(stream_key, idempotency_key),
UNIQUE(stream_key, sequence)
);
CREATE INDEX IF NOT EXISTS idx_message_handoff_target_status
ON message_handoff(target, status, created_at);
CREATE INDEX IF NOT EXISTS idx_message_handoff_scope
ON message_handoff(tenant_id, project_id, user_id, conversation_id, run_id);
"""
class DurableHandoffStore:
def __init__(self, path: str | Path) -> None:
self.path = Path(path)
self.path.parent.mkdir(parents=True, exist_ok=True)
with self._connect() as conn:
conn.executescript(_SCHEMA)
columns = {str(row[1]) for row in conn.execute("PRAGMA table_info(message_handoff)")}
if "correlation_id" not in columns:
conn.execute("ALTER TABLE message_handoff ADD COLUMN correlation_id TEXT")
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_message_handoff_lineage "
"ON message_handoff(correlation_id, causation_id)"
)
def _connect(self) -> sqlite3.Connection:
conn = sqlite3.connect(self.path, timeout=30.0)
conn.row_factory = sqlite3.Row
conn.execute("PRAGMA foreign_keys=ON")
conn.execute("PRAGMA busy_timeout=30000")
return conn
@staticmethod
def _from_row(row: sqlite3.Row) -> MessageEnvelope:
return MessageEnvelope(
source=row["source"], target=row["target"], message_type=row["message_type"],
payload=json.loads(row["payload_json"]), tenant_id=row["tenant_id"], user_id=row["user_id"],
project_id=row["project_id"], conversation_id=row["conversation_id"], task_id=row["task_id"],
run_id=row["run_id"], correlation_id=row["correlation_id"],
causation_id=row["causation_id"], sequence=int(row["sequence"]),
capability_scope=tuple(json.loads(row["capability_scope_json"])),
confidentiality_scope=row["confidentiality_scope"], idempotency_key=row["idempotency_key"],
expires_at=row["expires_at"], created_at=row["created_at"], envelope_id=row["envelope_id"],
)
def put(self, envelope: MessageEnvelope) -> tuple[str, MessageEnvelope]:
if envelope.is_expired():
raise MessageExpired(MessageExpired.code)
record = envelope.as_record()
with self._connect() as conn:
conn.execute("BEGIN IMMEDIATE")
replay = conn.execute(
"SELECT * FROM message_handoff WHERE stream_key=? AND idempotency_key=?",
(envelope.stream_key, envelope.idempotency_key),
).fetchone()
if replay is not None:
existing = self._from_row(replay)
if existing.envelope_id != envelope.envelope_id or existing.payload_sha256 != envelope.payload_sha256:
raise IdempotencyConflict(IdempotencyConflict.code)
conn.commit()
return "REPLAY", existing
seq = conn.execute(
"SELECT * FROM message_handoff WHERE stream_key=? AND sequence=?",
(envelope.stream_key, envelope.sequence),
).fetchone()
if seq is not None:
existing = self._from_row(seq)
if existing.envelope_id == envelope.envelope_id:
conn.commit()
return "REPLAY", existing
raise SequenceConflict(SequenceConflict.code)
maximum = int(conn.execute(
"SELECT COALESCE(MAX(sequence),0) FROM message_handoff WHERE stream_key=?",
(envelope.stream_key,),
).fetchone()[0])
expected = maximum + 1
if envelope.sequence != expected:
raise SequenceGap(f"{SequenceGap.code}:expected={expected}:actual={envelope.sequence}")
columns = tuple(record)
placeholders = ",".join("?" for _ in columns)
conn.execute(
f"INSERT INTO message_handoff ({','.join(columns)}) VALUES ({placeholders})",
tuple(record[column] for column in columns),
)
conn.commit()
return "CREATED", envelope
def get(self, envelope_id: str, *, actor: Any) -> MessageEnvelope:
with self._connect() as conn:
row = conn.execute("SELECT * FROM message_handoff WHERE envelope_id=?", (envelope_id,)).fetchone()
if row is None:
raise KeyError(envelope_id)
envelope = self._from_row(row)
if envelope.is_expired():
conn.execute("UPDATE message_handoff SET status='EXPIRED' WHERE envelope_id=?", (envelope_id,))
conn.commit()
raise MessageExpired(MessageExpired.code)
authorize_delivery(actor, envelope)
return envelope
def pending(self, *, target: str, actor: Any, limit: int = 100) -> list[MessageEnvelope]:
if not 1 <= limit <= 1000:
raise ValueError("MESSAGE_PENDING_LIMIT_INVALID")
visible: list[MessageEnvelope] = []
with self._connect() as conn:
rows = conn.execute(
"SELECT * FROM message_handoff WHERE target=? AND status='PENDING' ORDER BY created_at, sequence LIMIT ?",
(target, limit),
).fetchall()
for row in rows:
envelope = self._from_row(row)
if envelope.is_expired():
conn.execute("UPDATE message_handoff SET status='EXPIRED' WHERE envelope_id=?", (envelope.envelope_id,))
continue
try:
authorize_delivery(actor, envelope)
except HandoffError:
continue
visible.append(envelope)
conn.commit()
return visible
def mark_delivered(self, envelope_id: str, *, actor: Any) -> MessageEnvelope:
envelope = self.get(envelope_id, actor=actor)
with self._connect() as conn:
conn.execute(
"UPDATE message_handoff SET status='DELIVERED', delivered_at=? "
"WHERE envelope_id=? AND status='PENDING'",
(_iso(_utcnow()), envelope_id),
)
conn.commit()
return envelope
def acknowledge(self, envelope_id: str, *, actor: Any) -> MessageEnvelope:
envelope = self.get(envelope_id, actor=actor)
with self._connect() as conn:
conn.execute(
"UPDATE message_handoff SET status='ACKED', acked_at=? WHERE envelope_id=?",
(_iso(_utcnow()), envelope_id),
)
conn.commit()
return envelope
class HandoffFabric:
def __init__(self, store: DurableHandoffStore) -> None:
self.store = store
def publish(self, envelope: MessageEnvelope, *, actor: Any | None = None) -> tuple[str, MessageEnvelope]:
return self.store.put(bind_envelope_scope(envelope, actor) if actor is not None else envelope)
@@ -0,0 +1,68 @@
from __future__ import annotations
from typing import Any, Mapping
from message_handoff import MessageEnvelope
def _get(value: Any, *names: str, default: Any = None) -> Any:
for name in names:
if isinstance(value, Mapping) and name in value:
return value[name]
if hasattr(value, name):
return getattr(value, name)
return default
def agent_event_to_envelope(event: Any, *, target: str | None = None, sequence: int = 1,
idempotency_key: str | None = None, **scope: Any) -> MessageEnvelope:
payload = _get(event, "payload", "data", default={}) or {}
target = target or _get(event, "target", "recipient")
if not target:
raise ValueError("MESSAGE_TARGET_REQUIRED")
return MessageEnvelope(
source=str(_get(event, "source", "producer", "agent_id", default="agent-event")),
target=str(target),
message_type=str(_get(event, "event_type", "type", "message_type", default="agent.event")),
payload=dict(payload),
sequence=sequence,
idempotency_key=idempotency_key or str(_get(event, "id", "event_id", default="")) or f"agent-event:{sequence}",
causation_id=_get(event, "causation_id"),
**scope,
)
def taskplane_event_to_envelope(event: Any, *, target: str, sequence: int = 1,
idempotency_key: str | None = None, **scope: Any) -> MessageEnvelope:
data = _get(event, "data", "payload", default={}) or {}
event_id = str(_get(event, "id", "event_id", default=""))
return MessageEnvelope(
source=str(_get(event, "source", default="task-plane")),
target=target,
message_type=str(_get(event, "type", "event_type", default="task.event")),
payload=dict(data),
sequence=sequence,
idempotency_key=idempotency_key or event_id or f"task-event:{sequence}",
causation_id=event_id or None,
**scope,
)
def legacy_message_to_envelope(message: Any, *, target: str | None = None, sequence: int = 1,
idempotency_key: str | None = None, **scope: Any) -> MessageEnvelope:
payload = _get(message, "payload", "data", "body", "content", default={})
if not isinstance(payload, Mapping):
payload = {"value": payload}
target = target or _get(message, "target", "recipient", "receiver")
if not target:
raise ValueError("MESSAGE_TARGET_REQUIRED")
source = _get(message, "source", "sender", "agent_id", default="legacy-message-bus")
kind = _get(message, "message_type", "type", "kind", default="legacy.message")
identifier = _get(message, "id", "message_id", "event_id")
return MessageEnvelope(
source=str(source), target=str(target), message_type=str(kind), payload=dict(payload),
sequence=sequence,
idempotency_key=idempotency_key or (str(identifier) if identifier else f"legacy:{sequence}"),
causation_id=str(identifier) if identifier else None,
**scope,
)
+9 -1
View File
@@ -11,6 +11,7 @@ from dataclasses import asdict, dataclass, field
from typing import Any
from .search_gateway_orchestrator import SearchGatewayOrchestrator
from .trace_tool_credential_governance import GovernanceActor, TraceContext
@dataclass(frozen=True)
@@ -43,6 +44,9 @@ class SearchGatewayAgentAdapter:
domain_profiles: list[str] | None = None,
user_scope: str = "public",
correlation_id: str | None = None,
*,
actor: GovernanceActor | dict[str, Any] | None = None,
trace: TraceContext | None = None,
) -> dict[str, Any]:
payload = {
"query": query,
@@ -59,7 +63,11 @@ class SearchGatewayAgentAdapter:
"session_profile": context.session_profile,
"correlation_id": correlation_id,
}
return self.orchestrator.execute(payload)
if actor is None and trace is None:
# Preserve adapter compatibility; the governed orchestrator itself
# still fails closed when trusted context is absent.
return self.orchestrator.execute(payload)
return self.orchestrator.execute(payload, actor=actor, trace=trace)
class OpenClawSearchClient(SearchGatewayAgentAdapter):
+70 -4
View File
@@ -13,10 +13,18 @@ from __future__ import annotations
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
import hashlib
import json
from typing import Any, Protocol
from uuid import uuid4
from .search_profile_resolver import DomainQueryProfileRegistry, SearchProfile, SearchProfileResolver
from .trace_tool_credential_governance import (
GovernanceActor,
GovernanceError,
GovernanceStore,
TraceContext,
)
def utc_now() -> str:
@@ -156,13 +164,25 @@ class SearchPolicyGuard:
class SearchGateway:
def __init__(self, providers: list[SearchProvider] | None = None):
def __init__(self, providers: list[SearchProvider] | None = None, *, governance_store: GovernanceStore | None = None):
self.providers = providers or []
self.governance_store = governance_store
self.profile_resolver = SearchProfileResolver()
self.planner = QueryPlanner()
self.guard = SearchPolicyGuard()
def search(self, request: SearchRequest) -> SearchResponse:
def search(self, request: SearchRequest, *, actor: GovernanceActor | dict[str, Any] | None = None, trace: TraceContext | None = None) -> SearchResponse:
trusted_actor: GovernanceActor | None = None
if self.governance_store is not None:
if actor is None or trace is None:
raise GovernanceError("TOOL_MCP_AUTHZ_DENY")
trusted_actor = GovernanceActor.from_value(actor)
if request.requester != trusted_actor.identity:
raise GovernanceError("TOOL_MCP_AUTHZ_DENY", field="requester")
if request.correlation_id is not None and request.correlation_id not in {
trace.trace_id, trace.correlation_id
}:
raise GovernanceError("TOOL_SCOPE_PROPAGATION", field="correlation_id")
request_id = request.correlation_id or f"search_{uuid4().hex[:12]}"
profile = self.profile_resolver.resolve(request.group_profile, request.user_profile, request.session_profile)
expanded_queries = self.planner.expand(request, profile)
@@ -184,14 +204,60 @@ class SearchGateway:
attempts: list[SearchProviderAttempt] = []
providers = self._ordered_providers(profile)
for provider in providers:
if request.authorized_sources and provider.name not in request.authorized_sources:
attempts.append(SearchProviderAttempt(provider=provider.name, status="denied", reason="SOURCE_AUTHZ_DENY"))
continue
if request.user_scope != "public" and provider.name not in request.authorized_sources:
attempts.append(SearchProviderAttempt(provider=provider.name, status="denied", reason="SOURCE_AUTHZ_REQUIRED"))
continue
try:
if self.governance_store is not None:
assert trusted_actor is not None and trace is not None
input_digest = hashlib.sha256(
json.dumps(
{"query": request.query, "intent": request.intent, "provider": provider.name},
sort_keys=True, separators=(",", ":"), ensure_ascii=False,
).encode("utf-8")
).hexdigest()
invocation_key = hashlib.sha256(
f"{trace.trace_id}:{trace.span_id}:{provider.name}:{input_digest}".encode("utf-8")
).hexdigest()
authorization = self.governance_store.authorize_tool(
actor=trusted_actor,
tool_name=f"search.{provider.name}",
trace=trace,
requested_scope=trace.scope,
input_digest=input_digest,
invocation_key=invocation_key,
)
if authorization.get("idempotent"):
raise GovernanceError("TOOL_INVOCATION_REPLAY")
provider_evidence, attempt = provider.search(request, expanded_queries)
if self.governance_store is not None:
result_digest = hashlib.sha256(
json.dumps(
{
"evidence": [item.to_dict() for item in provider_evidence],
"attempt": attempt.to_dict(),
},
sort_keys=True, separators=(",", ":"), ensure_ascii=False,
).encode("utf-8")
).hexdigest()
self.governance_store.record_tool_receipt(
invocation_key=invocation_key,
result_digest=result_digest,
actor=trusted_actor,
trace=trace,
)
evidence.extend(provider_evidence)
attempts.append(attempt)
if provider_evidence:
break
except Exception as exc:
attempts.append(SearchProviderAttempt(provider=getattr(provider, "name", "unknown"), status="failed", reason=str(exc)))
except GovernanceError:
raise
except Exception:
# Provider exceptions may contain URLs, tokens, or local paths.
attempts.append(SearchProviderAttempt(provider=getattr(provider, "name", "unknown"), status="failed", reason="PROVIDER_FAILURE"))
if evidence:
return SearchResponse(
+8 -6
View File
@@ -7,27 +7,29 @@ try:
except Exception:
APIRouter = None
from .search_gateway import SearchGateway, SearchRequest
from .search_gateway import SearchRequest
from .search_gateway_orchestrator import SearchGatewayOrchestrator
from .trace_tool_credential_governance import GovernanceActor, TraceContext
from .search_gateway_dashboard import SearchGatewayDashboard
gateway = SearchGateway()
orchestrator = SearchGatewayOrchestrator()
dashboard = SearchGatewayDashboard()
def search_gateway_request(payload: dict[str, Any]) -> dict[str, Any]:
request = SearchRequest(**payload)
return gateway.search(request).to_dict()
def search_gateway_request(payload: dict[str, Any], *, actor: GovernanceActor | dict[str, Any] | None = None, trace: TraceContext | None = None) -> dict[str, Any]:
return orchestrator.execute(payload, actor=actor, trace=trace)
def get_search_gateway_status() -> dict[str, Any]:
return {
"module": "search-gateway",
"status": "ready",
"providers_configured": len(gateway.providers),
"providers_configured": len(orchestrator.gateway.providers),
"policy_guard": "enabled",
"profile_customization": "enabled",
"fallback_required_when_provider_unavailable": True,
"governance": "required",
}
@@ -3,24 +3,49 @@
from __future__ import annotations
from typing import Any
import os
from .search_audit_store import SearchAuditStore
from .search_event_bridge import SearchEventBridge
from .search_gateway import SearchGateway, SearchRequest
from .search_provider_registry import SearchProviderRegistry
from .trace_tool_credential_governance import (
GovernanceActor,
GovernanceScope,
GovernanceStore,
ToolDescriptor,
TraceContext,
)
class SearchGatewayOrchestrator:
def __init__(self):
def __init__(self, *, governance_store: GovernanceStore | None = None):
self.registry = SearchProviderRegistry()
self.audit = SearchAuditStore()
self.events = SearchEventBridge()
self.gateway = SearchGateway(self.registry.ordered())
self.governance = governance_store or GovernanceStore(
os.environ.get("AIOS_SEARCH_GOVERNANCE_DB", "runtime/search-governance.sqlite3")
)
registrar = GovernanceActor(
"search-runtime-bootstrap", GovernanceScope("system"), ("system.admin",)
)
for provider in self.registry.ordered():
self.governance.register_tool(
ToolDescriptor(
f"search.{provider.name}", "tool", GovernanceScope("global"), ("tool.search",)
),
registrar,
)
self.gateway = SearchGateway(self.registry.ordered(), governance_store=self.governance)
def execute(self, payload: dict[str, Any]) -> dict[str, Any]:
def execute(self, payload: dict[str, Any], *, actor: GovernanceActor | dict[str, Any] | None = None, trace: TraceContext | None = None) -> dict[str, Any]:
if actor is None or trace is None:
from .trace_tool_credential_governance import GovernanceError
raise GovernanceError("TOOL_MCP_AUTHZ_DENY")
self.governance.record_trace(trace, actor)
request = SearchRequest(**payload)
self.events.search_started(request.to_dict())
response = self.gateway.search(request)
response = self.gateway.search(request, actor=actor, trace=trace)
result = response.to_dict()
for attempt in result.get("attempts", []):
self.events.provider_attempted(result["request_id"], attempt)
@@ -35,6 +60,7 @@ class SearchGatewayOrchestrator:
"registry": self.registry.status(),
"audit": "enabled",
"events": "enabled",
"governance": "required",
}
+56 -26
View File
@@ -34,6 +34,7 @@ from .models import (
)
from .policy import PolicyEngine
from .store import TaskPlaneStore
from .scope import bind_task_scope
from .worker_registry import WorkerRegistry
@@ -68,28 +69,54 @@ class TaskPlaneService:
self.delegation = DelegationEngine(store)
self.store.probe_attestation_verifier = self.policy.probe_attestation_verifier
def create_task(self, body: dict[str, Any], idempotency_key: str | None = None) -> dict[str, Any]:
def _authorized_task(self, task_id: str, actor: str, action: str) -> TaskRecord:
task = self.store.get_task(task_id, self.policy.actor_context(actor))
self.policy.assert_allowed(action, actor, task=task)
return task
def create_task(
self,
body: dict[str, Any],
idempotency_key: str | None = None,
*,
actor_identity: str | None = None,
) -> dict[str, Any]:
if body.get("parent_task_id"):
raise ForbiddenError("child tasks must use the delegate endpoint")
issuer = body.get("issuer", {})
actor = issuer.get("id", "anon") if isinstance(issuer, dict) else "anon"
declared_actor = issuer.get("id", "anon") if isinstance(issuer, dict) else "anon"
actor = actor_identity or declared_actor
self.policy.assert_allowed("create", actor)
try:
plan = TaskPlan.from_dict(body)
except (TypeError, ValueError) as exc:
raise ValidationError("invalid task plan") from exc
if actor_identity is not None:
plan = bind_task_scope(
plan, self.policy.actor_context(actor_identity), public_create=True
)
task = self.store.create_task_with_event(plan, idempotency_key=idempotency_key)
return task.to_dict()
def list_tasks(self, actor: str | None = None) -> list[dict[str, Any]]:
context = None
if actor is not None:
self.policy.assert_allowed("read", actor)
return [task.to_dict(include_task=False) for task in self.store.list_tasks()]
context = self.policy.actor_context(actor)
return [
task.to_dict(include_task=False)
for task in self.store.list_tasks(context)
]
def get_task(self, task_id: str, actor: str | None = None) -> dict[str, Any]:
context = None
if actor is not None:
self.policy.assert_allowed("read", actor, task=self.store.get_task(task_id))
return self.store.get_task(task_id).to_dict()
self.policy.assert_allowed("read", actor)
context = self.policy.actor_context(actor)
task = self.store.get_task(task_id, context)
if actor is not None:
self.policy.assert_allowed("read", actor, task=task)
return task.to_dict()
def get_worker(self, worker_id: str, actor: str | None = None) -> dict[str, Any]:
if actor is not None:
@@ -163,9 +190,8 @@ class TaskPlaneService:
if not worker_id:
raise ValidationError("worker_id required")
task = self.store.get_task(task_id)
task = self._authorized_task(task_id, actor, "claim")
self.policy.assert_readonly_worker_binding(actor, str(worker_id))
self.policy.assert_allowed("claim", actor, task=task)
lease_seconds = _bounded_int(
body.get("lease_seconds", 30), "lease_seconds", 1, 3600
)
@@ -184,10 +210,9 @@ class TaskPlaneService:
if not claim_id:
raise ValidationError("claim_id required")
actor = body.get("actor_identity") or body.get("worker_identity") or "worker"
task = self.store.get_task(task_id)
task = self._authorized_task(task_id, actor, "heartbeat")
claim = self.store.get_claim(claim_id)
self.policy.assert_readonly_worker_binding(actor, claim.worker_id)
self.policy.assert_allowed("heartbeat", actor, task=task)
if claim.worker_identity != actor:
raise ForbiddenError("claim identity mismatch")
lease_seconds = _bounded_int(
@@ -206,10 +231,9 @@ class TaskPlaneService:
message = body.get("message")
if progress is None:
raise ValidationError("progress required")
task = self.store.get_task(task_id)
task = self._authorized_task(task_id, actor, "progress")
claim = self.store.get_claim(claim_id)
self.policy.assert_readonly_worker_binding(actor, claim.worker_id)
self.policy.assert_allowed("progress", actor, task=task)
if task.task_plan.data_classification == "credential-secret":
if message not in {None, ""} or contains_secret_text(stage):
raise ValidationError(
@@ -234,7 +258,7 @@ class TaskPlaneService:
if not claim_id:
raise ValidationError("claim_id required")
actor = body.get("actor_identity") or body.get("worker_identity") or "worker"
task = self.store.get_task(task_id)
task = self._authorized_task(task_id, actor, "result")
status = body.get("status", "COMPLETED")
outputs = body.get("outputs", [])
@@ -247,7 +271,6 @@ class TaskPlaneService:
raise ValidationError("evidence items must be objects")
claim = self.store.get_claim(claim_id)
self.policy.assert_readonly_worker_binding(actor, claim.worker_id)
self.policy.assert_allowed("result", actor, task=task)
if claim.worker_identity != actor:
raise ForbiddenError("claim identity mismatch")
if task.task_plan.data_classification == "credential-secret":
@@ -326,8 +349,7 @@ class TaskPlaneService:
reason = body.get("reason", "failed")
actor = body.get("actor_identity") or body.get("worker_identity") or "worker"
self.policy.assert_allowed("fail", actor)
task = self.store.get_task(task_id)
self.policy.assert_allowed("fail", actor, task=task)
task = self._authorized_task(task_id, actor, "fail")
claim = self.store.get_claim(claim_id)
if claim.worker_identity != actor:
raise ForbiddenError("claim identity mismatch")
@@ -339,8 +361,7 @@ class TaskPlaneService:
actor = body.get("actor_identity") or body.get("worker_identity") or "owner"
self.policy.assert_allowed("cancel", actor)
reason = body.get("reason", "cancelled")
task = self.store.get_task(task_id)
self.policy.assert_allowed("cancel", actor, task=task)
task = self._authorized_task(task_id, actor, "cancel")
updated = self.store.cancel_task_with_event(
task_id, task.revision, reason=reason
)
@@ -349,8 +370,7 @@ class TaskPlaneService:
def delegate(self, task_id: str, body: dict[str, Any]) -> dict[str, Any]:
actor = body.get("actor_identity") or "owner"
self.policy.assert_allowed("delegate", actor)
parent = self.store.get_task(task_id)
self.policy.assert_allowed("delegate", actor, task=parent)
parent = self._authorized_task(task_id, actor, "delegate")
if "task" in body:
try:
child_plan = TaskPlan.from_dict(body["task"])
@@ -364,25 +384,31 @@ class TaskPlaneService:
except (TypeError, ValueError) as exc:
raise ValidationError("invalid child task plan") from exc
child_plan.parent_task_id = task_id
child_plan = bind_task_scope(
child_plan, self.policy.actor_context(actor), public_create=False
)
child = self.delegation.delegate(task_id, child_plan, caller=actor)
return child.to_dict()
def get_events(self, task_id: str, actor: str | None = None) -> list[dict[str, Any]]:
if actor is not None:
self.policy.assert_allowed("audit", actor, task=self.store.get_task(task_id))
self.store.get_task(task_id)
self._authorized_task(task_id, actor, "audit")
else:
self.store.get_task(task_id)
return self.store.list_events(task_id)
def get_evidence(self, task_id: str, actor: str | None = None) -> list[dict[str, Any]]:
if actor is not None:
self.policy.assert_allowed("audit", actor, task=self.store.get_task(task_id))
self.store.get_task(task_id)
self._authorized_task(task_id, actor, "audit")
else:
self.store.get_task(task_id)
return self.evidence.list(task_id)
def get_children(self, task_id: str, actor: str | None = None) -> list[str]:
if actor is not None:
self.policy.assert_allowed("read", actor, task=self.store.get_task(task_id))
self.store.get_task(task_id)
self._authorized_task(task_id, actor, "read")
else:
self.store.get_task(task_id)
return self.store.list_children(task_id)
@@ -506,7 +532,11 @@ class TaskPlaneRequestHandler(BaseHTTPRequestHandler):
):
raise ForbiddenError("issuer identity mismatch")
body["issuer"] = {"type": "authenticated", "id": identity}
return 201, self.service.create_task(body or {}, headers.get("idempotency-key"))
return 201, self.service.create_task(
body or {},
headers.get("idempotency-key"),
actor_identity=identity,
)
if body.get("actor_identity") not in {None, identity}:
raise ForbiddenError("actor identity mismatch")
body["actor_identity"] = identity
+5
View File
@@ -106,3 +106,8 @@ class DurableOutboxDispatcher:
receipt=str(receipt) if receipt else None,
error=None if receipt else f"{adapter.name} receipt unavailable",
)
# R4.2 P0-B compatibility adapter.
def to_handoff_envelope(event, *, target, sequence=1, idempotency_key=None, **scope):
from message_handoff_adapters import taskplane_event_to_envelope
return taskplane_event_to_envelope(event, target=target, sequence=sequence, idempotency_key=idempotency_key, **scope)
+911
View File
@@ -0,0 +1,911 @@
"""Durable multi-agent fleet scheduling for TaskPlane.
The fleet layer decides *which* eligible worker should receive *which* queued task.
It never replaces TaskPlane claim/lease ownership: callers must still perform the
authoritative TaskPlane claim CAS after a reservation is returned.
"""
from __future__ import annotations
from contextlib import contextmanager
from dataclasses import dataclass
import hashlib
import json
import math
import sqlite3
import threading
import time
from typing import Any, Iterable, Mapping, Sequence
DEFAULT_RESERVATION_SECONDS = 30
DEFAULT_CIRCUIT_THRESHOLD = 3
DEFAULT_CIRCUIT_SECONDS = 30
MAX_CIRCUIT_SECONDS = 15 * 60
MAX_SLOTS = 128
class FleetError(RuntimeError):
pass
class FleetConflict(FleetError):
pass
def _canonical(value: object) -> bytes:
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode("utf-8")
def _sha(value: object) -> str:
return hashlib.sha256(_canonical(value)).hexdigest()
def _text(value: object, default: str = "") -> str:
if value is None:
return default
return str(value).strip() or default
def _int(value: object, default: int, minimum: int, maximum: int) -> int:
if value is None:
return default
if isinstance(value, bool):
raise FleetError("FLEET_INTEGER_INVALID")
try:
result = int(value)
except (TypeError, ValueError) as exc:
raise FleetError("FLEET_INTEGER_INVALID") from exc
if result < minimum or result > maximum:
raise FleetError("FLEET_INTEGER_OUT_OF_RANGE")
return result
def _float(value: object, default: float, minimum: float, maximum: float) -> float:
if value is None:
return default
try:
result = float(value)
except (TypeError, ValueError) as exc:
raise FleetError("FLEET_NUMBER_INVALID") from exc
if not math.isfinite(result) or result < minimum or result > maximum:
raise FleetError("FLEET_NUMBER_OUT_OF_RANGE")
return result
def _mapping(value: object) -> Mapping[str, Any]:
if isinstance(value, Mapping):
return value
if hasattr(value, "to_dict"):
data = value.to_dict()
if isinstance(data, Mapping):
return data
if hasattr(value, "__dict__"):
return vars(value)
return {}
def _attr(value: object, name: str, default: Any = None) -> Any:
if isinstance(value, Mapping):
return value.get(name, default)
return getattr(value, name, default)
def _task_plan(task: object) -> object:
return _attr(task, "task_plan", _attr(task, "plan", task))
def _task_id(task: object) -> str:
return _text(_attr(task, "task_id", _attr(_task_plan(task), "task_id", "")))
def _task_status(task: object) -> str:
return _text(_attr(task, "status", "")).upper()
def _task_revision(task: object) -> int:
return _int(_attr(task, "revision", 0), 0, 0, 2**31 - 1)
def _task_created_at(task: object) -> float:
value = _attr(task, "created_at", _attr(task, "updated_at", 0.0))
try:
return float(value or 0.0)
except (TypeError, ValueError):
return 0.0
def _task_required_capabilities(task: object) -> frozenset[str]:
plan = _task_plan(task)
value = _attr(plan, "required_capabilities", ())
if not isinstance(value, Sequence) or isinstance(value, (str, bytes)):
return frozenset()
return frozenset(_text(item) for item in value if _text(item))
def _task_metadata(task: object) -> Mapping[str, Any]:
plan = _task_plan(task)
value = _attr(plan, "metadata", {})
return value if isinstance(value, Mapping) else {}
def _task_scope_key(task: object) -> str:
plan = _task_plan(task)
scope_type = _text(_attr(plan, "scope_type", "legacy"), "legacy")
tenant = _text(_attr(plan, "tenant_id", ""))
project = _text(_attr(plan, "project_id", ""))
user = _text(_attr(plan, "user_id", ""))
if scope_type == "tenant":
return "tenant:" + tenant + "|project:" + (project or "*")
if scope_type == "system":
return "system"
return "legacy:" + (user or "*")
def _worker_id(worker: object) -> str:
return _text(_attr(worker, "worker_id", ""))
def _worker_capabilities(worker: object) -> frozenset[str]:
value = _attr(worker, "capabilities", ())
if isinstance(value, Mapping):
value = value.keys()
if not isinstance(value, Iterable) or isinstance(value, (str, bytes)):
return frozenset()
return frozenset(_text(item) for item in value if _text(item))
@dataclass(frozen=True)
class FleetWorkerPolicy:
worker_id: str
max_slots: int = 1
failure_domain: str = "default"
weight: float = 1.0
labels: Mapping[str, str] | None = None
revision: int = 1
@dataclass(frozen=True)
class FleetDispatchDecision:
reservation_id: str
task_id: str
worker_id: str
task_revision: int
scope_key: str
failure_domain: str
reserved_until: float
score: tuple[float, float, float, str]
def to_dict(self) -> dict[str, Any]:
return {
"reservation_id": self.reservation_id,
"task_id": self.task_id,
"worker_id": self.worker_id,
"task_revision": self.task_revision,
"scope_key": self.scope_key,
"failure_domain": self.failure_domain,
"reserved_until": self.reserved_until,
"score": list(self.score),
}
class FleetStateStore:
"""Small durable fleet journal colocated in the existing TaskPlane SQLite database."""
def __init__(self, connection: sqlite3.Connection, lock: threading.RLock | None = None) -> None:
self.conn = connection
self.lock = lock or threading.RLock()
self._savepoint_counter = 0
self._ensure_schema()
@contextmanager
def _transaction(self):
with self.lock:
self._savepoint_counter += 1
name = f"fleet_{threading.get_ident()}_{self._savepoint_counter}"
self.conn.execute(f"SAVEPOINT {name}")
try:
yield
except Exception:
self.conn.execute(f"ROLLBACK TO {name}")
self.conn.execute(f"RELEASE {name}")
raise
else:
self.conn.execute(f"RELEASE {name}")
def _ensure_schema(self) -> None:
with self.lock:
self.conn.executescript(
"""
CREATE TABLE IF NOT EXISTS fleet_worker_capacity (
worker_id TEXT PRIMARY KEY,
max_slots INTEGER NOT NULL,
failure_domain TEXT NOT NULL,
weight REAL NOT NULL,
labels_json TEXT NOT NULL,
revision INTEGER NOT NULL,
updated_at REAL NOT NULL
);
CREATE TABLE IF NOT EXISTS fleet_scope_state (
scope_key TEXT PRIMARY KEY,
virtual_finish REAL NOT NULL,
dispatch_count INTEGER NOT NULL,
last_dispatched_at REAL,
revision INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS fleet_dispatches (
reservation_id TEXT PRIMARY KEY,
task_id TEXT NOT NULL UNIQUE,
worker_id TEXT NOT NULL,
task_revision INTEGER NOT NULL,
scope_key TEXT NOT NULL,
failure_domain TEXT NOT NULL,
state TEXT NOT NULL,
reserved_until REAL NOT NULL,
claim_id TEXT,
created_at REAL NOT NULL,
updated_at REAL NOT NULL
);
CREATE INDEX IF NOT EXISTS ix_fleet_dispatch_worker_state
ON fleet_dispatches(worker_id, state, reserved_until);
CREATE TABLE IF NOT EXISTS fleet_worker_failures (
worker_id TEXT PRIMARY KEY,
consecutive_failures INTEGER NOT NULL,
circuit_open_until REAL,
last_failure_class TEXT,
revision INTEGER NOT NULL,
updated_at REAL NOT NULL
);
CREATE TABLE IF NOT EXISTS fleet_task_worker_failures (
task_id TEXT NOT NULL,
worker_id TEXT NOT NULL,
failure_count INTEGER NOT NULL,
blocked_until REAL,
last_failure_class TEXT,
revision INTEGER NOT NULL,
updated_at REAL NOT NULL,
PRIMARY KEY(task_id, worker_id)
);
"""
)
def configure_worker(
self,
worker_id: str,
*,
max_slots: int = 1,
failure_domain: str = "default",
weight: float = 1.0,
labels: Mapping[str, str] | None = None,
expected_revision: int | None = None,
now: float | None = None,
) -> FleetWorkerPolicy:
worker_id = _text(worker_id)
if not worker_id:
raise FleetError("FLEET_WORKER_ID_REQUIRED")
max_slots = _int(max_slots, 1, 1, MAX_SLOTS)
weight = _float(weight, 1.0, 0.05, 100.0)
failure_domain = _text(failure_domain, "default")
normalized_labels = {
_text(k): _text(v)
for k, v in dict(labels or {}).items()
if _text(k) and _text(v)
}
now = float(time.time() if now is None else now)
with self._transaction():
row = self.conn.execute(
"SELECT revision FROM fleet_worker_capacity WHERE worker_id=?",
(worker_id,),
).fetchone()
current = int(row[0]) if row else 0
if expected_revision is not None and current != int(expected_revision):
raise FleetConflict("FLEET_WORKER_POLICY_CAS_MISMATCH")
revision = current + 1
self.conn.execute(
"""
INSERT INTO fleet_worker_capacity(
worker_id,max_slots,failure_domain,weight,labels_json,revision,updated_at
) VALUES (?,?,?,?,?,?,?)
ON CONFLICT(worker_id) DO UPDATE SET
max_slots=excluded.max_slots,
failure_domain=excluded.failure_domain,
weight=excluded.weight,
labels_json=excluded.labels_json,
revision=excluded.revision,
updated_at=excluded.updated_at
""",
(
worker_id,
max_slots,
failure_domain,
weight,
json.dumps(normalized_labels, sort_keys=True, separators=(",", ":")),
revision,
now,
),
)
return FleetWorkerPolicy(
worker_id, max_slots, failure_domain, weight, normalized_labels, revision
)
def worker_policy(self, worker_id: str) -> FleetWorkerPolicy:
worker_id = _text(worker_id)
with self.lock:
row = self.conn.execute(
"""
SELECT worker_id,max_slots,failure_domain,weight,labels_json,revision
FROM fleet_worker_capacity WHERE worker_id=?
""",
(worker_id,),
).fetchone()
if row is None:
return FleetWorkerPolicy(worker_id=worker_id)
return FleetWorkerPolicy(
worker_id=str(row[0]),
max_slots=int(row[1]),
failure_domain=str(row[2]),
weight=float(row[3]),
labels=json.loads(row[4] or "{}"),
revision=int(row[5]),
)
def _cleanup_expired(self, now: float) -> None:
self.conn.execute(
"""
UPDATE fleet_dispatches
SET state='EXPIRED', updated_at=?
WHERE state='RESERVED' AND reserved_until<=?
""",
(now, now),
)
def reserved_slots(self, worker_id: str, now: float) -> int:
with self.lock:
self._cleanup_expired(now)
row = self.conn.execute(
"""
SELECT COUNT(*) FROM fleet_dispatches
WHERE worker_id=? AND state='RESERVED' AND reserved_until>?
""",
(worker_id, now),
).fetchone()
return int(row[0] if row else 0)
def scope_virtual_finish(self, scope_key: str) -> float:
with self.lock:
row = self.conn.execute(
"SELECT virtual_finish FROM fleet_scope_state WHERE scope_key=?",
(scope_key,),
).fetchone()
return float(row[0]) if row else 0.0
def reserve(
self,
*,
task_id: str,
worker_id: str,
task_revision: int,
scope_key: str,
failure_domain: str,
score: tuple[float, float, float, str],
reservation_seconds: int,
now: float,
scope_weight: float,
) -> FleetDispatchDecision:
reservation_seconds = _int(
reservation_seconds, DEFAULT_RESERVATION_SECONDS, 1, 300
)
reservation_id = "fleet-" + _sha(
{
"task_id": task_id,
"worker_id": worker_id,
"task_revision": task_revision,
"scope_key": scope_key,
}
)[:32]
reserved_until = now + reservation_seconds
with self._transaction():
self._cleanup_expired(now)
existing = self.conn.execute(
"""
SELECT reservation_id,worker_id,task_revision,scope_key,failure_domain,
reserved_until,state
FROM fleet_dispatches WHERE task_id=?
""",
(task_id,),
).fetchone()
# Capacity is rechecked inside the same durable reservation transaction.
# This closes the cross-thread/process race between eligibility scoring and reserve.
policy_row = self.conn.execute(
"SELECT max_slots FROM fleet_worker_capacity WHERE worker_id=?",
(worker_id,),
).fetchone()
max_slots = int(policy_row[0]) if policy_row else 1
if existing and str(existing[6]) in {"RESERVED", "CLAIMED"} and float(existing[5]) > now:
if str(existing[1]) != worker_id or int(existing[2]) != task_revision:
raise FleetConflict("FLEET_TASK_ALREADY_RESERVED")
return FleetDispatchDecision(
str(existing[0]), task_id, worker_id, task_revision, str(existing[3]),
str(existing[4]), float(existing[5]), score
)
claim_columns = {
str(row[1])
for row in self.conn.execute("PRAGMA table_info(claims)").fetchall()
}
active_claims = 0
if {"worker_id", "status"}.issubset(claim_columns):
sql = "SELECT COUNT(*) FROM claims WHERE worker_id=? AND status='ACTIVE'"
params: list[Any] = [worker_id]
if "lease_expires_at" in claim_columns:
sql += " AND lease_expires_at>?"
params.append(now)
row = self.conn.execute(sql, tuple(params)).fetchone()
active_claims = int(row[0] if row else 0)
row = self.conn.execute(
"""
SELECT COUNT(*) FROM fleet_dispatches
WHERE worker_id=? AND task_id<>?
AND state='RESERVED' AND reserved_until>?
""",
(worker_id, task_id, now),
).fetchone()
active_reservations = int(row[0] if row else 0)
if active_claims + active_reservations >= max_slots:
raise FleetConflict("FLEET_WORKER_CAPACITY_EXHAUSTED")
self.conn.execute(
"""
INSERT INTO fleet_dispatches(
reservation_id,task_id,worker_id,task_revision,scope_key,failure_domain,
state,reserved_until,claim_id,created_at,updated_at
) VALUES (?,?,?,?,?,?,'RESERVED',?,NULL,?,?)
ON CONFLICT(task_id) DO UPDATE SET
reservation_id=excluded.reservation_id,
worker_id=excluded.worker_id,
task_revision=excluded.task_revision,
scope_key=excluded.scope_key,
failure_domain=excluded.failure_domain,
state='RESERVED',
reserved_until=excluded.reserved_until,
claim_id=NULL,
updated_at=excluded.updated_at
""",
(
reservation_id, task_id, worker_id, task_revision, scope_key,
failure_domain, reserved_until, now, now,
),
)
row = self.conn.execute(
"SELECT virtual_finish,dispatch_count,revision FROM fleet_scope_state WHERE scope_key=?",
(scope_key,),
).fetchone()
current_finish = float(row[0]) if row else 0.0
dispatch_count = int(row[1]) if row else 0
revision = int(row[2]) if row else 0
next_finish = max(current_finish, 0.0) + 1.0 / max(scope_weight, 0.05)
self.conn.execute(
"""
INSERT INTO fleet_scope_state(
scope_key,virtual_finish,dispatch_count,last_dispatched_at,revision
) VALUES (?,?,?,?,?)
ON CONFLICT(scope_key) DO UPDATE SET
virtual_finish=excluded.virtual_finish,
dispatch_count=excluded.dispatch_count,
last_dispatched_at=excluded.last_dispatched_at,
revision=excluded.revision
""",
(scope_key, next_finish, dispatch_count + 1, now, revision + 1),
)
return FleetDispatchDecision(
reservation_id, task_id, worker_id, task_revision, scope_key,
failure_domain, reserved_until, score
)
def bind_claim(self, reservation_id: str, claim_id: str, *, now: float | None = None) -> None:
now = float(time.time() if now is None else now)
with self._transaction():
row = self.conn.execute(
"SELECT state FROM fleet_dispatches WHERE reservation_id=?",
(reservation_id,),
).fetchone()
if row is None:
raise FleetError("FLEET_RESERVATION_NOT_FOUND")
if str(row[0]) not in {"RESERVED", "CLAIMED"}:
raise FleetConflict("FLEET_RESERVATION_NOT_ACTIVE")
self.conn.execute(
"""
UPDATE fleet_dispatches
SET state='CLAIMED', claim_id=?, updated_at=?
WHERE reservation_id=?
""",
(claim_id, now, reservation_id),
)
def release(
self,
task_id: str,
*,
state: str = "RELEASED",
now: float | None = None,
) -> None:
if state not in {"RELEASED", "COMPLETED", "FAILED", "EXPIRED"}:
raise FleetError("FLEET_RESERVATION_STATE_INVALID")
now = float(time.time() if now is None else now)
with self._transaction():
self.conn.execute(
"UPDATE fleet_dispatches SET state=?,updated_at=? WHERE task_id=?",
(state, now, task_id),
)
def worker_circuit_open(self, worker_id: str, now: float) -> bool:
with self.lock:
row = self.conn.execute(
"SELECT circuit_open_until FROM fleet_worker_failures WHERE worker_id=?",
(worker_id,),
).fetchone()
return bool(row and row[0] is not None and float(row[0]) > now)
def task_worker_blocked(self, task_id: str, worker_id: str, now: float) -> bool:
with self.lock:
row = self.conn.execute(
"""
SELECT blocked_until FROM fleet_task_worker_failures
WHERE task_id=? AND worker_id=?
""",
(task_id, worker_id),
).fetchone()
return bool(row and row[0] is not None and float(row[0]) > now)
def record_worker_failure(
self,
worker_id: str,
failure_class: str,
*,
threshold: int = DEFAULT_CIRCUIT_THRESHOLD,
base_circuit_seconds: int = DEFAULT_CIRCUIT_SECONDS,
now: float | None = None,
) -> dict[str, Any]:
now = float(time.time() if now is None else now)
threshold = _int(threshold, DEFAULT_CIRCUIT_THRESHOLD, 1, 20)
base_circuit_seconds = _int(base_circuit_seconds, DEFAULT_CIRCUIT_SECONDS, 1, 3600)
with self._transaction():
row = self.conn.execute(
"""
SELECT consecutive_failures,revision FROM fleet_worker_failures
WHERE worker_id=?
""",
(worker_id,),
).fetchone()
failures = (int(row[0]) if row else 0) + 1
revision = (int(row[1]) if row else 0) + 1
open_until = None
if failures >= threshold:
exponent = min(failures - threshold, 8)
open_until = now + min(
MAX_CIRCUIT_SECONDS, base_circuit_seconds * (2**exponent)
)
self.conn.execute(
"""
INSERT INTO fleet_worker_failures(
worker_id,consecutive_failures,circuit_open_until,last_failure_class,
revision,updated_at
) VALUES (?,?,?,?,?,?)
ON CONFLICT(worker_id) DO UPDATE SET
consecutive_failures=excluded.consecutive_failures,
circuit_open_until=excluded.circuit_open_until,
last_failure_class=excluded.last_failure_class,
revision=excluded.revision,
updated_at=excluded.updated_at
""",
(worker_id, failures, open_until, _text(failure_class, "unknown"), revision, now),
)
return {
"worker_id": worker_id,
"consecutive_failures": failures,
"circuit_open_until": open_until,
"revision": revision,
}
def record_worker_success(self, worker_id: str, *, now: float | None = None) -> None:
now = float(time.time() if now is None else now)
with self._transaction():
self.conn.execute(
"""
INSERT INTO fleet_worker_failures(
worker_id,consecutive_failures,circuit_open_until,last_failure_class,
revision,updated_at
) VALUES (?,0,NULL,NULL,1,?)
ON CONFLICT(worker_id) DO UPDATE SET
consecutive_failures=0,
circuit_open_until=NULL,
last_failure_class=NULL,
revision=revision+1,
updated_at=excluded.updated_at
""",
(worker_id, now),
)
def record_task_worker_failure(
self,
task_id: str,
worker_id: str,
failure_class: str,
*,
quarantine_seconds: int = 60,
now: float | None = None,
) -> dict[str, Any]:
now = float(time.time() if now is None else now)
quarantine_seconds = _int(quarantine_seconds, 60, 1, 3600)
with self._transaction():
row = self.conn.execute(
"""
SELECT failure_count,revision FROM fleet_task_worker_failures
WHERE task_id=? AND worker_id=?
""",
(task_id, worker_id),
).fetchone()
count = (int(row[0]) if row else 0) + 1
revision = (int(row[1]) if row else 0) + 1
blocked_until = now + quarantine_seconds
self.conn.execute(
"""
INSERT INTO fleet_task_worker_failures(
task_id,worker_id,failure_count,blocked_until,last_failure_class,
revision,updated_at
) VALUES (?,?,?,?,?,?,?)
ON CONFLICT(task_id,worker_id) DO UPDATE SET
failure_count=excluded.failure_count,
blocked_until=excluded.blocked_until,
last_failure_class=excluded.last_failure_class,
revision=excluded.revision,
updated_at=excluded.updated_at
""",
(
task_id, worker_id, count, blocked_until,
_text(failure_class, "unknown"), revision, now,
),
)
return {
"task_id": task_id,
"worker_id": worker_id,
"failure_count": count,
"blocked_until": blocked_until,
"revision": revision,
}
class FleetScheduler:
"""Deterministic capacity/fairness scheduler over existing TaskPlane records."""
def __init__(
self,
store: Any,
workers: Any,
*,
now_fn: Any = time.time,
reservation_seconds: int = DEFAULT_RESERVATION_SECONDS,
) -> None:
self.store = store
self.workers = workers
self.now_fn = now_fn
self.reservation_seconds = _int(
reservation_seconds, DEFAULT_RESERVATION_SECONDS, 1, 300
)
connection = getattr(store, "conn", None)
if not isinstance(connection, sqlite3.Connection):
raise FleetError("FLEET_TASKPLANE_SQLITE_CONNECTION_REQUIRED")
lock = getattr(store, "_lock", None)
self.state = FleetStateStore(connection, lock)
def configure_worker(self, worker_id: str, **kwargs: Any) -> FleetWorkerPolicy:
return self.state.configure_worker(worker_id, **kwargs)
def _online(self, worker_id: str, now: float) -> bool:
method = getattr(self.workers, "is_online", None)
if not callable(method):
raise FleetError("FLEET_WORKER_REGISTRY_ONLINE_CHECK_REQUIRED")
try:
return bool(method(worker_id, now=now))
except TypeError:
return bool(method(worker_id))
def _worker_record(self, worker_id: str) -> object:
getter = getattr(self.store, "get_worker", None)
if not callable(getter):
raise FleetError("FLEET_WORKER_LOOKUP_REQUIRED")
return getter(worker_id)
def _active_claims(self, worker_id: str, now: float) -> int:
# The TaskPlane store owns one SQLite connection in the current runtime.
# Fleet reads must share its lock just like fleet writes do, otherwise two
# concurrent pullers can issue overlapping operations on the same handle.
with self.state.lock:
columns = {
str(row[1])
for row in self.store.conn.execute("PRAGMA table_info(claims)").fetchall()
}
required = {"worker_id", "status"}
if not required.issubset(columns):
raise FleetError("FLEET_CLAIMS_SCHEMA_UNSUPPORTED")
sql = "SELECT COUNT(*) FROM claims WHERE worker_id=? AND status='ACTIVE'"
params: list[Any] = [worker_id]
if "lease_expires_at" in columns:
sql += " AND lease_expires_at>?"
params.append(now)
row = self.store.conn.execute(sql, tuple(params)).fetchone()
return int(row[0] if row else 0)
@staticmethod
def _labels_match(policy: FleetWorkerPolicy, task: object) -> bool:
metadata = _task_metadata(task)
affinity = metadata.get("fleet_affinity_labels") or {}
if affinity and not isinstance(affinity, Mapping):
return False
labels = dict(policy.labels or {})
for key, value in dict(affinity).items():
if labels.get(_text(key)) != _text(value):
return False
avoid = metadata.get("fleet_avoid_failure_domains") or []
if isinstance(avoid, Sequence) and not isinstance(avoid, (str, bytes)):
if policy.failure_domain in {_text(item) for item in avoid}:
return False
return True
def _eligible_worker(self, worker_id: str, task: object, now: float) -> tuple[bool, int, FleetWorkerPolicy]:
if not self._online(worker_id, now):
return False, 0, self.state.worker_policy(worker_id)
worker = self._worker_record(worker_id)
policy = self.state.worker_policy(worker_id)
if self.state.worker_circuit_open(worker_id, now):
return False, 0, policy
if self.state.task_worker_blocked(_task_id(task), worker_id, now):
return False, 0, policy
if not self._labels_match(policy, task):
return False, 0, policy
required = _task_required_capabilities(task)
capabilities = _worker_capabilities(worker)
if required and not required.issubset(capabilities):
return False, 0, policy
active = self._active_claims(worker_id, now)
reserved = self.state.reserved_slots(worker_id, now)
load = active + reserved
if load >= policy.max_slots:
return False, load, policy
return True, load, policy
@staticmethod
def _task_priority(task: object, now: float) -> float:
metadata = _task_metadata(task)
base = _int(metadata.get("fleet_priority"), 0, -100, 100)
created = _task_created_at(task)
waited = max(0.0, now - created) if created > 0 else 0.0
aging = min(100.0, waited / 60.0)
return float(base) + aging
def _task_score(self, task: object, now: float) -> tuple[float, float, float, str]:
scope = _task_scope_key(task)
virtual_finish = self.state.scope_virtual_finish(scope)
priority = self._task_priority(task, now)
return (virtual_finish, -priority, _task_created_at(task), _task_id(task))
def _tasks(self, actor_context: Any = None) -> list[object]:
lister = getattr(self.store, "list_tasks", None)
if not callable(lister):
raise FleetError("FLEET_TASK_LIST_REQUIRED")
try:
tasks = lister(actor_context)
except TypeError:
tasks = lister()
return [
task for task in tasks
if _task_status(task) in {"CREATED", "QUEUED"}
and _task_id(task)
]
def rank_for_worker(
self,
worker_id: str,
*,
actor_context: Any = None,
now: float | None = None,
) -> list[object]:
"""Return fleet-ranked tasks without reserving or claiming anything."""
now = float(self.now_fn() if now is None else now)
worker_id = _text(worker_id)
if not worker_id:
return []
ranked: list[object] = []
for task in self._tasks(actor_context):
eligible, _, _ = self._eligible_worker(worker_id, task, now)
if eligible:
ranked.append(task)
return sorted(ranked, key=lambda task: self._task_score(task, now))
def reserve_for_worker(
self,
worker_id: str,
task: object,
*,
now: float | None = None,
) -> FleetDispatchDecision:
"""Reserve exactly one already-ranked task, rechecking eligibility and capacity."""
now = float(self.now_fn() if now is None else now)
worker_id = _text(worker_id)
eligible, _, policy = self._eligible_worker(worker_id, task, now)
if not eligible:
raise FleetConflict("FLEET_WORKER_NOT_ELIGIBLE")
metadata = _task_metadata(task)
scope_weight = _float(metadata.get("fleet_scope_weight"), 1.0, 0.05, 100.0)
return self.state.reserve(
task_id=_task_id(task),
worker_id=worker_id,
task_revision=_task_revision(task),
scope_key=_task_scope_key(task),
failure_domain=policy.failure_domain,
score=self._task_score(task, now),
reservation_seconds=self.reservation_seconds,
now=now,
scope_weight=scope_weight,
)
def select(
self,
worker_ids: Sequence[str],
*,
actor_context: Any = None,
limit: int = 1,
now: float | None = None,
) -> list[FleetDispatchDecision]:
now = float(self.now_fn() if now is None else now)
limit = _int(limit, 1, 1, 100)
worker_ids = tuple(dict.fromkeys(_text(item) for item in worker_ids if _text(item)))
tasks = sorted(self._tasks(actor_context), key=lambda task: self._task_score(task, now))
decisions: list[FleetDispatchDecision] = []
local_load: dict[str, int] = {}
for task in tasks:
if len(decisions) >= limit:
break
candidates: list[tuple[int, float, str, FleetWorkerPolicy]] = []
for worker_id in worker_ids:
eligible, load, policy = self._eligible_worker(worker_id, task, now)
if not eligible:
continue
effective_load = local_load.get(worker_id, load)
if effective_load >= policy.max_slots:
continue
candidates.append(
(
effective_load,
-policy.weight,
worker_id,
policy,
)
)
if not candidates:
continue
candidates.sort(key=lambda item: (item[0], item[1], item[2]))
_, _, worker_id, policy = candidates[0]
score = self._task_score(task, now)
metadata = _task_metadata(task)
scope_weight = _float(metadata.get("fleet_scope_weight"), 1.0, 0.05, 100.0)
try:
decision = self.state.reserve(
task_id=_task_id(task),
worker_id=worker_id,
task_revision=_task_revision(task),
scope_key=_task_scope_key(task),
failure_domain=policy.failure_domain,
score=score,
reservation_seconds=self.reservation_seconds,
now=now,
scope_weight=scope_weight,
)
except FleetConflict:
continue
local_load[worker_id] = local_load.get(
worker_id,
self._active_claims(worker_id, now)
+ self.state.reserved_slots(worker_id, now)
- 1,
) + 1
decisions.append(decision)
return decisions
+27
View File
@@ -13,6 +13,8 @@ from dataclasses import dataclass, field
from datetime import datetime, timezone
from typing import Any
from .scope import normalize_task_scope
SENSITIVE_KEYS = {
"secret",
@@ -66,6 +68,12 @@ TASK_PLAN_KEYS = {
"allowed_input_paths",
"allowed_output_paths",
"output_policy",
"scope_type",
"tenant_id",
"user_id",
"project_id",
"conversation_id",
"run_id",
}
@@ -237,6 +245,12 @@ class TaskPlan:
schema_version: int = 1
task_id: str | None = None
task_type: str = "code.change"
scope_type: str = "legacy"
tenant_id: str | None = None
user_id: str | None = None
project_id: str | None = None
conversation_id: str | None = None
run_id: str | None = None
issuer: dict[str, Any] = field(default_factory=dict)
intent: str | None = None
required_capabilities: list[str] = field(default_factory=list)
@@ -311,10 +325,17 @@ class TaskPlan:
raise ValueError("credential-secret requires metadata-only logging")
if output_policy != "hash-count-status-only":
raise ValueError("credential-secret requires restricted output")
scope = normalize_task_scope(data)
return cls(
schema_version=schema_version,
task_id=validate_identifier(data["task_id"], "task_id"),
task_type=validate_type(data.get("task_type", "code.change"), "task_type"),
scope_type=scope.scope_type,
tenant_id=scope.tenant_id,
user_id=scope.user_id,
project_id=scope.project_id,
conversation_id=scope.conversation_id,
run_id=scope.run_id,
issuer=issuer,
intent=data.get("intent"),
required_capabilities=_normalize_list(data.get("required_capabilities", [])),
@@ -348,6 +369,12 @@ class TaskPlan:
"schema_version": self.schema_version,
"task_id": self.task_id,
"task_type": self.task_type,
"scope_type": self.scope_type,
"tenant_id": self.tenant_id,
"user_id": self.user_id,
"project_id": self.project_id,
"conversation_id": self.conversation_id,
"run_id": self.run_id,
"issuer": self.issuer,
"intent": self.intent,
"required_capabilities": list(self.required_capabilities),
+20 -1
View File
@@ -3,10 +3,11 @@
from __future__ import annotations
from dataclasses import dataclass
from typing import Callable
from typing import Any, Callable, Mapping
from .errors import ForbiddenError, LockedError
from .models import ProbeEvidence, ResultEnvelope, TaskRecord
from .scope import TaskActorContext, actor_context as build_actor_context
RISK_RANK = {"low": 1, "medium": 2, "high": 3}
@@ -93,6 +94,10 @@ class PolicyEngine:
result_signature_verifier: Callable[[ResultEnvelope], bool] | None = None,
probe_attestation_verifier: Callable[[ProbeEvidence], bool] | None = None,
identity_capabilities: dict[str, set[str]] | None = None,
identity_actor_contexts: Mapping[
str, TaskActorContext | Mapping[str, Any]
]
| None = None,
admin_identities: set[str] | None = None,
readonly_worker_execution_enabled: bool = False,
readonly_worker_profiles: list[ReadonlyWorkerProfile] | None = None,
@@ -109,12 +114,26 @@ class PolicyEngine:
identity: set(capabilities)
for identity, capabilities in (identity_capabilities or {}).items()
}
self.identity_actor_contexts = dict(identity_actor_contexts or {})
self.admin_identities = set(admin_identities or ())
self.readonly_worker_execution_enabled = readonly_worker_execution_enabled
self.readonly_worker_profiles = {
profile.identity: profile for profile in (readonly_worker_profiles or [])
}
def actor_context(self, identity: str) -> TaskActorContext:
configured = self.identity_actor_contexts.get(
identity, self.identity_actor_contexts.get("*")
)
capabilities = set(
self.identity_capabilities.get(
identity, self.identity_capabilities.get("*", set())
)
)
if identity in self.admin_identities:
capabilities.add("task.admin")
return build_actor_context(identity, configured, capabilities)
def _readonly_worker_allows(
self, action: str, actor: str, task: TaskRecord | None
) -> bool:
+200
View File
@@ -74,3 +74,203 @@ class TaskScheduler:
operation_digest=task.task_plan.operation_digest(),
)
return claim
# R4.2 P2 durable multi-agent fleet scheduling public surface.
from .fleet import (
FleetConflict,
FleetDispatchDecision,
FleetError,
FleetScheduler,
FleetStateStore,
FleetWorkerPolicy,
)
# R4.2 P2-B fleet-aware pull scheduling.
#
# The original TaskScheduler remains the security oracle and claim-CAS implementation.
# Fleet ordering only controls the order presented to that oracle.
import inspect as _fleet_inspect
import threading as _fleet_threading
from collections.abc import Mapping as _FleetMapping
from .worker_registry import WorkerRegistry as _FleetWorkerRegistry
from .errors import NotFoundError as _FleetNotFoundError
_LegacyTaskScheduler = TaskScheduler
_LegacyScheduleParameters = _fleet_inspect.signature(
_LegacyTaskScheduler.schedule_for_worker
).parameters
_LegacyNextParameters = _fleet_inspect.signature(
_LegacyTaskScheduler.next_queued_task
).parameters
class _FleetOrderedStoreProxy:
def __init__(self, delegate, tasks):
self._delegate = delegate
self._tasks = tuple(tasks)
def list_tasks(self, *args, **kwargs):
return list(self._tasks)
def __getattr__(self, name):
return getattr(self._delegate, name)
def _fleet_claim_field(claim, name, default=None):
if isinstance(claim, _FleetMapping):
return claim.get(name, default)
return getattr(claim, name, default)
class TaskScheduler(_LegacyTaskScheduler):
"""Fleet-integrated pull scheduler preserving legacy security and claim semantics."""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
store = getattr(self, "store", args[0] if args else None)
if store is None:
raise FleetError("FLEET_SCHEDULER_STORE_REQUIRED")
registry = getattr(self, "workers", None)
if registry is None:
registry = _FleetWorkerRegistry(store)
self.fleet_scheduler = FleetScheduler(store, registry)
self._fleet_local = _fleet_threading.local()
def _fleet_legacy_next(self, worker_id, tasks, *args, **kwargs):
proxy = _FleetOrderedStoreProxy(self.store, tasks)
legacy = _LegacyTaskScheduler(proxy)
legacy_kwargs = dict(kwargs)
if "actor_context" not in _LegacyNextParameters:
legacy_kwargs.pop("actor_context", None)
return legacy.next_queued_task(worker_id, *args, **legacy_kwargs)
def next_queued_task(self, worker_id, *args, **kwargs):
actor_context = kwargs.get(
"actor_context", getattr(self._fleet_local, "actor_context", None)
)
# Preserve legacy fail-closed worker checks before fleet ordering.
probe_kwargs = dict(kwargs)
if "actor_context" not in _LegacyNextParameters:
probe_kwargs.pop("actor_context", None)
legacy_probe = _LegacyTaskScheduler(self.store).next_queued_task(
worker_id, *args, **probe_kwargs
)
if legacy_probe is None:
return None
ranked = self.fleet_scheduler.rank_for_worker(
worker_id, actor_context=actor_context
)
# Preserve FleetScheduler fairness order. The legacy scheduler is used
# only as a fail-closed security oracle: present one fleet-ranked task
# at a time so its internal sort cannot reorder across scopes.
# Keep the legacy priority dimension ahead of Fleet fairness for
# compatibility with existing TaskPlane scheduling contracts; the
# Fleet order remains authoritative among equal-priority tasks.
def _legacy_priority(task):
plan = getattr(task, "task_plan", task)
metadata = getattr(plan, "metadata", {}) or {}
if not isinstance(metadata, _FleetMapping):
return 0
try:
return int(metadata.get("priority", 0))
except (TypeError, ValueError):
return 0
ranked = sorted(ranked, key=lambda task: -_legacy_priority(task))
for ranked_candidate in ranked:
try:
candidate = self._fleet_legacy_next(
worker_id, [ranked_candidate], *args, **kwargs
)
except _FleetNotFoundError:
continue
if candidate is None:
continue
ranked_id = getattr(
ranked_candidate,
"task_id",
getattr(getattr(ranked_candidate, "task_plan", None), "task_id", None),
)
candidate_id = getattr(
candidate,
"task_id",
getattr(getattr(candidate, "task_plan", None), "task_id", None),
)
if candidate_id != ranked_id:
raise FleetConflict("FLEET_LEGACY_ORACLE_ORDER_VIOLATION")
try:
decision = self.fleet_scheduler.reserve_for_worker(
worker_id, candidate
)
except FleetConflict:
continue
self._fleet_local.pending = decision
return candidate
return None
def schedule_for_worker(self, worker_id, *args, **kwargs):
actor_context = kwargs.get("actor_context")
legacy_kwargs = dict(kwargs)
if "actor_context" not in _LegacyScheduleParameters:
legacy_kwargs.pop("actor_context", None)
self._fleet_local.actor_context = actor_context
self._fleet_local.pending = None
try:
claim = _LegacyTaskScheduler.schedule_for_worker(
self, worker_id, *args, **legacy_kwargs
)
decision = getattr(self._fleet_local, "pending", None)
if claim is None:
if decision is not None:
self.fleet_scheduler.state.release(
decision.task_id, state="RELEASED"
)
return None
if decision is not None:
claim_id = _fleet_claim_field(claim, "claim_id")
claim_task_id = _fleet_claim_field(claim, "task_id", decision.task_id)
if not claim_id:
self.fleet_scheduler.state.release(
decision.task_id, state="RELEASED"
)
raise FleetError("FLEET_CLAIM_ID_REQUIRED")
if claim_task_id != decision.task_id:
self.fleet_scheduler.state.release(
decision.task_id, state="RELEASED"
)
raise FleetConflict("FLEET_CLAIM_TASK_MISMATCH")
self.fleet_scheduler.state.bind_claim(
decision.reservation_id, str(claim_id)
)
return claim
except Exception:
decision = getattr(self._fleet_local, "pending", None)
if decision is not None:
self.fleet_scheduler.state.release(
decision.task_id, state="RELEASED"
)
raise
finally:
self._fleet_local.pending = None
self._fleet_local.actor_context = None
def record_worker_failure(self, worker_id, failure_class, **kwargs):
return self.fleet_scheduler.state.record_worker_failure(
worker_id, failure_class, **kwargs
)
def record_worker_success(self, worker_id, **kwargs):
return self.fleet_scheduler.state.record_worker_success(
worker_id, **kwargs
)
def quarantine_task_worker(
self, task_id, worker_id, failure_class, **kwargs
):
return self.fleet_scheduler.state.record_task_worker_failure(
task_id, worker_id, failure_class, **kwargs
)
+181
View File
@@ -0,0 +1,181 @@
"""Trusted TaskPlane scope model and SQL authorization helpers."""
from __future__ import annotations
from dataclasses import dataclass, replace
from typing import Any, Mapping
from .errors import ForbiddenError, ValidationError
SCOPE_FIELDS = ("tenant_id", "user_id", "project_id", "conversation_id", "run_id")
PRIVILEGED_CAPABILITIES = frozenset(
{"*", "admin", "system.admin", "task.admin", "task.read.all"}
)
def _value(value: Any, field: str) -> Any:
if hasattr(value, "task_plan"):
value = value.task_plan
if isinstance(value, Mapping):
return value.get(field)
return getattr(value, field, None)
def _text(value: Any) -> str | None:
if value is None:
return None
result = str(value).strip()
return result or None
@dataclass(frozen=True)
class TaskScope:
scope_type: str
tenant_id: str | None = None
user_id: str | None = None
project_id: str | None = None
conversation_id: str | None = None
run_id: str | None = None
@dataclass(frozen=True)
class TaskActorContext:
identity: str
scope_type: str = "legacy"
tenant_id: str | None = None
user_id: str | None = None
project_id: str | None = None
conversation_id: str | None = None
run_id: str | None = None
capabilities: frozenset[str] = frozenset()
def normalize_task_scope(value: Any) -> TaskScope:
fields = {field: _text(_value(value, field)) for field in SCOPE_FIELDS}
declared = _text(_value(value, "scope_type"))
if declared is None:
declared = "tenant" if fields["tenant_id"] else "legacy"
if declared == "tenant":
if fields["tenant_id"] is None:
raise ValidationError(
"tenant scope requires tenant_id", code="TASK_SCOPE_TENANT_ID_REQUIRED"
)
elif declared == "legacy":
if any(fields.values()):
raise ValidationError(
"legacy scope cannot contain scoped ownership",
code="TASK_SCOPE_LEGACY_WITH_SCOPED_FIELDS",
)
elif declared == "system":
if any(fields[field] for field in ("tenant_id", "user_id", "project_id", "conversation_id")):
raise ValidationError(
"system scope cannot contain tenant ownership",
code="TASK_SCOPE_SYSTEM_WITH_TENANT_FIELDS",
)
else:
raise ValidationError("invalid task scope type", code="TASK_SCOPE_TYPE_INVALID")
return TaskScope(declared, **fields)
def actor_is_privileged(actor: TaskActorContext) -> bool:
return bool(actor.capabilities & PRIVILEGED_CAPABILITIES)
def actor_context(
identity: str,
configured: TaskActorContext | Mapping[str, Any] | None,
capabilities: set[str] | frozenset[str],
) -> TaskActorContext:
if isinstance(configured, TaskActorContext):
context = configured
else:
data = dict(configured or {})
data.pop("identity", None)
data.pop("capabilities", None)
context = TaskActorContext(identity=str(identity), **data)
scope = normalize_task_scope(context)
return replace(
context,
identity=str(identity),
scope_type=scope.scope_type,
tenant_id=scope.tenant_id,
user_id=scope.user_id,
project_id=scope.project_id,
conversation_id=scope.conversation_id,
run_id=scope.run_id,
capabilities=frozenset(str(item) for item in capabilities),
)
def bind_task_scope(plan: Any, actor: TaskActorContext, *, public_create: bool) -> Any:
requested = normalize_task_scope(plan)
if actor_is_privileged(actor):
trusted = normalize_task_scope(actor)
if requested.scope_type == "legacy" and trusted.scope_type != "legacy":
plan.scope_type = trusted.scope_type
for field in SCOPE_FIELDS:
setattr(plan, field, getattr(actor, field))
return plan
trusted = normalize_task_scope(actor)
if public_create and trusted.scope_type == "legacy":
raise ForbiddenError(
"authenticated identity has no tenant scope",
code="TASK_ACTOR_SCOPE_UNAVAILABLE",
)
if trusted.scope_type != "tenant":
if requested.scope_type == "legacy":
plan.scope_type = trusted.scope_type
for field in SCOPE_FIELDS:
setattr(plan, field, getattr(actor, field))
return plan
if requested.scope_type != trusted.scope_type:
raise ForbiddenError("task scope mismatch", code="TASK_SCOPE_MISMATCH_DENY")
return plan
if requested.scope_type not in {"legacy", "tenant"}:
raise ForbiddenError("task scope mismatch", code="TASK_SCOPE_MISMATCH_DENY")
updates: dict[str, Any] = {"scope_type": "tenant"}
for field in SCOPE_FIELDS:
declared = getattr(requested, field)
authoritative = getattr(actor, field)
if declared is not None and declared != authoritative:
raise ForbiddenError(
"task scope mismatch",
code="TASK_SCOPE_MISMATCH_DENY",
details={"field": field},
)
updates[field] = authoritative
for field, item in updates.items():
setattr(plan, field, item)
return plan
def actor_scope_sql(actor: TaskActorContext | None) -> tuple[str, tuple[Any, ...]]:
if actor is None or actor_is_privileged(actor):
return "", ()
scope = normalize_task_scope(actor)
if scope.scope_type == "legacy":
return "scope_type = 'legacy'", ()
if scope.scope_type == "system":
return "scope_type = 'system'", ()
clauses = ["scope_type = 'tenant'", "tenant_id = ?"]
params: list[Any] = [scope.tenant_id]
for field in ("user_id", "project_id", "conversation_id", "run_id"):
value = getattr(scope, field)
if value is None:
clauses.append(f"{field} IS NULL")
else:
clauses.append(f"({field} IS NULL OR {field} = ?)")
params.append(value)
return " AND ".join(clauses), tuple(params)
def task_scope_values(value: Any) -> tuple[str | None, ...]:
scope = normalize_task_scope(value)
return (
scope.scope_type,
scope.tenant_id,
scope.user_id,
scope.project_id,
scope.conversation_id,
scope.run_id,
)
+95 -11
View File
@@ -32,6 +32,7 @@ from .models import (
redact_value,
validate_identifier,
)
from .scope import TaskActorContext, actor_scope_sql, task_scope_values
EventEnvelope = tuple[str, str, str, str, str, float]
@@ -134,6 +135,12 @@ class TaskPlaneStore:
status TEXT NOT NULL,
task_json TEXT NOT NULL,
task_type TEXT NOT NULL,
scope_type TEXT NOT NULL DEFAULT 'legacy',
tenant_id TEXT,
user_id TEXT,
project_id TEXT,
conversation_id TEXT,
run_id TEXT,
root_task_id TEXT NOT NULL,
parent_task_id TEXT,
correlation_id TEXT,
@@ -272,6 +279,21 @@ class TaskPlaneStore:
)
"""
)
self._ensure_columns(
"tasks",
{
"scope_type": "TEXT NOT NULL DEFAULT 'legacy'",
"tenant_id": "TEXT",
"user_id": "TEXT",
"project_id": "TEXT",
"conversation_id": "TEXT",
"run_id": "TEXT",
},
)
self.conn.execute(
"UPDATE tasks SET scope_type='legacy' "
"WHERE scope_type IS NULL OR TRIM(scope_type)=''"
)
self._ensure_columns(
"workers",
{
@@ -299,6 +321,14 @@ class TaskPlaneStore:
)
self.conn.execute("CREATE INDEX IF NOT EXISTS ix_tasks_status ON tasks(status)")
self.conn.execute("CREATE INDEX IF NOT EXISTS ix_tasks_parent ON tasks(parent_task_id)")
self.conn.execute(
"CREATE INDEX IF NOT EXISTS ix_tasks_scope_tenant_project "
"ON tasks(scope_type, tenant_id, project_id)"
)
self.conn.execute(
"CREATE INDEX IF NOT EXISTS ix_tasks_scope_owner "
"ON tasks(scope_type, tenant_id, user_id, conversation_id, run_id)"
)
self.conn.execute("CREATE INDEX IF NOT EXISTS ix_workers_status ON workers(status)")
self.conn.execute("CREATE INDEX IF NOT EXISTS ix_claims_task ON claims(task_id)")
self.conn.execute(
@@ -347,6 +377,42 @@ class TaskPlaneStore:
raise NotFoundError("task")
task_json = json.loads(row["task_json"])
task_plan = TaskPlan.from_dict(task_json)
stored_scope = tuple(
row[field]
for field in (
"scope_type",
"tenant_id",
"user_id",
"project_id",
"conversation_id",
"run_id",
)
)
json_has_scope = any(
field in task_json
for field in (
"scope_type",
"tenant_id",
"user_id",
"project_id",
"conversation_id",
"run_id",
)
)
if json_has_scope and stored_scope != task_scope_values(task_plan):
raise ConflictError(
"task scope persistence mismatch",
code="TASK_SCOPE_PERSISTENCE_MISMATCH",
)
if not json_has_scope:
(
task_plan.scope_type,
task_plan.tenant_id,
task_plan.user_id,
task_plan.project_id,
task_plan.conversation_id,
task_plan.run_id,
) = stored_scope
task_plan.parent_task_id = row["parent_task_id"]
task_plan.correlation_id = row["correlation_id"]
return TaskRecord(
@@ -389,6 +455,7 @@ class TaskPlaneStore:
task_id = task_plan.task_id or f"task-{uuid.uuid4()}"
task_plan.task_id = task_id
task_json = redact_value(task_plan.to_dict(redact=False))
scope_values = task_scope_values(task_plan)
now = _now()
with self._transaction():
if idempotency_key:
@@ -397,7 +464,11 @@ class TaskPlaneStore:
(idempotency_key,),
).fetchone()
if existing:
if _canonical(json.loads(existing["task_json"])) != _canonical(task_json):
existing_plan = TaskPlan.from_dict(
json.loads(existing["task_json"])
)
existing_json = redact_value(existing_plan.to_dict(redact=False))
if _canonical(existing_json) != _canonical(task_json):
raise ConflictError("idempotency key payload mismatch")
return self.get_task(existing["task_id"])
root_task_id = task_id
@@ -412,14 +483,18 @@ class TaskPlaneStore:
self.conn.execute(
"""
INSERT INTO tasks
(task_id, revision, status, task_json, task_type, root_task_id, parent_task_id, correlation_id, idempotency_key, created_at, updated_at, failure_reason)
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL)
(task_id, revision, status, task_json, task_type,
scope_type, tenant_id, user_id, project_id, conversation_id, run_id,
root_task_id, parent_task_id, correlation_id, idempotency_key,
created_at, updated_at, failure_reason)
VALUES (?, 1, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, NULL)
""",
(
task_id,
"CREATED",
json.dumps(task_json, ensure_ascii=False),
task_plan.task_type,
*scope_values,
root_task_id,
task_plan.parent_task_id,
task_plan.correlation_id,
@@ -451,21 +526,30 @@ class TaskPlaneStore:
)
return self.get_task(task.task_id)
def get_task(self, task_id: str) -> TaskRecord:
def get_task(
self, task_id: str, actor_context: TaskActorContext | None = None
) -> TaskRecord:
task_id = _validated_id(task_id, "task_id")
predicate, scope_params = actor_scope_sql(actor_context)
sql = "SELECT * FROM tasks WHERE task_id=?"
if predicate:
sql += f" AND ({predicate})"
with self._lock:
row = self.conn.execute(
"SELECT * FROM tasks WHERE task_id=?", (task_id,)
).fetchone()
row = self.conn.execute(sql, (task_id, *scope_params)).fetchone()
if row is None:
raise NotFoundError(f"task {task_id} not found")
return self._row_to_task(row)
def list_tasks(self) -> list[TaskRecord]:
def list_tasks(
self, actor_context: TaskActorContext | None = None
) -> list[TaskRecord]:
predicate, scope_params = actor_scope_sql(actor_context)
sql = "SELECT * FROM tasks"
if predicate:
sql += f" WHERE ({predicate})"
sql += " ORDER BY created_at DESC"
with self._lock:
rows = self.conn.execute(
"SELECT * FROM tasks ORDER BY created_at DESC"
).fetchall()
rows = self.conn.execute(sql, scope_params).fetchall()
return [self._row_to_task(row) for row in rows]
def get_task_budget(self, task_id: str) -> Budget:
@@ -0,0 +1,759 @@
"""Scope-bound governance for traces, Tool/MCP calls, and credentials.
This module deliberately stores references and audit metadata only. Secret
material remains in the configured credential provider boundary.
"""
from __future__ import annotations
from dataclasses import asdict, dataclass, field
from datetime import datetime, timezone
from contextlib import contextmanager
import hashlib
import json
import re
import sqlite3
from pathlib import Path
from typing import Any, Iterable, Mapping
SCOPE_FIELDS = ("tenant_id", "user_id", "project_id", "conversation_id", "run_id")
PRIVILEGED_CAPABILITIES = frozenset({"*", "admin", "system.admin", "governance.admin"})
REFERENCE_RE = re.compile(r"^(systemd-credential|vault|secret-manager):[A-Za-z0-9][A-Za-z0-9._:/#-]{0,255}$")
IDENTIFIER_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,179}$")
class GovernanceError(RuntimeError):
def __init__(self, code: str, *, field: str | None = None) -> None:
super().__init__(code if field is None else f"{code}:{field}")
self.code = code
self.field = field
def _clean(value: Any) -> str | None:
if value is None:
return None
text = str(value).strip()
return text or None
def _scope_tuple(value: Iterable[str] | str | None) -> tuple[str, ...]:
if value is None:
return ()
if isinstance(value, str):
value = value.split(",")
return tuple(sorted({_clean(item) for item in value if _clean(item)}))
def _canonical(value: Any) -> str:
return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False, default=str)
def _sha(value: Any) -> str:
return hashlib.sha256(_canonical(value).encode("utf-8")).hexdigest()
def _now() -> str:
return datetime.now(timezone.utc).isoformat()
def _identifier(value: Any, code: str) -> str:
text = _clean(value)
if text is None or IDENTIFIER_RE.fullmatch(text) is None:
raise GovernanceError(code)
return text
@dataclass(frozen=True)
class GovernanceScope:
"""Explicit resource scope; system/global resources never get fake tenants."""
scope_type: str
tenant_id: str | None = None
user_id: str | None = None
project_id: str | None = None
conversation_id: str | None = None
run_id: str | None = None
def __post_init__(self) -> None:
object.__setattr__(self, "scope_type", (_clean(self.scope_type) or "").lower())
for name in SCOPE_FIELDS:
object.__setattr__(self, name, _clean(getattr(self, name)))
if self.scope_type == "tenant":
if self.tenant_id is None:
raise GovernanceError("GOVERNANCE_TENANT_ID_REQUIRED")
elif self.scope_type in {"system", "global"}:
if any(getattr(self, name) is not None for name in SCOPE_FIELDS):
raise GovernanceError("GOVERNANCE_NON_TENANT_SCOPE_INVALID")
elif self.scope_type == "legacy":
if any(getattr(self, name) is not None for name in SCOPE_FIELDS):
raise GovernanceError("GOVERNANCE_LEGACY_SCOPE_INVALID")
else:
raise GovernanceError("GOVERNANCE_SCOPE_TYPE_INVALID")
@classmethod
def from_value(cls, value: Any) -> "GovernanceScope":
def get(name: str) -> Any:
return value.get(name) if isinstance(value, Mapping) else getattr(value, name, None)
declared = _clean(get("scope_type"))
tenant_id = _clean(get("tenant_id"))
return cls(
scope_type=declared or ("tenant" if tenant_id else "legacy"),
tenant_id=tenant_id,
user_id=get("user_id"),
project_id=get("project_id"),
conversation_id=get("conversation_id"),
run_id=get("run_id"),
)
def as_dict(self) -> dict[str, Any]:
return asdict(self)
@dataclass(frozen=True)
class GovernanceActor:
identity: str
scope: GovernanceScope
capabilities: tuple[str, ...] = ()
def __post_init__(self) -> None:
object.__setattr__(self, "identity", _identifier(self.identity, "GOVERNANCE_ACTOR_INVALID"))
object.__setattr__(self, "capabilities", _scope_tuple(self.capabilities))
@classmethod
def from_value(cls, value: Any) -> "GovernanceActor":
if isinstance(value, cls):
return value
if not isinstance(value, Mapping):
raise GovernanceError("GOVERNANCE_ACTOR_INVALID")
return cls(
identity=value.get("identity") or value.get("actor_id") or value.get("user_id"),
scope=GovernanceScope.from_value(value),
capabilities=_scope_tuple(value.get("capabilities") or value.get("capability_scope")),
)
@property
def privileged(self) -> bool:
return bool(set(self.capabilities) & PRIVILEGED_CAPABILITIES)
def authorize_scope(actor: GovernanceActor | Mapping[str, Any], resource: GovernanceScope | Mapping[str, Any]) -> None:
actor = GovernanceActor.from_value(actor)
resource = resource if isinstance(resource, GovernanceScope) else GovernanceScope.from_value(resource)
if actor.privileged:
return
if resource.scope_type in {"system", "global"}:
raise GovernanceError("GOVERNANCE_SYSTEM_SCOPE_AUTHZ_DENY")
if resource.scope_type == "legacy":
if actor.scope.scope_type != "legacy":
raise GovernanceError("GOVERNANCE_LEGACY_SCOPE_AUTHZ_DENY")
return
if actor.scope.scope_type != "tenant":
raise GovernanceError("GOVERNANCE_SCOPE_MISMATCH_DENY", field="scope_type")
for name in SCOPE_FIELDS:
expected = getattr(resource, name)
actual = getattr(actor.scope, name)
if expected is not None and expected != actual:
raise GovernanceError("GOVERNANCE_SCOPE_MISMATCH_DENY", field=name)
@dataclass(frozen=True)
class TraceContext:
trace_id: str
span_id: str
scope: GovernanceScope
parent_span_id: str | None = None
correlation_id: str | None = None
causation_id: str | None = None
task_id: str | None = None
claim_id: str | None = None
message_id: str | None = None
def __post_init__(self) -> None:
object.__setattr__(self, "trace_id", _identifier(self.trace_id, "TRACE_ID_INVALID"))
object.__setattr__(self, "span_id", _identifier(self.span_id, "TRACE_SPAN_ID_INVALID"))
for name in ("parent_span_id", "correlation_id", "causation_id", "task_id", "claim_id", "message_id"):
value = _clean(getattr(self, name))
if value is not None and IDENTIFIER_RE.fullmatch(value) is None:
raise GovernanceError("TRACE_LINEAGE_IDENTIFIER_INVALID", field=name)
object.__setattr__(self, name, value)
def child(self, *, span_id: str, actor: GovernanceActor | Mapping[str, Any], **lineage: Any) -> "TraceContext":
authorize_scope(actor, self.scope)
for name in ("trace_id", "task_id", "claim_id", "message_id", "correlation_id"):
supplied = _clean(lineage.get(name))
expected = getattr(self, name)
if supplied is not None and expected is not None and supplied != expected:
raise GovernanceError("TRACE_LINEAGE_CONTINUITY", field=name)
requested_scope = lineage.pop("scope", self.scope)
requested_scope = requested_scope if isinstance(requested_scope, GovernanceScope) else GovernanceScope.from_value(requested_scope)
if requested_scope != self.scope:
raise GovernanceError("TRACE_SCOPE_MISMATCH_DENY")
return TraceContext(
trace_id=self.trace_id,
span_id=span_id,
parent_span_id=self.span_id,
correlation_id=self.correlation_id,
causation_id=_clean(lineage.get("causation_id")) or self.span_id,
task_id=self.task_id,
claim_id=self.claim_id,
message_id=self.message_id,
scope=self.scope,
)
def as_dict(self) -> dict[str, Any]:
return {
"trace_id": self.trace_id,
"span_id": self.span_id,
"parent_span_id": self.parent_span_id,
"correlation_id": self.correlation_id,
"causation_id": self.causation_id,
"task_id": self.task_id,
"claim_id": self.claim_id,
"message_id": self.message_id,
**self.scope.as_dict(),
}
def bind_trace_lineage(
*,
actor: GovernanceActor | Mapping[str, Any],
task: Any,
message: Any,
trace_id: str,
span_id: str,
parent: TraceContext | None = None,
) -> TraceContext:
actor = GovernanceActor.from_value(actor)
task_scope = GovernanceScope.from_value(task)
message_scope = GovernanceScope.from_value(message)
if task_scope != message_scope:
raise GovernanceError("TRACE_SCOPE_MISMATCH_DENY")
authorize_scope(actor, task_scope)
def get(value: Any, name: str) -> Any:
return value.get(name) if isinstance(value, Mapping) else getattr(value, name, None)
task_id = _clean(get(task, "task_id"))
message_task = _clean(get(message, "task_id"))
if task_id and message_task and task_id != message_task:
raise GovernanceError("TRACE_LINEAGE_CONTINUITY", field="task_id")
context = TraceContext(
trace_id=trace_id,
span_id=span_id,
parent_span_id=parent.span_id if parent else None,
correlation_id=_clean(get(task, "correlation_id")) or _clean(get(message, "correlation_id")),
causation_id=_clean(get(message, "causation_id")) or (parent.span_id if parent else None),
task_id=task_id or message_task,
claim_id=_clean(get(task, "claim_id")) or _clean(get(message, "claim_id")),
message_id=_clean(get(message, "envelope_id")) or _clean(get(message, "id")),
scope=task_scope,
)
if parent is not None:
if parent.trace_id != context.trace_id or parent.scope != context.scope:
raise GovernanceError("TRACE_LINEAGE_CONTINUITY")
if parent.task_id and context.task_id and parent.task_id != context.task_id:
raise GovernanceError("TRACE_LINEAGE_CONTINUITY", field="task_id")
return context
@dataclass(frozen=True)
class ToolDescriptor:
name: str
kind: str
scope: GovernanceScope
required_capabilities: tuple[str, ...] = ()
credential_ref: str | None = None
def __post_init__(self) -> None:
object.__setattr__(self, "name", _identifier(self.name, "TOOL_NAME_INVALID"))
object.__setattr__(self, "kind", (_clean(self.kind) or "").lower())
if self.kind not in {"tool", "mcp"}:
raise GovernanceError("TOOL_KIND_INVALID")
object.__setattr__(self, "required_capabilities", _scope_tuple(self.required_capabilities))
if self.credential_ref is not None and REFERENCE_RE.fullmatch(self.credential_ref) is None:
raise GovernanceError("CREDENTIAL_SECRET_EXPOSURE_DENY")
@dataclass(frozen=True)
class CredentialGrant:
grant_id: str
credential_ref: str
scope: GovernanceScope
operations: tuple[str, ...]
task_id: str | None = None
tool_name: str | None = None
expires_at: str | None = None
def __post_init__(self) -> None:
object.__setattr__(self, "grant_id", _identifier(self.grant_id, "CREDENTIAL_GRANT_ID_INVALID"))
if REFERENCE_RE.fullmatch(self.credential_ref or "") is None:
raise GovernanceError("CREDENTIAL_SECRET_EXPOSURE_DENY")
object.__setattr__(self, "operations", _scope_tuple(self.operations))
if not self.operations:
raise GovernanceError("CREDENTIAL_OPERATIONS_REQUIRED")
if self.expires_at is not None:
parsed = datetime.fromisoformat(self.expires_at.replace("Z", "+00:00"))
if parsed.tzinfo is None:
raise GovernanceError("CREDENTIAL_EXPIRY_INVALID")
def authorize(
self,
actor: GovernanceActor | Mapping[str, Any],
*,
operation: str,
task_id: str | None = None,
tool_name: str | None = None,
now: datetime | None = None,
) -> str:
try:
authorize_scope(actor, self.scope)
except GovernanceError as exc:
raise GovernanceError("CREDENTIAL_SCOPE_DENY", field=exc.field) from exc
if operation not in self.operations:
raise GovernanceError("CREDENTIAL_SCOPE_DENY", field="operation")
if self.task_id is not None and self.task_id != _clean(task_id):
raise GovernanceError("CREDENTIAL_SCOPE_DENY", field="task_id")
if self.tool_name is not None and self.tool_name != _clean(tool_name):
raise GovernanceError("CREDENTIAL_SCOPE_DENY", field="tool_name")
if self.expires_at is not None:
current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc)
expiry = datetime.fromisoformat(self.expires_at.replace("Z", "+00:00")).astimezone(timezone.utc)
if current >= expiry:
raise GovernanceError("CREDENTIAL_SCOPE_DENY", field="expires_at")
return self.credential_ref
def public(self) -> dict[str, Any]:
return {
"grant_id": self.grant_id,
"credential_ref": self.credential_ref,
"operations": list(self.operations),
"task_id": self.task_id,
"tool_name": self.tool_name,
"expires_at": self.expires_at,
**self.scope.as_dict(),
}
_SCHEMA = """
CREATE TABLE IF NOT EXISTS governance_schema_metadata (
singleton INTEGER PRIMARY KEY CHECK(singleton=1),
schema_version INTEGER NOT NULL
);
CREATE TABLE IF NOT EXISTS governed_tools (
name TEXT PRIMARY KEY,
kind TEXT NOT NULL,
scope_json TEXT NOT NULL,
required_capabilities_json TEXT NOT NULL,
credential_ref TEXT,
descriptor_sha256 TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS governed_trace_spans (
trace_id TEXT NOT NULL,
span_id TEXT NOT NULL,
parent_span_id TEXT,
context_json TEXT NOT NULL,
context_sha256 TEXT NOT NULL,
created_at TEXT NOT NULL,
PRIMARY KEY(trace_id, span_id)
);
CREATE TABLE IF NOT EXISTS governed_credential_grants (
grant_id TEXT PRIMARY KEY,
grant_json TEXT NOT NULL,
grant_sha256 TEXT NOT NULL,
created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS governed_tool_invocations (
invocation_key TEXT PRIMARY KEY,
request_sha256 TEXT NOT NULL,
tool_name TEXT NOT NULL,
task_id TEXT NOT NULL,
claim_id TEXT NOT NULL,
trace_id TEXT NOT NULL,
span_id TEXT NOT NULL,
status TEXT NOT NULL,
receipt_sha256 TEXT,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS governance_audit_evidence (
sequence INTEGER PRIMARY KEY AUTOINCREMENT,
evidence_id TEXT NOT NULL UNIQUE,
action TEXT NOT NULL,
actor_identity TEXT NOT NULL,
trace_id TEXT,
span_id TEXT,
resource_id TEXT NOT NULL,
outcome TEXT NOT NULL,
details_json TEXT NOT NULL,
previous_sha256 TEXT NOT NULL,
evidence_sha256 TEXT NOT NULL,
created_at TEXT NOT NULL
);
"""
class GovernanceStore:
def __init__(self, path: str | Path) -> None:
self.path = Path(path)
self.path.parent.mkdir(parents=True, exist_ok=True)
with self._connection() as connection:
has_metadata = connection.execute(
"SELECT 1 FROM sqlite_master WHERE type='table' AND name='governance_schema_metadata'"
).fetchone()
if has_metadata:
row = connection.execute(
"SELECT schema_version FROM governance_schema_metadata WHERE singleton=1"
).fetchone()
if row is not None and int(row[0]) != 1:
raise GovernanceError("GOVERNANCE_SCHEMA_VERSION_UNSUPPORTED")
connection.executescript(_SCHEMA)
row = connection.execute(
"SELECT schema_version FROM governance_schema_metadata WHERE singleton=1"
).fetchone()
if row is None:
connection.execute(
"INSERT INTO governance_schema_metadata(singleton,schema_version) VALUES(1,1)"
)
connection.commit()
def _connect(self) -> sqlite3.Connection:
connection = sqlite3.connect(self.path, timeout=30.0)
connection.row_factory = sqlite3.Row
connection.execute("PRAGMA foreign_keys=ON")
connection.execute("PRAGMA busy_timeout=30000")
return connection
@contextmanager
def _connection(self):
connection = self._connect()
try:
yield connection
connection.commit()
except Exception:
connection.rollback()
raise
finally:
connection.close()
def _audit(
self,
connection: sqlite3.Connection,
*,
action: str,
actor: GovernanceActor,
resource_id: str,
outcome: str,
trace: TraceContext | None,
details: Mapping[str, Any],
) -> dict[str, Any]:
previous = connection.execute(
"SELECT evidence_sha256 FROM governance_audit_evidence ORDER BY sequence DESC LIMIT 1"
).fetchone()
previous_sha = str(previous[0]) if previous else ""
created_at = _now()
material = {
"action": action,
"actor_identity": actor.identity,
"trace_id": trace.trace_id if trace else None,
"span_id": trace.span_id if trace else None,
"resource_id": resource_id,
"outcome": outcome,
"details": dict(details),
"previous_sha256": previous_sha,
"created_at": created_at,
}
digest = _sha(material)
evidence_id = "gev-" + digest[:32]
connection.execute(
"""INSERT INTO governance_audit_evidence
(evidence_id,action,actor_identity,trace_id,span_id,resource_id,outcome,
details_json,previous_sha256,evidence_sha256,created_at)
VALUES (?,?,?,?,?,?,?,?,?,?,?)""",
(
evidence_id, action, actor.identity, material["trace_id"], material["span_id"],
resource_id, outcome, _canonical(material["details"]), previous_sha, digest, created_at,
),
)
return {"evidence_id": evidence_id, "evidence_sha256": digest, **material}
def record_trace(self, trace: TraceContext, actor: GovernanceActor | Mapping[str, Any]) -> dict[str, Any]:
actor = GovernanceActor.from_value(actor)
authorize_scope(actor, trace.scope)
context = trace.as_dict()
digest = _sha(context)
with self._connection() as connection:
existing = connection.execute(
"SELECT context_sha256 FROM governed_trace_spans WHERE trace_id=? AND span_id=?",
(trace.trace_id, trace.span_id),
).fetchone()
if existing is not None and existing[0] != digest:
raise GovernanceError("TRACE_LINEAGE_CONFLICT")
if existing is not None:
return {"context": context, "context_sha256": digest, "idempotent": True}
if trace.parent_span_id:
parent = connection.execute(
"SELECT context_json FROM governed_trace_spans WHERE trace_id=? AND span_id=?",
(trace.trace_id, trace.parent_span_id),
).fetchone()
if parent is None:
raise GovernanceError("TRACE_LINEAGE_CONTINUITY", field="parent_span_id")
parent_context = json.loads(parent[0])
if GovernanceScope.from_value(parent_context) != trace.scope:
raise GovernanceError("TRACE_SCOPE_MISMATCH_DENY")
connection.execute(
"INSERT OR IGNORE INTO governed_trace_spans VALUES (?,?,?,?,?,?)",
(trace.trace_id, trace.span_id, trace.parent_span_id, _canonical(context), digest, _now()),
)
evidence = self._audit(
connection, action="TRACE_RECORDED", actor=actor,
resource_id=f"{trace.trace_id}:{trace.span_id}", outcome="ALLOW", trace=trace,
details={"context_sha256": digest},
)
return {"context": context, "context_sha256": digest, "evidence": evidence}
def register_tool(self, descriptor: ToolDescriptor, actor: GovernanceActor | Mapping[str, Any]) -> dict[str, Any]:
actor = GovernanceActor.from_value(actor)
if not actor.privileged:
raise GovernanceError("TOOL_MCP_AUTHZ_DENY")
public = {
"name": descriptor.name,
"kind": descriptor.kind,
"required_capabilities": list(descriptor.required_capabilities),
"credential_ref": descriptor.credential_ref,
**descriptor.scope.as_dict(),
}
digest = _sha(public)
with self._connection() as connection:
existing = connection.execute(
"SELECT descriptor_sha256 FROM governed_tools WHERE name=?", (descriptor.name,)
).fetchone()
if existing is not None and existing[0] != digest:
raise GovernanceError("TOOL_DESCRIPTOR_CONFLICT")
if existing is not None:
return {"descriptor": public, "descriptor_sha256": digest, "idempotent": True}
connection.execute(
"INSERT OR IGNORE INTO governed_tools VALUES (?,?,?,?,?,?)",
(descriptor.name, descriptor.kind, _canonical(descriptor.scope.as_dict()),
_canonical(descriptor.required_capabilities), descriptor.credential_ref, digest),
)
evidence = self._audit(
connection, action="TOOL_REGISTERED", actor=actor, resource_id=descriptor.name,
outcome="ALLOW", trace=None, details={"descriptor_sha256": digest},
)
return {"descriptor": public, "descriptor_sha256": digest, "evidence": evidence}
def authorize_tool(
self,
*,
actor: GovernanceActor | Mapping[str, Any],
tool_name: str,
trace: TraceContext,
requested_scope: GovernanceScope | Mapping[str, Any],
input_digest: str,
invocation_key: str,
) -> dict[str, Any]:
actor = GovernanceActor.from_value(actor)
invocation_key = _identifier(invocation_key, "TOOL_INVOCATION_KEY_INVALID")
if not re.fullmatch(r"[0-9a-f]{64}", input_digest):
raise GovernanceError("TOOL_INPUT_DIGEST_INVALID")
if trace.task_id is None or trace.claim_id is None:
raise GovernanceError("TOOL_MCP_AUTHZ_DENY", field="active_claim")
requested_scope = requested_scope if isinstance(requested_scope, GovernanceScope) else GovernanceScope.from_value(requested_scope)
if requested_scope != trace.scope:
raise GovernanceError("TOOL_SCOPE_PROPAGATION")
authorize_scope(actor, requested_scope)
with self._connection() as connection:
persisted_trace = connection.execute(
"SELECT context_sha256 FROM governed_trace_spans WHERE trace_id=? AND span_id=?",
(trace.trace_id, trace.span_id),
).fetchone()
if persisted_trace is None or persisted_trace[0] != _sha(trace.as_dict()):
raise GovernanceError("TOOL_MCP_AUTHZ_DENY", field="trace")
row = connection.execute("SELECT * FROM governed_tools WHERE name=?", (tool_name,)).fetchone()
if row is None:
raise GovernanceError("TOOL_MCP_AUTHZ_DENY")
descriptor_scope = GovernanceScope.from_value(json.loads(row["scope_json"]))
if descriptor_scope.scope_type == "tenant" and descriptor_scope != requested_scope:
raise GovernanceError("TOOL_MCP_AUTHZ_DENY")
required = set(json.loads(row["required_capabilities_json"]))
if not required.issubset(set(actor.capabilities)):
raise GovernanceError("TOOL_MCP_AUTHZ_DENY")
request_sha = _sha({
"tool_name": tool_name, "input_digest": input_digest,
"task_id": trace.task_id, "claim_id": trace.claim_id,
"trace_id": trace.trace_id, "span_id": trace.span_id,
"scope": requested_scope.as_dict(),
})
replay = connection.execute(
"SELECT * FROM governed_tool_invocations WHERE invocation_key=?",
(invocation_key,),
).fetchone()
if replay is not None:
if replay["request_sha256"] != request_sha:
raise GovernanceError("TOOL_INVOCATION_IDEMPOTENCY_CONFLICT")
if replay["status"] != "VERIFIED":
raise GovernanceError("TOOL_INVOCATION_RECONCILIATION_REQUIRED")
return {
"tool_name": tool_name, "trace_id": trace.trace_id,
"span_id": trace.span_id, "invocation_key": invocation_key,
"status": "VERIFIED", "receipt_sha256": replay["receipt_sha256"],
"idempotent": True,
}
created_at = _now()
connection.execute(
"""INSERT INTO governed_tool_invocations
(invocation_key,request_sha256,tool_name,task_id,claim_id,trace_id,span_id,
status,receipt_sha256,created_at,updated_at)
VALUES(?,?,?,?,?,?,?,?,?,?,?)""",
(invocation_key, request_sha, tool_name, trace.task_id, trace.claim_id,
trace.trace_id, trace.span_id, "AUTHORIZED", None, created_at, created_at),
)
evidence = self._audit(
connection, action="TOOL_AUTHORIZED", actor=actor, resource_id=tool_name,
outcome="ALLOW", trace=trace,
details={"input_sha256": input_digest, "scope_sha256": _sha(requested_scope.as_dict()),
"invocation_key": invocation_key},
)
return {
"tool_name": tool_name,
"trace_id": trace.trace_id,
"span_id": trace.span_id,
"invocation_key": invocation_key,
"status": "AUTHORIZED",
"credential_required": row["credential_ref"] is not None,
"evidence": evidence,
}
def record_tool_receipt(
self,
*,
invocation_key: str,
result_digest: str,
actor: GovernanceActor | Mapping[str, Any],
trace: TraceContext,
) -> dict[str, Any]:
actor = GovernanceActor.from_value(actor)
invocation_key = _identifier(invocation_key, "TOOL_INVOCATION_KEY_INVALID")
if not re.fullmatch(r"[0-9a-f]{64}", result_digest):
raise GovernanceError("TOOL_RECEIPT_DIGEST_INVALID")
authorize_scope(actor, trace.scope)
with self._connection() as connection:
row = connection.execute(
"SELECT * FROM governed_tool_invocations WHERE invocation_key=?",
(invocation_key,),
).fetchone()
if row is None or row["trace_id"] != trace.trace_id or row["span_id"] != trace.span_id:
raise GovernanceError("TOOL_RECEIPT_BINDING_INVALID")
if row["status"] == "VERIFIED":
if row["receipt_sha256"] != result_digest:
raise GovernanceError("TOOL_RECEIPT_CONFLICT")
return {"invocation_key": invocation_key, "status": "VERIFIED", "idempotent": True}
if row["status"] != "AUTHORIZED":
raise GovernanceError("TOOL_INVOCATION_RECONCILIATION_REQUIRED")
connection.execute(
"UPDATE governed_tool_invocations SET status='VERIFIED',receipt_sha256=?,updated_at=? "
"WHERE invocation_key=? AND status='AUTHORIZED'",
(result_digest, _now(), invocation_key),
)
evidence = self._audit(
connection, action="TOOL_RESULT_VERIFIED", actor=actor,
resource_id=row["tool_name"], outcome="ALLOW", trace=trace,
details={"invocation_key": invocation_key, "result_sha256": result_digest},
)
return {"invocation_key": invocation_key, "status": "VERIFIED", "evidence": evidence}
def store_grant(self, grant: CredentialGrant, actor: GovernanceActor | Mapping[str, Any]) -> dict[str, Any]:
actor = GovernanceActor.from_value(actor)
if not actor.privileged:
raise GovernanceError("CREDENTIAL_SCOPE_DENY")
public = grant.public()
digest = _sha(public)
with self._connection() as connection:
existing = connection.execute(
"SELECT grant_sha256 FROM governed_credential_grants WHERE grant_id=?", (grant.grant_id,)
).fetchone()
if existing is not None and existing[0] != digest:
raise GovernanceError("CREDENTIAL_GRANT_CONFLICT")
connection.execute(
"INSERT OR IGNORE INTO governed_credential_grants VALUES (?,?,?,?)",
(grant.grant_id, _canonical(public), digest, _now()),
)
evidence = self._audit(
connection, action="CREDENTIAL_GRANT_STORED", actor=actor,
resource_id=grant.grant_id, outcome="ALLOW", trace=None,
details={"grant_sha256": digest},
)
return {"grant": public, "grant_sha256": digest, "evidence": evidence}
def authorize_credential(
self,
*,
grant_id: str,
actor: GovernanceActor | Mapping[str, Any],
trace: TraceContext,
operation: str,
task_id: str | None,
tool_name: str | None,
now: datetime | None = None,
) -> dict[str, Any]:
actor = GovernanceActor.from_value(actor)
with self._connection() as connection:
row = connection.execute(
"SELECT grant_json FROM governed_credential_grants WHERE grant_id=?", (grant_id,)
).fetchone()
if row is None:
raise GovernanceError("CREDENTIAL_SCOPE_DENY")
raw = json.loads(row[0])
grant = CredentialGrant(
grant_id=raw["grant_id"], credential_ref=raw["credential_ref"],
scope=GovernanceScope.from_value(raw), operations=tuple(raw["operations"]),
task_id=raw.get("task_id"), tool_name=raw.get("tool_name"),
expires_at=raw.get("expires_at"),
)
if trace.scope != grant.scope:
raise GovernanceError("CREDENTIAL_SCOPE_DENY", field="trace_scope")
reference = grant.authorize(
actor, operation=operation, task_id=task_id, tool_name=tool_name, now=now
)
evidence = self._audit(
connection, action="CREDENTIAL_AUTHORIZED", actor=actor,
resource_id=grant_id, outcome="ALLOW", trace=trace,
details={"operation": operation, "task_id": task_id, "tool_name": tool_name,
"credential_ref_sha256": _sha(reference)},
)
return {"credential_ref": reference, "evidence": evidence}
def verify_audit_chain(self, required_actions: Iterable[str] = ()) -> bool:
ordered_actions: list[str] = []
previous = ""
with self._connection() as connection:
rows = connection.execute("SELECT * FROM governance_audit_evidence ORDER BY sequence").fetchall()
for row in rows:
details = json.loads(row["details_json"])
material = {
"action": row["action"], "actor_identity": row["actor_identity"],
"trace_id": row["trace_id"], "span_id": row["span_id"],
"resource_id": row["resource_id"], "outcome": row["outcome"],
"details": details, "previous_sha256": row["previous_sha256"],
"created_at": row["created_at"],
}
if row["previous_sha256"] != previous or row["evidence_sha256"] != _sha(material):
return False
previous = row["evidence_sha256"]
ordered_actions.append(row["action"])
position = 0
for required in required_actions:
try:
position = ordered_actions.index(required, position) + 1
except ValueError:
return False
return True
__all__ = [
"CredentialGrant", "GovernanceActor", "GovernanceError", "GovernanceScope",
"GovernanceStore", "ToolDescriptor", "TraceContext", "authorize_scope",
"bind_trace_lineage",
]
@@ -0,0 +1,9 @@
providers:
oneapi:
name: oneapi
base_url: http://172.18.0.2:3000/v1
key_env: OPENAI_API_KEY
default_model: deepseek-v4-flash
model:
provider: oneapi
name: deepseek-v4-flash
@@ -0,0 +1,13 @@
{
"argv": [
"/usr/local/bin/hermes",
"run",
"--non-interactive"
],
"input_mode": "stdin_text",
"max_output_bytes": 131072,
"output_mode": "stdout_text",
"runtime": "hermes",
"schema": "aios.external-agent-runtime.profile.v1",
"timeout_seconds": 900
}
@@ -0,0 +1,38 @@
{
"argv": [
"/opt/aios-hermes-agent/hermes-agent/.venv/bin/hermes",
"-z",
"{prompt}",
"--provider",
"oneapi",
"-m",
"{model_id}",
"--no-restore-cwd"
],
"credential_env": {
"OPENAI_API_KEY": {
"credential": "aios-external-agent-oneapi-env",
"format": "dotenv",
"key": "ONEAPI_REGULAR_TOKEN"
}
},
"environment": {
"HERMES_HOME": "/etc/aios/external-agent-runtime.d/hermes-home"
},
"input_mode": "stdin_text",
"max_output_bytes": 131072,
"model_governance": {
"max_output_tokens": 512,
"model_capabilities": [
"text"
],
"provider_ids": [
"oneapi"
],
"required": true
},
"output_mode": "stdout_text",
"runtime": "hermes",
"schema": "aios.external-agent-runtime.profile.v1",
"timeout_seconds": 180
}
@@ -0,0 +1,41 @@
{
"tools": {
"profile": "minimal"
},
"memory": {
"search": {
"enabled": false
}
},
"agents": {
"defaults": {
"workspace": "/var/lib/aios-external-agent-executor/openclaw-workspace",
"model": {
"primary": "openai/deepseek-v4-flash"
},
"thinkingDefault": "off"
},
"entries": {
"main": {
"default": true,
"workspace": "/var/lib/aios-external-agent-executor/openclaw-workspace"
}
}
},
"models": {
"providers": {
"openai": {
"baseUrl": "http://172.18.0.2:3000/v1",
"api": "openai-completions",
"models": [
{
"id": "deepseek-v4-flash",
"name": "deepseek-v4-flash",
"contextWindow": 32768,
"maxTokens": 256
}
]
}
}
}
}
@@ -0,0 +1,13 @@
{
"argv": [
"/usr/local/bin/openclaw",
"run",
"--non-interactive"
],
"input_mode": "stdin_text",
"max_output_bytes": 131072,
"output_mode": "stdout_text",
"runtime": "openclaw",
"schema": "aios.external-agent-runtime.profile.v1",
"timeout_seconds": 900
}
@@ -0,0 +1,48 @@
{
"argv": [
"/usr/bin/node",
"/opt/aios/openclaw-engineering/v2026.7.2-beta.7/openclaw.mjs",
"agent",
"--local",
"--json",
"--agent",
"main",
"--thinking",
"off",
"--message",
"{prompt}",
"--model",
"{model_id}",
"--timeout",
"180"
],
"credential_env": {
"OPENAI_API_KEY": {
"credential": "aios-external-agent-oneapi-env",
"format": "dotenv",
"key": "ONEAPI_REGULAR_TOKEN"
}
},
"environment": {
"OPENCLAW_CONFIG_PATH": "/etc/aios/external-agent-runtime.d/openclaw-config.json",
"OPENCLAW_HOME": "/var/lib/aios-external-agent-executor/openclaw-home",
"OPENCLAW_STATE_DIR": "/var/lib/aios-external-agent-executor/openclaw-state"
},
"input_mode": "stdin_text",
"max_output_bytes": 131072,
"model_governance": {
"input_reservation_floor_tokens": 16384,
"max_output_tokens": 256,
"model_capabilities": [
"text"
],
"provider_ids": [
"openai"
],
"required": true
},
"output_mode": "stdout_json",
"runtime": "openclaw",
"schema": "aios.external-agent-runtime.profile.v1",
"timeout_seconds": 240
}
@@ -0,0 +1,95 @@
{
"models": [
{
"capabilities": [
"agent.runtime.hermes",
"text"
],
"enabled": true,
"evaluation_sha256": "43ae0db1640daeb6d33e2787769732844f366d36070065d3a2239905bd9a823b",
"evaluation_state": "APPROVED",
"input_cost_micros_per_1k": 0,
"max_input_tokens": 32768,
"max_output_tokens": 512,
"model_id": "deepseek-v4-flash",
"output_cost_micros_per_1k": 0,
"priority": 10,
"provider_id": "oneapi",
"version": "profile-sha:54615e533bfdb6a2e8b870ae74bb8cd7fc45177495bcf6e8c45a50837cefb85a"
},
{
"capabilities": [
"agent.runtime.openclaw",
"text"
],
"enabled": true,
"evaluation_sha256": "7b1c89634eb9c0e7cb5a3d75323d904ebbd91b3ae5494ce938bd8316a4cf5b12",
"evaluation_state": "APPROVED",
"input_cost_micros_per_1k": 0,
"max_input_tokens": 32768,
"max_output_tokens": 256,
"model_id": "openai/deepseek-v4-flash",
"output_cost_micros_per_1k": 0,
"priority": 10,
"provider_id": "openai",
"version": "profile-sha:2a31c480226aa78d44ed9692bddcedc99c3aa03c23c0dd5f031a5cbbb8bbd7ba"
}
],
"policies": [
{
"allowed_models": [
"deepseek-v4-flash",
"openai/deepseek-v4-flash"
],
"daily_cost_budget_micros": 0,
"daily_token_budget": 250000,
"denied_providers": [],
"fallback_limit": 1,
"max_input_tokens_per_request": 16384,
"max_output_tokens_per_request": 512,
"project_id": null,
"required_capabilities": [
"text"
],
"scope_type": "legacy",
"tenant_id": null
},
{
"allowed_models": [
"deepseek-v4-flash",
"openai/deepseek-v4-flash"
],
"daily_cost_budget_micros": 0,
"daily_token_budget": 250000,
"denied_providers": [],
"fallback_limit": 1,
"max_input_tokens_per_request": 16384,
"max_output_tokens_per_request": 512,
"project_id": "staging",
"required_capabilities": [
"text"
],
"scope_type": "tenant",
"tenant_id": "model-governance-live"
}
],
"providers": [
{
"cooldown_seconds": 60,
"credential_ref": "runtime-profile:hermes",
"enabled": true,
"failure_domain": "oneapi",
"max_failures": 2,
"provider_id": "oneapi"
},
{
"cooldown_seconds": 60,
"credential_ref": "runtime-profile:openclaw",
"enabled": true,
"failure_domain": "oneapi",
"max_failures": 2,
"provider_id": "openai"
}
],
"schema": "aios.model-governance.bootstrap.v1"
}
@@ -0,0 +1,36 @@
-- R4.2 P0-B durable Message/Handoff Fabric.
-- Forward-only. The existing 20260707 message_outbox remains a compatibility source.
CREATE TABLE IF NOT EXISTS message_handoff (
envelope_id TEXT PRIMARY KEY,
stream_key TEXT NOT NULL,
source TEXT NOT NULL,
target TEXT NOT NULL,
message_type TEXT NOT NULL,
payload_json TEXT NOT NULL,
payload_sha256 TEXT NOT NULL,
tenant_id TEXT,
user_id TEXT,
project_id TEXT,
conversation_id TEXT,
task_id TEXT,
run_id TEXT,
causation_id TEXT,
sequence INTEGER NOT NULL CHECK(sequence >= 1),
capability_scope_json TEXT NOT NULL,
confidentiality_scope TEXT NOT NULL,
idempotency_key TEXT NOT NULL,
expires_at TEXT,
created_at TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'PENDING',
delivered_at TEXT,
acked_at TEXT,
UNIQUE(stream_key, idempotency_key),
UNIQUE(stream_key, sequence)
);
CREATE INDEX IF NOT EXISTS idx_message_handoff_target_status
ON message_handoff(target, status, created_at);
CREATE INDEX IF NOT EXISTS idx_message_handoff_scope
ON message_handoff(tenant_id, project_id, user_id, conversation_id, run_id);
@@ -0,0 +1,37 @@
[Unit]
Description=AIOS owner-controlled Task execution-rebind API
After=aios-agent-task-api.service
Wants=aios-agent-task-api.service
[Service]
Type=simple
User=aios-agent-task
Group=aios-agent-task
LoadCredential=agent_task_control_token:/etc/aios-gitops/agent-task-control.token
EnvironmentFile=/etc/aios-gitops/agent-task-execution-rebind.env
Environment=AIOS_AGENT_TASK_DB=/var/lib/aios-agent-task/state.sqlite3
Environment=AIOS_AGENT_TASK_CONTROL_TOKEN_FILE=%d/agent_task_control_token
Environment=AIOS_GOVERNED_WRITE_API_URL=http://127.0.0.1:18892
Environment=AIOS_TASK_EXECUTION_REBIND_HOST=127.0.0.1
Environment=AIOS_TASK_EXECUTION_REBIND_PORT=18942
ExecStart=/usr/bin/python3 -B /opt/aios/current/active-runtime/app/aios_agent_task_execution_rebind.py
Restart=on-failure
RestartSec=2
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
LockPersonality=true
RestrictSUIDSGID=true
RestrictRealtime=true
RestrictNamespaces=true
MemoryDenyWriteExecute=true
RestrictAddressFamilies=AF_UNIX AF_INET
ReadWritePaths=/var/lib/aios-agent-task
UMask=0077
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,37 @@
[Unit]
Description=AIOS owner-controlled Task initial execution projection API
After=aios-agent-task-api.service aios-chatgpt-bridge-v2-staging.service
Wants=aios-agent-task-api.service aios-chatgpt-bridge-v2-staging.service
[Service]
Type=simple
User=aios-agent-task
Group=aios-agent-task
LoadCredential=agent_task_control_token:/etc/aios-gitops/agent-task-control.token
Environment=AIOS_AGENT_TASK_DB=/var/lib/aios-agent-task/state.sqlite3
Environment=AIOS_AGENT_TASK_CONTROL_TOKEN_FILE=%d/agent_task_control_token
Environment=AIOS_GOVERNED_WRITE_API_URL=http://127.0.0.1:18892
Environment=AIOS_AGENT_TASK_INITIAL_EXECUTION_PROJECTION_AUTHORIZATION_ID=owner-initial-execution-projection-recovery-20260820
Environment=AIOS_TASK_INITIAL_EXECUTION_PROJECTION_HOST=127.0.0.1
Environment=AIOS_TASK_INITIAL_EXECUTION_PROJECTION_PORT=18943
ExecStart=/usr/bin/python3 -B /opt/aios/current/active-runtime/app/aios_agent_task_initial_execution_projection.py
Restart=on-failure
RestartSec=2
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
LockPersonality=true
RestrictSUIDSGID=true
RestrictRealtime=true
RestrictNamespaces=true
MemoryDenyWriteExecute=true
RestrictAddressFamilies=AF_UNIX AF_INET
ReadWritePaths=/var/lib/aios-agent-task
UMask=0077
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,36 @@
[Unit]
Description=AIOS owner-controlled Task retry-extension API
After=aios-agent-task-api.service
Wants=aios-agent-task-api.service
[Service]
Type=simple
User=aios-agent-task
Group=aios-agent-task
LoadCredential=agent_task_control_token:/etc/aios-gitops/agent-task-control.token
EnvironmentFile=/etc/aios-gitops/agent-task-retry-extension.env
Environment=AIOS_AGENT_TASK_DB=/var/lib/aios-agent-task/state.sqlite3
Environment=AIOS_AGENT_TASK_CONTROL_TOKEN_FILE=%d/agent_task_control_token
Environment=AIOS_TASK_RETRY_HOST=127.0.0.1
Environment=AIOS_TASK_RETRY_PORT=18941
ExecStart=/usr/bin/python3 -B /opt/aios/current/active-runtime/app/aios_agent_task_retry_extension.py
Restart=on-failure
RestartSec=2
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
LockPersonality=true
RestrictSUIDSGID=true
RestrictRealtime=true
RestrictNamespaces=true
MemoryDenyWriteExecute=true
RestrictAddressFamilies=AF_UNIX AF_INET
ReadWritePaths=/var/lib/aios-agent-task
UMask=0077
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,43 @@
[Unit]
Description=AIOS credential-isolated external Agent runtime executor
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=aios-engineering-executor
Group=aios-executor-exchange
ExecStart=/usr/bin/python3 -B /opt/aios/current/active-runtime/app/aios_external_agent_runtime.py --root /var/lib/aios-external-agent-executor --profile-root /etc/aios/external-agent-runtime.d
Restart=always
RestartSec=2
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
LockPersonality=true
RestrictSUIDSGID=true
RestrictRealtime=true
RestrictNamespaces=true
MemoryDenyWriteExecute=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
IPAddressDeny=127.0.0.0/8
IPAddressDeny=::1/128
IPAddressAllow=172.18.0.2/32
LoadCredential=aios-external-agent-oneapi-env:/data/agent/secrets/oneapi-model-ready.env
InaccessiblePaths=/etc/aios-gitops
InaccessiblePaths=/data/aios-gitops
InaccessiblePaths=/data/agent/secrets
ReadOnlyPaths=/etc/aios/external-agent-runtime.d
ReadOnlyPaths=/opt/aios-hermes-agent
ReadOnlyPaths=/opt/aios/openclaw-engineering
Environment=AIOS_MODEL_GOVERNANCE_DB=/var/lib/aios-model-governance/state.sqlite3
Environment=AIOS_MODEL_GOVERNANCE_ENABLED=1
ReadWritePaths=/var/lib/aios-external-agent-executor
ReadWritePaths=/var/lib/aios-model-governance
UMask=0027
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,41 @@
[Unit]
Description=AIOS disabled-by-default HTTPS edge for GPT read-only Online Task ingress
After=network-online.target aios-gpt-online-task-ingress.service
Wants=network-online.target aios-gpt-online-task-ingress.service
ConditionPathExists=/etc/aios-gitops/gpt-https-edge.approved
[Service]
Type=simple
DynamicUser=yes
LoadCredential=tls_cert:/etc/aios-gitops/gpt-https-edge-tls.crt
LoadCredential=tls_key:/etc/aios-gitops/gpt-https-edge-tls.key
Environment=AIOS_GPT_EDGE_TLS_CERT_FILE=%d/tls_cert
Environment=AIOS_GPT_EDGE_TLS_KEY_FILE=%d/tls_key
Environment=AIOS_GPT_EDGE_HOST=127.0.0.1
Environment=AIOS_GPT_EDGE_PORT=18443
Environment=AIOS_GPT_EDGE_PUBLIC_BIND_APPROVED=0
Environment=AIOS_GPT_EDGE_PUBLIC_DOMAIN=
ExecStart=/usr/bin/python3 -B /opt/aios/current/active-runtime/app/aios_gpt_https_edge.py
Restart=on-failure
RestartSec=2
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectKernelLogs=yes
ProtectClock=yes
LockPersonality=yes
RestrictSUIDSGID=yes
RestrictRealtime=yes
RestrictNamespaces=yes
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_INET AF_INET6
InaccessiblePaths=/data/aios-gitops /etc/aios-gitops /home /var/lib/aios-agent-task
UMask=0077
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,45 @@
[Unit]
Description=AIOS external GPT read-only Online Task ingress
After=network.target aios-chatgpt-bridge-v2-staging.service aios-agent-task-api.service
Wants=aios-chatgpt-bridge-v2-staging.service aios-agent-task-api.service
[Service]
Type=simple
DynamicUser=yes
StateDirectory=aios-gpt-online-task-ingress
StateDirectoryMode=0750
LoadCredential=gpt_ingress_token:/etc/aios-gitops/gpt-online-task-ingress.token
LoadCredential=agent_task_control_token:/etc/aios-gitops/agent-task-control.token
Environment=AIOS_GPT_INGRESS_TOKEN_FILE=%d/gpt_ingress_token
Environment=AIOS_AGENT_TASK_CONTROL_TOKEN_FILE=%d/agent_task_control_token
Environment=AIOS_GPT_INGRESS_STATE_ROOT=%S/aios-gpt-online-task-ingress
Environment=AIOS_GPT_INGRESS_HOST=127.0.0.1
Environment=AIOS_GPT_INGRESS_PORT=18945
Environment=AIOS_GOVERNED_WRITE_API_URL=http://127.0.0.1:18892
Environment=AIOS_AGENT_TASK_API_URL=http://127.0.0.1:18940
ExecStart=/usr/bin/python3 -B /opt/aios/current/active-runtime/app/aios_gpt_online_task_ingress.py
Restart=on-failure
RestartSec=2
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectKernelLogs=yes
ProtectClock=yes
LockPersonality=yes
RestrictSUIDSGID=yes
RestrictRealtime=yes
RestrictNamespaces=yes
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_UNIX AF_INET
IPAddressDeny=any
IPAddressAllow=127.0.0.0/8
InaccessiblePaths=/data/aios-gitops /etc/aios-gitops /home
UMask=0077
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,48 @@
[Unit]
Description=AIOS Hermes Online Agent Worker
After=aios-agent-task-api.service aios-chatgpt-bridge-v2-staging.service aios-external-agent-executor.service
Wants=aios-agent-task-api.service aios-chatgpt-bridge-v2-staging.service aios-external-agent-executor.service
[Service]
Type=simple
User=aios-engineering-worker
Group=aios-engineering-worker
SupplementaryGroups=gitops aios-executor-exchange
LoadCredentialEncrypted=agent_token:/etc/credstore.encrypted/aios-online-agent-worker-20260815/agent_token
Environment=PYTHONUNBUFFERED=1
Environment=AIOS_AGENT_TASK_API_URL=http://127.0.0.1:18940
Environment=AIOS_GOVERNED_WRITE_API_URL=http://127.0.0.1:18892
Environment=AIOS_AGENT_ID=online-hermes-worker-cn-network-mkt
Environment=AIOS_AGENT_TYPE=external-agent-runtime-worker
Environment=AIOS_AGENT_VERSION=2026-08-18.1-hermes-runtime
Environment=AIOS_WORKER_CAPABILITIES=agent.runtime.hermes
Environment=AIOS_WORKER_ALLOWED_ADAPTERS=hermes
Environment=AIOS_AUTHORITY_GIT_DIR=/data/aios-gitops/repo.git
Environment=AIOS_EXTERNAL_AGENT_EXECUTOR_ROOT=/var/lib/aios-external-agent-executor
Environment=AIOS_WORKER_STATE_ROOT=/var/lib/aios-online-agent-hermes
Environment=AIOS_WORKER_CHECKPOINT_DB=/var/lib/aios-online-agent-hermes/checkpoints.sqlite3
Environment=AIOS_WORKER_WORKSPACE_ROOT=/var/lib/aios-online-agent-hermes/workspaces
ExecStart=/usr/bin/python3 -B /opt/aios/current/active-runtime/app/aios_online_agent_worker.py
Restart=always
RestartSec=5
TimeoutStopSec=20
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
LockPersonality=true
RestrictSUIDSGID=true
RestrictRealtime=true
RestrictNamespaces=true
MemoryDenyWriteExecute=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
ReadOnlyPaths=/data/aios-gitops/repo.git
ReadWritePaths=/var/lib/aios-online-agent-hermes
ReadWritePaths=/var/lib/aios-external-agent-executor
UMask=0027
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,33 @@
[Unit]
Description=AIOS uncredentialed deterministic Planner/Executor
After=local-fs.target
[Service]
Type=simple
User=aios-engineering-executor
Group=aios-executor-exchange
ExecStart=/usr/bin/python3 -B /opt/aios/current/active-runtime/app/aios_isolated_executor.py --root /var/lib/aios-online-agent-executor
Restart=always
RestartSec=2
NoNewPrivileges=true
PrivateTmp=true
PrivateNetwork=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
LockPersonality=true
RestrictSUIDSGID=true
RestrictRealtime=true
RestrictNamespaces=true
MemoryDenyWriteExecute=true
RestrictAddressFamilies=AF_UNIX AF_INET
IPAddressDeny=any
IPAddressAllow=localhost
InaccessiblePaths=/run/credentials
ReadWritePaths=/var/lib/aios-online-agent-executor
UMask=0027
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,46 @@
[Unit]
Description=AIOS governed Online Agent Worker
After=aios-agent-task-api.service aios-chatgpt-bridge-v2-staging.service
Wants=aios-agent-task-api.service aios-chatgpt-bridge-v2-staging.service
[Service]
Type=simple
User=aios-engineering-worker
Group=aios-engineering-worker
SupplementaryGroups=gitops aios-executor-exchange
LoadCredentialEncrypted=agent_token:/etc/credstore.encrypted/aios-online-agent-worker-20260815/agent_token
Environment=PYTHONUNBUFFERED=1
Environment=AIOS_AGENT_TASK_API_URL=http://127.0.0.1:18940
Environment=AIOS_GOVERNED_WRITE_API_URL=http://127.0.0.1:18892
Environment=AIOS_AGENT_ID=online-engineering-worker
Environment=AIOS_AGENT_TYPE=trusted-online-engineering-worker
Environment=AIOS_AGENT_VERSION=2026-08-17.2-durable
Environment=AIOS_WORKER_CAPABILITIES=git.read,git.test,governed.write.branch,governed.write.file
Environment=AIOS_AUTHORITY_GIT_DIR=/data/aios-gitops/repo.git
Environment=AIOS_ISOLATED_EXECUTOR_ROOT=/var/lib/aios-online-agent-executor
Environment=AIOS_WORKER_STATE_ROOT=/var/lib/aios-online-agent-worker
Environment=AIOS_WORKER_CHECKPOINT_DB=/var/lib/aios-online-agent-worker/checkpoints.sqlite3
Environment=AIOS_WORKER_WORKSPACE_ROOT=/var/lib/aios-online-agent-worker/workspaces
ExecStart=/usr/bin/python3 -B /opt/aios/current/active-runtime/app/aios_online_agent_worker.py
Restart=always
RestartSec=5
TimeoutStopSec=20
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
LockPersonality=true
RestrictSUIDSGID=true
RestrictRealtime=true
RestrictNamespaces=true
MemoryDenyWriteExecute=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
ReadOnlyPaths=/data/aios-gitops/repo.git
ReadWritePaths=/var/lib/aios-online-agent-worker
UMask=0027
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,48 @@
[Unit]
Description=AIOS OpenClaw Online Agent Worker
After=aios-agent-task-api.service aios-chatgpt-bridge-v2-staging.service aios-external-agent-executor.service
Wants=aios-agent-task-api.service aios-chatgpt-bridge-v2-staging.service aios-external-agent-executor.service
[Service]
Type=simple
User=aios-engineering-worker
Group=aios-engineering-worker
SupplementaryGroups=gitops aios-executor-exchange
LoadCredentialEncrypted=agent_token:/etc/credstore.encrypted/aios-online-agent-worker-20260815/agent_token
Environment=PYTHONUNBUFFERED=1
Environment=AIOS_AGENT_TASK_API_URL=http://127.0.0.1:18940
Environment=AIOS_GOVERNED_WRITE_API_URL=http://127.0.0.1:18892
Environment=AIOS_AGENT_ID=online-openclaw-worker-cn-network-mkt
Environment=AIOS_AGENT_TYPE=external-agent-runtime-worker
Environment=AIOS_AGENT_VERSION=2026-08-18.1-openclaw-runtime
Environment=AIOS_WORKER_CAPABILITIES=agent.runtime.openclaw
Environment=AIOS_WORKER_ALLOWED_ADAPTERS=openclaw
Environment=AIOS_AUTHORITY_GIT_DIR=/data/aios-gitops/repo.git
Environment=AIOS_EXTERNAL_AGENT_EXECUTOR_ROOT=/var/lib/aios-external-agent-executor
Environment=AIOS_WORKER_STATE_ROOT=/var/lib/aios-online-agent-openclaw
Environment=AIOS_WORKER_CHECKPOINT_DB=/var/lib/aios-online-agent-openclaw/checkpoints.sqlite3
Environment=AIOS_WORKER_WORKSPACE_ROOT=/var/lib/aios-online-agent-openclaw/workspaces
ExecStart=/usr/bin/python3 -B /opt/aios/current/active-runtime/app/aios_online_agent_worker.py
Restart=always
RestartSec=5
TimeoutStopSec=20
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
LockPersonality=true
RestrictSUIDSGID=true
RestrictRealtime=true
RestrictNamespaces=true
MemoryDenyWriteExecute=true
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
ReadOnlyPaths=/data/aios-gitops/repo.git
ReadWritePaths=/var/lib/aios-online-agent-openclaw
ReadWritePaths=/var/lib/aios-external-agent-executor
UMask=0027
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,47 @@
[Unit]
Description=AIOS durable Owner authorization broker
After=network.target aios-chatgpt-bridge-v2-staging.service
Wants=aios-chatgpt-bridge-v2-staging.service
[Service]
Type=simple
User=root
Group=aios-executor-exchange
RuntimeDirectory=aios-owner-authorization-broker
RuntimeDirectoryMode=0750
StateDirectory=aios-owner-authorization-broker
StateDirectoryMode=0750
LoadCredential=agent_task_control_token:/etc/aios-gitops/agent-task-control.token
Environment=AIOS_AGENT_TASK_CONTROL_TOKEN_FILE=%d/agent_task_control_token
Environment=AIOS_OWNER_DURABLE_AUTHORIZATION_POLICY=/etc/aios-gitops/owner-durable-authorization.json
Environment=AIOS_OWNER_DURABLE_AUTHORIZATION_REVOCATIONS=/etc/aios-gitops/owner-durable-authorization-revocations.json
Environment=AIOS_OWNER_AUTH_BROKER_ACCOUNT=chatgpt-governed-implementer-singlehost
Environment=AIOS_OWNER_AUTH_BROKER_SOCKET=%t/aios-owner-authorization-broker/broker.sock
ExecStart=/usr/bin/python3 -B /opt/aios/current/active-runtime/app/aios_durable_owner_authorization.py
Restart=on-failure
RestartSec=2
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectKernelLogs=yes
ProtectClock=yes
LockPersonality=yes
RestrictSUIDSGID=yes
RestrictRealtime=yes
RestrictNamespaces=yes
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_UNIX AF_INET
IPAddressDeny=any
IPAddressAllow=127.0.0.0/8
ReadOnlyPaths=/etc/aios-gitops/owner-durable-authorization.json
ReadOnlyPaths=/etc/aios-gitops/owner-durable-authorization-revocations.json
InaccessiblePaths=/data/aios-gitops /home
UMask=0007
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,50 @@
[Unit]
Description=AIOS release-candidate Online Agent worker
After=network.target aios-agent-task-api.service
Wants=aios-agent-task-api.service
[Service]
Type=notify
NotifyAccess=main
WatchdogSec=90
User=root
Group=root
StateDirectory=aios-online-agent-release
StateDirectoryMode=0750
LoadCredentialEncrypted=agent_token:/etc/credstore.encrypted/aios-release-online-agent-worker/agent_token
LoadCredentialEncrypted=release_candidate_token:/etc/credstore.encrypted/aios-release-online-agent-worker/release_candidate_token
Environment=AIOS_RELEASE_WORKER_AGENT_ID=online-release-worker-cn-network-mkt-20260819
Environment=AIOS_RELEASE_WORKER_AGENT_VERSION=2026-08-19.1-release-candidate
Environment=AIOS_AGENT_TASK_API_URL=http://127.0.0.1:18940
Environment=AIOS_RELEASE_WORKER_STATE_ROOT=/var/lib/aios-online-agent-release
Environment=AIOS_WORKER_STATE_ROOT=/var/lib/aios-online-agent-release
Environment=AIOS_RELEASE_WORKER_POLL_SECONDS=3
Environment=AIOS_RELEASE_WORKER_LEASE_SECONDS=60
Environment=AIOS_RELEASE_WORKER_HEARTBEAT_SECONDS=15
Environment=AIOS_RELEASE_API_URL=http://127.0.0.1:18890
ExecStart=/usr/bin/python3 -B /opt/aios/current/active-runtime/app/aios_release_online_agent_worker.py
Restart=always
RestartSec=2
NoNewPrivileges=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ProtectKernelLogs=yes
ProtectClock=yes
LockPersonality=yes
RestrictSUIDSGID=yes
RestrictRealtime=yes
RestrictNamespaces=yes
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_UNIX AF_INET
IPAddressDeny=any
IPAddressAllow=127.0.0.0/8
InaccessiblePaths=/data/aios-gitops /home
UMask=0077
[Install]
WantedBy=multi-user.target
@@ -0,0 +1,27 @@
[Unit]
Description=AIOS persistent Online Agent worker availability guard
After=local-fs.target
[Service]
Type=oneshot
User=root
Group=root
Environment=PYTHONUNBUFFERED=1
Environment=AIOS_WORKER_AVAILABILITY_MAX_STALE_SECONDS=90
Environment=AIOS_WORKER_AVAILABILITY_RESTART_COOLDOWN_SECONDS=60
ExecStart=/usr/bin/python3 -B /opt/aios/current/active-runtime/app/aios_worker_availability_guard.py --heal
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectControlGroups=true
LockPersonality=true
RestrictSUIDSGID=true
RestrictRealtime=true
RestrictNamespaces=true
MemoryDenyWriteExecute=true
RestrictAddressFamilies=AF_UNIX
ReadWritePaths=/run/aios-worker-availability
UMask=0022
@@ -0,0 +1,13 @@
[Unit]
Description=Periodically ensure AIOS Online Agent workers remain available
[Timer]
OnBootSec=15s
OnUnitActiveSec=30s
AccuracySec=5s
RandomizedDelaySec=3s
Persistent=true
Unit=aios-worker-availability-guard.service
[Install]
WantedBy=timers.target
@@ -0,0 +1,4 @@
g aios-executor-exchange -
u aios-engineering-executor - "AIOS isolated engineering executor" /var/lib/aios-online-agent-executor /usr/sbin/nologin
m aios-engineering-executor aios-executor-exchange
m aios-engineering-worker aios-executor-exchange
@@ -0,0 +1,12 @@
d /var/lib/aios-external-agent-executor 2770 aios-engineering-executor aios-executor-exchange -
d /var/lib/aios-external-agent-executor/requests 2770 aios-engineering-worker aios-executor-exchange -
d /var/lib/aios-external-agent-executor/results 2770 aios-engineering-executor aios-executor-exchange -
d /var/lib/aios-external-agent-executor/failed 2770 aios-engineering-executor aios-executor-exchange -
d /var/lib/aios-external-agent-executor/openclaw-workspace 0750 aios-engineering-executor aios-executor-exchange -
d /var/lib/aios-external-agent-executor/openclaw-state 0750 aios-engineering-executor aios-executor-exchange -
d /var/lib/aios-external-agent-executor/openclaw-home 0750 aios-engineering-executor aios-executor-exchange -
d /var/lib/aios-online-agent-hermes 0750 aios-engineering-worker aios-engineering-worker -
d /var/lib/aios-online-agent-hermes/workspaces 0750 aios-engineering-worker aios-engineering-worker -
d /var/lib/aios-online-agent-openclaw 0750 aios-engineering-worker aios-engineering-worker -
d /var/lib/aios-online-agent-openclaw/workspaces 0750 aios-engineering-worker aios-engineering-worker -
d /etc/aios/external-agent-runtime.d 0755 root root -
@@ -0,0 +1 @@
d /var/lib/aios-model-governance 0750 aios-engineering-executor aios-executor-exchange -
@@ -0,0 +1,5 @@
d /var/lib/aios-online-agent-executor 2770 aios-engineering-executor aios-executor-exchange -
d /var/lib/aios-online-agent-executor/requests 2770 aios-engineering-worker aios-executor-exchange -
d /var/lib/aios-online-agent-executor/results 2770 aios-engineering-executor aios-executor-exchange -
d /var/lib/aios-online-agent-executor/failed 2770 aios-engineering-executor aios-executor-exchange -
d /var/lib/aios-online-agent-executor/sources 2770 aios-engineering-worker aios-executor-exchange -
@@ -0,0 +1 @@
d /run/aios-worker-availability 0755 root root -
+223
View File
@@ -240,6 +240,114 @@ components:
additionalProperties: false
properties:
candidate_id: {type: string}
BridgeWriterLeaseRecoveryRequest:
type: object
additionalProperties: false
required: [schema, task_id, repository, branch_mode, base_oid, expected_branch_oid, target_branch, generation, lease_seconds]
properties:
schema:
type: string
enum: [aios.task-writer-lease.v2, aios.task-writer-lease-request.v2]
task_id: {type: string, minLength: 1, maxLength: 256}
job_id: {type: string, pattern: '^job-[A-Za-z0-9][A-Za-z0-9._-]{7,255}$'}
repository: {type: string, const: aios/aios}
branch_mode: {type: string, enum: [CREATE_NEW, CONTINUE_EXISTING]}
base_oid: {type: string, pattern: '^[0-9a-f]{40}$'}
expected_branch_oid: {type: string, pattern: '^[0-9a-f]{40}$'}
target_branch: {type: string, pattern: '^agent/[A-Za-z0-9][A-Za-z0-9._/-]{2,179}$'}
generation: {type: integer, minimum: 1}
lease_seconds: {type: integer, minimum: 1, maximum: 86400}
BridgeWriterLeaseResponse:
type: object
additionalProperties: false
required: [schema, task_id, job_id, repository, branch_mode, base_oid, expected_branch_oid, target_branch, generation, lease_seconds, create_target_branch, worktree_mode, cas_ref_update_required]
properties:
schema: {type: string, enum: [aios.task-writer-lease.v2]}
task_id: {type: string}
job_id: {type: string, pattern: '^job-[A-Za-z0-9][A-Za-z0-9._-]{7,255}$'}
repository: {type: string, const: aios/aios}
branch_mode: {type: string, enum: [CREATE_NEW, CONTINUE_EXISTING]}
base_oid: {type: string, pattern: '^[0-9a-f]{40}$'}
expected_branch_oid: {type: string, pattern: '^[0-9a-f]{40}$'}
target_branch: {type: string, pattern: '^agent/[A-Za-z0-9][A-Za-z0-9._/-]{2,179}$'}
generation: {type: integer, minimum: 1}
lease_seconds: {type: integer, minimum: 1, maximum: 86400}
create_target_branch: {type: boolean}
worktree_mode: {type: string, enum: [detached_at_base_oid, detached_at_expected_head]}
cas_ref_update_required: {type: boolean, const: true}
lease_expires_at: {type: string, format: date-time}
BridgeWriterLeaseRecoveryResponse:
type: object
additionalProperties: false
required: [schema, task_id, job_id, repository, branch_mode, base_oid, expected_branch_oid, target_branch, generation, lease_seconds, lease_expires_at, create_target_branch, worktree_mode, cas_ref_update_required, recovered, bootstrapped, idempotent, runtime_rehydrated, durable_branch_lease_present]
properties:
schema: {type: string, enum: [aios.task-writer-lease.v2]}
task_id: {type: string}
job_id: {type: string, pattern: '^job-[A-Za-z0-9][A-Za-z0-9._-]{7,255}$'}
repository: {type: string, const: aios/aios}
branch_mode: {type: string, enum: [CREATE_NEW, CONTINUE_EXISTING]}
base_oid: {type: string, pattern: '^[0-9a-f]{40}$'}
expected_branch_oid: {type: string, pattern: '^[0-9a-f]{40}$'}
target_branch: {type: string, pattern: '^agent/[A-Za-z0-9][A-Za-z0-9._/-]{2,179}$'}
generation: {type: integer, minimum: 1}
lease_seconds: {type: integer, minimum: 1, maximum: 86400}
lease_expires_at: {type: string, format: date-time}
create_target_branch: {type: boolean, const: false}
worktree_mode: {type: string, enum: [detached_at_base_oid, detached_at_expected_head]}
cas_ref_update_required: {type: boolean, const: true}
recovered: {type: boolean}
bootstrapped: {type: boolean}
idempotent: {type: boolean}
runtime_rehydrated: {type: boolean}
durable_branch_lease_present: {type: boolean}
BridgeCommitFileRequest:
type: object
additionalProperties: false
required: [task_id, repository, expected_branch_oid, target_branch, generation, idempotency_key, path, content_base64, message]
properties:
task_id: {type: string, minLength: 1}
job_id: {type: string, pattern: '^job-[A-Za-z0-9][A-Za-z0-9._-]{7,255}$'}
repository: {type: string, const: aios/aios}
expected_branch_oid: {type: string, pattern: '^[0-9a-f]{40}$'}
target_branch: {type: string, pattern: '^agent/[A-Za-z0-9][A-Za-z0-9._/-]{2,179}$'}
generation: {type: integer, minimum: 1}
idempotency_key: {type: string, minLength: 1, maxLength: 256}
path: {type: string, minLength: 1, maxLength: 512}
content_base64: {type: string, format: byte}
message: {type: string, minLength: 1, maxLength: 200}
BridgeCommitFilesRequest:
type: object
additionalProperties: false
required: [task_id, repository, expected_branch_oid, target_branch, generation, idempotency_key, files, message]
properties:
task_id: {type: string, minLength: 1}
job_id: {type: string, pattern: '^job-[A-Za-z0-9][A-Za-z0-9._-]{7,255}$'}
repository: {type: string, const: aios/aios}
expected_branch_oid: {type: string, pattern: '^[0-9a-f]{40}$'}
target_branch: {type: string, pattern: '^agent/[A-Za-z0-9][A-Za-z0-9._/-]{2,179}$'}
generation: {type: integer, minimum: 1}
idempotency_key: {type: string, minLength: 1, maxLength: 256}
files: {type: object, additionalProperties: {type: string, format: byte}, minProperties: 1}
message: {type: string, minLength: 1, maxLength: 200}
BridgeCommitResponse:
type: object
additionalProperties: true
required: [job_id, commit_oid, commit_visible, branch_mode, target_branch, generation]
properties:
job_id: {type: string, pattern: '^job-[A-Za-z0-9][A-Za-z0-9._-]{7,255}$'}
commit_oid: {type: string, pattern: '^[0-9a-f]{40}$'}
commit_visible: {type: boolean}
branch_mode: {type: string, enum: [CONTINUE_EXISTING]}
target_branch: {type: string, pattern: '^agent/[A-Za-z0-9][A-Za-z0-9._/-]{2,179}$'}
generation: {type: integer, minimum: 1}
BridgeGovernedCapabilities:
type: object
required: [bridge_schema_version, governed_writes_enabled, operations, operation_contracts]
properties:
bridge_schema_version: {type: string, const: aios.chatgpt-gitops-bridge.v2}
governed_writes_enabled: {type: boolean, const: true}
operations: {type: array, items: {type: string}}
operation_contracts: {type: object, additionalProperties: true}
paths:
/createReleaseIntent:
post:
@@ -567,3 +675,118 @@ paths:
'200': {description: Safe GitOps plan}
'401': {description: Unauthorized}
x-openai-isConsequential: true
/v2/capabilities:
get:
operationId: getGovernedWriteCapabilities
summary: Read the governed Bridge v2 capability contract
responses:
'200':
description: Governed operations and their request/response contracts
content:
application/json:
schema: {$ref: '#/components/schemas/BridgeGovernedCapabilities'}
/v2/writer-lease:
post:
operationId: acquireWriterLease
summary: Acquire the exact governed writer lease for a task generation
security: [{bearerAuth: []}]
requestBody:
required: true
content:
application/json:
schema: {$ref: '#/components/schemas/BridgeWriterLeaseRecoveryRequest'}
responses:
'200':
description: Exact detached worktree lease binding
content:
application/json:
schema: {$ref: '#/components/schemas/BridgeWriterLeaseResponse'}
'400': {description: Invalid lease binding}
'409': {description: Branch, base, or generation conflict}
x-openai-isConsequential: true
/v2/writer-lease/bootstrap:
post:
operationId: bootstrapWriterLease
summary: Bootstrap a governed writer runtime without creating a durable lease
description: >
Staging-only runtime recovery. It validates the exact task, repository,
base OID, expected branch OID, target branch, and generation. It must
not create a branch, insert a branch lease, or consume a generation.
security: [{bearerAuth: []}]
requestBody:
required: true
content:
application/json:
schema: {$ref: '#/components/schemas/BridgeWriterLeaseRecoveryRequest'}
responses:
'200':
description: Runtime binding bootstrapped or returned idempotently
content:
application/json:
schema: {$ref: '#/components/schemas/BridgeWriterLeaseRecoveryResponse'}
'400': {description: Invalid lease binding}
'409': {description: A matching durable lease already exists or binding conflicts}
'503': {description: Runtime or durable state unavailable}
x-openai-isConsequential: true
/v2/writer-lease/renew:
post:
operationId: renewWriterLease
summary: Recover an expired governed writer lease without changing its binding
description: >
Renews only the current task generation and exact repository, base OID,
expected branch OID, and target branch. Durable renewal is revision-CAS
protected; runtime rehydration is reported separately.
security: [{bearerAuth: []}]
requestBody:
required: true
content:
application/json:
schema: {$ref: '#/components/schemas/BridgeWriterLeaseRecoveryRequest'}
responses:
'200':
description: Lease recovered or idempotently observed as already active
content:
application/json:
schema: {$ref: '#/components/schemas/BridgeWriterLeaseRecoveryResponse'}
'400': {description: Invalid schema, generation, or lease duration}
'409': {description: Task, base, repository, branch, generation, or durable lease conflict}
'503': {description: Durable job store or runtime unavailable}
x-openai-isConsequential: true
/v2/commit-file:
post:
operationId: commitFileGoverned
summary: Commit one file through the governed writer lease
security: [{bearerAuth: []}]
requestBody:
required: true
content:
application/json:
schema: {$ref: '#/components/schemas/BridgeCommitFileRequest'}
responses:
'200':
description: Authority and remote visibility receipt
content:
application/json:
schema: {$ref: '#/components/schemas/BridgeCommitResponse'}
'409': {description: Lease, expected-head, idempotency, or repository conflict}
'503': {description: Reconciliation required}
x-openai-isConsequential: true
/v2/commit-files:
post:
operationId: commitFilesGoverned
summary: Atomically commit a bounded set of files through the governed writer lease
security: [{bearerAuth: []}]
requestBody:
required: true
content:
application/json:
schema: {$ref: '#/components/schemas/BridgeCommitFilesRequest'}
responses:
'200':
description: Authority and remote visibility receipt
content:
application/json:
schema: {$ref: '#/components/schemas/BridgeCommitResponse'}
'409': {description: Lease, expected-head, idempotency, or repository conflict}
'503': {description: Reconciliation required}
x-openai-isConsequential: true
@@ -0,0 +1,54 @@
# R4.2 P0-A TaskPlane Scope Isolation Design
Status: approved candidate design. This document is not evidence of a protected-main merge or production deployment.
## Boundary
- Task: `r42-p0a-taskplane-scope-isolation-20260816`
- Governed Job: `job-3fd743b51971c6007923699c1233f84c`
- Repository: `aios/aios`
- Base: `0eb6e1a864edfdaeef37b438991707a110e21558`
- Target branch: `agent/r42-p0a-taskplane-scope-isolation-20260816`
- Generation: `1`
The implementation reuses this exact boundary. It does not create a replacement Task, Job, branch, generation, or idempotency key.
## Model and invariants
`TaskPlan` gains first-class `scope_type`, `tenant_id`, `user_id`, `project_id`, `conversation_id`, and `run_id` fields. The fields participate in accepted-key validation, parsing, serialization, persistence, and authorization.
Historical rows without scope data normalize to `scope_type=legacy`; no tenant is inferred. Tenant-scoped tasks require a non-empty `tenant_id`. System/global resources use an explicit `scope_type=system` invariant and do not receive a fabricated tenant. A task may add user, project, conversation, and run ownership within its tenant.
## Trusted ActorContext
TaskPlane defines a small immutable `TaskActorContext` used only for authorization. The HTTP authenticator returns an identity. `PolicyEngine` resolves that identity through server-owned policy configuration into the effective context and capabilities. Request fields such as `tenant_id`, `user_id`, `group_id`, and `role` are declarations only and never expand authority.
Public create, list, and get paths always use the resolved context. Internal compatibility calls may omit a context and retain explicit legacy behavior, but the HTTP adapter never uses that path.
## Persistence
SQLite remains the current runtime. `_create_schema()` performs an idempotent forward migration that adds the six scope columns and indexes, then backfills only missing `scope_type` values to `legacy`.
New writes persist scope in both the canonical `task_json` representation and first-class columns. Reads accept historical JSON without scope, using legacy columns, but reject a new-row mismatch between JSON and columns. SQL list/get predicates are parameterized and applied before row hydration.
PostgreSQL remains a separate migration target and is not claimed by this change.
## Authorization flow
1. HTTP bearer authentication yields an identity.
2. `PolicyEngine` verifies capability and resolves the identity to `TaskActorContext`.
3. Create binds the requested scope to the trusted context and rejects mismatches.
4. List/get pass the trusted context to `TaskPlaneStore`.
5. Store applies tenant and ownership predicates in SQL; privileged capabilities may explicitly cross scope.
Tenant actors cannot read legacy tasks, another tenant, or mismatched owned resources. Legacy actors can only read legacy tasks. System actors require explicit privileged capability for cross-scope access.
## Failure behavior
Missing tenant scope, forged scope, unknown scope types, JSON/column divergence, and unauthorized list/get operations fail closed with canonical TaskPlane errors. Public get does not reveal whether an out-of-scope task exists.
## Verification
Tests exercise real model round trips, SQLite migration/restart, Service/Policy/Store wiring, tenant A versus tenant B list/get denial, ownership filtering, forged create payload rejection, legacy isolation, privileged access, and SQL parameterization. Existing TaskPlane API, RBAC, recovery/security, and security-contract suites remain mandatory.
The candidate is complete only after real `TASK_CLAIMED`, `TASK_HEARTBEAT`, exact WriterLease, governed commit, OID convergence, result submission, and `TASK_VERIFIED`. Protected main and production remain unchanged.
@@ -0,0 +1,90 @@
# R4.2 P0-C Trace / Tool-MCP / Credential Governance
## Status boundary
This change is a candidate implementation on the governed P0-C branch. It does
not promote protected `main`, deploy Production, or assert that all R4.2 target
architecture is complete.
## Scope model
Governance resources use an explicit `scope_type`:
- `tenant` requires `tenant_id` and may narrow to user, project,
conversation, and run;
- `system` and `global` prohibit tenant ownership fields;
- `legacy` preserves unscoped historical records without inventing ownership.
System and global tools therefore never receive a synthetic tenant merely to
satisfy a tenant-oriented schema.
## Trace lineage
`TaskPlan` and `MessageEnvelope` scope is bound to a `TraceContext`. The trace
records `trace_id`, `span_id`, parent span, correlation, causation, task,
message, and the canonical P0-A scope. Child spans must keep the parent's trace,
task, and scope. Privileged visibility does not permit cross-scope trace
splicing.
`MessageEnvelope.correlation_id` is persisted and identity-bound. Existing
handoff databases are upgraded in place with an idempotent column migration;
legacy rows retain their prior event IDs because a null correlation field is
not added to the historical identity material.
Observability keeps its legacy presentation fields, but new governed records
also carry immutable trace and scope fields. Scoped records require a trusted
server actor for queries. A repeated execution ID is an idempotent replay only
when the complete record matches; conflicting overwrites are denied.
## Tool and MCP governance
`ToolDescriptor` binds a tool or MCP server to:
- an explicit resource scope;
- required actor capabilities;
- an optional credential reference.
Invocation authorization requires exact propagation of the trace scope and a
capability superset. Unknown tools, cross-tenant calls, scope substitution, and
non-privileged access to system/global resources fail closed. Every successful
registration and invocation appends hash-chained audit evidence.
## Credential governance
Credentials are represented only by references whose schemes are
`systemd-credential`, `vault`, or `secret-manager`. Raw bearer tokens, passwords,
and arbitrary values are rejected. Grants are bound to scope, operation, task,
tool, and expiry. Public/audit records contain a reference hash where disclosure
is unnecessary; secret material is never persisted by this module.
The existing runtime credential broker remains backward compatible for system
leases, adds explicit tenant/system/global invariants, and rejects providers
that attempt to return raw token/secret/password environment values instead of
file/reference indirection.
## Durable evidence
The SQLite governance store is restart-safe and idempotently initializes four
tables: governed tools, trace spans, credential grants, and audit evidence.
Audit entries form a SHA-256 chain. Acceptance requires the chain to verify and
contain the required actions; missing evidence is not treated as success.
## Acceptance tests
The candidate must pass:
- `TRACE_LINEAGE_CONTINUITY`;
- `TRACE_SCOPE_MISMATCH_DENY`;
- `TOOL_MCP_AUTHZ_DENY`;
- `TOOL_SCOPE_PROPAGATION`;
- `CREDENTIAL_SCOPE_DENY`;
- `CREDENTIAL_SECRET_EXPOSURE_DENY`;
- `AUDIT_EVIDENCE_COMPLETENESS`;
- P0-A scope and P0-B handoff regression suites;
- Python compilation, `git diff --check`, and a diff-scoped secret scan.
The repository-wide baseline currently has seven unrelated failures in the
isolated host environment (five stale continuation-contract fixtures and two
host-permission fixtures). The same seven failures reproduce at the untouched
`a642b045775b9154a00ae78543037fbef4fb447b` base; they are not suppressed or
modified by P0-C.
@@ -0,0 +1,75 @@
# R4.2 P1 — Durable Online Agent Execution
Status: candidate implementation. This document does not imply protected-main promotion or Production deployment.
## State model
The generic worker uses two local durable layers:
1. `tasks/<task_id>.json`: atomic fast-state cache for compatibility and quick restart.
2. `checkpoints.sqlite3`: append-only attempt/phase journal with `synchronous=FULL`.
Checkpoint identity is bound to:
- task_id
- attempt
- execution_digest
- job_id
- repository
- base_oid
- expected_branch_oid
- target_branch
- generation
A checkpoint from another repository/branch/generation is never reused.
Phases are monotonic within an attempt:
`claimed → workspace_prepared → verification_passed → source_capture_done|commit_done → result_pending → result_submitted`
`reconciliation_required` is an explicit recovery phase.
## Attempt-aware result idempotency
Result submission idempotency keys include the official Task attempt and the execution digest. The same
attempt reuses the same key after a timeout/restart; a later official attempt cannot collide with an
earlier result.
Before submitting a result the worker durably writes `result_pending`, including the result hash and
idempotency key. After a successful or exact-idempotency-reconciled API response it writes
`result_submitted`.
A 409 is accepted only if the Task API already contains the exact same result body and the Task is
VERIFYING/SUCCEEDED. A conflicting result remains a hard failure.
## Resume rules
- `source_capture_done` and `result_pending` may be reused across later official attempts only when the
execution/scope binding is identical. This avoids repeating read-only source capture.
- A deterministic task that crashed before governed commit replays from a new clean detached worktree.
This is safe because there is no external side effect yet.
- `commit_done` is never recommitted. Recovery requires exact Job receipt OID convergence plus exactly
one ACTIVE WriterLease whose `current_head_oid` equals the governed commit.
- delayed Git visibility remains `RECONCILIATION_REQUIRED`, preserving an explicit recovery entry.
## Crash cleanup
On worker startup, stale `task-*` workspaces under the configured workspace root are removed before
polling begins. The path boundary is verified before deletion.
## Contracts
P1 gates include:
- CHECKPOINT_DURABLE_WRITE
- RESUME_FROM_CHECKPOINT
- RETRY_IDEMPOTENT_NO_DUPLICATE_SIDE_EFFECT
- CRASH_RECOVERY_CONVERGENCE
- WORKER_RESTART_RESUME
- CHECKPOINT_SCOPE_ISOLATION
- RESULT_SUBMISSION_REPLAY_SAFE
- WORKSPACE_CLEANUP_RECOVERY
- LEASE_RENEWAL_RECOVERY
WriterLease and governed Job authority remain in the existing Bridge/GitOps durable stores; the worker
checkpoint DB is not a competing authority for Git state.
@@ -0,0 +1,18 @@
# R4.2 P2 Agent Fleet / Multi-Agent Scheduling / Fault Isolation
P2 adds a durable fleet decision layer inside TaskPlane without replacing TaskPlane claim/lease
ownership. Worker registration/heartbeat remains authoritative for liveness. TaskPlane claim CAS
remains authoritative for ownership. Probe-before-reassignment remains mandatory after an expired
lease. P1 checkpoint/result replay remains authoritative after worker restart.
`FleetScheduler` adds durable worker capacity slots, weighted scope fairness, starvation aging,
affinity/anti-affinity labels, worker circuit breakers, per-task/worker quarantine and idempotent
dispatch reservations. Fleet state is colocated in the existing TaskPlane SQLite database and is
created idempotently; no new daemon or database is introduced.
A fleet reservation is only a scheduling decision. It does not claim a Task. Callers must perform
the normal TaskPlane claim and may then bind the returned claim ID to the reservation. This preserves
the existing ownership and recovery invariants.
Failure recovery never bypasses probe evidence. When TaskPlane recovery places work back in a
claimable state, P2 may quarantine the failed worker and prefer another worker/failure domain.
@@ -0,0 +1,79 @@
# R4.2 P2 Compact Interface Map Runtime
## Purpose
The persistent Online Agent already supports the canonical read-only `source_capture` action, but
its `SourceCaptureExecutor` always returns an `aios.engineering.source-snapshot.v2` tar.gz+base64
payload. That is correct for implementation source transfer, but too large for control-plane
consumers that only need a bounded implementation map.
P2 adds a new **adapter**, not a new execution action:
```text
action=source_capture
adapter=compact_interface_map
```
This preserves the existing execution-v2 schema and Bridge policy:
- read-only repository;
- no change set;
- no verification commands;
- `writer_lease=false`;
- no branch or commit;
- existing capability and execution-digest validation.
## Compact map contract
`aios_online_agent_compact_map.py` scans only approved repository roots and source suffixes. It
returns:
- at most 40 relevant modules;
- at most 80 selected Python symbols in total;
- at most 40 test symbols;
- at most 25 excerpts;
- at most 1200 characters per excerpt;
- SQL table names;
- systemd unit names;
- current invariants, failure windows and reuse points;
- P2 fleet contract-to-surface mappings.
The result is hard-bounded to 120 KiB and contains no source archive or `payload_b64`.
Source excerpts apply a conservative sensitive-pattern redaction before result construction.
## P2 contracts
The map supplies evidence surfaces for:
- `FLEET_AGENT_REGISTRATION`
- `FLEET_CAPACITY_SCHEDULING`
- `FLEET_FAIRNESS`
- `FLEET_FAULT_ISOLATION`
- `FLEET_WORKER_REASSIGNMENT`
- `FLEET_SCOPE_ISOLATION`
- `FLEET_IDEMPOTENT_DISPATCH`
- `FLEET_FAILURE_RECOVERY`
Missing first-class scheduling/fleet primitives are reported as gaps rather than invented.
## Result
When the adapter is selected, the worker returns:
```text
status=SUCCEEDED
classification=P2_INTERFACE_MAP_CAPTURED
execution_mode=compact_interface_map
writer_lease=NONE
```
The existing `source_capture_done` / P1 checkpoint/retry path is reused, so the compact analysis
inherits attempt-aware result idempotency and restart-safe result submission.
## Governance
This capability is runtime infrastructure only. The implementation candidate may use a governed
branch/WriterLease for its own code change, but any task executed through `compact_interface_map`
is read-only and must not create a WriterLease, branch, commit, protected-ref write, or Production
side effect.
@@ -0,0 +1,17 @@
# R4.2 P2-B Fleet Order Preservation Repair
P2-C live acceptance exposed a real integration defect in the verified P2-B candidate. The fleet
scheduler correctly produced durable cross-scope fairness order, but the integration passed the
entire ranked list to the legacy scheduler. The legacy scheduler then applied its own priority /
created_at / task_id sort and silently discarded fleet ordering.
The repair preserves the legacy scheduler as the fail-closed security oracle while changing how it
is called: one fleet-ranked candidate is presented at a time. A candidate rejected by legacy
security/policy advances to the next fleet candidate. A candidate returned by the oracle must have
the same task ID as the candidate presented, otherwise execution fails closed.
Task ownership remains unchanged: the original schedule_for_worker/claim_task CAS remains
authoritative, and a fleet reservation is still only bound after the claim succeeds.
This repair does not modify FleetScheduler state semantics, capacity, WriterLease, checkpoint,
protected refs or Production behavior.
@@ -0,0 +1,23 @@
# R4.2 P2-B Fleet Runtime Integration
P2-B connects the verified P2 fleet primitives to the existing TaskPlane pull scheduler without
replacing its security or ownership logic.
The integration uses two stages. `FleetScheduler.rank_for_worker()` performs only read-only
capacity/fairness/fault-isolation ranking. The original `TaskScheduler.next_queued_task()` is then
run over that ordered list and remains the fail-closed security oracle for worker liveness,
capabilities, data classification, execution zone and other existing policy checks. Only a task
accepted by the original scheduler receives a short durable fleet reservation.
The original `TaskScheduler.schedule_for_worker()` remains the sole claim path and continues to call
the existing TaskPlane `claim_task` CAS. After a successful claim, the claim ID is bound to the
fleet reservation. A claim error releases the reservation. A fleet reservation is never treated as
task ownership.
P2-B also makes the reservation capacity check atomic inside the fleet SQLite transaction so two
concurrent pullers cannot oversubscribe the same worker slot. Circuit and task/worker quarantine
state are enforced on the real pull path. Explicit execution-failure signals can be recorded through
the integrated scheduler's failure/success/quarantine methods; policy/claim failures are not
automatically mislabeled as worker crashes.
No new daemon, database or production write path is introduced.
@@ -0,0 +1,123 @@
# External Agent Attempt-Scoped Request Identity
## Problem
External Agent execution previously derived its durable request ID from:
```text
task_id
job_id
execution_digest
runtime
```
The Online Task attempt was missing.
That is incorrect for retryable external side effects. A failed Model Governance route reservation is
durable and terminal for that request ID. Reusing the same external request ID on a later official
Task attempt therefore reuses the old route decision/reservation and can fail settlement with:
```text
MODEL_RESERVATION_STATE_INVALID
```
Archiving an exchange failed marker does not fix the root cause because the Model Governance
reservation remains intentionally durable.
## Contract
External request identity is now:
```text
hash(
task_id,
job_id,
execution_digest,
runtime,
attempt
)
```
The attempt is a **transient execution coordinate**. It is not added to the signed governed execution
envelope and does not alter the Task/Job binding or execution digest.
The worker gets the authoritative attempt from the successful claim response and passes it through:
```text
process_claim
→ RuntimeAdapter.execute(..., attempt=N)
→ ExternalAgentExecutor
→ ExternalAgentExchange
→ external request JSON
→ external result JSON
```
The external result must bind the same attempt.
## Idempotency
This gives the intended two-level behavior:
```text
same Task/Job/execution/runtime + same attempt
→ same external request ID
→ same external result/reservation can be replayed safely
same Task/Job/execution/runtime + next official attempt
→ different external request ID
→ different Model Governance request ID
→ new reservation
```
Old FAILED reservations are retained for audit. No Model Governance DB cleanup is required or
permitted.
## Model Governance
Model Governance already derives its route request ID from:
```text
external_request_id + ":model"
```
Therefore making the external request ID attempt-scoped automatically makes route decisions and
reservations attempt-scoped without creating another state authority.
## Recovery
For the current Hermes live Task:
```text
attempt 6
→ old request/reservation remains FAILED
official NEEDS_REWORK
→ bounded retry extension 6 → 7
attempt 7
→ new external request ID
→ new route request ID
→ new reservation
→ provider execution
→ settlement
```
No replacement Task/Job, direct DB rewrite, failed-reservation deletion or marker deletion is
required.
## Worker Availability
The repair must preserve the already verified Worker Availability runtime. Runtime switches use the
maintenance inhibit and must finish with the guard healthy and all three workers fresh.
## Verification
Required regression coverage:
- request ID stable within one attempt;
- request ID different across attempts;
- request schema requires a valid attempt;
- external result binds attempt;
- old failed Model Governance reservation remains FAILED;
- next attempt gets a new reservation and can settle;
- existing external-agent and Model Governance runtime tests still pass.
@@ -0,0 +1,18 @@
# Generic Compact Interface-Map Profiles
The compact interface-map runtime is a reusable read-only control-plane capability. It keeps the
canonical `source_capture` action and `compact_interface_map` adapter, but selects a bounded semantic
profile from the task's source signals.
The existing `fleet` profile remains backward compatible and returns
`P2_INTERFACE_MAP_CAPTURED` with the `FLEET_*` contracts.
The new `model_governance` profile is selected when model/provider/routing/budget/evaluation terms
are present. It returns `MODEL_GOVERNANCE_INTERFACE_MAP_CAPTURED` and model-specific contracts for
provider registry, selection policy, version pinning, budget accounting, fallback, scope isolation,
credential boundary, trace/audit, evaluation gates and routing resilience.
Profile and classification are emitted by the compact-map module and revalidated by the worker
before result submission. Unknown or mismatched profile/classification pairs fail closed.
No new Bridge action, daemon, database, WriterLease, mutation path or credential surface is added.
@@ -0,0 +1,142 @@
# Hermes / OpenClaw Online Agent Runtime
## Goal
Promote Hermes and OpenClaw from shadow-only adapter descriptions to real Online Agent workers
without creating a second Task/claim protocol.
Both workers reuse the existing P1/P2 Online Agent lifecycle:
```text
Online Task projected
→ capability-filtered poll
→ automatic claim
→ heartbeat
→ durable checkpoint
→ external Agent execution
→ attempt-aware result submission
→ official verification
```
They do **not** change the existing TaskPlane `OpenClawAdapter/HermesAdapter` shadow contract. Those
adapters remain side-effect-free compatibility surfaces.
## Execution action
The canonical execution envelope adds a read-only action:
```text
action=agent_runtime
adapter=hermes | openclaw
```
The action requires capability `agent.runtime.<adapter>`, has an empty change set, no verification
commands, and `writer_lease=false`.
`project_online_task()` copies only bounded task fields into the signed execution envelope:
- goal;
- instructions;
- success criteria;
- non-secret constraints.
The task may not choose a command, executable, URL, credential path, environment, or shell string.
## Worker separation
The generic engineering worker defaults to:
```text
AIOS_WORKER_ALLOWED_ADAPTERS=deterministic,source_capture,compact_interface_map,codex
```
so it skips Hermes/OpenClaw tasks before claim.
Dedicated services advertise:
```text
Hermes:
AIOS_WORKER_CAPABILITIES=agent.runtime.hermes
AIOS_WORKER_ALLOWED_ADAPTERS=hermes
OpenClaw:
AIOS_WORKER_CAPABILITIES=agent.runtime.openclaw
AIOS_WORKER_ALLOWED_ADAPTERS=openclaw
```
This makes adapter selection a pre-claim capability/fleet decision instead of a post-claim failure.
## Credential-isolated executor
The Online Agent worker itself still owns only Task API/Bridge coordination credentials. It writes
a hash-bound request to `/var/lib/aios-external-agent-executor`.
`aios-external-agent-executor.service` runs as the existing uncredentialed engineering executor user.
It has:
- one provider-only `LoadCredential` projection for the model-ready OneAPI environment;
- `/etc/aios-gitops`, `/data/aios-gitops`, and the original provider-secret directory inaccessible;
- no repository path;
- no shell execution;
- loopback denied, preventing direct access to the AIOS Task/Bridge control plane;
- only the staging OneAPI container address allowed for the installed Agent runtime/provider;
- only the exchange directory writable.
The root-owned runtime profile in `/etc/aios/external-agent-runtime.d/<runtime>.json` decides the
absolute executable, fixed non-secret environment, provider-only credential mapping, and I/O mode.
A task cannot override any of these values. The profile may select `OPENAI_API_KEY` only from the
systemd credential named `aios-external-agent-oneapi-env`; it cannot name a credential, inject an
environment variable, or carry an inline secret. The runtime executor parses only the selected
`ONEAPI_REGULAR_TOKEN` entry and does not expose the source file or other provider tiers to the
child.
## Runtime profile
A profile supports:
- `argv`: fixed argv, absolute allowlisted executable;
- `input_mode`: `stdin_text`, `stdin_json`, or `file_json`;
- `output_mode`: `stdout_text`, `stdout_json`, or `file_json`;
- timeout;
- bounded output size.
`{input_json}` and `{output_json}` placeholders are allowed only for file modes. `{prompt}` is a
root-profile-only argv placeholder populated from the bounded, signed task projection.
The repository includes the staging profile templates validated against the installed Hermes
v0.20.0 and OpenClaw v2026.7.2-beta.7 CLIs. Before activation, the Agent must re-run the actual
`--help`/version probes and install the profiles and static configs root-owned. If the installed
CLI differs, update only the root-owned profile/config projection and record the observed command;
do not modify the Task contract to compensate.
The provider projection is staging-only and uses the existing model-ready OneAPI source. It is not
an AIOS Task, Bridge, GitOps, SSH, or Writer credential. Live activation remains blocked if the
model-ready source cannot be projected through systemd or if the selected provider credential is
missing/quota-invalid. This worker never passes API keys through task metadata or arbitrary
environment variables.
## Retry/restart
The external request ID is deterministic over Task/Job/execution digest/runtime. The exchange keeps
the request/result across worker restart. A repeated attempt with the exact execution digest reuses
the durable external result instead of invoking the Agent twice.
The normal P1 result idempotency/checkpoint path remains authoritative for Task result submission.
## Result
Successful execution returns:
```text
status=SUCCEEDED
classification=EXTERNAL_AGENT_RUNTIME_SUCCEEDED
execution_mode=agent_runtime
agent_runtime=hermes | openclaw
writer_lease=NONE
changed_files=[]
credential_material_exposed=false
control_plane_access_granted_to_runtime=false
```
This first runtime version is intentionally read-only with respect to the repository. Engineering
changes continue to use assistant-authored deterministic plans and governed WriterLease/commit.
@@ -0,0 +1,109 @@
# Model Governance Policy / Profile Floor Compatibility
## Problem
The root-owned OpenClaw runtime profile reserves a minimum of 16,384 input tokens so Model
Governance can cover OpenClaw-injected workspace, skills and system context before provider
execution.
The staging scope policies still limited input to 16,000 tokens per request. Route selection
therefore failed before reservation with:
```text
MODEL_INPUT_TOKEN_LIMIT_DENY
```
This was a configuration invariant violation:
```text
scope max_input_tokens_per_request
<
root-owned profile input_reservation_floor_tokens
```
The registered OpenClaw model itself supports 32,768 input tokens, so the model capability was not
the blocker.
## Invariant
For every Model Governance scope that allows a model/runtime whose root-owned profile declares an
input reservation floor:
```text
scope.max_input_tokens_per_request
>=
profile.model_governance.input_reservation_floor_tokens
```
The profile floor remains root-owned. Task payloads cannot alter it.
## Staging policy
The two staging policies that allow `openai/deepseek-v4-flash` are updated from:
```text
16000
```
to:
```text
16384
```
This is the minimum compatible value. It intentionally remains below the model's 32,768-token
technical input ceiling.
The daily token budget is unchanged.
## Governance path
The live SQLite policy must be changed only through the repository bootstrap and
`aios_model_governance_admin.py`.
The admin reconciler:
- reads the governed repository config;
- uses current policy revision as CAS;
- calls `ModelGovernanceStore.set_scope_policy`;
- appends normal Model Governance audit evidence;
- verifies the audit chain.
Direct SQL updates remain forbidden.
## Exhausted OpenClaw recovery
The fixed OpenClaw live Task has exhausted its normal three attempts.
After the governed policy candidate is officially verified and the live policy is reconciled:
```text
attempt 3 VERIFYING/FAILED result
→ official NEEDS_REWORK
→ Task becomes FAILED 3/3
→ official bounded retry extension 3 → 4
→ same Task / same Job returns QUEUED
→ persistent OpenClaw worker auto-claims attempt 4
```
AR1 attempt-scoped external request identity ensures attempt 4 receives a new external request and a
new Model Governance reservation.
No historical reservation is deleted or rewritten.
## Acceptance
Attempt 4 must prove:
```text
scope max_input_tokens_per_request=16384
profile input_reservation_floor_tokens=16384
reserved_estimated_input_tokens>=16384
actual input <= reserved input
settlement=SETTLED
usage_source=runtime
audit_chain_valid=true
```
If attempt 4 exposes a new failure, stop. No additional retry extension is authorized by this
repair.
@@ -0,0 +1,211 @@
# AIOS Model Governance Runtime
## Status and purpose
This component turns the previously `PARTIAL` Model Governance interface map into an enforceable
runtime policy for the already verified Hermes/OpenClaw Online Agent path.
It is intentionally a **single SQLite library**, not a new network service. The existing external
Agent executor calls the library immediately before launching a model-backed runtime and settles the
same reservation immediately after the result.
The authoritative execution chain remains:
```text
Online Task scope / trace
→ dedicated worker claim + heartbeat
→ external Agent executor
→ ModelGovernanceStore.route()
→ root-owned runtime profile binds provider/model/version
→ Hermes / OpenClaw process
→ ModelGovernanceStore.record_success() / record_failure()
→ durable Online Agent checkpoint/result
```
P0 scope/trace/credential, P1 checkpoint/retry, P2 fleet scheduling and the existing external Agent
credential boundary remain authoritative.
## Ten Model Governance contracts
### MODEL_PROVIDER_REGISTRY
`model_governance_providers` stores provider identity, failure domain, circuit policy and an opaque
credential reference. Raw credentials are rejected.
Allowed reference prefixes:
- `systemd-credential:`
- `runtime-profile:`
- `credential-broker:`
### MODEL_SELECTION_POLICY
`model_governance_scope_policies` constrains eligible models/providers/capabilities. Routing is
deterministic: explicit preferred models, then configured priority, then stable provider/model IDs.
### MODEL_VERSION_PINNING
A registry row always includes a model version. A scope can additionally pin a model ID + exact
version. Every route decision persists both model ID and version plus model/policy revisions.
### MODEL_BUDGET_ACCOUNTING
A route first reserves estimated input/output tokens and estimated cost inside `BEGIN IMMEDIATE`.
Settled usage and outstanding reservations both count against the daily scope budget, preventing two
concurrent requests from oversubscribing the same budget.
Runtime-reported usage is preferred. When telemetry is absent, the reserved upper bound is charged
rather than silently under-counting.
OpenClaw's verified result shape is supported directly:
```text
meta.agentMeta.usage.input
meta.agentMeta.usage.output
meta.agentMeta.costUsd
```
Hermes currently has no structured usage in the verified output, so it uses conservative
reservation settlement.
### MODEL_FALLBACK_POLICY
Provider failures are durable. Repeated failures open a circuit for the configured cooldown. A
fallback request excludes the failed provider and increments `fallback_index`; the scope policy
bounds the number of fallback attempts.
### MODEL_SCOPE_ISOLATION
The signed `agent_runtime` execution envelope now carries authoritative Task scope + trace identity.
The projection layer always overwrites `runtime_input.scope` and `runtime_input.trace_id` from the
Online Task, even if an execution request attempts to supply different values.
Tenant/project lookup may fall back only to the same tenant's policy. It never falls back to a
system or legacy policy. `system` and `legacy` require their own explicit policies, avoiding fake
tenant identities.
### MODEL_CREDENTIAL_BOUNDARY
Model governance stores credential references only. The root-owned external-runtime profile still
owns the actual provider credential projection. Task payloads cannot select executable, argv,
environment, provider credential, or credential path.
The external child process remains unable to reach the AIOS control plane.
### MODEL_TRACE_AUDIT
Every route, failure, provider/model/policy update and settlement appends to a SHA-256 hash-chained
audit table with task ID, trace ID and scope key. `verify_audit_chain()` detects tampering.
### MODEL_EVALUATION_GATE
A model cannot enter the registry unless its state is `APPROVED` and it has a 64-hex evaluation
evidence digest.
The staging bootstrap uses the already verified Hermes/OpenClaw E2E external-result SHA256 values as
evaluation evidence, rather than a configuration-file hash.
### MODEL_ROUTING_RESILIENCE
Route request IDs are idempotent and bound to a canonical request hash. Conflicting replay is
rejected. Provider health, reservations, usage settlement and policy/provider/model updates use
transaction/CAS semantics.
## Runtime profile contract
A root-owned Hermes/OpenClaw profile may declare:
```json
{
"model_governance": {
"required": true,
"provider_ids": ["provider-a"],
"model_capabilities": ["text"],
"max_output_tokens": 512
}
}
```
When `required=true`:
- `provider_ids` must be non-empty;
- `max_output_tokens` must be positive;
- argv/environment must contain at least one governance-controlled marker:
`{provider_id}`, `{model_id}`, or `{model_version}`;
- `AIOS_MODEL_GOVERNANCE_ENABLED` must be enabled;
- a matching scope policy/model/provider must exist;
- budget reservation must succeed before the child process is launched.
The task cannot provide these markers or their values.
## SQLite authority
Default database:
```text
/var/lib/aios-model-governance/state.sqlite3
```
The executor process already owns the only runtime mutation path. The directory is created by
tmpfiles and is the only new writable path added to that executor.
No direct Task/Job/Git database writes are introduced.
## Bootstrap/reconciliation
`aios_model_governance_admin.py` applies a root-owned JSON bootstrap idempotently. It compares desired
state to current rows and uses the current revision as the CAS input only when a change is actually
required.
The repository staging bootstrap is not a Production policy. It exists only to reproduce and
govern the already verified non-production Hermes/OpenClaw runtimes.
## Cost caveat
The current verified OneAPI-backed runtime evidence reports zero authoritative provider cost for
OpenClaw and no structured cost for Hermes. The staging bootstrap therefore does **not** invent
pricing and sets cost rates/budget to zero/unbounded.
Token budgets remain enforced. Real monetary budget enforcement becomes effective as soon as
authoritative per-model pricing is entered into the root-owned registry.
## OpenClaw JIT/WASM compatibility
The previous live acceptance proved that OpenClaw Node/WASM requires
`MemoryDenyWriteExecute=no`. Model Governance must preserve that already-evidenced runtime exception;
it must not further weaken `NoNewPrivileges`, `ProtectSystem=strict`, `ProtectHome`, loopback deny,
credential isolation or repository isolation.
## Failure/recovery invariants
- Budget denial occurs before provider execution.
- Missing scope policy fails closed.
- Missing/disabled governance fails closed when a profile requires it.
- Actual usage cannot exceed the reservation.
- Provider failure releases that reservation into `FAILED` state before fallback.
- Settlement is idempotent.
- Route replay is idempotent only for the exact canonical request.
- Audit corruption is detectable.
- Online Agent checkpoint/result retry remains unchanged and remains the outer recovery authority.
## Verification
Required unit/integration coverage includes:
- registry/evaluation/CAS;
- scope isolation and no system→tenant fallback;
- model/version pinning;
- atomic budget reservation;
- settlement/replay;
- missing telemetry conservative settlement;
- provider circuit/fallback;
- trace/audit tamper detection;
- authoritative scope/trace projection;
- task runtime-input scope override attack;
- root-owned provider/model binding;
- OpenClaw nested usage extraction;
- real Hermes/OpenClaw staging live tasks with Model Governance evidence.
The governed implementation is not complete without the unified all-modified-files bundle for the
candidate delta and the cumulative lineage bundle.
@@ -0,0 +1,80 @@
# OpenClaw Input Reservation Floor
## Problem
Model Governance originally estimated input reservation from the user-visible prompt only.
The verified OpenClaw runtime injects additional root-owned/system-controlled context including
workspace, skills and system instructions. In live acceptance the prompt estimate reserved only
153 input tokens while OpenClaw reported approximately 7,770 actual input tokens.
The settlement correctly failed closed with:
```text
MODEL_USAGE_EXCEEDS_RESERVATION
```
The provider call itself had succeeded. The defect was therefore pre-execution reservation
underestimation, not provider connectivity.
## Contract
A root-owned external Agent profile may now declare:
```json
{
"model_governance": {
"input_reservation_floor_tokens": 16384
}
}
```
The effective input reservation is:
```text
max(
byte-based prompt estimate,
profile input_reservation_floor_tokens
)
```
Only the root-owned runtime profile can choose this floor. Online Task payloads cannot lower or
override it.
The OpenClaw staging profile uses 16,384 tokens. This is greater than twice the observed 7,770-token
live input high-water mark while remaining below the registered 32,768-token model input limit.
Hermes has no floor unless explicitly configured and therefore preserves its existing behavior.
## Fail-closed behavior
The floor is not an unlimited post-execution top-up.
If actual runtime usage exceeds the reserved amount, settlement still fails with
`MODEL_USAGE_EXCEEDS_RESERVATION`.
This keeps budget enforcement pre-execution and avoids silently authorizing usage after it has
already happened.
## Evidence
External results expose:
```text
prompt_estimated_input_tokens
input_reservation_floor_tokens
reserved_estimated_input_tokens
```
alongside the normal settlement and audit evidence.
## Retry
The current OpenClaw attempt 1 failure remains preserved. After this code repair is governed and
activated, official `NEEDS_REWORK` returns the same fixed Task to the normal retry path. Because AR1
already made external request identity attempt-scoped, attempt 2 receives a new external request ID
and a new reservation.
No retry-extension is required because the Task is currently attempt 1 of 3.
No Model Governance reservation is deleted or rewritten.
@@ -0,0 +1,36 @@
# Owner-Controlled Same-Binding Execution Rebind
## Purpose
`aios_agent_task_execution_rebind.py` closes a projection/plan race without replacing the Task,
Governed Job, target branch, generation, or idempotency lineage.
The only supported transition is:
```text
official NEEDS_REWORK
→ Task QUEUED with no active lease
→ persisted source_capture envelope
→ canonical Bridge resolves an already-registered deterministic plan
→ CAS-bound metadata.execution + execution_projection replacement
→ audit event
→ normal worker claim
```
The service rejects arbitrary metadata patches. It derives the new envelope from canonical Bridge
`/v2/execution-envelope`, validates every governed binding coordinate against the persisted envelope,
requires `source_capture → deterministic`, requires exact old/new execution digests supplied by the
Owner recovery request, and increments the Task revision under `BEGIN IMMEDIATE`.
It records `TASK_EXECUTION_REBIND_AUTHORIZED` and an idempotent `task_execution_rebinds` receipt.
No Task/Job/branch replacement is created and the worker still performs the real claim/heartbeat,
WriterLease and governed commit.
The endpoint is loopback-only:
`POST /v1/tasks/{task_id}/execution-rebind`
and requires the existing Owner control credential plus
`AIOS_AGENT_TASK_EXECUTION_REBIND_AUTHORIZATION_ID`.
Direct/manual Task database edits remain forbidden; the service is the audited mutation boundary.
@@ -0,0 +1,212 @@
# AIOS Worker Availability Runtime
## Priority
This runtime exists to make the already verified Online Agent execution path continuously usable
without repeated human intervention.
The system already proved that generic, Hermes and OpenClaw workers can claim and complete real
Online Tasks. The remaining availability defect was operational: evidence-freeze workflows left the
worker services stopped, while registry `ACTIVE` records remained stale.
The availability contract therefore separates:
```text
AUTO_CLAIM_MECHANISM=VERIFIED
CURRENT_WORKER_AVAILABILITY=<fresh runtime state>
```
A historical successful claim is not current liveness.
## Source of truth for idle liveness
An idle worker may have no Task lease heartbeat. Therefore Task heartbeat alone cannot prove that
the worker is continuously polling.
`aios_online_agent_worker.py` now writes:
```text
<AIOS_WORKER_STATE_ROOT>/worker-liveness.json
```
on:
- process start: `STARTING`;
- every successful poll: `POLL_OK`;
- poll failure: `POLL_FAILED`;
- every successful Task lease heartbeat while executing: `BUSY`.
The file contains:
- agent ID/type/version;
- process PID;
- monotonically increasing local sequence;
- `last_seen_epoch`;
- poll/heartbeat intervals;
- current Task/stage while busy.
The liveness file is worker-owned observability state only. It does not write or mutate Task, Job,
Git, WriterLease or registry databases.
## Self-healing guard
`aios-worker-availability-guard.timer` invokes a root-owned oneshot guard every 30 seconds.
The guard manages:
```text
aios-online-agent-worker-20260815.service
aios-hermes-online-agent-worker.service
aios-openclaw-online-agent-worker.service
```
and the required external runtime dependency:
```text
aios-external-agent-executor.service
```
For each worker:
1. missing systemd unit → fail closed;
2. disabled → `systemctl enable`;
3. inactive → `systemctl start`;
4. active but liveness older than 90 seconds → restart;
5. repeated stale restart within 60 seconds → do not restart again; report degraded.
Long-running Tasks are safe because the Task lease-heartbeat thread refreshes `BUSY` liveness.
The guard never claims a Task and never touches Task/Job/Git databases.
## Maintenance inhibit
Planned maintenance creates:
```text
/run/aios-worker-availability/maintenance.lock
```
While this file exists the guard performs no start/restart actions and reports:
```text
MAINTENANCE_INHIBITED
```
Maintenance tooling must remove the file when done. This prevents the self-healer from fighting
intentional service changes.
## Status
The guard writes:
```text
/run/aios-worker-availability/status.json
```
with current worker/dependency states, liveness ages and recovery actions.
A worker is currently available only when:
```text
systemd load_state=loaded
systemd active_state=active
unit enabled/static
worker-liveness.json present
last_seen_age <= 90s
```
Registry `ACTIVE` without fresh poll/busy liveness is not sufficient.
## Boot/restart behavior
The timer is:
```text
OnBootSec=15s
OnUnitActiveSec=30s
Persistent=true
```
Therefore reboot, manual stop left behind by another workflow, process crash, and hung poll loop are
all repaired without requiring a human to notice and intervene.
Normal worker units retain their own `Restart=always`; the availability guard handles the cases that
`Restart=always` cannot, especially a deliberate `systemctl stop` left after an evidence workflow.
## Acceptance
The live acceptance requires all of the following:
```text
guard timer active/enabled
external executor active/enabled
generic worker active/enabled + fresh liveness
Hermes worker active/enabled + fresh liveness
OpenClaw worker active/enabled + fresh liveness
```
Then a new read-only Online Task is projected through canonical Bridge `18892`.
Success requires:
```text
TASK_CREATED
→ generic worker auto claim
→ TASK_HEARTBEAT
→ RESULT_SUBMITTED
→ official TASK_VERIFIED(SUCCEEDED)
```
After the Task, the services must remain active and liveness must continue advancing. The acceptance
also stops a worker intentionally once and proves the guard starts it again with a new fresh
liveness sequence.
Only after these gates may current status be upgraded from:
```text
OVERALL=PARTIAL
```
to:
```text
PERSISTENT_WORKER_LIVENESS=VERIFIED
NEW_TASK_AUTO_CONSUMPTION=VERIFIED
WORKER_AVAILABILITY_RUNTIME=VERIFIED_CANDIDATE
```
Protected main and Production remain out of scope.
## Forced source-capture projection compatibility
The deployed canonical Bridge currently projects a default `source_capture` envelope even when a
governed binding request asks for binding-only behavior. This must not be allowed to make always-on
workers consume engineering Tasks before their deterministic plan is registered.
The worker therefore distinguishes a real read-only source capture from an engineering placeholder:
```text
source_capture
+ capability_scope contains governed.write.branch or governed.write.file
= PENDING_DETERMINISTIC_PLAN
```
For such a placeholder the worker:
1. re-resolves through the canonical plan registry **without replaying the stale source-capture
envelope**;
2. if no deterministic plan exists, emits `TASK_SKIPPED_PENDING_DETERMINISTIC_PLAN` and does not
claim;
3. if the registered canonical envelope is deterministic and the governed binding still matches,
uses that deterministic envelope and claims normally.
A real read-only source-capture Task with only `git.read`/`git.test` still replays its exact persisted
payload and remains claimable.
This makes persistent workers safe with the currently deployed Bridge while preserving the same
Task/Job binding. No Task DB rewrite, cancellation, replacement Task, fake NEEDS_REWORK, or
pre-claim metadata mutation is required.
A later Bridge cleanup may implement true binding-only projection, but Worker Availability does not
depend on that cleanup.
@@ -0,0 +1,96 @@
# Canonical all-modified-files packager and secret scanning
## Problem
The historical all-modified-files evidence packager lived only as a copied handoff script.
Its private-key scanner matched the fragment:
```text
BEGIN PRIVATE KEY
```
instead of a complete PEM header. A regression test containing that literal therefore caused a
false secret finding even though the candidate contained no credential.
A task had to use a temporary evidence-only compatibility packager to complete its bundle. That is
acceptable as an emergency workaround but not as a durable unattended path.
## Repository source of truth
The canonical implementation is now:
```text
active-runtime/app/aios_all_modified_files_packager.py
```
It preserves the historical output/bundle schemas:
```text
aios.all-modified-files.bundle.v1
aios.all-modified-files.bundle-summary.v1
aios.modified-documents.index.v1
```
and the host-visible status lines:
```text
ALL_MODIFIED_FILES_BUNDLE=PASS
ALL_MODIFIED_FILES_BUNDLE_PATH=...
ALL_MODIFIED_FILES_BUNDLE_SHA256=...
```
Future governed tasks should use this repository version instead of copying an unversioned scanner.
## Secret scanning invariant
Private-key detection requires a complete PEM header:
```text
-----BEGIN [RSA|EC|DSA|OPENSSH ]PRIVATE KEY-----
```
The following source/test/document text is not secret material and must remain packageable:
```text
BEGIN PRIVATE KEY
PRIVATE KEY
PEM private-key header
```
Bearer and key/value credential detection remains fail-closed:
```text
Authorization: Bearer <20+ token chars>
password=<12+ non-space chars>
passwd=<12+ non-space chars>
secret=<12+ non-space chars>
api_key=<12+ non-space chars>
api-key=<12+ non-space chars>
```
Tests construct the real PEM header dynamically so the scanner's own test source does not contain a
literal credential-looking header.
## Bundle integrity
The packager continues to enforce:
- exact base→candidate ancestry;
- before/after file capture;
- rename/copy/delete handling;
- path traversal denial;
- execution evidence symlink denial;
- secret scan over repository before/after and execution evidence;
- document index;
- SHA-256 closure;
- ZIP without symlinks.
No secret finding is downgraded to a warning.
## Migration
The old handoff script may remain for historical reproducibility, but new handoffs should call the
repository packager from the verified candidate lineage.
A temporary compatibility scanner is no longer needed once this candidate is in the lineage.
@@ -0,0 +1,210 @@
# AIOS Autonomous Control Plane — Admission + Invariant + Repair
## Why this exists
Recent Agent registration, Task claim, release-worker, credential and retry failures shared one
failure mode:
> The control plane accepted an intent before the runtime had proved it could execute the intent.
This design does not add another fleet of daemons. It adds one shared library contract that Task API,
workers and governed host recovery consume.
## Three layers
```text
Intent
↓
Runtime Admission
↓
System Invariants
↓
Repair Policy
↓
Execution
```
### Runtime Admission
Admission occurs before Task claim.
A capability such as `release:candidate` is not considered executable merely because its name
matches. The capability contract also binds:
- supported contract version;
- allowed permission;
- forbidden operations;
- runtime dependencies;
- health requirements.
For `release:candidate`:
```text
permission:
prepare_release
forbidden:
production_deploy
submit_owner_approval
tag_publish
baseline_registry_finalize
runtime_switch
dependencies:
credential_projection
trusted_builder
health:
authenticated_probe
release_api_health
worker_liveness
```
Only a fully passing decision reaches:
```text
EXECUTION_READY
```
A failed decision remains:
```text
EXECUTION_READY_FAILED
```
and the worker does not call `/claim`.
Therefore an infrastructure defect discovered by admission does not consume a business attempt.
### System Invariants
The same admission checks are exposed as a machine-readable invariant snapshot.
Examples:
```text
capability_contract_registry
task_capabilities_registered
worker_capabilities_registered
capability_match
forbidden_operations_absent
credential_projection
trusted_builder
authenticated_probe
release_api_health
worker_liveness
```
A runtime can expose:
```text
HEALTHY
DEGRADED
```
without relying on a human reconstructing state from logs.
### Repair Policy
Failure classification maps to bounded recovery:
```text
DEPENDENCY_UNAVAILABLE → level 0 / auto local
RESOURCE_UNAVAILABLE → level 0 / auto local
WORKER_CRASH → level 0 / auto local
LEASE_LOST → level 0 / auto local
CAPABILITY_CONTRACT_ERROR → level 1 / same-lineage repair task
CREDENTIAL_PROJECTION_ERROR → level 1 / same-lineage repair task
UNKNOWN → level 1 / diagnostic repair task
POLICY_DENIED → level 2 / Owner authorization
BUSINESS_PERMANENT → level 3 / terminal/manual decision
```
Only business-permanent failure consumes the business retry budget in this first contract.
The current Task API does not yet have a first-class `EXECUTION_READY_FAILED` durable state. Until it
does, the Release Worker enforces the most important invariant by refusing to claim a Task whose
admission fails.
## Capability Contract Registry
`aios_capability_contract.py` remains backward-compatible with existing aliases but now validates
every alias target against a versioned semantic contract registry.
Unknown capabilities and wildcard capabilities remain fail-closed.
Agent registration and Task creation should next be changed to call `registered_capability()` so a
capability cannot be accepted at registration and rejected only later by the matcher.
## Credential rule
Admission probes credential usability, not merely credential existence.
For release candidate creation the authenticated probe intentionally sends an invalid empty
`/createReleaseCandidate` request.
Expected:
```text
HTTP 400
```
This proves:
```text
credential readable
→ authentication succeeded
→ scope accepted
→ request validation rejected empty business input
```
A permission failure/401/403/5xx is `EXECUTION_READY_FAILED`.
## Lineage
Repair actions are descriptive, not permission grants. Any repair Task must retain the existing
root/parent/attempt/authority chain and use governed same-lineage recovery.
No repair action may silently create a replacement business Task.
## Authority separation
Capability answers:
```text
can this runtime perform the operation?
```
Authority answers:
```text
is this operation currently authorized?
```
`release:candidate` never implies Owner approval, deployment, tag publication, baseline finalization
or runtime switch.
## Deployment sequence
Phase 1 (this candidate):
- Capability Contract Registry;
- shared Autonomous Control Plane library;
- Release Worker pre-claim admission;
- failure classification + repair evidence.
Phase 2:
- Task API uses the same library before returning Tasks from poll and again during claim CAS;
- Task schema gains durable `ADMITTED/EXECUTION_READY/EXECUTION_READY_FAILED` evidence without
counting pre-side-effect infrastructure failures against business attempts.
Phase 3:
- Agent registration validates against the same Capability Contract Registry;
- external connector writes only durable Intent/Command Inbox records and leaves execution to the
local control plane.
This sequencing avoids a second controller stack while converging every decision onto one contract.
@@ -0,0 +1,228 @@
# Durable Owner Authorization and capability projection
## Problem
AIOS already supports unattended worker execution after a Task is executable, but Owner intent was
not durable at the capability layer. The same Owner repeatedly had to authorize or repair:
- `governed-task:project`;
- `release:candidate`.
This caused valid one-time Owner intent to be lost between conversations and forced repeated human
interaction.
## Design
Durable authorization is represented by a root-owned mode `0600` policy outside Git. The policy is
persistent until explicitly revoked.
The first persistable capabilities are deliberately narrow:
```text
governed-task:project
release:candidate
```
They cover preparation work only.
The following capabilities can never be inherited from this durable grant:
```text
submitOwnerApproval
startDeployment
tag_publish
baseline_registry_finalize
runtime_switch
main_write
production_deploy
force_push
credential_admin
```
Those remain separate high-risk release/Owner-signature gates.
## Policy
Example schema:
```json
{
"schema": "aios.owner-durable-authorization.v1",
"grant_id": "owner-permanent-preparation-20260819",
"owner_identity": "owner@example",
"account": "chatgpt-governed-implementer-singlehost",
"capabilities": [
"governed-task:project",
"release:candidate"
],
"issued_at": "2026-08-19T05:00:00Z",
"not_before": "2026-08-19T05:00:00Z",
"persistent_until_revoked": true,
"constraints": {
"repository": "aios/aios",
"allowed_target_prefix": "agent/",
"release_version_prefix": "V2R2C",
"max_task_risk": "engineering"
}
}
```
The repository never stores the live Owner policy or release credential.
## Revocation
A root-owned mode `0600` revocation file lists `revoked_grant_ids`.
Revocation is checked on every operation, not only at service startup. A running broker therefore
fails closed immediately after revocation without needing a new conversation.
## Broker boundary
The broker exposes only an AF_UNIX socket:
```text
/run/aios-owner-authorization-broker/broker.sock
```
The socket is mode `0660` and owned by the service's `aios-executor-exchange` group. This avoids introducing another long-lived bearer secret solely for local broker access.
It exposes only:
```text
GET /v1/capabilities
POST /v1/project-governed-task
POST /v1/authorize-release-candidate
```
`project-governed-task` calls only the canonical Bridge.
`authorize-release-candidate` persists an Owner authorization decision only. It deliberately does
**not** invent a local trusted-builder endpoint. The external release connector remains responsible
for projecting the account's real `release:candidate` scope. Until that external projection exists,
the decision state is `AUTHORIZED_PENDING_EXTERNAL_CAPABILITY_PROJECTION`.
The broker holds only the existing host control credential needed for canonical governed-task
projection. It never holds an Owner signing key or a release/deployment credential. The caller also supplies the expected account identifier over the Unix transport; the OS socket permission is the local access boundary.
## Canonical Online Task projection
A durable `governed-task:project` grant may create a complete canonical Online Task rather than a
bare Task. It requires:
```text
require_online_task_projection=true
generation=1
branch_mode=CREATE_NEW
max_attempts<=3
risk<=engineering
initial action=source_capture
```
Projected task capabilities are limited to:
```text
git.read
git.test
governed.write.branch
governed.write.file
```
A write-capable task still begins as `source_capture`. Existing placeholder-deferral prevents the
generic worker from claiming it until a deterministic plan is registered. Repository mutation
therefore still requires the normal deterministic plan, WriterLease and governed commit path.
This fixes the previous split-brain behavior where a bare Task could exist without the canonical
governed execution projection.
## External capability projection outbox
A `release:candidate` authorization now creates a durable machine-readable outbox record:
```text
schema=aios.external-capability-projection.request.v1
state=PENDING_EXTERNAL_CONNECTOR
capability=release:candidate
```
The projection identifier is deterministic from:
```text
grant_id
account
capability
request_sha256
```
The outbox stores no credentials or Owner signing material. It retains only the already-authorized
release-candidate request, policy/request hashes, account, grant id and timestamps.
The broker exposes read-only Unix-socket endpoints:
```text
GET /v1/projections
GET /v1/projections/{projection_id}
```
These endpoints re-check the current Owner policy on every read. If the durable grant has been
revoked, enumeration fails closed with `OWNER_GRANT_REVOKED`. The historical outbox record is
retained for audit and is not silently deleted.
The repository intentionally does **not** provide a connector acknowledgement/write endpoint yet.
No in-repository code can truthfully assert that the external ChatGPT connector account has gained
`release:candidate`. Until the platform consumes the outbox and performs its own identity-scope
projection, the only valid state is:
```text
AUTHORIZED_PENDING_EXTERNAL_CAPABILITY_PROJECTION
PENDING_EXTERNAL_CONNECTOR
```
A future external projector must preserve:
```text
release:candidate
!= submitOwnerApproval
!= startDeployment
!= tag_publish
!= baseline_registry_finalize
!= runtime_switch
```
## Idempotency and audit
Every broker operation requires an idempotency key. A receipt SQLite database stores only:
- idempotency key;
- capability;
- request SHA-256;
- response SHA-256;
- sanitized upstream response;
- timestamp.
An exact retry returns the same receipt. Reusing an idempotency key with different content fails.
## Unattended invariant
Once the Owner policy and narrow service credentials are provisioned:
```text
Owner durable grant
→ broker decision
→ canonical governed task projection / release candidate preparation
→ worker / trusted builder
```
No repeated Owner interaction is required for those preparation capabilities.
Production execution remains:
```text
release candidate
→ immutable approval envelope
→ Owner signature
→ deployment
```
The durable grant never substitutes for the Owner signature.
@@ -0,0 +1,114 @@
# Owner-approved GPT HTTPS edge activation and rollback
## Scope
This controller is the final host-side control mechanism before the disabled-by-default GPT HTTPS
edge may bind publicly.
Repository implementation and public activation remain separate states:
```text
IMPLEMENTED
≠
OWNER_APPROVED
≠
ACTIVATED
≠
RUNTIME_VERIFIED
```
The controller never creates owner approval, TLS assets, DNS, firewall rules or ingress bearer
credentials.
## Approval binding
The owner approval file is root-owned mode `0600` and has schema:
```json
{
"schema": "aios.gpt-https-edge.public-approval.v1",
"approval_id": "owner-public-edge-...",
"candidate_oid": "<exact governed edge candidate>",
"domain": "tasks.example.com",
"public_bind": true,
"bind_host": "0.0.0.0",
"port": 18443,
"tls_cert_sha256": "<exact certificate sha256>",
"issued_at": "2026-08-19T00:00:00Z",
"not_before": "2026-08-19T00:00:00Z",
"expires_at": "2026-08-20T00:00:00Z",
"owner": "<owner identity>"
}
```
Approval lifetime is at most seven days. Candidate/domain/bind/port/certificate are immutable within
one approval.
Changing any of those requires a new owner approval.
## Check
```text
aios_gpt_https_edge_activation.py check --candidate <GE>
```
Checks:
- approval ownership/mode/schema/time window;
- exact candidate binding;
- exact certificate hash;
- certificate/private-key pair;
- certificate hostname;
- certificate remaining lifetime;
- loopback GPT ingress service health.
It performs no public network mutation.
## Activate
`activate` additionally:
- writes one exact systemd public-bind drop-in atomically;
- enables/restarts only the HTTPS edge unit;
- verifies effective ExecStart points at the immutable candidate release;
- verifies effective public-bind/domain environment;
- performs a TLS handshake to `127.0.0.1:18443` using the approved public hostname;
- writes a durable activation receipt.
Any failure after drop-in mutation automatically invokes rollback.
Exact replay of an already-active matching approval returns the durable receipt without restarting
or creating duplicate state.
## Rollback
`rollback`:
- stops and disables only the HTTPS edge;
- removes only the controller-owned public-bind drop-in;
- daemon-reloads systemd;
- verifies edge inactive;
- writes a durable rollback receipt.
It does not delete:
- owner approval;
- TLS certificate/private key;
- ingress bearer token;
- internal loopback ingress;
- Task/Job state;
- Git branches.
Rollback is therefore replay-safe and does not require reconstructing secrets.
## Not covered
This controller intentionally does not own:
- DNS records;
- host firewall/NAT/security-group changes;
- certificate issuance/renewal;
- OpenAI Custom GPT domain verification;
- public Internet acceptance from outside the host.
Those remain separate owner/environment gates.
@@ -0,0 +1,120 @@
# Disabled-by-default HTTPS edge for external GPT Online Task access
## Purpose
The internal ingress (`127.0.0.1:18945`) remains loopback-only. This edge is the only component
allowed to become Internet-facing, and only after a separate owner approval.
It forwards only:
```text
POST /v1/read-only-tasks
GET /v1/read-only-tasks/{request_id}
GET /openapi.json
GET /health
```
It never proxies:
```text
18892 canonical Bridge
18940 Task API
WriterLease
commit
verification
release
tag
baseline
arbitrary URLs
```
## Default state
The systemd unit is disabled by policy unless the owner creates:
```text
/etc/aios-gitops/gpt-https-edge.approved
```
The base unit still binds only to:
```text
127.0.0.1:18443
AIOS_GPT_EDGE_PUBLIC_BIND_APPROVED=0
```
Non-loopback bind is rejected by code unless an owner-controlled drop-in sets:
```text
AIOS_GPT_EDGE_PUBLIC_BIND_APPROVED=1
AIOS_GPT_EDGE_HOST=0.0.0.0
AIOS_GPT_EDGE_PUBLIC_DOMAIN=<approved hostname>
```
This repository candidate does not create that drop-in or approval marker.
## TLS
TLS certificate/key are owner-managed systemd credentials:
```text
/etc/aios-gitops/gpt-https-edge-tls.crt
/etc/aios-gitops/gpt-https-edge-tls.key
```
The service requires TLS 1.2 or newer, disables TLS compression, and never logs or returns key
material.
Certificate/key issuance and renewal are external owner responsibilities until a separately
governed ACME workflow is implemented.
## Request protections
The edge provides:
- 64 KiB request cap;
- 256 KiB upstream response cap;
- JSON validation before proxying;
- fixed upstream `http://127.0.0.1:18945`;
- Authorization-only header forwarding;
- no Cookie/X-Forwarded-* trust;
- route/method allowlist;
- per-source-IP in-memory rate limiting;
- bounded concurrent request handling;
- optional exact Host-header enforcement;
- HSTS / no-store / nosniff / no-referrer response headers.
The rate limiter is intentionally simple and local. Multi-node/global abuse protection is outside
the current one-host architecture.
## Authentication
The HTTPS edge does not create a new authentication scheme. It forwards the external Bearer
credential to the loopback GPT ingress, which verifies `gpt_ingress_token`.
The edge never receives the AIOS control-plane token.
## Deployment boundary
Public activation is a high-risk network operation and requires explicit owner approval. Before
activation verify:
```text
host-side GPT ingress E2E=VERIFIED
owner ingress token provisioned
TLS certificate/key provisioned
public hostname owned and DNS configured
firewall/network exposure reviewed
explicit public-bind approval recorded
rollback plan available
```
Only after those gates may an owner-authorized runtime drop-in bind this service publicly.
Until then:
```text
GPT_HTTPS_EDGE_IMPLEMENTATION=candidate
PUBLIC_NETWORK_EXPOSURE=NONE
PUBLIC_CUSTOM_GPT_DIRECT_AI0S_ACCESS=NOT_VERIFIED
```
@@ -0,0 +1,138 @@
# External GPT → Online Task ingress
## Scope
This component closes the current architectural gap between an external Custom GPT and AIOS
without exposing the internal Task API (`18940`) or canonical Bridge (`18892`) directly.
The first supported public-facing capability is intentionally read-only:
```text
authenticated external intent
→ host-side ingress
→ canonical /v2/governed-task
→ canonical /v2/execution-envelope
→ persistent generic worker
→ source_capture result
→ official control-plane verification
```
The ingress itself never claims, heartbeats, verifies, commits, acquires WriterLease, or writes
protected refs.
## Security boundary
The service binds only to loopback (`127.0.0.1:18945`). A future owner-approved TLS reverse proxy
or tunnel may expose this service externally. Direct non-loopback binding is denied by code.
Two independent credentials are projected by systemd:
- `gpt_ingress_token`: authenticates the external caller to this ingress.
- `agent_task_control_token`: lets the ingress call the loopback AIOS control plane.
Neither credential is stored in Git, SQLite, task payloads, results, logs, or API responses.
The ingress has no repository filesystem access.
## Caller-controlled fields
Allowed:
```text
request_id
goal
instructions
success_criteria
constraints.max_files
constraints.focus_paths
constraints.notes
```
Not caller-controlled:
```text
repository
base_oid
expected_branch_oid
target_branch
branch_mode
generation
risk
priority
max_attempts
capabilities
action/adapter
WriterLease
worker identity
credentials
verification decision
main/production/tag/baseline permissions
```
All external tasks are hard-bound to:
```text
repository=aios/aios
capabilities=[git.read, git.test]
action=source_capture
writer_lease=false
repository_mutation=forbidden
control_plane_side_effects=forbidden
credential_access=forbidden
```
## Idempotency
`request_id` is the durable external idempotency boundary.
The ingress stores only:
```text
request hash
derived AIOS task id
canonical job id
base oid
sanitized receipt
state
```
An exact retry returns the same receipt. Reusing a `request_id` with different content returns
`IDEMPOTENCY_CONFLICT`.
A crash after canonical task creation can recover the same binding from the Task API rather than
creating a replacement Task.
## Status
`GET /v1/read-only-tasks/{request_id}` returns only sanitized state:
```text
task/job ids
status
attempt
whether claimed
result classification
official verification decision/classification
source snapshot hash/count when available
```
It does not proxy raw Task metadata, execution envelopes, result text, credentials, or source files.
## Public exposure
This candidate does **not** make AIOS publicly reachable. Public exposure requires a separately
approved HTTPS ingress/reverse proxy with:
- TLS;
- external authentication lifecycle;
- rate limiting / abuse controls;
- request size limits;
- observability;
- explicit owner approval for the network exposure.
Until that step is verified:
```text
HOST_SIDE_EXTERNAL_GPT_ADAPTER=implemented/staging-candidate
PUBLIC_CUSTOM_GPT_DIRECT_AI0S_ACCESS=not verified
```

Some files were not shown because too many files have changed in this diff Show More