Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c3984ca2b4 | ||
|
|
0c7e2d3b89 | ||
|
|
1b3b74fd62 | ||
|
|
378cc4d56a | ||
|
|
95fc4d37f0 | ||
|
|
9195470ceb | ||
|
|
cbe33ecab8 | ||
|
|
e3fc48e5b9 | ||
|
|
ed3bcf76c7 |
@@ -1,412 +0,0 @@
|
|||||||
# OpenClaw 在线执行指令:启动执行器、配置 Secret 引用并确认 Gitea 同步
|
|
||||||
|
|
||||||
- 任务 ID:`AIOS-V2R1-PRODUCTION-DEPLOY-20260720`
|
|
||||||
- 执行器:OpenClaw
|
|
||||||
- 任务分支:`task/openclaw/AIOS-V2R1-PRODUCTION-DEPLOY-20260720`
|
|
||||||
- 候选分支:`integration/aios-v2r1-cto-production-candidate-20260720`
|
|
||||||
- 语言:中文
|
|
||||||
- 执行模式:无人值守、证据驱动、默认拒绝
|
|
||||||
|
|
||||||
## 1. 任务目标
|
|
||||||
|
|
||||||
一次性完成以下工作:
|
|
||||||
|
|
||||||
1. 发现并启动环境中已注册的 OpenClaw 在线执行器;
|
|
||||||
2. 领取本任务并持续写入 claim、heartbeat、progress、blocker 和 result;
|
|
||||||
3. 将任务分支同步到最新 Candidate HEAD;
|
|
||||||
4. 确认 Candidate、OpenClaw Task、Codex Task 三个分支已同步到 Gitea,且本地与远端 OID 完全一致;
|
|
||||||
5. 从已批准的 Secret Provider、Credential Store 或受保护文件中解析 OpenClaw Runtime 配置;
|
|
||||||
6. 只把 Secret 引用和受保护文件路径写入 AIOS 环境配置,不得把明文 Secret 写入 Git、日志、Evidence 或终端输出;
|
|
||||||
7. 执行文档—代码检查、候选测试、全量验收和预发布健康检查;
|
|
||||||
8. 对低风险部署或 Runtime 缺陷创建 Patch 并修复;
|
|
||||||
9. 把所有修改和阻塞提交到系统,交由 Codex 统一审视。
|
|
||||||
|
|
||||||
## 2. 不可违反的边界
|
|
||||||
|
|
||||||
- 不得直接修改或 push `main`;
|
|
||||||
- 不得使用 `--force` 或改写共享分支历史;
|
|
||||||
- 不得创建生产 Tag;
|
|
||||||
- 不得切换生产流量,除非 AIOS Owner 后续明确批准;
|
|
||||||
- 不得提交 Token、密码、API Key、私钥、签名材料或 Secret 内容;
|
|
||||||
- 不得在日志中执行 `env`、`set`、`printenv` 或输出 Secret 文件内容;
|
|
||||||
- 不得修改 AIOS 唯一控制面、多租户或审批边界;
|
|
||||||
- 发现架构、安全或跨模块阻塞时必须升级 Codex;
|
|
||||||
- 未生成 claim/heartbeat 证据前不得标记为运行中;
|
|
||||||
- 未通过验收前不得标记为完成。
|
|
||||||
|
|
||||||
## 3. 第一阶段:执行器发现与领取
|
|
||||||
|
|
||||||
### 3.1 发现执行器入口
|
|
||||||
|
|
||||||
使用当前环境中已经注册和批准的 OpenClaw 执行入口。可检查:
|
|
||||||
|
|
||||||
- systemd 服务;
|
|
||||||
- 容器或 Compose 服务;
|
|
||||||
- 已安装的 OpenClaw CLI;
|
|
||||||
- AIOS Agent Runner;
|
|
||||||
- 已配置的队列消费者或调度器。
|
|
||||||
|
|
||||||
不得臆造不存在的命令、服务名或路径。
|
|
||||||
|
|
||||||
如果找不到可执行入口,立即创建 Blocker:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"blocker_id": "BLK-EXECUTOR-ENTRYPOINT-MISSING",
|
|
||||||
"severity": "high",
|
|
||||||
"description": "环境中未发现已注册的 OpenClaw 执行器入口",
|
|
||||||
"owner": "AIOS 运维负责人",
|
|
||||||
"exit_condition": "提供或启用已批准的 OpenClaw systemd、容器、CLI 或 Agent Runner 入口"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
状态更新为 `blocked`,不得伪报已经启动。
|
|
||||||
|
|
||||||
### 3.2 启动执行器
|
|
||||||
|
|
||||||
发现入口后:
|
|
||||||
|
|
||||||
1. 验证配置来源和运行用户;
|
|
||||||
2. 启动或恢复执行器;
|
|
||||||
3. 验证进程/容器处于运行状态;
|
|
||||||
4. 验证执行器能够读取本任务分支;
|
|
||||||
5. 立即写入 `.agent/claim.json`;
|
|
||||||
6. 更新 `.agent/status.json`:`queued -> claimed -> running`;
|
|
||||||
7. 写入首次 `.agent/heartbeat.json`;
|
|
||||||
8. 将证据提交到任务分支并同步 Gitea。
|
|
||||||
|
|
||||||
`claim.json` 至少包含:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"schema": "aios.v2.agent-claim.v1",
|
|
||||||
"task_id": "AIOS-V2R1-PRODUCTION-DEPLOY-20260720",
|
|
||||||
"executor": "openclaw",
|
|
||||||
"task_branch": "task/openclaw/AIOS-V2R1-PRODUCTION-DEPLOY-20260720",
|
|
||||||
"candidate_branch": "integration/aios-v2r1-cto-production-candidate-20260720",
|
|
||||||
"candidate_commit": "<领取时实际 Candidate OID>",
|
|
||||||
"claimed_at": "<UTC ISO-8601>",
|
|
||||||
"executor_instance": "<非敏感实例标识>"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
Heartbeat 间隔不得超过 300 秒。
|
|
||||||
|
|
||||||
## 4. 第二阶段:同步最新 Candidate
|
|
||||||
|
|
||||||
### 4.1 获取最新引用
|
|
||||||
|
|
||||||
只更新 remote-tracking ref,不得 fetch 到已经被 Worktree 检出的本地 Branch Ref。
|
|
||||||
|
|
||||||
执行原则:
|
|
||||||
|
|
||||||
```text
|
|
||||||
fetch remote candidate -> refs/remotes/<remote>/<candidate>
|
|
||||||
```
|
|
||||||
|
|
||||||
不得使用会触发以下错误的流程:
|
|
||||||
|
|
||||||
```text
|
|
||||||
refusing to fetch into branch checked out at worktree
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4.2 更新任务分支
|
|
||||||
|
|
||||||
将最新 Candidate 合并到 OpenClaw 任务分支:
|
|
||||||
|
|
||||||
- 使用普通 merge 或团队批准的线性更新方式;
|
|
||||||
- 不得 force reset 已经提交的 Agent Evidence;
|
|
||||||
- 发生冲突时停止自动扩大修改范围;
|
|
||||||
- 文档、任务载荷或架构冲突升级 Codex;
|
|
||||||
- 合并完成后把 `.agent/task.json` 与 `.agent/status.json` 中的 `candidate_commit` 更新为实际 Candidate OID。
|
|
||||||
|
|
||||||
### 4.3 必须记录
|
|
||||||
|
|
||||||
在 `.agent/progress.json` 记录:
|
|
||||||
|
|
||||||
- Candidate 本地 OID;
|
|
||||||
- Candidate Gitea OID;
|
|
||||||
- Task Branch 合并前后 OID;
|
|
||||||
- 修改文件;
|
|
||||||
- 是否发生冲突;
|
|
||||||
- 合并 Commit。
|
|
||||||
|
|
||||||
## 5. 第三阶段:确认 Gitea 同步
|
|
||||||
|
|
||||||
需要验证以下分支:
|
|
||||||
|
|
||||||
```text
|
|
||||||
integration/aios-v2r1-cto-production-candidate-20260720
|
|
||||||
task/openclaw/AIOS-V2R1-PRODUCTION-DEPLOY-20260720
|
|
||||||
task/codex/AIOS-V2R1-PRODUCTION-SUPERVISOR-20260720
|
|
||||||
```
|
|
||||||
|
|
||||||
目标仓库:
|
|
||||||
|
|
||||||
```text
|
|
||||||
g.cnnetmkt.top/git/aios/aios
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5.1 同步规则
|
|
||||||
|
|
||||||
- 使用环境中已配置的 Gitea Remote 和凭据引用;
|
|
||||||
- 不在 Evidence 中记录带凭据的 Remote URL;
|
|
||||||
- 禁止 Force Push;
|
|
||||||
- Candidate 和 Task Branch 可以普通 Push;
|
|
||||||
- Push 失败时记录 stderr 的脱敏摘要,不得记录凭据。
|
|
||||||
|
|
||||||
### 5.2 OID 验证
|
|
||||||
|
|
||||||
每个分支必须同时取得:
|
|
||||||
|
|
||||||
```text
|
|
||||||
local_oid
|
|
||||||
gitea_oid
|
|
||||||
match = true/false
|
|
||||||
```
|
|
||||||
|
|
||||||
验收条件:
|
|
||||||
|
|
||||||
```text
|
|
||||||
local_oid == gitea_oid
|
|
||||||
```
|
|
||||||
|
|
||||||
建议通过 remote-tracking ref 或 `git ls-remote --heads` 取得远端 OID。输出中不得泄露认证信息。
|
|
||||||
|
|
||||||
如果 Push 或 OID 校验失败,创建 Blocker:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"blocker_id": "BLK-GITEA-SYNC",
|
|
||||||
"severity": "high",
|
|
||||||
"description": "本地与 Gitea 分支 OID 不一致或无推送权限",
|
|
||||||
"owner": "GitOps/Gitea 运维负责人",
|
|
||||||
"exit_condition": "三个目标分支均完成普通推送且 local_oid 等于 gitea_oid"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
在 Gitea OID 一致前,不得继续预发布部署。
|
|
||||||
|
|
||||||
## 6. 第四阶段:配置 Secret 引用
|
|
||||||
|
|
||||||
### 6.1 需要解析的配置
|
|
||||||
|
|
||||||
至少确认:
|
|
||||||
|
|
||||||
```text
|
|
||||||
AIOS_OPENCLAW_URL
|
|
||||||
AIOS_OPENCLAW_CREDENTIAL_REF
|
|
||||||
AIOS_OPENCLAW_SIGNING_MATERIAL_FILE
|
|
||||||
AIOS_OPENCLAW_SIGNING_KEY_ID
|
|
||||||
```
|
|
||||||
|
|
||||||
其中:
|
|
||||||
|
|
||||||
- `AIOS_OPENCLAW_URL` 是经过批准的 Runtime Endpoint;
|
|
||||||
- `AIOS_OPENCLAW_CREDENTIAL_REF` 只保存 Secret Provider 引用;
|
|
||||||
- `AIOS_OPENCLAW_SIGNING_MATERIAL_FILE` 指向仓库外的受保护文件;
|
|
||||||
- `AIOS_OPENCLAW_SIGNING_KEY_ID` 是 Key 标识,不是 Key 内容。
|
|
||||||
|
|
||||||
### 6.2 Secret 来源优先级
|
|
||||||
|
|
||||||
按以下顺序查找:
|
|
||||||
|
|
||||||
1. 已批准的运行时 Secret Store;
|
|
||||||
2. 已批准的 Credential Provider;
|
|
||||||
3. systemd Credential;
|
|
||||||
4. 只读 Secret Volume;
|
|
||||||
5. 仓库外受保护文件。
|
|
||||||
|
|
||||||
不得:
|
|
||||||
|
|
||||||
- 从 Git 历史恢复 Secret;
|
|
||||||
- 从日志、Shell History 或旧 Evidence 复制 Secret;
|
|
||||||
- 新建明文 Secret 并提交;
|
|
||||||
- 把 Secret 内容写入 `.env` 模板或任务 JSON。
|
|
||||||
|
|
||||||
### 6.3 配置文件与权限
|
|
||||||
|
|
||||||
生产候选环境文件:
|
|
||||||
|
|
||||||
```text
|
|
||||||
/etc/aios-v2/aios-v2.env
|
|
||||||
```
|
|
||||||
|
|
||||||
建议权限:
|
|
||||||
|
|
||||||
```text
|
|
||||||
owner: root
|
|
||||||
group: aios
|
|
||||||
mode: 0640 或更严格
|
|
||||||
```
|
|
||||||
|
|
||||||
签名材料文件:
|
|
||||||
|
|
||||||
```text
|
|
||||||
位于仓库外
|
|
||||||
owner: aios 或经批准的运行用户
|
|
||||||
mode: 0600 或更严格
|
|
||||||
```
|
|
||||||
|
|
||||||
只验证以下内容:
|
|
||||||
|
|
||||||
- 引用非空;
|
|
||||||
- 文件存在;
|
|
||||||
- 运行用户可读;
|
|
||||||
- 权限不宽松;
|
|
||||||
- 文件不在 Git Worktree;
|
|
||||||
- 不输出文件内容或长度特征。
|
|
||||||
|
|
||||||
### 6.4 缺少 Secret 时的行为
|
|
||||||
|
|
||||||
若无法从已批准来源解析任一必需配置,创建:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"blocker_id": "BLK-SECRET-REF-MISSING",
|
|
||||||
"severity": "high",
|
|
||||||
"description": "缺少已批准的 OpenClaw Runtime Endpoint、Credential Ref、Signing File 或 Key ID",
|
|
||||||
"owner": "AIOS Secret/运维负责人",
|
|
||||||
"exit_condition": "四项配置均可从批准来源解析,且权限检查通过"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
不得要求用户在聊天或 Git 中提供 Secret 明文。
|
|
||||||
|
|
||||||
## 7. 第五阶段:候选验收与预发布
|
|
||||||
|
|
||||||
在 Gitea OID 和 Secret 引用检查都通过后,依次执行:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python3 tools/aios_v2_docs_code_check.py \
|
|
||||||
--base-ref integration/aios-v2r1-cto-production-candidate-20260720
|
|
||||||
|
|
||||||
python3 -m unittest discover \
|
|
||||||
-s tests/aios_v2_candidate \
|
|
||||||
-p 'test_*.py' \
|
|
||||||
-v
|
|
||||||
|
|
||||||
bash deploy/aios-v2/acceptance/run-all.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
部署后执行:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
AIOS_RUN_LIVE_HEALTHCHECK=1 \
|
|
||||||
bash deploy/aios-v2/acceptance/run-all.sh
|
|
||||||
|
|
||||||
bash deploy/aios-v2/healthcheck/check.sh
|
|
||||||
```
|
|
||||||
|
|
||||||
必须验证:
|
|
||||||
|
|
||||||
- AIOS HTTP Health;
|
|
||||||
- SQLite `quick_check=ok`;
|
|
||||||
- GitOps Controller detached + clean;
|
|
||||||
- OpenClaw execute/status/cancel/timeout/health;
|
|
||||||
- Hermes claim/heartbeat/complete/fail/lease recovery;
|
|
||||||
- 服务重启后 Tenant/Agent/Task/Audit/Idempotency 恢复;
|
|
||||||
- 回滚后健康检查通过;
|
|
||||||
- 日志与 Evidence 无 Secret。
|
|
||||||
|
|
||||||
## 8. Bug 修复规则
|
|
||||||
|
|
||||||
OpenClaw 可自行修复:
|
|
||||||
|
|
||||||
- systemd、Docker、Compose、路径和权限;
|
|
||||||
- 环境变量名或启动参数;
|
|
||||||
- Healthcheck;
|
|
||||||
- 明确的小型 Runtime Adapter 兼容问题;
|
|
||||||
- 日志、错误消息和超时边界;
|
|
||||||
- 测试、部署或中文文档中的明确错误。
|
|
||||||
|
|
||||||
每个修复必须:
|
|
||||||
|
|
||||||
1. 创建 `patch/bugfix/*`;
|
|
||||||
2. 记录复现步骤;
|
|
||||||
3. 添加或更新测试;
|
|
||||||
4. 修改代码;
|
|
||||||
5. 同步 `docs/aios-v2/` 和追踪矩阵;
|
|
||||||
6. 执行文档门禁和相关测试;
|
|
||||||
7. 提交 Git;
|
|
||||||
8. 更新 `.agent/blockers.json` 和 `.agent/progress.json`;
|
|
||||||
9. 提交 Codex 审查。
|
|
||||||
|
|
||||||
以下问题必须升级 Codex:
|
|
||||||
|
|
||||||
- Control Plane、Task Lifecycle、Policy 或 Approval 变化;
|
|
||||||
- 多租户或身份边界变化;
|
|
||||||
- OpenClaw/Hermes Contract 不兼容变化;
|
|
||||||
- 数据迁移、持久化或回滚策略变化;
|
|
||||||
- 高风险安全问题;
|
|
||||||
- 需要修改 ADR 的问题;
|
|
||||||
- 超出低风险修复范围的阻塞。
|
|
||||||
|
|
||||||
## 9. 证据文件
|
|
||||||
|
|
||||||
任务分支必须持续维护:
|
|
||||||
|
|
||||||
```text
|
|
||||||
.agent/claim.json
|
|
||||||
.agent/heartbeat.json
|
|
||||||
.agent/progress.json
|
|
||||||
.agent/blockers.json
|
|
||||||
.agent/result.json
|
|
||||||
.agent/status.json
|
|
||||||
```
|
|
||||||
|
|
||||||
`result.json` 至少包含:
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"schema": "aios.v2.agent-result.v1",
|
|
||||||
"task_id": "AIOS-V2R1-PRODUCTION-DEPLOY-20260720",
|
|
||||||
"final_status": "complete|failed|blocked",
|
|
||||||
"candidate_commit": "<OID>",
|
|
||||||
"task_commit": "<OID>",
|
|
||||||
"gitea_sync": {
|
|
||||||
"candidate_match": true,
|
|
||||||
"openclaw_task_match": true,
|
|
||||||
"codex_task_match": true
|
|
||||||
},
|
|
||||||
"secret_reference_status": "configured|blocked",
|
|
||||||
"executor_status": "running|blocked",
|
|
||||||
"test_status": "passed|failed|not_run",
|
|
||||||
"deployment_status": "passed|failed|not_run",
|
|
||||||
"rollback_status": "passed|failed|not_run",
|
|
||||||
"documentation_sync_status": "passed|failed",
|
|
||||||
"changed_files": [],
|
|
||||||
"commits": [],
|
|
||||||
"open_blockers": [],
|
|
||||||
"remaining_risks": [],
|
|
||||||
"completed_at": "<UTC ISO-8601>"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## 10. 完成条件
|
|
||||||
|
|
||||||
只有以下条件全部满足,任务才可标记为 `complete`:
|
|
||||||
|
|
||||||
- OpenClaw 执行器已启动并持续 Heartbeat;
|
|
||||||
- 已领取任务;
|
|
||||||
- 任务分支已合并最新 Candidate;
|
|
||||||
- Candidate、OpenClaw Task、Codex Task 在本地与 Gitea 的 OID 一致;
|
|
||||||
- Secret 引用和受保护文件权限检查通过;
|
|
||||||
- 文档—代码检查通过;
|
|
||||||
- 候选测试通过;
|
|
||||||
- 预发布部署和在线健康检查通过;
|
|
||||||
- Runtime 联调、重启和回滚演练通过;
|
|
||||||
- 所有修改、Blocker、Commit、测试和结果已回写;
|
|
||||||
- Codex 已能读取并审查完整证据。
|
|
||||||
|
|
||||||
如果缺少外部权限、执行入口或已批准 Secret,任务必须标记为 `blocked`,并准确说明 Owner 与解除条件,不得伪报完成。
|
|
||||||
|
|
||||||
## 11. Owner Gate
|
|
||||||
|
|
||||||
完成本指令不等于自动批准生产发布。以下动作仍需 AIOS Owner 明确批准:
|
|
||||||
|
|
||||||
- 合并 `main`;
|
|
||||||
- 创建生产 Tag;
|
|
||||||
- 正式生产部署;
|
|
||||||
- 生产流量切换;
|
|
||||||
- 接受高风险剩余问题。
|
|
||||||
@@ -1,23 +0,0 @@
|
|||||||
{
|
|
||||||
"schema": "aios.v2.credential-status.v1",
|
|
||||||
"task_id": "AIOS-V2R1-PRODUCTION-DEPLOY-20260720",
|
|
||||||
"credential_name": "openclaw-deploy-cli",
|
|
||||||
"credential_type": "gitea-api-token-reference",
|
|
||||||
"endpoints": [
|
|
||||||
"http://127.0.0.1:3000",
|
|
||||||
"https://g.cnnetmkt.top/git"
|
|
||||||
],
|
|
||||||
"reported_scope": "all",
|
|
||||||
"reported_status": "working",
|
|
||||||
"verification_source": "owner-reported",
|
|
||||||
"secret_value_stored": false,
|
|
||||||
"security_status": "rotation_required_before_production",
|
|
||||||
"reason": "令牌值曾在聊天中以明文出现,必须按已暴露凭据处理",
|
|
||||||
"required_actions": [
|
|
||||||
"立即撤销当前令牌",
|
|
||||||
"创建新令牌并使用最小必要作用域",
|
|
||||||
"仅通过 Secret Provider、systemd Credential 或受保护文件向 OpenClaw 提供",
|
|
||||||
"更新 AIOS_OPENCLAW_CREDENTIAL_REF,不在 Git、日志或 Evidence 中保存令牌值",
|
|
||||||
"旋转后重新验证本地与外部 Gitea 端点"
|
|
||||||
]
|
|
||||||
}
|
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
{
|
||||||
|
"contract_commit": "cbe33ecab8e9794ba5f318bea43aeaf447a56fbc",
|
||||||
|
"created_at": "2026-07-17T14:13:38+00:00",
|
||||||
|
"schema_version": 1,
|
||||||
|
"state_writer": "openclaw",
|
||||||
|
"work_units": [
|
||||||
|
{
|
||||||
|
"fallback": "codex",
|
||||||
|
"gate": "R0_RUNNER_LIVE_RECOVERY",
|
||||||
|
"id": "R0-DISCOVER-SECRET-PROVIDER",
|
||||||
|
"primary": "openclaw",
|
||||||
|
"status": "READY",
|
||||||
|
"track": "R_RUNNER"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fallback": "codex",
|
||||||
|
"gate": "P2_REAL_B3_EVIDENCE",
|
||||||
|
"id": "P2-INVENTORY-REAL-EVIDENCE",
|
||||||
|
"primary": "openclaw",
|
||||||
|
"status": "READY",
|
||||||
|
"track": "P_PERSONAL_WEIXIN"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"fallback": "openclaw",
|
||||||
|
"gate": "P1_REPAIR3_CODE",
|
||||||
|
"id": "P1-POST-CONTRACT-REACK",
|
||||||
|
"primary": "codex",
|
||||||
|
"status": "READY",
|
||||||
|
"track": "P_PERSONAL_WEIXIN"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -0,0 +1,157 @@
|
|||||||
|
{
|
||||||
|
"adaptive_dispatch": {
|
||||||
|
"codex_first": [
|
||||||
|
"complex_code",
|
||||||
|
"security_boundary",
|
||||||
|
"architecture",
|
||||||
|
"high_risk_review",
|
||||||
|
"strong_model_materially_better"
|
||||||
|
],
|
||||||
|
"default_orchestrator": "openclaw",
|
||||||
|
"dual_required": [
|
||||||
|
"release_gate",
|
||||||
|
"crypto_or_identity_change",
|
||||||
|
"production_migration_or_rollback"
|
||||||
|
],
|
||||||
|
"max_attempts_per_work_unit": 2,
|
||||||
|
"on_timeout": [
|
||||||
|
"mark_TIMEOUT",
|
||||||
|
"preserve_logs",
|
||||||
|
"switch_agent"
|
||||||
|
],
|
||||||
|
"openclaw_first": [
|
||||||
|
"quality_close",
|
||||||
|
"large_or_repetitive",
|
||||||
|
"long_running",
|
||||||
|
"infra_or_git_inventory",
|
||||||
|
"cost_sensitive"
|
||||||
|
],
|
||||||
|
"single_state_writer_lease_required": true,
|
||||||
|
"strict_role_partition": false,
|
||||||
|
"timeout_can_pass": false
|
||||||
|
},
|
||||||
|
"authoritative_state": {
|
||||||
|
"b3": "BLOCKED_EVIDENCE_REQUIRED",
|
||||||
|
"b4": "NOT_STARTED",
|
||||||
|
"baseline": "V1R4C6",
|
||||||
|
"production_release": "NOT_STARTED",
|
||||||
|
"release": "paused",
|
||||||
|
"repair3": {
|
||||||
|
"base": "8d9c6da90fc8caa69b691d243c6fefb40b6b0575",
|
||||||
|
"candidate": "PASS_CANDIDATE_PREPARED",
|
||||||
|
"commit": "e110cda9968cadc5c7b7179ec5003b05aad11ce4",
|
||||||
|
"exact4": "PASS",
|
||||||
|
"result": "49a9c2e0cc898ecf2bbd45f454e007644d7f3045"
|
||||||
|
},
|
||||||
|
"runner": {
|
||||||
|
"artifact_sha256": "85c472c05a1e55a5899b215d3cd41df53912787de529ac9cb3d5308da4158130",
|
||||||
|
"commit": "bbd27cf06acf1f589eef291eab54d4b127aa3aea",
|
||||||
|
"live": "BLOCKED_GATEWAY_SECRET_REF_UNAVAILABLE",
|
||||||
|
"offline": "PASS",
|
||||||
|
"result": "e789632e359e9232d8ae081e7c4f23ed7b615a9d"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"contract_id": "V2R1-PERSONAL-WEIXIN-PRODUCTION-EXECUTION-CONTRACT",
|
||||||
|
"facts_policy": {
|
||||||
|
"accept": [
|
||||||
|
"committed_git_tree",
|
||||||
|
"full_oid",
|
||||||
|
"worktree_bare_gitea_oid_match",
|
||||||
|
"test_log_with_rc",
|
||||||
|
"immutable_digest",
|
||||||
|
"result_only_commit",
|
||||||
|
"live_evidence"
|
||||||
|
],
|
||||||
|
"reject": [
|
||||||
|
"verbal_claim",
|
||||||
|
"readme_only",
|
||||||
|
"local_only_commit",
|
||||||
|
"environment_variable_as_secret_ref",
|
||||||
|
"compile_only",
|
||||||
|
"timeout_as_pass",
|
||||||
|
"unrelated_repo",
|
||||||
|
"fabricated_evidence"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"forbidden_git": [
|
||||||
|
"reset",
|
||||||
|
"amend",
|
||||||
|
"rebase",
|
||||||
|
"force_push",
|
||||||
|
"direct_main_push",
|
||||||
|
"direct_bare_ref_mutation",
|
||||||
|
"unapproved_gitea_db_mutation",
|
||||||
|
"reuse_accident_worktree",
|
||||||
|
"delete_live_worktree_to_clear_collision"
|
||||||
|
],
|
||||||
|
"generated_at": "2026-07-17T14:13:38+00:00",
|
||||||
|
"handoff": {
|
||||||
|
"ack": "CONTRACT_ACK=<full_contract_commit_oid>",
|
||||||
|
"advance": "independent_verification_required",
|
||||||
|
"all_agents_match": true
|
||||||
|
},
|
||||||
|
"join_gate": {
|
||||||
|
"requires": [
|
||||||
|
"RUNNER_RECOVERED",
|
||||||
|
"B3_PASS",
|
||||||
|
"B4_PASS",
|
||||||
|
"INTEGRATION_PASS"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"model_policy": {
|
||||||
|
"codex": "strongest_model_actually_available_in_runtime",
|
||||||
|
"gpt_5_6": "use_only_if_exposed",
|
||||||
|
"no_unverified_model_claims": true,
|
||||||
|
"openclaw_default": "DeepSeek V4 Flash"
|
||||||
|
},
|
||||||
|
"owner_authorization": {
|
||||||
|
"adaptive_agent_assignment_authorized": true,
|
||||||
|
"agent_substitution_authorized": true,
|
||||||
|
"autonomous_execution": true,
|
||||||
|
"may_bypass_release_controls": false,
|
||||||
|
"may_bypass_secret_controls": false,
|
||||||
|
"may_fabricate_evidence": false,
|
||||||
|
"user_step_by_step_confirmation_required": false
|
||||||
|
},
|
||||||
|
"release_gates": [
|
||||||
|
"MAIN_MERGE",
|
||||||
|
"STAGING_AND_CANARY",
|
||||||
|
"PRODUCTION"
|
||||||
|
],
|
||||||
|
"schema_version": 3,
|
||||||
|
"state_rules": [
|
||||||
|
"tracks_run_parallel",
|
||||||
|
"runner_block_does_not_block_offline_work_or_evidence_inventory",
|
||||||
|
"join_requires_both_tracks",
|
||||||
|
"failure_stops_dependents",
|
||||||
|
"production_remains_V1R4C6_paused_until_production_pass"
|
||||||
|
],
|
||||||
|
"status": "ACTIVE",
|
||||||
|
"tracks": {
|
||||||
|
"P_PERSONAL_WEIXIN": {
|
||||||
|
"b3": "BLOCKED_EVIDENCE_REQUIRED",
|
||||||
|
"b4": "NOT_STARTED",
|
||||||
|
"parallel": true,
|
||||||
|
"repair3": "PASS_CANDIDATE_PREPARED",
|
||||||
|
"repair3_commit": "e110cda9968cadc5c7b7179ec5003b05aad11ce4"
|
||||||
|
},
|
||||||
|
"R_RUNNER": {
|
||||||
|
"commit": "bbd27cf06acf1f589eef291eab54d4b127aa3aea",
|
||||||
|
"current": "BLOCKED_GATEWAY_SECRET_REF_UNAVAILABLE",
|
||||||
|
"fallback": "codex",
|
||||||
|
"parallel": true,
|
||||||
|
"primary": "openclaw",
|
||||||
|
"required": [
|
||||||
|
"artifact_verified",
|
||||||
|
"approved_secret_provider_contract",
|
||||||
|
"owner_probe_verified",
|
||||||
|
"service_healthy",
|
||||||
|
"heartbeat_healthy",
|
||||||
|
"gateway_auth",
|
||||||
|
"durable_task_create_read_cancel",
|
||||||
|
"execution_channel",
|
||||||
|
"live_rollback"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
{
|
||||||
|
"b3": "BLOCKED_EVIDENCE_REQUIRED",
|
||||||
|
"b4": "NOT_STARTED",
|
||||||
|
"baseline": "V1R4C6",
|
||||||
|
"contract_branch": "task/governance/v2r1-personal-weixin-production-contract-20260717",
|
||||||
|
"contract_commit": "cbe33ecab8e9794ba5f318bea43aeaf447a56fbc",
|
||||||
|
"dispatch_commit": null,
|
||||||
|
"production_release": "NOT_STARTED",
|
||||||
|
"release": "paused",
|
||||||
|
"repair3": "PASS_CANDIDATE_PREPARED",
|
||||||
|
"runner": "BLOCKED_GATEWAY_SECRET_REF_UNAVAILABLE",
|
||||||
|
"schema_version": 1,
|
||||||
|
"single_state_writer": "openclaw",
|
||||||
|
"updated_at": "2026-07-17T14:13:38+00:00"
|
||||||
|
}
|
||||||
@@ -1,18 +0,0 @@
|
|||||||
{
|
|
||||||
"schema": "aios.v2.agent-status.v1",
|
|
||||||
"task_id": "AIOS-V2R1-PRODUCTION-DEPLOY-20260720",
|
|
||||||
"executor": "openclaw",
|
|
||||||
"state": "queued",
|
|
||||||
"current_step": "等待外部 OpenClaw 执行器领取",
|
|
||||||
"progress_completed": 0,
|
|
||||||
"progress_total": 8,
|
|
||||||
"candidate_commit": "634f6b164f5b20372ab9ec8144bbf76df97de335",
|
|
||||||
"task_branch_commit": "c6e7edcf54ef48c827a690ce9987284507eb173b",
|
|
||||||
"claimed_at": null,
|
|
||||||
"heartbeat_at": null,
|
|
||||||
"last_commit": null,
|
|
||||||
"test_status": "not_run",
|
|
||||||
"blockers": [],
|
|
||||||
"result": null,
|
|
||||||
"updated_at": "2026-07-20T00:00:00Z"
|
|
||||||
}
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
{
|
|
||||||
"schema": "aios.v2.agent-task.v1",
|
|
||||||
"task_id": "AIOS-V2R1-PRODUCTION-DEPLOY-20260720",
|
|
||||||
"display_name": "AIOS V2R1 OpenClaw 预发布部署与小型缺陷修复任务",
|
|
||||||
"language": "zh-CN",
|
|
||||||
"executor": "openclaw",
|
|
||||||
"role": "deployment-operator",
|
|
||||||
"state": "queued",
|
|
||||||
"candidate_branch": "integration/aios-v2r1-cto-production-candidate-20260720",
|
|
||||||
"candidate_commit": "634f6b164f5b20372ab9ec8144bbf76df97de335",
|
|
||||||
"canonical_task": "tasks/openclaw-production-deploy-task.yaml",
|
|
||||||
"supervisor_task": "task/codex/AIOS-V2R1-PRODUCTION-SUPERVISOR-20260720",
|
|
||||||
"claim_required": true,
|
|
||||||
"heartbeat_interval_seconds": 300,
|
|
||||||
"evidence_directory": ".agent/evidence",
|
|
||||||
"instructions": [
|
|
||||||
"领取前读取候选分支完整中文文档和生产候选报告",
|
|
||||||
"领取后写入 .agent/claim.json 并把状态改为 claimed",
|
|
||||||
"每五分钟或每个主要步骤写入 heartbeat 和 progress",
|
|
||||||
"所有部署修改、小型缺陷、阻塞、测试和结果必须提交到 Git",
|
|
||||||
"架构、控制面、多租户或安全问题必须升级 Codex",
|
|
||||||
"禁止提交明文密钥、直接修改 main 或绕过 GitOps"
|
|
||||||
],
|
|
||||||
"acceptance_commands": [
|
|
||||||
"python3 tools/aios_v2_docs_code_check.py --base-ref integration/aios-v2r1-cto-production-candidate-20260720",
|
|
||||||
"python3 -m unittest discover -s tests/aios_v2_candidate -p test_*.py -v",
|
|
||||||
"bash deploy/aios-v2/acceptance/run-all.sh",
|
|
||||||
"bash deploy/aios-v2/healthcheck/check.sh"
|
|
||||||
],
|
|
||||||
"created_at": "2026-07-20T00:00:00Z"
|
|
||||||
}
|
|
||||||
@@ -0,0 +1,69 @@
|
|||||||
|
# Codex task: apply and verify AIOS Orchestrator Gateway v1
|
||||||
|
|
||||||
|
Contract: `cbe33ecab8e9794ba5f318bea43aeaf447a56fbc`
|
||||||
|
|
||||||
|
Use the strongest actually available Codex model for the initial security review.
|
||||||
|
If unavailable or timed out, record TIMEOUT/NOT_EXECUTED and let OpenClaw run
|
||||||
|
the deterministic test path. Do not count a timeout as PASS.
|
||||||
|
|
||||||
|
## Apply
|
||||||
|
|
||||||
|
1. Create a new clean append-only branch and isolated worktree.
|
||||||
|
2. Validate the delivery tarball and `SHA256SUMS`.
|
||||||
|
3. Run the external `scripts/apply_gateway_patch.sh` with `SOURCE_DIR` and
|
||||||
|
`WORKTREE`; do not alter unrelated paths.
|
||||||
|
4. Review all added code, especially:
|
||||||
|
- exact contract and optimistic state-version checks;
|
||||||
|
- event/decision idempotency;
|
||||||
|
- SQLite leases and retry behavior;
|
||||||
|
- Workspace Agent `202` asynchronous callback flow;
|
||||||
|
- Responses background creation and polling;
|
||||||
|
- file-inbox and Git-ref polling;
|
||||||
|
- traversal/symlink protections;
|
||||||
|
- OpenClaw lifecycle synchronization;
|
||||||
|
- systemd sandbox;
|
||||||
|
- install and rollback transactions.
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
Run from the final source tree:
|
||||||
|
|
||||||
|
- Python compile for all gateway and helper files;
|
||||||
|
- unit tests;
|
||||||
|
- JSON schema/config parsing;
|
||||||
|
- Bash syntax;
|
||||||
|
- `systemd-analyze verify`;
|
||||||
|
- event duplicate and payload-conflict tests;
|
||||||
|
- stale state-version rejection;
|
||||||
|
- lease/double-claim test;
|
||||||
|
- Workspace trigger failure and Responses fallback;
|
||||||
|
- response queued/in-progress/completed polling;
|
||||||
|
- file-inbox recovery after HTTP failure;
|
||||||
|
- Git-ref initial baseline and subsequent-change event;
|
||||||
|
- task start/end and project summary synchronization;
|
||||||
|
- dispatch traversal and symlink rejection;
|
||||||
|
- retry/backoff/dead-letter;
|
||||||
|
- install dry-run;
|
||||||
|
- rollback test in an isolated fixture;
|
||||||
|
- `git diff --check`;
|
||||||
|
- full repository tests;
|
||||||
|
- final Gitleaks.
|
||||||
|
|
||||||
|
## Commit and review
|
||||||
|
|
||||||
|
1. Create an append-only implementation commit.
|
||||||
|
2. Push without force.
|
||||||
|
3. Confirm `WORKTREE_OID == BARE_OID == GITEA_OID`.
|
||||||
|
4. Create an independent RESULT-only branch and exact-one result commit.
|
||||||
|
5. Record the source bundle SHA-256, implementation OID, RESULT OID, tests,
|
||||||
|
Gitleaks, systemd validation and remaining external configuration blockers.
|
||||||
|
|
||||||
|
## Forbidden
|
||||||
|
|
||||||
|
No reset, amend, rebase, force push, direct main push, controller mutation,
|
||||||
|
secret disclosure, fabricated test result, production deployment or B3 evidence
|
||||||
|
fabrication.
|
||||||
|
|
||||||
|
Success:
|
||||||
|
|
||||||
|
`GATEWAY_CODE_REVIEW=PASS IMPLEMENTATION=<oid> RESULT=<oid> OID_MATCH=YES`
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
# Codex V2R1 adaptive execution
|
||||||
|
CONTRACT_ACK=cbe33ecab8e9794ba5f318bea43aeaf447a56fbc
|
||||||
|
Read `.agent/policies/V2R1-PERSONAL-WEIXIN-PRODUCTION-EXECUTION-CONTRACT.yaml`.
|
||||||
|
Execute P1-POST-CONTRACT-REACK now:
|
||||||
|
- read-only verify Repair3 e110cda9968cadc5c7b7179ec5003b05aad11ce4
|
||||||
|
- read-only verify RESULT 49a9c2e0cc898ecf2bbd45f454e007644d7f3045
|
||||||
|
- verify lineage from 8d9c6da90fc8caa69b691d243c6fefb40b6b0575, exact4, tests, Gitleaks and RESULT-only scope
|
||||||
|
- do not modify Repair3 code in this unit
|
||||||
|
Act as fallback for OpenClaw only on timeout/unavailability. Use GPT-5.6 only if actually exposed; otherwise strongest visible model. Prefer OpenClaw when quality is comparable or volume is high; prefer Codex for complex/security-critical work.
|
||||||
|
Forbidden: reset, amend, rebase, force push, direct main push, controller mutation, secret disclosure, fabricated evidence, skipped gates.
|
||||||
|
Report:
|
||||||
|
CONTRACT=cbe33ecab8e9794ba5f318bea43aeaf447a56fbc WORK_UNIT=<id> TRACK=<id> GATE=<id> AGENT=codex STATUS=<status> IMPLEMENTATION=<oid-or-NONE> RESULT=<oid-or-NONE> OID_MATCH=<YES-or-NO> BLOCKER=<reason-or-NONE>
|
||||||
|
Keep BASELINE=V1R4C6 RELEASE=paused PRODUCTION_RELEASE=NOT_STARTED until production gate passes.
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
# OpenClaw task: deploy and supervise AIOS Orchestrator Gateway v1
|
||||||
|
|
||||||
|
Contract: `cbe33ecab8e9794ba5f318bea43aeaf447a56fbc`
|
||||||
|
|
||||||
|
You are the default orchestrator and deployment supervisor.
|
||||||
|
|
||||||
|
## Preconditions
|
||||||
|
|
||||||
|
- Codex has applied and independently reviewed this source bundle on a clean,
|
||||||
|
append-only branch.
|
||||||
|
- Implementation and RESULT OIDs are identical across worktree, bare repo and
|
||||||
|
Gitea.
|
||||||
|
- Final Gitleaks and repository tests are PASS.
|
||||||
|
- No force push, reset, amend or rebase occurred.
|
||||||
|
|
||||||
|
## Responsibilities
|
||||||
|
|
||||||
|
1. Validate the source bundle SHA-256 and `SHA256SUMS`.
|
||||||
|
2. Review the generated systemd service, poll timer, tmpfiles and sysusers files.
|
||||||
|
3. Install through `scripts/install_aios_orchestrator_gateway.sh`.
|
||||||
|
4. Configure one or both central-model channels:
|
||||||
|
- published Workspace Agent API channel with `agtch_...` trigger ID and a
|
||||||
|
Workspace Agent access token stored outside Git; or
|
||||||
|
- Responses API fallback with an approved model and API key stored outside Git.
|
||||||
|
5. Publish the callback endpoint securely and configure the Workspace Agent
|
||||||
|
action from `workspace-agent/orchestrator-decision-action.openapi.yaml`.
|
||||||
|
6. Confirm that the Workspace Agent instructions match
|
||||||
|
`workspace-agent/CENTRAL_AGENT_INSTRUCTIONS.md`.
|
||||||
|
7. Enable both:
|
||||||
|
- the continuously running gateway service;
|
||||||
|
- the independent `aios-orchestrator-gateway-poll.timer`.
|
||||||
|
8. Verify HTTP event ingestion, file-inbox polling, Git-ref polling,
|
||||||
|
Workspace Agent trigger acceptance, callback decision, Responses background
|
||||||
|
polling fallback, dispatch queue generation, retry leases and loop guards.
|
||||||
|
9. Verify OpenClaw receives and synchronizes:
|
||||||
|
- TASK_DISPATCHED
|
||||||
|
- TASK_STARTED
|
||||||
|
- TASK_ENDED
|
||||||
|
- PROJECT_SUMMARY
|
||||||
|
- PROJECT_COMPLETED
|
||||||
|
10. Run a harmless bootstrap work unit through OpenClaw and Codex and confirm
|
||||||
|
duplicate event submission is idempotent.
|
||||||
|
11. Run rollback, reinstall and heartbeat/health verification.
|
||||||
|
12. Create one RESULT-only review recording exact commits, artifact digest,
|
||||||
|
tests, systemd security, polling recovery, lifecycle synchronization and
|
||||||
|
rollback.
|
||||||
|
|
||||||
|
## Fail-closed boundaries
|
||||||
|
|
||||||
|
- Do not put Workspace Agent tokens, API keys, callback tokens or gateway
|
||||||
|
credentials in Git, logs, command arguments or RESULT files.
|
||||||
|
- Do not claim the gateway resolves the existing B3 credential/raw-wire blocker.
|
||||||
|
- Do not advance B4 or production merely because this gateway is healthy.
|
||||||
|
- Do not bypass owner/admin production release approvals.
|
||||||
|
- Do not use reset, amend, rebase, force push, direct main push or bare-ref edits.
|
||||||
|
|
||||||
|
## Required final status
|
||||||
|
|
||||||
|
Success:
|
||||||
|
|
||||||
|
`ORCHESTRATOR_GATEWAY_LIVE=PASS POLLING=PASS OPENCLAW_SYNC=PASS ROLLBACK=PASS`
|
||||||
|
|
||||||
|
Blocker:
|
||||||
|
|
||||||
|
`ORCHESTRATOR_GATEWAY_LIVE=BLOCKED reason=<stable_reason>`
|
||||||
|
|
||||||
|
Keep `BASELINE=V1R4C6 RELEASE=paused PRODUCTION_RELEASE=NOT_STARTED` until the
|
||||||
|
existing production contract advances all release gates.
|
||||||
+13
@@ -0,0 +1,13 @@
|
|||||||
|
# OpenClaw V2R1 adaptive orchestration
|
||||||
|
CONTRACT_ACK=cbe33ecab8e9794ba5f318bea43aeaf447a56fbc
|
||||||
|
Read `.agent/policies/V2R1-PERSONAL-WEIXIN-PRODUCTION-EXECUTION-CONTRACT.yaml`.
|
||||||
|
You are default orchestrator and sole authoritative state writer.
|
||||||
|
Execute immediately:
|
||||||
|
1. R0-DISCOVER-SECRET-PROVIDER: discover approved provider metadata and owner-probe source without reading or outputting secrets. Environment variables alone are not approved secret refs.
|
||||||
|
2. P2-INVENTORY-REAL-EVIDENCE: inventory approved restricted B3 evidence locations; validate existing facts; never synthesize evidence.
|
||||||
|
3. Independently verify Codex results before advancing.
|
||||||
|
Prefer OpenClaw for comparable-quality, large, repetitive, long-running, infrastructure and cost-sensitive work. Prefer Codex strongest actually available model for complex code, security boundaries, architecture and high-risk review. On timeout mark TIMEOUT and switch agent; never infer PASS.
|
||||||
|
Forbidden: reset, amend, rebase, force push, direct main push, controller mutation, secret disclosure, fabricated evidence, skipped gates.
|
||||||
|
Report:
|
||||||
|
CONTRACT=cbe33ecab8e9794ba5f318bea43aeaf447a56fbc WORK_UNIT=<id> TRACK=<id> GATE=<id> AGENT=openclaw STATUS=<status> COMMIT=<oid-or-NONE> BLOCKER=<reason-or-NONE>
|
||||||
|
Keep BASELINE=V1R4C6 RELEASE=paused PRODUCTION_RELEASE=NOT_STARTED until production gate passes.
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
# AIOS Codex automatic command watcher
|
||||||
|
|
||||||
|
The watcher polls the governed `commands/active` queue without a chat trigger. It uses SQLite transactions for exclusive leases, records the exact command digest, rejects contract drift, retries transient failures, dead-letters exhausted work, and resumes from the last durable phase so an end-event retry does not execute a work unit twice.
|
||||||
|
|
||||||
|
Commands are JSON documents validated by `schemas/codex-command.schema.json`. `acceptance_probe` is the only built-in action and has no external side effect. Production `codex` work requires an administrator-provisioned absolute executor path; the instruction is passed on standard input, never argv or environment. The executor receives a sanitized environment and must obtain credentials through its approved file or exec provider. If the executor is absent, the task fails closed and follows the retry/dead-letter policy.
|
||||||
|
|
||||||
|
The watcher provides at-most-once execution. It writes the `EXECUTING` phase before launching an external executor; if the host dies after an external effect but before a result is durably recorded, recovery does not replay the work and instead dead-letters the ambiguity for human review. Start and end event JSON is persisted before transmission, so response-loss retries reuse the exact event ID and payload and are deduplicated by the Gateway.
|
||||||
|
|
||||||
|
Lifecycle evidence is appended to `/var/log/aios-codex-watcher/audit.jsonl` as `CODEX_AUTO_CLAIM`, `TASK_STARTED`, `TASK_EXECUTED`, `TASK_ENDED`, and `EVENT_POSTED`. Inputs, tokens, stdout, and stderr are not logged; only result digests are retained.
|
||||||
|
|
||||||
|
Install from a reviewed commit with `sudo scripts/install_aios_codex_watcher.sh`. The installer creates a content-addressed release, atomically updates `current`, writes a receipt, and enables the five-second systemd timer. The existing Gateway inbound token file is referenced in place and never copied.
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
# AIOS Orchestrator Gateway v0.2
|
||||||
|
|
||||||
|
This component turns the current passive GPT orchestration pattern into a
|
||||||
|
durable, unattended loop:
|
||||||
|
|
||||||
|
```text
|
||||||
|
Gitea/OpenClaw/Codex event
|
||||||
|
-> HTTP event endpoint or polled file inbox
|
||||||
|
-> durable SQLite state machine
|
||||||
|
-> Workspace Agent API trigger
|
||||||
|
-> strict callback decision
|
||||||
|
-> Responses API background fallback and polling
|
||||||
|
-> OpenClaw/Codex dispatch queues
|
||||||
|
-> start/end/result events
|
||||||
|
-> next orchestration cycle
|
||||||
|
```
|
||||||
|
|
||||||
|
## Reliability mechanisms
|
||||||
|
|
||||||
|
The gateway does not depend on one timer, webhook or model call:
|
||||||
|
|
||||||
|
- HTTP event ingestion.
|
||||||
|
- Periodic file-inbox polling for missed webhooks and offline producers.
|
||||||
|
- Local bare-Git ref polling for `main`, governance, patch and result refs.
|
||||||
|
- Workspace Agent retries with a stable event `Idempotency-Key`.
|
||||||
|
- Workspace Agent callback timeout and Responses API background fallback.
|
||||||
|
- Responses status polling until completion or fail-closed terminal status.
|
||||||
|
- SQLite optimistic state versions, idempotency records and processing leases.
|
||||||
|
- Exponential retry with a maximum retry count and dead-letter state.
|
||||||
|
- Same-blocker and maximum-hop guards to stop non-progress loops.
|
||||||
|
- A systemd timer runs `--poll-once` as an independent reliability watchdog.
|
||||||
|
|
||||||
|
Workspace Agent trigger runs are asynchronous and do not expose a retrievable
|
||||||
|
final response. The central Workspace Agent must therefore call the
|
||||||
|
`submit_orchestration_decision` action.
|
||||||
|
|
||||||
|
## OpenClaw synchronization
|
||||||
|
|
||||||
|
Every lifecycle stage is durably written to the OpenClaw synchronization queue:
|
||||||
|
|
||||||
|
- `TASK_DISPATCHED`
|
||||||
|
- `TASK_STARTED`
|
||||||
|
- `TASK_ENDED`
|
||||||
|
- `DECISION_APPLIED`
|
||||||
|
- `PROJECT_SUMMARY`
|
||||||
|
- `PROJECT_COMPLETED`
|
||||||
|
|
||||||
|
An optional authenticated OpenClaw HTTP endpoint can receive the same records.
|
||||||
|
A failed HTTP delivery remains in the lifecycle retry queue. OpenClaw remains
|
||||||
|
the default project-status synchronizer and authoritative progress presenter.
|
||||||
|
|
||||||
|
## Agent execution contract
|
||||||
|
|
||||||
|
Each dispatched JSON envelope contains:
|
||||||
|
|
||||||
|
- exact contract commit;
|
||||||
|
- source event;
|
||||||
|
- clean base commit;
|
||||||
|
- primary and fallback agent;
|
||||||
|
- allowed paths;
|
||||||
|
- required checks;
|
||||||
|
- forbidden operations;
|
||||||
|
- completion event callback format.
|
||||||
|
|
||||||
|
OpenClaw and Codex must send a `work_unit_started` event before execution and a
|
||||||
|
`work_unit_completed`, `work_unit_blocked`, or `work_unit_failed` event when
|
||||||
|
they stop. This guarantees that OpenClaw can synchronize every task start,
|
||||||
|
finish, and overall project summary.
|
||||||
|
|
||||||
|
## Security
|
||||||
|
|
||||||
|
- Model and callback credentials are read only from root-owned secret files.
|
||||||
|
- Credentials never enter Git, dispatch payloads or logs.
|
||||||
|
- The gateway runs as a dedicated unprivileged user.
|
||||||
|
- Dispatch and sync paths are protected against traversal and symlink targets.
|
||||||
|
- All state changes require the exact contract commit and current state version.
|
||||||
|
- Production evidence, credentials and approvals stay fail-closed.
|
||||||
|
- Git history operations remain controlled by the committed governance contract.
|
||||||
|
|
||||||
|
## Deployment sequence
|
||||||
|
|
||||||
|
1. Apply this source bundle to a clean append-only branch.
|
||||||
|
2. Run unit, repository, shell, JSON, systemd and Gitleaks checks.
|
||||||
|
3. Configure one or both model channels:
|
||||||
|
- published Workspace Agent API channel and Workspace Agent access token;
|
||||||
|
- Responses API key and approved fallback model.
|
||||||
|
4. Publish the decision callback as a Workspace Agent action or MCP tool.
|
||||||
|
5. Install the service through `scripts/install_aios_orchestrator_gateway.sh`.
|
||||||
|
6. Run `scripts/verify_aios_orchestrator_gateway.sh`.
|
||||||
|
7. Submit a harmless event and verify:
|
||||||
|
- trigger acceptance;
|
||||||
|
- strict decision callback;
|
||||||
|
- dispatch file;
|
||||||
|
- OpenClaw task-start/task-end/project-summary synchronization;
|
||||||
|
- duplicate idempotency;
|
||||||
|
- polling recovery after suppressing the HTTP webhook.
|
||||||
|
|
||||||
|
The installer copies an explicit gateway-only allowlist into a release directory
|
||||||
|
named by its SHA-256 digest, atomically updates `/opt/aios-orchestrator-gateway/current`,
|
||||||
|
and persists `/var/lib/aios-orchestrator-gateway/install-receipt.json` with the
|
||||||
|
current and previous release. `--dry-run` performs source validation and emits
|
||||||
|
`AIOS_ORCHESTRATOR_INSTALL_DRY_RUN=PASS` without changing installed state.
|
||||||
|
|
||||||
|
Rollback reads that root-owned `0600` receipt, verifies both release directory
|
||||||
|
names and `SOURCE_DIGEST` files, coordinates the gateway and poll units, and
|
||||||
|
atomically switches `current` to the previous release. Repeating the same
|
||||||
|
rollback is a no-op and emits `NOOP_ALREADY_ROLLED_BACK`.
|
||||||
@@ -1,25 +0,0 @@
|
|||||||
"""AIOS-managed Hermes runtime boundary."""
|
|
||||||
|
|
||||||
from .adapter import (
|
|
||||||
HermesAdapter,
|
|
||||||
HermesConflict,
|
|
||||||
HermesNotFound,
|
|
||||||
HermesRuntimeError,
|
|
||||||
InMemoryHermesStore,
|
|
||||||
)
|
|
||||||
from .contract import HermesHealth, HermesJob, HermesJobState, HermesResult
|
|
||||||
from .worker import HermesWorker, WorkerRunResult
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"HermesAdapter",
|
|
||||||
"HermesConflict",
|
|
||||||
"HermesHealth",
|
|
||||||
"HermesJob",
|
|
||||||
"HermesJobState",
|
|
||||||
"HermesNotFound",
|
|
||||||
"HermesResult",
|
|
||||||
"HermesRuntimeError",
|
|
||||||
"HermesWorker",
|
|
||||||
"InMemoryHermesStore",
|
|
||||||
"WorkerRunResult",
|
|
||||||
]
|
|
||||||
@@ -1,437 +0,0 @@
|
|||||||
"""Tenant-partitioned Hermes queue and worker lease boundary."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import threading
|
|
||||||
from dataclasses import replace
|
|
||||||
from datetime import datetime, timedelta, timezone
|
|
||||||
from typing import Iterable, Mapping
|
|
||||||
|
|
||||||
from .contract import (
|
|
||||||
ALLOWED_TRANSITIONS,
|
|
||||||
TERMINAL_STATES,
|
|
||||||
HermesHealth,
|
|
||||||
HermesJob,
|
|
||||||
HermesJobState,
|
|
||||||
HermesResult,
|
|
||||||
utc_now,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class HermesRuntimeError(RuntimeError):
|
|
||||||
"""Base Hermes runtime-boundary failure."""
|
|
||||||
|
|
||||||
|
|
||||||
class HermesNotFound(HermesRuntimeError):
|
|
||||||
"""A tenant-scoped job was not found."""
|
|
||||||
|
|
||||||
|
|
||||||
class HermesConflict(HermesRuntimeError):
|
|
||||||
"""A state, lease, idempotency, or revision conflict occurred."""
|
|
||||||
|
|
||||||
|
|
||||||
class InMemoryHermesStore:
|
|
||||||
"""Thread-safe reference store; production backends implement this contract."""
|
|
||||||
|
|
||||||
def __init__(self) -> None:
|
|
||||||
self.lock = threading.RLock()
|
|
||||||
self.jobs: dict[str, HermesJob] = {}
|
|
||||||
self.results: dict[str, HermesResult] = {}
|
|
||||||
self.idempotency: dict[tuple[str, str], tuple[str, str]] = {}
|
|
||||||
self.worker_heartbeats: dict[str, tuple[str, str | None]] = {}
|
|
||||||
|
|
||||||
|
|
||||||
class HermesAdapter:
|
|
||||||
"""Queue and reconcile Hermes jobs without owning AIOS policy."""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
store: InMemoryHermesStore | None = None,
|
|
||||||
*,
|
|
||||||
runtime_id: str = "hermes-runtime",
|
|
||||||
runtime_version: str = "v1",
|
|
||||||
default_lease_seconds: int = 60,
|
|
||||||
) -> None:
|
|
||||||
if default_lease_seconds < 5:
|
|
||||||
raise ValueError("default_lease_seconds must be >= 5")
|
|
||||||
self.store = store or InMemoryHermesStore()
|
|
||||||
self.runtime_id = runtime_id
|
|
||||||
self.runtime_version = runtime_version
|
|
||||||
self.default_lease_seconds = int(default_lease_seconds)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _digest(job: HermesJob) -> str:
|
|
||||||
raw = json.dumps(
|
|
||||||
{
|
|
||||||
"tenant_id": job.tenant_id,
|
|
||||||
"cell_id": job.cell_id,
|
|
||||||
"profile_id": job.profile_id,
|
|
||||||
"task_id": job.task_id,
|
|
||||||
"trace_id": job.trace_id,
|
|
||||||
"action": job.action,
|
|
||||||
"payload": dict(job.payload),
|
|
||||||
"priority": job.priority,
|
|
||||||
},
|
|
||||||
ensure_ascii=False,
|
|
||||||
sort_keys=True,
|
|
||||||
separators=(",", ":"),
|
|
||||||
default=str,
|
|
||||||
).encode("utf-8")
|
|
||||||
return hashlib.sha256(raw).hexdigest()
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _parse(value: str | None) -> datetime | None:
|
|
||||||
if value is None:
|
|
||||||
return None
|
|
||||||
normalized = value[:-1] + "+00:00" if value.endswith("Z") else value
|
|
||||||
parsed = datetime.fromisoformat(normalized)
|
|
||||||
if parsed.tzinfo is None:
|
|
||||||
raise ValueError("timestamp must include timezone")
|
|
||||||
return parsed.astimezone(timezone.utc)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _format(value: datetime) -> str:
|
|
||||||
return value.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _assert_tenant(job: HermesJob, tenant_id: str) -> None:
|
|
||||||
if job.tenant_id != tenant_id:
|
|
||||||
raise HermesNotFound("job not found")
|
|
||||||
|
|
||||||
def enqueue(self, job: HermesJob) -> HermesJob:
|
|
||||||
if job.state != HermesJobState.QUEUED or job.revision != 1:
|
|
||||||
raise HermesConflict("new job must be QUEUED at revision 1")
|
|
||||||
digest = self._digest(job)
|
|
||||||
idem_ref = (job.tenant_id, job.idempotency_key)
|
|
||||||
with self.store.lock:
|
|
||||||
existing = self.store.jobs.get(job.job_id)
|
|
||||||
if existing is not None:
|
|
||||||
if self._digest(existing) != digest:
|
|
||||||
raise HermesConflict("job_id reused with different content")
|
|
||||||
return existing
|
|
||||||
prior = self.store.idempotency.get(idem_ref)
|
|
||||||
if prior is not None:
|
|
||||||
prior_id, prior_digest = prior
|
|
||||||
if prior_digest != digest:
|
|
||||||
raise HermesConflict("idempotency key reused with different content")
|
|
||||||
return self.store.jobs[prior_id]
|
|
||||||
self.store.jobs[job.job_id] = job
|
|
||||||
self.store.idempotency[idem_ref] = (job.job_id, digest)
|
|
||||||
return job
|
|
||||||
|
|
||||||
def get(self, job_id: str, *, tenant_id: str) -> HermesJob:
|
|
||||||
with self.store.lock:
|
|
||||||
job = self.store.jobs.get(job_id)
|
|
||||||
if job is None:
|
|
||||||
raise HermesNotFound("job not found")
|
|
||||||
self._assert_tenant(job, tenant_id)
|
|
||||||
return job
|
|
||||||
|
|
||||||
def list_jobs(
|
|
||||||
self,
|
|
||||||
*,
|
|
||||||
tenant_id: str,
|
|
||||||
cell_id: str | None = None,
|
|
||||||
profile_id: str | None = None,
|
|
||||||
states: Iterable[HermesJobState | str] | None = None,
|
|
||||||
) -> list[HermesJob]:
|
|
||||||
state_filter = {HermesJobState(item) for item in states} if states else None
|
|
||||||
with self.store.lock:
|
|
||||||
self._requeue_expired_locked()
|
|
||||||
jobs = [
|
|
||||||
job
|
|
||||||
for job in self.store.jobs.values()
|
|
||||||
if job.tenant_id == tenant_id
|
|
||||||
and (cell_id is None or job.cell_id == cell_id)
|
|
||||||
and (profile_id is None or job.profile_id == profile_id)
|
|
||||||
and (state_filter is None or job.state in state_filter)
|
|
||||||
]
|
|
||||||
return sorted(jobs, key=lambda item: (item.priority, item.created_at, item.job_id))
|
|
||||||
|
|
||||||
def claim(
|
|
||||||
self,
|
|
||||||
*,
|
|
||||||
tenant_id: str,
|
|
||||||
cell_id: str,
|
|
||||||
profile_id: str,
|
|
||||||
worker_id: str,
|
|
||||||
lease_seconds: int | None = None,
|
|
||||||
) -> HermesJob | None:
|
|
||||||
worker_id = str(worker_id or "").strip()
|
|
||||||
if not worker_id:
|
|
||||||
raise ValueError("worker_id is required")
|
|
||||||
duration = int(lease_seconds or self.default_lease_seconds)
|
|
||||||
if duration < 5:
|
|
||||||
raise ValueError("lease_seconds must be >= 5")
|
|
||||||
with self.store.lock:
|
|
||||||
self._requeue_expired_locked()
|
|
||||||
candidates = [
|
|
||||||
job
|
|
||||||
for job in self.store.jobs.values()
|
|
||||||
if job.tenant_id == tenant_id
|
|
||||||
and job.cell_id == cell_id
|
|
||||||
and job.profile_id == profile_id
|
|
||||||
and job.state == HermesJobState.QUEUED
|
|
||||||
]
|
|
||||||
self.store.worker_heartbeats[worker_id] = (utc_now(), None)
|
|
||||||
if not candidates:
|
|
||||||
return None
|
|
||||||
current = sorted(
|
|
||||||
candidates,
|
|
||||||
key=lambda item: (item.priority, item.created_at, item.job_id),
|
|
||||||
)[0]
|
|
||||||
claimed = self._transition_locked(
|
|
||||||
current,
|
|
||||||
HermesJobState.CLAIMED,
|
|
||||||
assigned_worker_id=worker_id,
|
|
||||||
lease_expires_at=self._format(
|
|
||||||
datetime.now(timezone.utc) + timedelta(seconds=duration)
|
|
||||||
),
|
|
||||||
)
|
|
||||||
self.store.worker_heartbeats[worker_id] = (utc_now(), claimed.job_id)
|
|
||||||
return claimed
|
|
||||||
|
|
||||||
def start(
|
|
||||||
self,
|
|
||||||
job_id: str,
|
|
||||||
*,
|
|
||||||
tenant_id: str,
|
|
||||||
worker_id: str,
|
|
||||||
expected_revision: int | None = None,
|
|
||||||
) -> HermesJob:
|
|
||||||
with self.store.lock:
|
|
||||||
current = self._owned_job_locked(job_id, tenant_id, worker_id)
|
|
||||||
self._assert_revision(current, expected_revision)
|
|
||||||
return self._transition_locked(current, HermesJobState.RUNNING)
|
|
||||||
|
|
||||||
def heartbeat(
|
|
||||||
self,
|
|
||||||
job_id: str,
|
|
||||||
*,
|
|
||||||
tenant_id: str,
|
|
||||||
worker_id: str,
|
|
||||||
lease_seconds: int | None = None,
|
|
||||||
) -> HermesJob:
|
|
||||||
duration = int(lease_seconds or self.default_lease_seconds)
|
|
||||||
with self.store.lock:
|
|
||||||
current = self._owned_job_locked(job_id, tenant_id, worker_id)
|
|
||||||
if current.state not in {HermesJobState.CLAIMED, HermesJobState.RUNNING}:
|
|
||||||
raise HermesConflict("job does not accept heartbeat")
|
|
||||||
updated = replace(
|
|
||||||
current,
|
|
||||||
revision=current.revision + 1,
|
|
||||||
lease_expires_at=self._format(
|
|
||||||
datetime.now(timezone.utc) + timedelta(seconds=duration)
|
|
||||||
),
|
|
||||||
updated_at=utc_now(),
|
|
||||||
)
|
|
||||||
self.store.jobs[job_id] = updated
|
|
||||||
self.store.worker_heartbeats[worker_id] = (utc_now(), job_id)
|
|
||||||
return updated
|
|
||||||
|
|
||||||
def complete(
|
|
||||||
self,
|
|
||||||
job_id: str,
|
|
||||||
*,
|
|
||||||
tenant_id: str,
|
|
||||||
worker_id: str,
|
|
||||||
output: Mapping[str, object],
|
|
||||||
expected_revision: int | None = None,
|
|
||||||
) -> HermesResult:
|
|
||||||
return self._finish(
|
|
||||||
job_id,
|
|
||||||
tenant_id=tenant_id,
|
|
||||||
worker_id=worker_id,
|
|
||||||
state=HermesJobState.COMPLETED,
|
|
||||||
output=output,
|
|
||||||
expected_revision=expected_revision,
|
|
||||||
)
|
|
||||||
|
|
||||||
def fail(
|
|
||||||
self,
|
|
||||||
job_id: str,
|
|
||||||
*,
|
|
||||||
tenant_id: str,
|
|
||||||
worker_id: str,
|
|
||||||
error: Mapping[str, object],
|
|
||||||
expected_revision: int | None = None,
|
|
||||||
) -> HermesResult:
|
|
||||||
return self._finish(
|
|
||||||
job_id,
|
|
||||||
tenant_id=tenant_id,
|
|
||||||
worker_id=worker_id,
|
|
||||||
state=HermesJobState.FAILED,
|
|
||||||
error=error,
|
|
||||||
expected_revision=expected_revision,
|
|
||||||
)
|
|
||||||
|
|
||||||
def cancel(
|
|
||||||
self,
|
|
||||||
job_id: str,
|
|
||||||
*,
|
|
||||||
tenant_id: str,
|
|
||||||
expected_revision: int | None = None,
|
|
||||||
) -> HermesResult:
|
|
||||||
with self.store.lock:
|
|
||||||
current = self.store.jobs.get(job_id)
|
|
||||||
if current is None:
|
|
||||||
raise HermesNotFound("job not found")
|
|
||||||
self._assert_tenant(current, tenant_id)
|
|
||||||
self._assert_revision(current, expected_revision)
|
|
||||||
if current.state in TERMINAL_STATES:
|
|
||||||
result = self.store.results.get(job_id)
|
|
||||||
if result is None:
|
|
||||||
raise HermesConflict("terminal result missing")
|
|
||||||
return result
|
|
||||||
updated = self._transition_locked(current, HermesJobState.CANCELLED)
|
|
||||||
result = HermesResult(
|
|
||||||
job_id=updated.job_id,
|
|
||||||
tenant_id=updated.tenant_id,
|
|
||||||
task_id=updated.task_id,
|
|
||||||
trace_id=updated.trace_id,
|
|
||||||
state=HermesJobState.CANCELLED,
|
|
||||||
worker_id=updated.assigned_worker_id,
|
|
||||||
error={"code": "cancelled", "message": "cancelled by AIOS"},
|
|
||||||
finished_at=utc_now(),
|
|
||||||
)
|
|
||||||
self.store.results[job_id] = result
|
|
||||||
return result
|
|
||||||
|
|
||||||
def result(self, job_id: str, *, tenant_id: str) -> HermesResult:
|
|
||||||
with self.store.lock:
|
|
||||||
job = self.store.jobs.get(job_id)
|
|
||||||
if job is None:
|
|
||||||
raise HermesNotFound("job not found")
|
|
||||||
self._assert_tenant(job, tenant_id)
|
|
||||||
result = self.store.results.get(job_id)
|
|
||||||
if result is None:
|
|
||||||
raise HermesConflict("job has no terminal result")
|
|
||||||
return result
|
|
||||||
|
|
||||||
def health(self) -> HermesHealth:
|
|
||||||
with self.store.lock:
|
|
||||||
self._requeue_expired_locked()
|
|
||||||
active_jobs = sum(
|
|
||||||
1 for job in self.store.jobs.values()
|
|
||||||
if job.state in {HermesJobState.CLAIMED, HermesJobState.RUNNING}
|
|
||||||
)
|
|
||||||
queued_jobs = sum(
|
|
||||||
1 for job in self.store.jobs.values()
|
|
||||||
if job.state == HermesJobState.QUEUED
|
|
||||||
)
|
|
||||||
partitions = {
|
|
||||||
(job.tenant_id, job.cell_id, job.profile_id)
|
|
||||||
for job in self.store.jobs.values()
|
|
||||||
}
|
|
||||||
active_workers = sum(
|
|
||||||
1 for heartbeat, _ in self.store.worker_heartbeats.values()
|
|
||||||
if self._parse(heartbeat)
|
|
||||||
and self._parse(heartbeat) >= datetime.now(timezone.utc) - timedelta(seconds=120)
|
|
||||||
)
|
|
||||||
return HermesHealth(
|
|
||||||
runtime_id=self.runtime_id,
|
|
||||||
status="healthy",
|
|
||||||
version=self.runtime_version,
|
|
||||||
heartbeat_at=utc_now(),
|
|
||||||
active_workers=active_workers,
|
|
||||||
active_jobs=active_jobs,
|
|
||||||
queued_jobs=queued_jobs,
|
|
||||||
tenant_partitions=len(partitions),
|
|
||||||
profile_isolation_enforced=True,
|
|
||||||
details={"store": type(self.store).__name__},
|
|
||||||
)
|
|
||||||
|
|
||||||
def _finish(
|
|
||||||
self,
|
|
||||||
job_id: str,
|
|
||||||
*,
|
|
||||||
tenant_id: str,
|
|
||||||
worker_id: str,
|
|
||||||
state: HermesJobState,
|
|
||||||
output: Mapping[str, object] | None = None,
|
|
||||||
error: Mapping[str, object] | None = None,
|
|
||||||
expected_revision: int | None,
|
|
||||||
) -> HermesResult:
|
|
||||||
with self.store.lock:
|
|
||||||
current = self._owned_job_locked(job_id, tenant_id, worker_id)
|
|
||||||
self._assert_revision(current, expected_revision)
|
|
||||||
updated = self._transition_locked(current, state)
|
|
||||||
result = HermesResult(
|
|
||||||
job_id=updated.job_id,
|
|
||||||
tenant_id=updated.tenant_id,
|
|
||||||
task_id=updated.task_id,
|
|
||||||
trace_id=updated.trace_id,
|
|
||||||
state=state,
|
|
||||||
worker_id=worker_id,
|
|
||||||
output=output,
|
|
||||||
error=error,
|
|
||||||
started_at=current.updated_at,
|
|
||||||
finished_at=utc_now(),
|
|
||||||
)
|
|
||||||
self.store.results[job_id] = result
|
|
||||||
self.store.worker_heartbeats[worker_id] = (utc_now(), None)
|
|
||||||
return result
|
|
||||||
|
|
||||||
def _owned_job_locked(self, job_id: str, tenant_id: str, worker_id: str) -> HermesJob:
|
|
||||||
current = self.store.jobs.get(job_id)
|
|
||||||
if current is None:
|
|
||||||
raise HermesNotFound("job not found")
|
|
||||||
self._assert_tenant(current, tenant_id)
|
|
||||||
if current.assigned_worker_id != worker_id:
|
|
||||||
raise HermesConflict("worker does not own job lease")
|
|
||||||
return current
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _assert_revision(job: HermesJob, expected: int | None) -> None:
|
|
||||||
if expected is not None and job.revision != expected:
|
|
||||||
raise HermesConflict("job revision changed")
|
|
||||||
|
|
||||||
def _transition_locked(
|
|
||||||
self,
|
|
||||||
current: HermesJob,
|
|
||||||
target: HermesJobState,
|
|
||||||
*,
|
|
||||||
assigned_worker_id: str | None = None,
|
|
||||||
lease_expires_at: str | None = None,
|
|
||||||
) -> HermesJob:
|
|
||||||
if target not in ALLOWED_TRANSITIONS[current.state]:
|
|
||||||
raise HermesConflict(
|
|
||||||
f"invalid transition {current.state.value} -> {target.value}"
|
|
||||||
)
|
|
||||||
updated = replace(
|
|
||||||
current,
|
|
||||||
state=target,
|
|
||||||
revision=current.revision + 1,
|
|
||||||
assigned_worker_id=(
|
|
||||||
assigned_worker_id
|
|
||||||
if assigned_worker_id is not None
|
|
||||||
else current.assigned_worker_id
|
|
||||||
),
|
|
||||||
lease_expires_at=(
|
|
||||||
lease_expires_at
|
|
||||||
if lease_expires_at is not None
|
|
||||||
else (None if target in TERMINAL_STATES else current.lease_expires_at)
|
|
||||||
),
|
|
||||||
updated_at=utc_now(),
|
|
||||||
)
|
|
||||||
self.store.jobs[current.job_id] = updated
|
|
||||||
return updated
|
|
||||||
|
|
||||||
def _requeue_expired_locked(self) -> None:
|
|
||||||
now = datetime.now(timezone.utc)
|
|
||||||
for current in list(self.store.jobs.values()):
|
|
||||||
expiry = self._parse(current.lease_expires_at)
|
|
||||||
if (
|
|
||||||
current.state in {HermesJobState.CLAIMED, HermesJobState.RUNNING}
|
|
||||||
and expiry is not None
|
|
||||||
and expiry <= now
|
|
||||||
):
|
|
||||||
self.store.jobs[current.job_id] = replace(
|
|
||||||
current,
|
|
||||||
state=HermesJobState.QUEUED,
|
|
||||||
revision=current.revision + 1,
|
|
||||||
assigned_worker_id=None,
|
|
||||||
lease_expires_at=None,
|
|
||||||
updated_at=utc_now(),
|
|
||||||
)
|
|
||||||
@@ -1,153 +0,0 @@
|
|||||||
"""Tenant-scoped Hermes runtime contracts."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import uuid
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from enum import Enum
|
|
||||||
from typing import Any, Mapping
|
|
||||||
|
|
||||||
|
|
||||||
def utc_now() -> str:
|
|
||||||
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
|
|
||||||
|
|
||||||
|
|
||||||
def _required(value: str, label: str) -> str:
|
|
||||||
normalized = str(value or "").strip()
|
|
||||||
if not normalized:
|
|
||||||
raise ValueError(f"{label} is required")
|
|
||||||
return normalized
|
|
||||||
|
|
||||||
|
|
||||||
def _safe_mapping(value: Mapping[str, Any] | None) -> dict[str, Any]:
|
|
||||||
return json.loads(json.dumps(dict(value or {}), ensure_ascii=False, sort_keys=True, default=str))
|
|
||||||
|
|
||||||
|
|
||||||
class HermesJobState(str, Enum):
|
|
||||||
QUEUED = "QUEUED"
|
|
||||||
CLAIMED = "CLAIMED"
|
|
||||||
RUNNING = "RUNNING"
|
|
||||||
COMPLETED = "COMPLETED"
|
|
||||||
FAILED = "FAILED"
|
|
||||||
CANCELLED = "CANCELLED"
|
|
||||||
|
|
||||||
|
|
||||||
ALLOWED_TRANSITIONS: Mapping[HermesJobState, frozenset[HermesJobState]] = {
|
|
||||||
HermesJobState.QUEUED: frozenset({HermesJobState.CLAIMED, HermesJobState.CANCELLED}),
|
|
||||||
HermesJobState.CLAIMED: frozenset({HermesJobState.RUNNING, HermesJobState.CANCELLED}),
|
|
||||||
HermesJobState.RUNNING: frozenset(
|
|
||||||
{HermesJobState.COMPLETED, HermesJobState.FAILED, HermesJobState.CANCELLED}
|
|
||||||
),
|
|
||||||
HermesJobState.COMPLETED: frozenset(),
|
|
||||||
HermesJobState.FAILED: frozenset(),
|
|
||||||
HermesJobState.CANCELLED: frozenset(),
|
|
||||||
}
|
|
||||||
TERMINAL_STATES = frozenset(
|
|
||||||
{HermesJobState.COMPLETED, HermesJobState.FAILED, HermesJobState.CANCELLED}
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class HermesJob:
|
|
||||||
job_id: str
|
|
||||||
tenant_id: str
|
|
||||||
cell_id: str
|
|
||||||
profile_id: str
|
|
||||||
task_id: str
|
|
||||||
trace_id: str
|
|
||||||
action: str
|
|
||||||
payload: Mapping[str, Any] = field(default_factory=dict)
|
|
||||||
idempotency_key: str = ""
|
|
||||||
priority: int = 100
|
|
||||||
state: HermesJobState = HermesJobState.QUEUED
|
|
||||||
revision: int = 1
|
|
||||||
assigned_worker_id: str | None = None
|
|
||||||
lease_expires_at: str | None = None
|
|
||||||
created_at: str = field(default_factory=utc_now)
|
|
||||||
updated_at: str = field(default_factory=utc_now)
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
for name in (
|
|
||||||
"job_id", "tenant_id", "cell_id", "profile_id", "task_id",
|
|
||||||
"trace_id", "action", "idempotency_key", "created_at", "updated_at",
|
|
||||||
):
|
|
||||||
object.__setattr__(self, name, _required(getattr(self, name), name))
|
|
||||||
object.__setattr__(self, "payload", _safe_mapping(self.payload))
|
|
||||||
object.__setattr__(self, "state", HermesJobState(self.state))
|
|
||||||
if not 0 <= int(self.priority) <= 1000:
|
|
||||||
raise ValueError("priority must be between 0 and 1000")
|
|
||||||
if self.revision < 1:
|
|
||||||
raise ValueError("revision must be >= 1")
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def create(
|
|
||||||
cls,
|
|
||||||
*,
|
|
||||||
tenant_id: str,
|
|
||||||
cell_id: str,
|
|
||||||
profile_id: str,
|
|
||||||
task_id: str,
|
|
||||||
trace_id: str,
|
|
||||||
action: str,
|
|
||||||
payload: Mapping[str, Any] | None,
|
|
||||||
idempotency_key: str,
|
|
||||||
priority: int = 100,
|
|
||||||
) -> "HermesJob":
|
|
||||||
return cls(
|
|
||||||
job_id=f"hermes_{uuid.uuid4().hex}",
|
|
||||||
tenant_id=tenant_id,
|
|
||||||
cell_id=cell_id,
|
|
||||||
profile_id=profile_id,
|
|
||||||
task_id=task_id,
|
|
||||||
trace_id=trace_id,
|
|
||||||
action=action,
|
|
||||||
payload=_safe_mapping(payload),
|
|
||||||
idempotency_key=idempotency_key,
|
|
||||||
priority=priority,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class HermesResult:
|
|
||||||
job_id: str
|
|
||||||
tenant_id: str
|
|
||||||
task_id: str
|
|
||||||
trace_id: str
|
|
||||||
state: HermesJobState
|
|
||||||
worker_id: str | None = None
|
|
||||||
output: Mapping[str, Any] | None = None
|
|
||||||
error: Mapping[str, Any] | None = None
|
|
||||||
started_at: str | None = None
|
|
||||||
finished_at: str | None = None
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
for name in ("job_id", "tenant_id", "task_id", "trace_id"):
|
|
||||||
object.__setattr__(self, name, _required(getattr(self, name), name))
|
|
||||||
object.__setattr__(self, "state", HermesJobState(self.state))
|
|
||||||
if self.output is not None:
|
|
||||||
object.__setattr__(self, "output", _safe_mapping(self.output))
|
|
||||||
if self.error is not None:
|
|
||||||
object.__setattr__(self, "error", _safe_mapping(self.error))
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class HermesHealth:
|
|
||||||
runtime_id: str
|
|
||||||
status: str
|
|
||||||
version: str
|
|
||||||
heartbeat_at: str
|
|
||||||
active_workers: int = 0
|
|
||||||
active_jobs: int = 0
|
|
||||||
queued_jobs: int = 0
|
|
||||||
tenant_partitions: int = 0
|
|
||||||
profile_isolation_enforced: bool = True
|
|
||||||
details: Mapping[str, Any] = field(default_factory=dict)
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
for name in ("runtime_id", "status", "version", "heartbeat_at"):
|
|
||||||
object.__setattr__(self, name, _required(getattr(self, name), name))
|
|
||||||
for name in ("active_workers", "active_jobs", "queued_jobs", "tenant_partitions"):
|
|
||||||
if int(getattr(self, name)) < 0:
|
|
||||||
raise ValueError(f"{name} must be >= 0")
|
|
||||||
object.__setattr__(self, "details", _safe_mapping(self.details))
|
|
||||||
@@ -1,155 +0,0 @@
|
|||||||
"""Hermes worker loop with lease renewal and bounded recovery."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import threading
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from typing import Any, Callable, Mapping
|
|
||||||
|
|
||||||
from .adapter import HermesAdapter, HermesConflict
|
|
||||||
from .contract import HermesJob, HermesJobState
|
|
||||||
|
|
||||||
|
|
||||||
JobHandler = Callable[[HermesJob], Mapping[str, Any]]
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class WorkerRunResult:
|
|
||||||
claimed: bool
|
|
||||||
job_id: str | None = None
|
|
||||||
state: str | None = None
|
|
||||||
error: str | None = None
|
|
||||||
|
|
||||||
|
|
||||||
class HermesWorker:
|
|
||||||
"""Poll one tenant/cell/profile partition and execute jobs safely."""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
adapter: HermesAdapter,
|
|
||||||
handler: JobHandler,
|
|
||||||
*,
|
|
||||||
worker_id: str,
|
|
||||||
tenant_id: str,
|
|
||||||
cell_id: str,
|
|
||||||
profile_id: str,
|
|
||||||
poll_seconds: float = 1.0,
|
|
||||||
heartbeat_seconds: float = 10.0,
|
|
||||||
) -> None:
|
|
||||||
if poll_seconds <= 0 or heartbeat_seconds <= 0:
|
|
||||||
raise ValueError("poll_seconds and heartbeat_seconds must be positive")
|
|
||||||
for name, value in {
|
|
||||||
"worker_id": worker_id,
|
|
||||||
"tenant_id": tenant_id,
|
|
||||||
"cell_id": cell_id,
|
|
||||||
"profile_id": profile_id,
|
|
||||||
}.items():
|
|
||||||
if not str(value or "").strip():
|
|
||||||
raise ValueError(f"{name} is required")
|
|
||||||
self.adapter = adapter
|
|
||||||
self.handler = handler
|
|
||||||
self.worker_id = worker_id
|
|
||||||
self.tenant_id = tenant_id
|
|
||||||
self.cell_id = cell_id
|
|
||||||
self.profile_id = profile_id
|
|
||||||
self.poll_seconds = float(poll_seconds)
|
|
||||||
self.heartbeat_seconds = float(heartbeat_seconds)
|
|
||||||
self._stop = threading.Event()
|
|
||||||
|
|
||||||
def stop(self) -> None:
|
|
||||||
self._stop.set()
|
|
||||||
|
|
||||||
def run_once(self) -> WorkerRunResult:
|
|
||||||
job = self.adapter.claim(
|
|
||||||
tenant_id=self.tenant_id,
|
|
||||||
cell_id=self.cell_id,
|
|
||||||
profile_id=self.profile_id,
|
|
||||||
worker_id=self.worker_id,
|
|
||||||
)
|
|
||||||
if job is None:
|
|
||||||
return WorkerRunResult(claimed=False)
|
|
||||||
|
|
||||||
running = self.adapter.start(
|
|
||||||
job.job_id,
|
|
||||||
tenant_id=self.tenant_id,
|
|
||||||
worker_id=self.worker_id,
|
|
||||||
expected_revision=job.revision,
|
|
||||||
)
|
|
||||||
|
|
||||||
heartbeat_stop = threading.Event()
|
|
||||||
heartbeat_thread = threading.Thread(
|
|
||||||
target=self._heartbeat_loop,
|
|
||||||
args=(running.job_id, heartbeat_stop),
|
|
||||||
name=f"hermes-heartbeat-{self.worker_id}",
|
|
||||||
daemon=True,
|
|
||||||
)
|
|
||||||
heartbeat_thread.start()
|
|
||||||
|
|
||||||
output: Mapping[str, Any] | None = None
|
|
||||||
execution_error: Exception | None = None
|
|
||||||
try:
|
|
||||||
output = dict(self.handler(running))
|
|
||||||
except Exception as exc: # runtime boundary: normalize below
|
|
||||||
execution_error = exc
|
|
||||||
finally:
|
|
||||||
# Stop the lease heartbeat before reading the final revision. This
|
|
||||||
# prevents a revision increment racing with complete/fail.
|
|
||||||
heartbeat_stop.set()
|
|
||||||
heartbeat_thread.join(timeout=max(self.heartbeat_seconds * 2, 1.0))
|
|
||||||
|
|
||||||
try:
|
|
||||||
current = self.adapter.get(running.job_id, tenant_id=self.tenant_id)
|
|
||||||
if execution_error is None:
|
|
||||||
result = self.adapter.complete(
|
|
||||||
running.job_id,
|
|
||||||
tenant_id=self.tenant_id,
|
|
||||||
worker_id=self.worker_id,
|
|
||||||
output=dict(output or {}),
|
|
||||||
expected_revision=current.revision,
|
|
||||||
)
|
|
||||||
return WorkerRunResult(
|
|
||||||
claimed=True,
|
|
||||||
job_id=result.job_id,
|
|
||||||
state=result.state.value,
|
|
||||||
)
|
|
||||||
|
|
||||||
result = self.adapter.fail(
|
|
||||||
running.job_id,
|
|
||||||
tenant_id=self.tenant_id,
|
|
||||||
worker_id=self.worker_id,
|
|
||||||
error={
|
|
||||||
"code": "worker_error",
|
|
||||||
"message": str(execution_error),
|
|
||||||
},
|
|
||||||
expected_revision=current.revision,
|
|
||||||
)
|
|
||||||
return WorkerRunResult(
|
|
||||||
claimed=True,
|
|
||||||
job_id=result.job_id,
|
|
||||||
state=result.state.value,
|
|
||||||
error=str(execution_error),
|
|
||||||
)
|
|
||||||
except HermesConflict as exc:
|
|
||||||
return WorkerRunResult(
|
|
||||||
claimed=True,
|
|
||||||
job_id=running.job_id,
|
|
||||||
state=HermesJobState.FAILED.value,
|
|
||||||
error=f"reconciliation_conflict: {exc}",
|
|
||||||
)
|
|
||||||
|
|
||||||
def run_forever(self) -> None:
|
|
||||||
while not self._stop.is_set():
|
|
||||||
result = self.run_once()
|
|
||||||
if not result.claimed:
|
|
||||||
self._stop.wait(self.poll_seconds)
|
|
||||||
|
|
||||||
def _heartbeat_loop(self, job_id: str, stop_event: threading.Event) -> None:
|
|
||||||
while not stop_event.wait(self.heartbeat_seconds):
|
|
||||||
try:
|
|
||||||
self.adapter.heartbeat(
|
|
||||||
job_id,
|
|
||||||
tenant_id=self.tenant_id,
|
|
||||||
worker_id=self.worker_id,
|
|
||||||
)
|
|
||||||
except Exception:
|
|
||||||
stop_event.set()
|
|
||||||
return
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
"""AIOS-managed OpenClaw execution adapter."""
|
|
||||||
|
|
||||||
from .adapter import (
|
|
||||||
AdapterError,
|
|
||||||
CommandConflict,
|
|
||||||
CommandNotFound,
|
|
||||||
CommandRejected,
|
|
||||||
InMemoryCallbackSink,
|
|
||||||
OpenClawAdapter,
|
|
||||||
)
|
|
||||||
from .contract import (
|
|
||||||
CommandState,
|
|
||||||
OpenClawCommand,
|
|
||||||
OpenClawResult,
|
|
||||||
RuntimeHealth,
|
|
||||||
sign_command,
|
|
||||||
verify_command_signature,
|
|
||||||
)
|
|
||||||
from .transport import HTTPRuntimeTransport, InProcessRuntimeTransport, RuntimeTransport
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"AdapterError",
|
|
||||||
"CommandConflict",
|
|
||||||
"CommandNotFound",
|
|
||||||
"CommandRejected",
|
|
||||||
"CommandState",
|
|
||||||
"HTTPRuntimeTransport",
|
|
||||||
"InMemoryCallbackSink",
|
|
||||||
"InProcessRuntimeTransport",
|
|
||||||
"OpenClawAdapter",
|
|
||||||
"OpenClawCommand",
|
|
||||||
"OpenClawResult",
|
|
||||||
"RuntimeHealth",
|
|
||||||
"RuntimeTransport",
|
|
||||||
"sign_command",
|
|
||||||
"verify_command_signature",
|
|
||||||
]
|
|
||||||
@@ -1,228 +0,0 @@
|
|||||||
"""Signed, tenant-isolated execution adapter for OpenClaw runtimes."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import threading
|
|
||||||
from dataclasses import replace
|
|
||||||
from typing import Callable, Mapping, Protocol
|
|
||||||
|
|
||||||
from .contract import (
|
|
||||||
CommandState,
|
|
||||||
OpenClawCommand,
|
|
||||||
OpenClawResult,
|
|
||||||
RuntimeHealth,
|
|
||||||
utc_now,
|
|
||||||
verify_command_signature,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class AdapterError(RuntimeError):
|
|
||||||
"""Base adapter failure."""
|
|
||||||
|
|
||||||
|
|
||||||
class CommandRejected(AdapterError):
|
|
||||||
"""A command failed signature or contract validation."""
|
|
||||||
|
|
||||||
|
|
||||||
class CommandNotFound(AdapterError):
|
|
||||||
"""A tenant-scoped command is not visible."""
|
|
||||||
|
|
||||||
|
|
||||||
class CommandConflict(AdapterError):
|
|
||||||
"""A command or idempotency key conflicts with an existing execution."""
|
|
||||||
|
|
||||||
|
|
||||||
class RuntimeTransport(Protocol):
|
|
||||||
def execute(self, command: OpenClawCommand, timeout_seconds: float) -> OpenClawResult: ...
|
|
||||||
def cancel(self, command_id: str) -> bool: ...
|
|
||||||
def health(self) -> RuntimeHealth: ...
|
|
||||||
|
|
||||||
|
|
||||||
class CallbackSink(Protocol):
|
|
||||||
def on_started(self, result: OpenClawResult) -> None: ...
|
|
||||||
def on_completed(self, result: OpenClawResult) -> None: ...
|
|
||||||
def on_failed(self, result: OpenClawResult) -> None: ...
|
|
||||||
|
|
||||||
|
|
||||||
class InMemoryCallbackSink:
|
|
||||||
def __init__(self) -> None:
|
|
||||||
self._lock = threading.RLock()
|
|
||||||
self.started: list[OpenClawResult] = []
|
|
||||||
self.completed: list[OpenClawResult] = []
|
|
||||||
self.failed: list[OpenClawResult] = []
|
|
||||||
|
|
||||||
def on_started(self, result: OpenClawResult) -> None:
|
|
||||||
with self._lock:
|
|
||||||
self.started.append(result)
|
|
||||||
|
|
||||||
def on_completed(self, result: OpenClawResult) -> None:
|
|
||||||
with self._lock:
|
|
||||||
self.completed.append(result)
|
|
||||||
|
|
||||||
def on_failed(self, result: OpenClawResult) -> None:
|
|
||||||
with self._lock:
|
|
||||||
self.failed.append(result)
|
|
||||||
|
|
||||||
|
|
||||||
SigningMaterialResolver = Callable[[str, str], bytes | str]
|
|
||||||
|
|
||||||
|
|
||||||
class OpenClawAdapter:
|
|
||||||
"""Verify, de-duplicate, execute, cancel and reconcile commands."""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
transport: RuntimeTransport,
|
|
||||||
signing_material_resolver: SigningMaterialResolver,
|
|
||||||
*,
|
|
||||||
callback_sink: CallbackSink | None = None,
|
|
||||||
default_timeout_seconds: float = 300.0,
|
|
||||||
) -> None:
|
|
||||||
if default_timeout_seconds <= 0:
|
|
||||||
raise ValueError("default_timeout_seconds must be positive")
|
|
||||||
self.transport = transport
|
|
||||||
self.signing_material_resolver = signing_material_resolver
|
|
||||||
self.callback_sink = callback_sink or InMemoryCallbackSink()
|
|
||||||
self.default_timeout_seconds = float(default_timeout_seconds)
|
|
||||||
self._lock = threading.RLock()
|
|
||||||
self._commands: dict[str, OpenClawCommand] = {}
|
|
||||||
self._results: dict[str, OpenClawResult] = {}
|
|
||||||
self._idempotency: dict[tuple[str, str], tuple[str, str]] = {}
|
|
||||||
|
|
||||||
def _verify(self, command: OpenClawCommand) -> None:
|
|
||||||
try:
|
|
||||||
key_material = self.signing_material_resolver(command.tenant_id, command.key_id)
|
|
||||||
except Exception as exc:
|
|
||||||
raise CommandRejected("signing material unavailable") from exc
|
|
||||||
if not verify_command_signature(command, key_material):
|
|
||||||
raise CommandRejected("invalid or expired command signature")
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _validate_result(command: OpenClawCommand, result: OpenClawResult) -> None:
|
|
||||||
if (
|
|
||||||
result.command_id != command.command_id
|
|
||||||
or result.tenant_id != command.tenant_id
|
|
||||||
or result.task_id != command.task_id
|
|
||||||
or result.trace_id != command.trace_id
|
|
||||||
):
|
|
||||||
raise AdapterError("runtime returned mismatched command identity")
|
|
||||||
if result.state not in {
|
|
||||||
CommandState.COMPLETED,
|
|
||||||
CommandState.FAILED,
|
|
||||||
CommandState.CANCELLED,
|
|
||||||
CommandState.TIMED_OUT,
|
|
||||||
}:
|
|
||||||
raise AdapterError("runtime returned a non-terminal result")
|
|
||||||
|
|
||||||
def execute(
|
|
||||||
self,
|
|
||||||
command: OpenClawCommand,
|
|
||||||
*,
|
|
||||||
timeout_seconds: float | None = None,
|
|
||||||
) -> OpenClawResult:
|
|
||||||
timeout = self.default_timeout_seconds if timeout_seconds is None else float(timeout_seconds)
|
|
||||||
if timeout <= 0:
|
|
||||||
raise ValueError("timeout_seconds must be positive")
|
|
||||||
self._verify(command)
|
|
||||||
digest = command.signature
|
|
||||||
idem_ref = (command.tenant_id, command.idempotency_key)
|
|
||||||
|
|
||||||
with self._lock:
|
|
||||||
prior_command = self._commands.get(command.command_id)
|
|
||||||
if prior_command is not None:
|
|
||||||
if prior_command.signature != command.signature:
|
|
||||||
raise CommandConflict("command_id reused with different content")
|
|
||||||
return self._results[command.command_id]
|
|
||||||
prior_idem = self._idempotency.get(idem_ref)
|
|
||||||
if prior_idem is not None:
|
|
||||||
prior_id, prior_digest = prior_idem
|
|
||||||
if prior_digest != digest:
|
|
||||||
raise CommandConflict("idempotency key reused with different content")
|
|
||||||
return self._results[prior_id]
|
|
||||||
running = OpenClawResult(
|
|
||||||
command_id=command.command_id,
|
|
||||||
tenant_id=command.tenant_id,
|
|
||||||
task_id=command.task_id,
|
|
||||||
trace_id=command.trace_id,
|
|
||||||
state=CommandState.RUNNING,
|
|
||||||
started_at=utc_now(),
|
|
||||||
)
|
|
||||||
self._commands[command.command_id] = command
|
|
||||||
self._results[command.command_id] = running
|
|
||||||
self._idempotency[idem_ref] = (command.command_id, digest)
|
|
||||||
|
|
||||||
self.callback_sink.on_started(running)
|
|
||||||
try:
|
|
||||||
result = self.transport.execute(command, timeout)
|
|
||||||
self._validate_result(command, result)
|
|
||||||
except TimeoutError as exc:
|
|
||||||
result = replace(
|
|
||||||
running,
|
|
||||||
state=CommandState.TIMED_OUT,
|
|
||||||
error={"code": "runtime_timeout", "message": str(exc) or "runtime timeout"},
|
|
||||||
finished_at=utc_now(),
|
|
||||||
)
|
|
||||||
except Exception as exc:
|
|
||||||
result = replace(
|
|
||||||
running,
|
|
||||||
state=CommandState.FAILED,
|
|
||||||
error={"code": "runtime_error", "message": str(exc)},
|
|
||||||
finished_at=utc_now(),
|
|
||||||
)
|
|
||||||
|
|
||||||
if result.started_at is None:
|
|
||||||
result = replace(result, started_at=running.started_at)
|
|
||||||
if result.finished_at is None:
|
|
||||||
result = replace(result, finished_at=utc_now())
|
|
||||||
with self._lock:
|
|
||||||
self._results[command.command_id] = result
|
|
||||||
if result.state == CommandState.COMPLETED:
|
|
||||||
self.callback_sink.on_completed(result)
|
|
||||||
else:
|
|
||||||
self.callback_sink.on_failed(result)
|
|
||||||
return result
|
|
||||||
|
|
||||||
def status(self, command_id: str, *, tenant_id: str) -> OpenClawResult:
|
|
||||||
with self._lock:
|
|
||||||
command = self._commands.get(command_id)
|
|
||||||
result = self._results.get(command_id)
|
|
||||||
if command is None or result is None or command.tenant_id != tenant_id:
|
|
||||||
raise CommandNotFound("command not found")
|
|
||||||
return result
|
|
||||||
|
|
||||||
def cancel(self, command_id: str, *, tenant_id: str) -> OpenClawResult:
|
|
||||||
with self._lock:
|
|
||||||
command = self._commands.get(command_id)
|
|
||||||
current = self._results.get(command_id)
|
|
||||||
if command is None or current is None or command.tenant_id != tenant_id:
|
|
||||||
raise CommandNotFound("command not found")
|
|
||||||
if current.state in {
|
|
||||||
CommandState.COMPLETED,
|
|
||||||
CommandState.FAILED,
|
|
||||||
CommandState.CANCELLED,
|
|
||||||
CommandState.TIMED_OUT,
|
|
||||||
CommandState.REJECTED,
|
|
||||||
}:
|
|
||||||
return current
|
|
||||||
if not self.transport.cancel(command_id):
|
|
||||||
raise AdapterError("runtime rejected cancellation")
|
|
||||||
cancelled = replace(
|
|
||||||
current,
|
|
||||||
state=CommandState.CANCELLED,
|
|
||||||
error={"code": "cancelled", "message": "cancelled by AIOS"},
|
|
||||||
finished_at=utc_now(),
|
|
||||||
)
|
|
||||||
with self._lock:
|
|
||||||
self._results[command_id] = cancelled
|
|
||||||
self.callback_sink.on_failed(cancelled)
|
|
||||||
return cancelled
|
|
||||||
|
|
||||||
def health(self) -> RuntimeHealth:
|
|
||||||
return self.transport.health()
|
|
||||||
|
|
||||||
def snapshot(self, *, tenant_id: str) -> Mapping[str, OpenClawResult]:
|
|
||||||
with self._lock:
|
|
||||||
return {
|
|
||||||
command_id: result
|
|
||||||
for command_id, result in self._results.items()
|
|
||||||
if self._commands[command_id].tenant_id == tenant_id
|
|
||||||
}
|
|
||||||
@@ -1,97 +0,0 @@
|
|||||||
"""Audit-oriented lifecycle callbacks for OpenClaw execution."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import threading
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from typing import Any, Mapping, Protocol
|
|
||||||
|
|
||||||
from .contract import OpenClawResult, utc_now
|
|
||||||
|
|
||||||
|
|
||||||
class EventPublisher(Protocol):
|
|
||||||
def publish(self, event: Mapping[str, Any]) -> None: ...
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class ExecutionEvent:
|
|
||||||
event_type: str
|
|
||||||
tenant_id: str
|
|
||||||
task_id: str
|
|
||||||
trace_id: str
|
|
||||||
command_id: str
|
|
||||||
state: str
|
|
||||||
created_at: str = field(default_factory=utc_now)
|
|
||||||
details: Mapping[str, Any] = field(default_factory=dict)
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"event_type": self.event_type,
|
|
||||||
"tenant_id": self.tenant_id,
|
|
||||||
"task_id": self.task_id,
|
|
||||||
"trace_id": self.trace_id,
|
|
||||||
"command_id": self.command_id,
|
|
||||||
"state": self.state,
|
|
||||||
"created_at": self.created_at,
|
|
||||||
"details": json.loads(
|
|
||||||
json.dumps(dict(self.details), ensure_ascii=False, default=str)
|
|
||||||
),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
class InMemoryEventPublisher:
|
|
||||||
"""Thread-safe publisher used by tests and first integration slice."""
|
|
||||||
|
|
||||||
def __init__(self) -> None:
|
|
||||||
self._lock = threading.RLock()
|
|
||||||
self.events: list[dict[str, Any]] = []
|
|
||||||
|
|
||||||
def publish(self, event: Mapping[str, Any]) -> None:
|
|
||||||
with self._lock:
|
|
||||||
self.events.append(
|
|
||||||
json.loads(json.dumps(dict(event), ensure_ascii=False, default=str))
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class AuditCallbackSink:
|
|
||||||
"""Convert runtime callbacks into tenant-scoped audit events."""
|
|
||||||
|
|
||||||
def __init__(self, publisher: EventPublisher) -> None:
|
|
||||||
self.publisher = publisher
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _event(
|
|
||||||
event_type: str,
|
|
||||||
result: OpenClawResult,
|
|
||||||
) -> ExecutionEvent:
|
|
||||||
details: dict[str, Any] = {
|
|
||||||
"runtime_id": result.runtime_id,
|
|
||||||
"attempts": result.attempts,
|
|
||||||
}
|
|
||||||
if result.error:
|
|
||||||
details["error"] = dict(result.error)
|
|
||||||
if result.output:
|
|
||||||
details["output_present"] = True
|
|
||||||
return ExecutionEvent(
|
|
||||||
event_type=event_type,
|
|
||||||
tenant_id=result.tenant_id,
|
|
||||||
task_id=result.task_id,
|
|
||||||
trace_id=result.trace_id,
|
|
||||||
command_id=result.command_id,
|
|
||||||
state=result.state.value,
|
|
||||||
details=details,
|
|
||||||
)
|
|
||||||
|
|
||||||
def on_started(self, result: OpenClawResult) -> None:
|
|
||||||
self.publisher.publish(self._event("OPENCLAW_STARTED", result).as_dict())
|
|
||||||
|
|
||||||
def on_completed(self, result: OpenClawResult) -> None:
|
|
||||||
self.publisher.publish(self._event("OPENCLAW_COMPLETED", result).as_dict())
|
|
||||||
|
|
||||||
def on_failed(self, result: OpenClawResult) -> None:
|
|
||||||
event_type = {
|
|
||||||
"CANCELLED": "OPENCLAW_CANCELLED",
|
|
||||||
"TIMED_OUT": "OPENCLAW_TIMED_OUT",
|
|
||||||
"REJECTED": "OPENCLAW_REJECTED",
|
|
||||||
}.get(result.state.value, "OPENCLAW_FAILED")
|
|
||||||
self.publisher.publish(self._event(event_type, result).as_dict())
|
|
||||||
@@ -1,213 +0,0 @@
|
|||||||
"""Versioned AIOS to OpenClaw execution contracts."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import hashlib
|
|
||||||
import hmac
|
|
||||||
import json
|
|
||||||
import uuid
|
|
||||||
from dataclasses import dataclass, field, replace
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from enum import Enum
|
|
||||||
from typing import Any, Mapping
|
|
||||||
|
|
||||||
CONTRACT_VERSION = "aios.openclaw.command.v1"
|
|
||||||
DEFAULT_CLOCK_SKEW_SECONDS = 300
|
|
||||||
|
|
||||||
|
|
||||||
def utc_now() -> str:
|
|
||||||
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
|
|
||||||
|
|
||||||
|
|
||||||
def _parse_timestamp(value: str) -> datetime:
|
|
||||||
normalized = str(value or "").strip()
|
|
||||||
if not normalized:
|
|
||||||
raise ValueError("timestamp is required")
|
|
||||||
if normalized.endswith("Z"):
|
|
||||||
normalized = normalized[:-1] + "+00:00"
|
|
||||||
parsed = datetime.fromisoformat(normalized)
|
|
||||||
if parsed.tzinfo is None:
|
|
||||||
raise ValueError("timestamp must include a timezone")
|
|
||||||
return parsed.astimezone(timezone.utc)
|
|
||||||
|
|
||||||
|
|
||||||
def _required(value: str, label: str) -> str:
|
|
||||||
normalized = str(value or "").strip()
|
|
||||||
if not normalized:
|
|
||||||
raise ValueError(f"{label} is required")
|
|
||||||
return normalized
|
|
||||||
|
|
||||||
|
|
||||||
def _safe_mapping(value: Mapping[str, Any] | None) -> dict[str, Any]:
|
|
||||||
return json.loads(json.dumps(dict(value or {}), ensure_ascii=False, sort_keys=True, default=str))
|
|
||||||
|
|
||||||
|
|
||||||
class CommandState(str, Enum):
|
|
||||||
ACCEPTED = "ACCEPTED"
|
|
||||||
RUNNING = "RUNNING"
|
|
||||||
COMPLETED = "COMPLETED"
|
|
||||||
FAILED = "FAILED"
|
|
||||||
CANCELLED = "CANCELLED"
|
|
||||||
TIMED_OUT = "TIMED_OUT"
|
|
||||||
REJECTED = "REJECTED"
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class OpenClawCommand:
|
|
||||||
command_id: str
|
|
||||||
tenant_id: str
|
|
||||||
task_id: str
|
|
||||||
trace_id: str
|
|
||||||
action: str
|
|
||||||
payload: Mapping[str, Any] = field(default_factory=dict)
|
|
||||||
idempotency_key: str = ""
|
|
||||||
created_at: str = field(default_factory=utc_now)
|
|
||||||
expires_at: str | None = None
|
|
||||||
contract_version: str = CONTRACT_VERSION
|
|
||||||
key_id: str = "default"
|
|
||||||
signature: str = ""
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
for field_name in (
|
|
||||||
"command_id", "tenant_id", "task_id", "trace_id", "action",
|
|
||||||
"idempotency_key", "created_at", "contract_version", "key_id",
|
|
||||||
):
|
|
||||||
object.__setattr__(self, field_name, _required(getattr(self, field_name), field_name))
|
|
||||||
if self.contract_version != CONTRACT_VERSION:
|
|
||||||
raise ValueError(f"unsupported contract_version: {self.contract_version}")
|
|
||||||
object.__setattr__(self, "payload", _safe_mapping(self.payload))
|
|
||||||
_parse_timestamp(self.created_at)
|
|
||||||
if self.expires_at is not None:
|
|
||||||
object.__setattr__(self, "expires_at", _required(self.expires_at, "expires_at"))
|
|
||||||
_parse_timestamp(self.expires_at)
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def create(
|
|
||||||
cls,
|
|
||||||
*,
|
|
||||||
tenant_id: str,
|
|
||||||
task_id: str,
|
|
||||||
trace_id: str,
|
|
||||||
action: str,
|
|
||||||
payload: Mapping[str, Any] | None,
|
|
||||||
idempotency_key: str,
|
|
||||||
expires_at: str | None = None,
|
|
||||||
key_id: str = "default",
|
|
||||||
command_id: str | None = None,
|
|
||||||
) -> "OpenClawCommand":
|
|
||||||
return cls(
|
|
||||||
command_id=command_id or f"cmd_{uuid.uuid4().hex}",
|
|
||||||
tenant_id=tenant_id,
|
|
||||||
task_id=task_id,
|
|
||||||
trace_id=trace_id,
|
|
||||||
action=action,
|
|
||||||
payload=_safe_mapping(payload),
|
|
||||||
idempotency_key=idempotency_key,
|
|
||||||
expires_at=expires_at,
|
|
||||||
key_id=key_id,
|
|
||||||
)
|
|
||||||
|
|
||||||
def unsigned_dict(self) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"command_id": self.command_id,
|
|
||||||
"tenant_id": self.tenant_id,
|
|
||||||
"task_id": self.task_id,
|
|
||||||
"trace_id": self.trace_id,
|
|
||||||
"action": self.action,
|
|
||||||
"payload": _safe_mapping(self.payload),
|
|
||||||
"idempotency_key": self.idempotency_key,
|
|
||||||
"created_at": self.created_at,
|
|
||||||
"expires_at": self.expires_at,
|
|
||||||
"contract_version": self.contract_version,
|
|
||||||
"key_id": self.key_id,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class OpenClawResult:
|
|
||||||
command_id: str
|
|
||||||
tenant_id: str
|
|
||||||
task_id: str
|
|
||||||
trace_id: str
|
|
||||||
state: CommandState
|
|
||||||
output: Mapping[str, Any] | None = None
|
|
||||||
error: Mapping[str, Any] | None = None
|
|
||||||
started_at: str | None = None
|
|
||||||
finished_at: str | None = None
|
|
||||||
runtime_id: str | None = None
|
|
||||||
attempts: int = 1
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
for field_name in ("command_id", "tenant_id", "task_id", "trace_id"):
|
|
||||||
object.__setattr__(self, field_name, _required(getattr(self, field_name), field_name))
|
|
||||||
object.__setattr__(self, "state", CommandState(self.state))
|
|
||||||
if self.output is not None:
|
|
||||||
object.__setattr__(self, "output", _safe_mapping(self.output))
|
|
||||||
if self.error is not None:
|
|
||||||
object.__setattr__(self, "error", _safe_mapping(self.error))
|
|
||||||
if self.started_at is not None:
|
|
||||||
_parse_timestamp(self.started_at)
|
|
||||||
if self.finished_at is not None:
|
|
||||||
_parse_timestamp(self.finished_at)
|
|
||||||
if self.attempts < 1:
|
|
||||||
raise ValueError("attempts must be >= 1")
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class RuntimeHealth:
|
|
||||||
runtime: str
|
|
||||||
runtime_id: str
|
|
||||||
status: str
|
|
||||||
version: str
|
|
||||||
heartbeat_at: str
|
|
||||||
active_tasks: int = 0
|
|
||||||
workspace_isolated: bool = False
|
|
||||||
credential_ref_configured: bool = False
|
|
||||||
details: Mapping[str, Any] = field(default_factory=dict)
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
for field_name in ("runtime", "runtime_id", "status", "version", "heartbeat_at"):
|
|
||||||
object.__setattr__(self, field_name, _required(getattr(self, field_name), field_name))
|
|
||||||
_parse_timestamp(self.heartbeat_at)
|
|
||||||
if self.active_tasks < 0:
|
|
||||||
raise ValueError("active_tasks must be >= 0")
|
|
||||||
object.__setattr__(self, "details", _safe_mapping(self.details))
|
|
||||||
|
|
||||||
|
|
||||||
def canonical_command_bytes(command: OpenClawCommand) -> bytes:
|
|
||||||
return json.dumps(
|
|
||||||
command.unsigned_dict(), ensure_ascii=False, sort_keys=True, separators=(",", ":")
|
|
||||||
).encode("utf-8")
|
|
||||||
|
|
||||||
|
|
||||||
def sign_command(command: OpenClawCommand, key_material: bytes | str) -> OpenClawCommand:
|
|
||||||
key = key_material.encode("utf-8") if isinstance(key_material, str) else bytes(key_material)
|
|
||||||
if not key:
|
|
||||||
raise ValueError("signing material is required")
|
|
||||||
signature = hmac.new(key, canonical_command_bytes(command), hashlib.sha256).hexdigest()
|
|
||||||
return replace(command, signature=signature)
|
|
||||||
|
|
||||||
|
|
||||||
def verify_command_signature(
|
|
||||||
command: OpenClawCommand,
|
|
||||||
key_material: bytes | str,
|
|
||||||
*,
|
|
||||||
now: datetime | None = None,
|
|
||||||
clock_skew_seconds: int = DEFAULT_CLOCK_SKEW_SECONDS,
|
|
||||||
) -> bool:
|
|
||||||
if not command.signature:
|
|
||||||
return False
|
|
||||||
key = key_material.encode("utf-8") if isinstance(key_material, str) else bytes(key_material)
|
|
||||||
if not key:
|
|
||||||
return False
|
|
||||||
expected = hmac.new(key, canonical_command_bytes(command), hashlib.sha256).hexdigest()
|
|
||||||
if not hmac.compare_digest(expected, command.signature):
|
|
||||||
return False
|
|
||||||
current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc)
|
|
||||||
created = _parse_timestamp(command.created_at)
|
|
||||||
if created.timestamp() - current.timestamp() > clock_skew_seconds:
|
|
||||||
return False
|
|
||||||
if command.expires_at is not None:
|
|
||||||
expires = _parse_timestamp(command.expires_at)
|
|
||||||
if current.timestamp() - expires.timestamp() > clock_skew_seconds:
|
|
||||||
return False
|
|
||||||
return True
|
|
||||||
@@ -1,103 +0,0 @@
|
|||||||
"""Health policy and self-healing decisions for OpenClaw runtimes."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from typing import Mapping
|
|
||||||
|
|
||||||
from .contract import RuntimeHealth
|
|
||||||
|
|
||||||
|
|
||||||
def _parse(value: str) -> datetime:
|
|
||||||
normalized = value[:-1] + "+00:00" if value.endswith("Z") else value
|
|
||||||
parsed = datetime.fromisoformat(normalized)
|
|
||||||
if parsed.tzinfo is None:
|
|
||||||
raise ValueError("heartbeat timestamp must include timezone")
|
|
||||||
return parsed.astimezone(timezone.utc)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class HealthPolicy:
|
|
||||||
max_heartbeat_age_seconds: int = 90
|
|
||||||
require_workspace_isolation: bool = True
|
|
||||||
require_credential_ref: bool = True
|
|
||||||
accepted_statuses: frozenset[str] = field(
|
|
||||||
default_factory=lambda: frozenset({"healthy", "ready"})
|
|
||||||
)
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
if self.max_heartbeat_age_seconds < 5:
|
|
||||||
raise ValueError("max_heartbeat_age_seconds must be >= 5")
|
|
||||||
if not self.accepted_statuses:
|
|
||||||
raise ValueError("accepted_statuses must not be empty")
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class HealthAssessment:
|
|
||||||
healthy: bool
|
|
||||||
reasons: tuple[str, ...]
|
|
||||||
recoverable: bool
|
|
||||||
recommended_action: str
|
|
||||||
details: Mapping[str, object] = field(default_factory=dict)
|
|
||||||
|
|
||||||
|
|
||||||
class RuntimeHealthMonitor:
|
|
||||||
"""Evaluate runtime health and recommend bounded recovery actions."""
|
|
||||||
|
|
||||||
def __init__(self, policy: HealthPolicy | None = None) -> None:
|
|
||||||
self.policy = policy or HealthPolicy()
|
|
||||||
|
|
||||||
def assess(
|
|
||||||
self,
|
|
||||||
health: RuntimeHealth,
|
|
||||||
*,
|
|
||||||
now: datetime | None = None,
|
|
||||||
) -> HealthAssessment:
|
|
||||||
current = (now or datetime.now(timezone.utc)).astimezone(timezone.utc)
|
|
||||||
heartbeat = _parse(health.heartbeat_at)
|
|
||||||
age = max(0.0, current.timestamp() - heartbeat.timestamp())
|
|
||||||
reasons: list[str] = []
|
|
||||||
|
|
||||||
if health.status.lower() not in self.policy.accepted_statuses:
|
|
||||||
reasons.append("runtime_status_unhealthy")
|
|
||||||
if age > self.policy.max_heartbeat_age_seconds:
|
|
||||||
reasons.append("heartbeat_stale")
|
|
||||||
if self.policy.require_workspace_isolation and not health.workspace_isolated:
|
|
||||||
reasons.append("workspace_not_isolated")
|
|
||||||
if self.policy.require_credential_ref and not health.credential_ref_configured:
|
|
||||||
reasons.append("credential_reference_missing")
|
|
||||||
|
|
||||||
if not reasons:
|
|
||||||
return HealthAssessment(
|
|
||||||
healthy=True,
|
|
||||||
reasons=(),
|
|
||||||
recoverable=True,
|
|
||||||
recommended_action="none",
|
|
||||||
details={"heartbeat_age_seconds": age},
|
|
||||||
)
|
|
||||||
|
|
||||||
non_recoverable = {
|
|
||||||
"workspace_not_isolated",
|
|
||||||
"credential_reference_missing",
|
|
||||||
}.intersection(reasons)
|
|
||||||
if non_recoverable:
|
|
||||||
action = "quarantine_runtime"
|
|
||||||
recoverable = False
|
|
||||||
elif "heartbeat_stale" in reasons:
|
|
||||||
action = "restart_runtime"
|
|
||||||
recoverable = True
|
|
||||||
else:
|
|
||||||
action = "drain_and_restart"
|
|
||||||
recoverable = True
|
|
||||||
|
|
||||||
return HealthAssessment(
|
|
||||||
healthy=False,
|
|
||||||
reasons=tuple(reasons),
|
|
||||||
recoverable=recoverable,
|
|
||||||
recommended_action=action,
|
|
||||||
details={
|
|
||||||
"heartbeat_age_seconds": age,
|
|
||||||
"runtime_id": health.runtime_id,
|
|
||||||
"active_tasks": health.active_tasks,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
@@ -1,214 +0,0 @@
|
|||||||
"""Runtime transport implementations for the OpenClaw adapter."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import threading
|
|
||||||
import urllib.error
|
|
||||||
import urllib.parse
|
|
||||||
import urllib.request
|
|
||||||
from typing import Any, Callable, Mapping, Protocol
|
|
||||||
|
|
||||||
from .contract import (
|
|
||||||
CommandState,
|
|
||||||
OpenClawCommand,
|
|
||||||
OpenClawResult,
|
|
||||||
RuntimeHealth,
|
|
||||||
utc_now,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class RuntimeTransport(Protocol):
|
|
||||||
def execute(self, command: OpenClawCommand, timeout_seconds: float) -> OpenClawResult: ...
|
|
||||||
def cancel(self, command_id: str) -> bool: ...
|
|
||||||
def health(self) -> RuntimeHealth: ...
|
|
||||||
|
|
||||||
|
|
||||||
class TransportError(RuntimeError):
|
|
||||||
"""Runtime transport or protocol failure."""
|
|
||||||
|
|
||||||
|
|
||||||
RuntimeHandler = Callable[[OpenClawCommand], Mapping[str, Any] | OpenClawResult]
|
|
||||||
|
|
||||||
|
|
||||||
class InProcessRuntimeTransport:
|
|
||||||
"""Deterministic runtime used by tests and controlled local integrations."""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
handler: RuntimeHandler,
|
|
||||||
*,
|
|
||||||
runtime_id: str = "openclaw-in-process",
|
|
||||||
version: str = "v1",
|
|
||||||
workspace_isolated: bool = True,
|
|
||||||
credential_ref_configured: bool = True,
|
|
||||||
) -> None:
|
|
||||||
self.handler = handler
|
|
||||||
self.runtime_id = runtime_id
|
|
||||||
self.version = version
|
|
||||||
self.workspace_isolated = workspace_isolated
|
|
||||||
self.credential_ref_configured = credential_ref_configured
|
|
||||||
self._lock = threading.RLock()
|
|
||||||
self._running: set[str] = set()
|
|
||||||
self._cancelled: set[str] = set()
|
|
||||||
|
|
||||||
def execute(self, command: OpenClawCommand, timeout_seconds: float) -> OpenClawResult:
|
|
||||||
if timeout_seconds <= 0:
|
|
||||||
raise ValueError("timeout_seconds must be positive")
|
|
||||||
with self._lock:
|
|
||||||
if command.command_id in self._cancelled:
|
|
||||||
return OpenClawResult(
|
|
||||||
command_id=command.command_id,
|
|
||||||
tenant_id=command.tenant_id,
|
|
||||||
task_id=command.task_id,
|
|
||||||
trace_id=command.trace_id,
|
|
||||||
state=CommandState.CANCELLED,
|
|
||||||
error={"code": "cancelled", "message": "cancelled before execution"},
|
|
||||||
finished_at=utc_now(),
|
|
||||||
runtime_id=self.runtime_id,
|
|
||||||
)
|
|
||||||
self._running.add(command.command_id)
|
|
||||||
try:
|
|
||||||
raw = self.handler(command)
|
|
||||||
if isinstance(raw, OpenClawResult):
|
|
||||||
return raw
|
|
||||||
if not isinstance(raw, Mapping):
|
|
||||||
raise TransportError("runtime handler must return a mapping or OpenClawResult")
|
|
||||||
return OpenClawResult(
|
|
||||||
command_id=command.command_id,
|
|
||||||
tenant_id=command.tenant_id,
|
|
||||||
task_id=command.task_id,
|
|
||||||
trace_id=command.trace_id,
|
|
||||||
state=CommandState(str(raw.get("state", "COMPLETED"))),
|
|
||||||
output=raw.get("output"),
|
|
||||||
error=raw.get("error"),
|
|
||||||
started_at=str(raw.get("started_at") or utc_now()),
|
|
||||||
finished_at=str(raw.get("finished_at") or utc_now()),
|
|
||||||
runtime_id=str(raw.get("runtime_id") or self.runtime_id),
|
|
||||||
attempts=int(raw.get("attempts", 1)),
|
|
||||||
)
|
|
||||||
finally:
|
|
||||||
with self._lock:
|
|
||||||
self._running.discard(command.command_id)
|
|
||||||
|
|
||||||
def cancel(self, command_id: str) -> bool:
|
|
||||||
with self._lock:
|
|
||||||
self._cancelled.add(command_id)
|
|
||||||
return command_id in self._running
|
|
||||||
|
|
||||||
def health(self) -> RuntimeHealth:
|
|
||||||
with self._lock:
|
|
||||||
active = len(self._running)
|
|
||||||
return RuntimeHealth(
|
|
||||||
runtime="openclaw",
|
|
||||||
runtime_id=self.runtime_id,
|
|
||||||
status="healthy",
|
|
||||||
version=self.version,
|
|
||||||
heartbeat_at=utc_now(),
|
|
||||||
active_tasks=active,
|
|
||||||
workspace_isolated=self.workspace_isolated,
|
|
||||||
credential_ref_configured=self.credential_ref_configured,
|
|
||||||
details={"transport": "in_process"},
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class HTTPRuntimeTransport:
|
|
||||||
"""Private HTTP transport for an OpenClaw runtime service."""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
base_url: str,
|
|
||||||
*,
|
|
||||||
runtime_id: str = "openclaw-http",
|
|
||||||
credential_ref: str | None = None,
|
|
||||||
max_response_bytes: int = 2_097_152,
|
|
||||||
) -> None:
|
|
||||||
normalized = str(base_url).rstrip("/")
|
|
||||||
if not normalized.startswith(("http://", "https://")):
|
|
||||||
raise ValueError("base_url must use http or https")
|
|
||||||
self.base_url = normalized
|
|
||||||
self.runtime_id = runtime_id
|
|
||||||
self.credential_ref = credential_ref
|
|
||||||
self.max_response_bytes = int(max_response_bytes)
|
|
||||||
|
|
||||||
def _request(
|
|
||||||
self,
|
|
||||||
method: str,
|
|
||||||
path: str,
|
|
||||||
*,
|
|
||||||
payload: Mapping[str, Any] | None,
|
|
||||||
timeout_seconds: float,
|
|
||||||
) -> Mapping[str, Any]:
|
|
||||||
body = None
|
|
||||||
headers = {"Accept": "application/json", "User-Agent": "AIOS-OpenClaw/1"}
|
|
||||||
if payload is not None:
|
|
||||||
body = json.dumps(dict(payload), separators=(",", ":")).encode("utf-8")
|
|
||||||
headers["Content-Type"] = "application/json"
|
|
||||||
if self.credential_ref:
|
|
||||||
headers["X-AIOS-Credential-Ref"] = self.credential_ref
|
|
||||||
request = urllib.request.Request(
|
|
||||||
f"{self.base_url}{path}",
|
|
||||||
data=body,
|
|
||||||
method=method,
|
|
||||||
headers=headers,
|
|
||||||
)
|
|
||||||
try:
|
|
||||||
with urllib.request.urlopen(request, timeout=timeout_seconds) as response:
|
|
||||||
raw = response.read(self.max_response_bytes + 1)
|
|
||||||
except urllib.error.HTTPError as exc:
|
|
||||||
raise TransportError(f"runtime returned HTTP {exc.code}") from exc
|
|
||||||
except urllib.error.URLError as exc:
|
|
||||||
raise TransportError(f"runtime unavailable: {exc.reason}") from exc
|
|
||||||
if len(raw) > self.max_response_bytes:
|
|
||||||
raise TransportError("runtime response exceeded size limit")
|
|
||||||
decoded = json.loads(raw.decode("utf-8"))
|
|
||||||
if not isinstance(decoded, Mapping):
|
|
||||||
raise TransportError("runtime response must be a JSON object")
|
|
||||||
return decoded
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _command_payload(command: OpenClawCommand) -> dict[str, Any]:
|
|
||||||
return {**command.unsigned_dict(), "signature": command.signature}
|
|
||||||
|
|
||||||
def execute(self, command: OpenClawCommand, timeout_seconds: float) -> OpenClawResult:
|
|
||||||
value = self._request(
|
|
||||||
"POST",
|
|
||||||
"/v1/execute",
|
|
||||||
payload=self._command_payload(command),
|
|
||||||
timeout_seconds=timeout_seconds,
|
|
||||||
)
|
|
||||||
return OpenClawResult(
|
|
||||||
command_id=str(value["command_id"]),
|
|
||||||
tenant_id=str(value["tenant_id"]),
|
|
||||||
task_id=str(value["task_id"]),
|
|
||||||
trace_id=str(value["trace_id"]),
|
|
||||||
state=CommandState(str(value["state"])),
|
|
||||||
output=value.get("output"),
|
|
||||||
error=value.get("error"),
|
|
||||||
started_at=value.get("started_at"),
|
|
||||||
finished_at=value.get("finished_at"),
|
|
||||||
runtime_id=str(value.get("runtime_id") or self.runtime_id),
|
|
||||||
attempts=int(value.get("attempts", 1)),
|
|
||||||
)
|
|
||||||
|
|
||||||
def cancel(self, command_id: str) -> bool:
|
|
||||||
value = self._request(
|
|
||||||
"POST",
|
|
||||||
f"/v1/commands/{urllib.parse.quote(command_id, safe='')}/cancel",
|
|
||||||
payload={},
|
|
||||||
timeout_seconds=10.0,
|
|
||||||
)
|
|
||||||
return bool(value.get("accepted"))
|
|
||||||
|
|
||||||
def health(self) -> RuntimeHealth:
|
|
||||||
value = self._request("GET", "/v1/health", payload=None, timeout_seconds=5.0)
|
|
||||||
return RuntimeHealth(
|
|
||||||
runtime=str(value.get("runtime") or "openclaw"),
|
|
||||||
runtime_id=str(value.get("runtime_id") or self.runtime_id),
|
|
||||||
status=str(value.get("status") or "unknown"),
|
|
||||||
version=str(value.get("version") or "unknown"),
|
|
||||||
heartbeat_at=str(value.get("heartbeat_at") or utc_now()),
|
|
||||||
active_tasks=int(value.get("active_tasks", 0)),
|
|
||||||
workspace_isolated=bool(value.get("workspace_isolated", False)),
|
|
||||||
credential_ref_configured=bool(value.get("credential_ref_configured", False)),
|
|
||||||
details=value.get("details") if isinstance(value.get("details"), Mapping) else {},
|
|
||||||
)
|
|
||||||
@@ -1,46 +0,0 @@
|
|||||||
"""AIOS V2 control plane and execution orchestration."""
|
|
||||||
|
|
||||||
from .control_plane import (
|
|
||||||
AgentRecord,
|
|
||||||
AuditEvent,
|
|
||||||
ControlPlaneError,
|
|
||||||
ControlPlaneService,
|
|
||||||
IdempotencyConflict,
|
|
||||||
InMemoryControlPlaneStore,
|
|
||||||
InvalidTransition,
|
|
||||||
NotFound,
|
|
||||||
PolicyDenied,
|
|
||||||
RequestContext,
|
|
||||||
RevisionConflict,
|
|
||||||
TaskRecord,
|
|
||||||
TaskState,
|
|
||||||
TenantIsolationError,
|
|
||||||
TenantRecord,
|
|
||||||
Unauthorized,
|
|
||||||
)
|
|
||||||
from .execution_tracking import ExecutionEvidence, ExecutionTracker
|
|
||||||
from .orchestrator import DispatchRecord, ExecutionOrchestrator, OrchestrationError
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"AgentRecord",
|
|
||||||
"AuditEvent",
|
|
||||||
"ControlPlaneError",
|
|
||||||
"ControlPlaneService",
|
|
||||||
"DispatchRecord",
|
|
||||||
"ExecutionEvidence",
|
|
||||||
"ExecutionOrchestrator",
|
|
||||||
"ExecutionTracker",
|
|
||||||
"IdempotencyConflict",
|
|
||||||
"InMemoryControlPlaneStore",
|
|
||||||
"InvalidTransition",
|
|
||||||
"NotFound",
|
|
||||||
"OrchestrationError",
|
|
||||||
"PolicyDenied",
|
|
||||||
"RequestContext",
|
|
||||||
"RevisionConflict",
|
|
||||||
"TaskRecord",
|
|
||||||
"TaskState",
|
|
||||||
"TenantIsolationError",
|
|
||||||
"TenantRecord",
|
|
||||||
"Unauthorized",
|
|
||||||
]
|
|
||||||
@@ -1,357 +0,0 @@
|
|||||||
"""Production-candidate AIOS V2 application entrypoint.
|
|
||||||
|
|
||||||
This process owns the public control-plane API and read-only web console. It
|
|
||||||
loads durable control-plane state, dispatches approved work through runtime
|
|
||||||
adapters, exposes tenant-scoped execution evidence and sanitized GitOps status,
|
|
||||||
and never exposes runtime credentials to the browser.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import dataclasses
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import stat
|
|
||||||
import urllib.parse
|
|
||||||
from enum import Enum
|
|
||||||
from http import HTTPStatus
|
|
||||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any, Mapping
|
|
||||||
|
|
||||||
from aios_hermes import HermesAdapter, HermesJobState, HermesResult
|
|
||||||
from aios_openclaw import HTTPRuntimeTransport, OpenClawAdapter
|
|
||||||
from aios_openclaw.callback import AuditCallbackSink, EventPublisher
|
|
||||||
|
|
||||||
from .console import CONSOLE_HTML
|
|
||||||
from .control_plane import AuditEvent, RequestContext
|
|
||||||
from .execution_tracking import ExecutionTracker
|
|
||||||
from .gateway import GatewayApplication, error_response
|
|
||||||
from .gitops_status import collect_gitops_status
|
|
||||||
from .orchestrator import ExecutionOrchestrator
|
|
||||||
from .persistence import DurableControlPlaneService, SQLiteSnapshotRepository
|
|
||||||
|
|
||||||
|
|
||||||
def _json_value(value: Any) -> Any:
|
|
||||||
if dataclasses.is_dataclass(value):
|
|
||||||
return {
|
|
||||||
item.name: _json_value(getattr(value, item.name))
|
|
||||||
for item in dataclasses.fields(value)
|
|
||||||
}
|
|
||||||
if isinstance(value, Enum):
|
|
||||||
return value.value
|
|
||||||
if isinstance(value, Mapping):
|
|
||||||
return {str(key): _json_value(item) for key, item in value.items()}
|
|
||||||
if isinstance(value, (list, tuple, set, frozenset)):
|
|
||||||
return [_json_value(item) for item in value]
|
|
||||||
return value
|
|
||||||
|
|
||||||
|
|
||||||
class FileSigningMaterialResolver:
|
|
||||||
"""Resolve runtime signing material from a root-owned file reference."""
|
|
||||||
|
|
||||||
def __init__(self, path: str | Path) -> None:
|
|
||||||
self.path = Path(path).resolve()
|
|
||||||
|
|
||||||
def __call__(self, tenant_id: str, key_id: str) -> bytes:
|
|
||||||
del tenant_id, key_id
|
|
||||||
metadata = self.path.stat()
|
|
||||||
if not stat.S_ISREG(metadata.st_mode):
|
|
||||||
raise RuntimeError("signing material reference is not a regular file")
|
|
||||||
if metadata.st_mode & 0o077:
|
|
||||||
raise RuntimeError("signing material file permissions must be 0600 or stricter")
|
|
||||||
value = self.path.read_bytes().strip()
|
|
||||||
if len(value) < 32:
|
|
||||||
raise RuntimeError("signing material is too short")
|
|
||||||
return value
|
|
||||||
|
|
||||||
|
|
||||||
class ControlPlaneEventPublisher(EventPublisher):
|
|
||||||
"""Append OpenClaw runtime events to the tenant audit stream."""
|
|
||||||
|
|
||||||
def __init__(self, service: DurableControlPlaneService) -> None:
|
|
||||||
self.service = service
|
|
||||||
|
|
||||||
def publish(self, event: Mapping[str, Any]) -> None:
|
|
||||||
tenant_id = str(event.get("tenant_id") or "").strip()
|
|
||||||
trace_id = str(event.get("trace_id") or "").strip()
|
|
||||||
command_id = str(event.get("command_id") or "").strip()
|
|
||||||
event_type = str(event.get("event_type") or "OPENCLAW_EVENT").strip()
|
|
||||||
if not tenant_id or not trace_id or not command_id:
|
|
||||||
raise ValueError("runtime audit event is missing tenant identity")
|
|
||||||
state_value = str(event.get("state") or "unknown").lower()
|
|
||||||
outcome = "allowed" if state_value == "completed" else state_value
|
|
||||||
audit = AuditEvent(
|
|
||||||
event_id=f"runtime_{command_id}_{len(self.service.store.audit_events) + 1}",
|
|
||||||
tenant_id=tenant_id,
|
|
||||||
principal_id="openclaw-runtime",
|
|
||||||
trace_id=trace_id,
|
|
||||||
action=event_type,
|
|
||||||
resource_type="openclaw_command",
|
|
||||||
resource_id=command_id,
|
|
||||||
outcome=outcome,
|
|
||||||
policy_version="runtime-v1",
|
|
||||||
details=dict(event.get("details") or {}),
|
|
||||||
)
|
|
||||||
with self.service.store.lock:
|
|
||||||
self.service.store.audit_events.append(audit)
|
|
||||||
self.service.repository.save(self.service._snapshot())
|
|
||||||
|
|
||||||
|
|
||||||
class CandidateApplication:
|
|
||||||
"""Compose durable control plane, runtimes, evidence and console routes."""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
gateway: GatewayApplication,
|
|
||||||
orchestrator: ExecutionOrchestrator,
|
|
||||||
tracker: ExecutionTracker,
|
|
||||||
hermes: HermesAdapter,
|
|
||||||
openclaw: OpenClawAdapter | None,
|
|
||||||
) -> None:
|
|
||||||
self.gateway = gateway
|
|
||||||
self.orchestrator = orchestrator
|
|
||||||
self.tracker = tracker
|
|
||||||
self.hermes = hermes
|
|
||||||
self.openclaw = openclaw
|
|
||||||
|
|
||||||
def context(self, headers: Mapping[str, str]) -> RequestContext:
|
|
||||||
return self.gateway.context_from_headers(headers)
|
|
||||||
|
|
||||||
def handle_json(
|
|
||||||
self,
|
|
||||||
method: str,
|
|
||||||
raw_path: str,
|
|
||||||
headers: Mapping[str, str],
|
|
||||||
body: Any,
|
|
||||||
) -> tuple[int, Mapping[str, Any]]:
|
|
||||||
parsed = urllib.parse.urlparse(raw_path)
|
|
||||||
path = parsed.path.rstrip("/") or "/"
|
|
||||||
method = method.upper()
|
|
||||||
|
|
||||||
if method == "GET" and path == "/api/v2/gitops":
|
|
||||||
context = self.context(headers)
|
|
||||||
return HTTPStatus.OK, {
|
|
||||||
"tenant_id": context.tenant_id,
|
|
||||||
"gitops": collect_gitops_status().as_dict(),
|
|
||||||
}
|
|
||||||
|
|
||||||
if method == "GET" and path == "/api/v2/runtime":
|
|
||||||
context = self.context(headers)
|
|
||||||
runtimes: list[dict[str, Any]] = []
|
|
||||||
if self.openclaw is None:
|
|
||||||
runtimes.append(
|
|
||||||
{
|
|
||||||
"name": "openclaw",
|
|
||||||
"status": "disabled",
|
|
||||||
"details": {"reason": "runtime_not_configured"},
|
|
||||||
}
|
|
||||||
)
|
|
||||||
else:
|
|
||||||
try:
|
|
||||||
health = self.openclaw.health()
|
|
||||||
runtimes.append(
|
|
||||||
{
|
|
||||||
"name": "openclaw",
|
|
||||||
"status": health.status,
|
|
||||||
"details": _json_value(health),
|
|
||||||
}
|
|
||||||
)
|
|
||||||
except Exception as exc:
|
|
||||||
runtimes.append(
|
|
||||||
{
|
|
||||||
"name": "openclaw",
|
|
||||||
"status": "unavailable",
|
|
||||||
"details": {"message": str(exc)},
|
|
||||||
}
|
|
||||||
)
|
|
||||||
hermes_health = self.hermes.health()
|
|
||||||
runtimes.append(
|
|
||||||
{
|
|
||||||
"name": "hermes",
|
|
||||||
"status": hermes_health.status,
|
|
||||||
"details": _json_value(hermes_health),
|
|
||||||
}
|
|
||||||
)
|
|
||||||
return HTTPStatus.OK, {
|
|
||||||
"tenant_id": context.tenant_id,
|
|
||||||
"runtimes": runtimes,
|
|
||||||
}
|
|
||||||
|
|
||||||
if method == "GET" and path == "/api/v2/executions":
|
|
||||||
context = self.context(headers)
|
|
||||||
return HTTPStatus.OK, {
|
|
||||||
"executions": _json_value(
|
|
||||||
self.tracker.list(tenant_id=context.tenant_id)
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
||||||
dispatch_match = __import__("re").fullmatch(
|
|
||||||
r"/api/v2/tasks/([^/]+)/dispatch",
|
|
||||||
path,
|
|
||||||
)
|
|
||||||
if method == "POST" and dispatch_match:
|
|
||||||
context = self.context(headers)
|
|
||||||
task_id = urllib.parse.unquote(dispatch_match.group(1))
|
|
||||||
payload = body if isinstance(body, Mapping) else {}
|
|
||||||
timeout = payload.get("timeout_seconds")
|
|
||||||
record = self.orchestrator.dispatch(
|
|
||||||
context,
|
|
||||||
task_id=task_id,
|
|
||||||
timeout_seconds=float(timeout) if timeout is not None else None,
|
|
||||||
)
|
|
||||||
return HTTPStatus.ACCEPTED, {"dispatch": _json_value(record)}
|
|
||||||
|
|
||||||
if method == "POST" and path == "/api/v2/hermes/results/reconcile":
|
|
||||||
context = self.context(headers)
|
|
||||||
if not isinstance(body, Mapping):
|
|
||||||
raise ValueError("JSON body must be an object")
|
|
||||||
result = HermesResult(
|
|
||||||
job_id=str(body["job_id"]),
|
|
||||||
tenant_id=str(body["tenant_id"]),
|
|
||||||
task_id=str(body["task_id"]),
|
|
||||||
trace_id=str(body["trace_id"]),
|
|
||||||
state=HermesJobState(str(body["state"])),
|
|
||||||
worker_id=(
|
|
||||||
str(body["worker_id"])
|
|
||||||
if body.get("worker_id") is not None
|
|
||||||
else None
|
|
||||||
),
|
|
||||||
output=body.get("output"),
|
|
||||||
error=body.get("error"),
|
|
||||||
started_at=body.get("started_at"),
|
|
||||||
finished_at=body.get("finished_at"),
|
|
||||||
)
|
|
||||||
task = self.orchestrator.reconcile_hermes(context, result)
|
|
||||||
return HTTPStatus.OK, {"task": _json_value(task)}
|
|
||||||
|
|
||||||
return self.gateway.handle(method, raw_path, headers, body)
|
|
||||||
|
|
||||||
|
|
||||||
class CandidateRequestHandler(BaseHTTPRequestHandler):
|
|
||||||
server_version = "AIOSV2Candidate/1.0"
|
|
||||||
max_body_bytes = 1_048_576
|
|
||||||
|
|
||||||
@property
|
|
||||||
def application(self) -> CandidateApplication:
|
|
||||||
return self.server.application # type: ignore[attr-defined]
|
|
||||||
|
|
||||||
def _read_body(self) -> Any:
|
|
||||||
length = int(self.headers.get("Content-Length", "0") or "0")
|
|
||||||
if length < 0 or length > self.max_body_bytes:
|
|
||||||
raise ValueError("request body is too large")
|
|
||||||
if length == 0:
|
|
||||||
return {}
|
|
||||||
content_type = self.headers.get("Content-Type", "").split(";", 1)[0].strip()
|
|
||||||
if content_type != "application/json":
|
|
||||||
raise ValueError("Content-Type must be application/json")
|
|
||||||
return json.loads(self.rfile.read(length).decode("utf-8"))
|
|
||||||
|
|
||||||
def _security_headers(self) -> None:
|
|
||||||
self.send_header("Cache-Control", "no-store")
|
|
||||||
self.send_header("X-Content-Type-Options", "nosniff")
|
|
||||||
self.send_header("X-Frame-Options", "DENY")
|
|
||||||
self.send_header("Referrer-Policy", "no-referrer")
|
|
||||||
self.send_header(
|
|
||||||
"Content-Security-Policy",
|
|
||||||
"default-src 'self'; style-src 'unsafe-inline'; script-src 'unsafe-inline'; connect-src 'self'",
|
|
||||||
)
|
|
||||||
|
|
||||||
def _send_json(self, status_code: int, payload: Mapping[str, Any]) -> None:
|
|
||||||
encoded = json.dumps(
|
|
||||||
_json_value(payload),
|
|
||||||
ensure_ascii=False,
|
|
||||||
separators=(",", ":"),
|
|
||||||
).encode("utf-8")
|
|
||||||
self.send_response(int(status_code))
|
|
||||||
self.send_header("Content-Type", "application/json; charset=utf-8")
|
|
||||||
self.send_header("Content-Length", str(len(encoded)))
|
|
||||||
self._security_headers()
|
|
||||||
self.end_headers()
|
|
||||||
self.wfile.write(encoded)
|
|
||||||
|
|
||||||
def _send_console(self) -> None:
|
|
||||||
encoded = CONSOLE_HTML.encode("utf-8")
|
|
||||||
self.send_response(HTTPStatus.OK)
|
|
||||||
self.send_header("Content-Type", "text/html; charset=utf-8")
|
|
||||||
self.send_header("Content-Length", str(len(encoded)))
|
|
||||||
self._security_headers()
|
|
||||||
self.end_headers()
|
|
||||||
self.wfile.write(encoded)
|
|
||||||
|
|
||||||
def _dispatch(self, method: str) -> None:
|
|
||||||
path = urllib.parse.urlparse(self.path).path.rstrip("/") or "/"
|
|
||||||
if method == "GET" and path in {"/", "/console"}:
|
|
||||||
self._send_console()
|
|
||||||
return
|
|
||||||
try:
|
|
||||||
body = self._read_body() if method == "POST" else {}
|
|
||||||
status_code, payload = self.application.handle_json(
|
|
||||||
method,
|
|
||||||
self.path,
|
|
||||||
{key: value for key, value in self.headers.items()},
|
|
||||||
body,
|
|
||||||
)
|
|
||||||
except BaseException as exc:
|
|
||||||
status_code, payload = error_response(exc)
|
|
||||||
self._send_json(status_code, payload)
|
|
||||||
|
|
||||||
def do_GET(self) -> None: # noqa: N802
|
|
||||||
self._dispatch("GET")
|
|
||||||
|
|
||||||
def do_POST(self) -> None: # noqa: N802
|
|
||||||
self._dispatch("POST")
|
|
||||||
|
|
||||||
|
|
||||||
class CandidateHTTPServer(ThreadingHTTPServer):
|
|
||||||
daemon_threads = True
|
|
||||||
allow_reuse_address = True
|
|
||||||
|
|
||||||
def __init__(self, address: tuple[str, int], application: CandidateApplication) -> None:
|
|
||||||
self.application = application
|
|
||||||
super().__init__(address, CandidateRequestHandler)
|
|
||||||
|
|
||||||
|
|
||||||
def build_application() -> CandidateApplication:
|
|
||||||
state_dir = Path(os.getenv("AIOS_V2_STATE_DIR", "/var/lib/aios-v2")).resolve()
|
|
||||||
state_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
repository = SQLiteSnapshotRepository(state_dir / "control-plane.sqlite3")
|
|
||||||
service = DurableControlPlaneService(repository)
|
|
||||||
tracker = ExecutionTracker(state_dir / "executions")
|
|
||||||
hermes = HermesAdapter()
|
|
||||||
|
|
||||||
openclaw_url = os.getenv("AIOS_OPENCLAW_URL", "").strip()
|
|
||||||
material_file = os.getenv("AIOS_OPENCLAW_SIGNING_MATERIAL_FILE", "").strip()
|
|
||||||
openclaw = None
|
|
||||||
resolver = None
|
|
||||||
if openclaw_url and material_file:
|
|
||||||
resolver = FileSigningMaterialResolver(material_file)
|
|
||||||
transport = HTTPRuntimeTransport(
|
|
||||||
openclaw_url,
|
|
||||||
credential_ref=os.getenv("AIOS_OPENCLAW_CREDENTIAL_REF") or None,
|
|
||||||
)
|
|
||||||
openclaw = OpenClawAdapter(
|
|
||||||
transport,
|
|
||||||
resolver,
|
|
||||||
callback_sink=AuditCallbackSink(ControlPlaneEventPublisher(service)),
|
|
||||||
)
|
|
||||||
|
|
||||||
gateway = GatewayApplication(service)
|
|
||||||
orchestrator = ExecutionOrchestrator(
|
|
||||||
service,
|
|
||||||
openclaw=openclaw,
|
|
||||||
hermes=hermes,
|
|
||||||
openclaw_signing_material_resolver=resolver,
|
|
||||||
)
|
|
||||||
return CandidateApplication(gateway, orchestrator, tracker, hermes, openclaw)
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
host = os.getenv("AIOS_V2_HOST", "127.0.0.1")
|
|
||||||
port = int(os.getenv("AIOS_V2_PORT", "18920"))
|
|
||||||
server = CandidateHTTPServer((host, port), build_application())
|
|
||||||
print(f"AIOS V2 candidate listening on {host}:{port}")
|
|
||||||
server.serve_forever()
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,69 +0,0 @@
|
|||||||
"""Dependency-free AIOS V2 web console assets.
|
|
||||||
|
|
||||||
Authentication and tenant identity are injected by the trusted reverse proxy.
|
|
||||||
The browser never selects a tenant and never calls OpenClaw or Hermes directly.
|
|
||||||
"""
|
|
||||||
|
|
||||||
CONSOLE_HTML = r'''<!doctype html>
|
|
||||||
<html lang="en">
|
|
||||||
<head>
|
|
||||||
<meta charset="utf-8">
|
|
||||||
<meta name="viewport" content="width=device-width,initial-scale=1">
|
|
||||||
<meta name="referrer" content="no-referrer">
|
|
||||||
<title>AIOS V2 Console</title>
|
|
||||||
<style>
|
|
||||||
:root{font-family:Inter,ui-sans-serif,system-ui,sans-serif;color:#152033;background:#f4f7fb}
|
|
||||||
*{box-sizing:border-box} body{margin:0} header{background:#111827;color:white;padding:18px 24px;display:flex;justify-content:space-between;align-items:center}
|
|
||||||
header h1{font-size:20px;margin:0} header span{font-size:12px;color:#cbd5e1}
|
|
||||||
main{padding:24px;max-width:1440px;margin:auto}.notice{background:#fff7ed;border:1px solid #fed7aa;padding:12px 14px;border-radius:10px;margin-bottom:18px;font-size:13px}
|
|
||||||
.cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(190px,1fr));gap:14px;margin-bottom:20px}.card,.panel{background:white;border:1px solid #e5e7eb;border-radius:12px;box-shadow:0 1px 2px rgba(0,0,0,.04)}
|
|
||||||
.card{padding:16px}.card b{display:block;font-size:28px;margin-top:8px}.card small{color:#64748b}.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(420px,1fr));gap:16px}.panel{padding:16px;overflow:auto}.panel h2{font-size:16px;margin:0 0 12px}
|
|
||||||
table{width:100%;border-collapse:collapse;font-size:12px}th,td{text-align:left;border-bottom:1px solid #edf0f4;padding:8px;vertical-align:top}th{color:#64748b;font-weight:600}.state{font-weight:700}.ok{color:#047857}.bad{color:#b91c1c}.muted{color:#64748b}
|
|
||||||
button{border:0;border-radius:8px;padding:9px 13px;background:#2563eb;color:white;cursor:pointer}button:disabled{opacity:.5}.toolbar{display:flex;gap:8px;align-items:center;margin-bottom:16px}.error{color:#b91c1c;white-space:pre-wrap}
|
|
||||||
</style>
|
|
||||||
</head>
|
|
||||||
<body>
|
|
||||||
<header><h1>AIOS V2 Control Plane</h1><span id="updated">Not loaded</span></header>
|
|
||||||
<main>
|
|
||||||
<div class="notice">Read-only candidate console. Identity and tenant context must be injected by the trusted reverse proxy. The console never calls OpenClaw or Hermes directly.</div>
|
|
||||||
<div class="toolbar"><button id="refresh">Refresh</button><span id="error" class="error"></span></div>
|
|
||||||
<section class="cards">
|
|
||||||
<div class="card"><small>Agents</small><b id="agentCount">—</b></div>
|
|
||||||
<div class="card"><small>Tasks</small><b id="taskCount">—</b></div>
|
|
||||||
<div class="card"><small>Audit events</small><b id="auditCount">—</b></div>
|
|
||||||
<div class="card"><small>Executions</small><b id="executionCount">—</b></div>
|
|
||||||
</section>
|
|
||||||
<section class="grid">
|
|
||||||
<div class="panel"><h2>Runtime health</h2><div id="runtime"></div></div>
|
|
||||||
<div class="panel"><h2>Tasks</h2><div id="tasks"></div></div>
|
|
||||||
<div class="panel"><h2>Agents</h2><div id="agents"></div></div>
|
|
||||||
<div class="panel"><h2>Execution evidence</h2><div id="executions"></div></div>
|
|
||||||
<div class="panel"><h2>Audit</h2><div id="audit"></div></div>
|
|
||||||
</section>
|
|
||||||
</main>
|
|
||||||
<script>
|
|
||||||
const esc=v=>String(v??'').replace(/[&<>"']/g,c=>({'&':'&','<':'<','>':'>','"':'"',"'":'''}[c]));
|
|
||||||
const table=(cols,rows)=>`<table><thead><tr>${cols.map(c=>`<th>${esc(c[0])}</th>`).join('')}</tr></thead><tbody>${rows.map(r=>`<tr>${cols.map(c=>`<td>${esc(c[1](r))}</td>`).join('')}</tr>`).join('')}</tbody></table>`;
|
|
||||||
async function api(path){const r=await fetch(path,{credentials:'same-origin',headers:{'Accept':'application/json'}});const data=await r.json().catch(()=>({}));if(!r.ok)throw new Error(`${r.status} ${data.message||data.error||path}`);return data}
|
|
||||||
async function refresh(){
|
|
||||||
const btn=document.getElementById('refresh');btn.disabled=true;document.getElementById('error').textContent='';
|
|
||||||
try{
|
|
||||||
const [agents,tasks,audit,runtime,executions]=await Promise.all([
|
|
||||||
api('/api/v2/agents'),api('/api/v2/tasks'),api('/api/v2/audit'),api('/api/v2/runtime'),api('/api/v2/executions')
|
|
||||||
]);
|
|
||||||
document.getElementById('agentCount').textContent=agents.agents.length;
|
|
||||||
document.getElementById('taskCount').textContent=tasks.tasks.length;
|
|
||||||
document.getElementById('auditCount').textContent=audit.events.length;
|
|
||||||
document.getElementById('executionCount').textContent=executions.executions.length;
|
|
||||||
document.getElementById('agents').innerHTML=table([['ID',x=>x.agent_id],['Runtime',x=>x.runtime],['Enabled',x=>x.enabled]],agents.agents);
|
|
||||||
document.getElementById('tasks').innerHTML=table([['Task',x=>x.task_id],['Action',x=>x.action],['State',x=>x.state],['Rev',x=>x.revision]],tasks.tasks);
|
|
||||||
document.getElementById('audit').innerHTML=table([['Time',x=>x.created_at],['Action',x=>x.action],['Outcome',x=>x.outcome],['Resource',x=>x.resource_id]],audit.events.slice(-100).reverse());
|
|
||||||
document.getElementById('executions').innerHTML=table([['Executor',x=>x.executor],['Task',x=>x.task_id],['Status',x=>x.status],['Progress',x=>`${x.progress_completed}/${x.progress_total}`],['Commit',x=>x.last_commit||'']],executions.executions);
|
|
||||||
document.getElementById('runtime').innerHTML=table([['Runtime',x=>x.name],['Status',x=>x.status],['Details',x=>JSON.stringify(x.details)]],runtime.runtimes);
|
|
||||||
document.getElementById('updated').textContent=`Updated ${new Date().toLocaleString()}`;
|
|
||||||
}catch(e){document.getElementById('error').textContent=e.message||String(e)}finally{btn.disabled=false}
|
|
||||||
}
|
|
||||||
document.getElementById('refresh').addEventListener('click',refresh);refresh();setInterval(refresh,30000);
|
|
||||||
</script>
|
|
||||||
</body>
|
|
||||||
</html>'''
|
|
||||||
@@ -1,627 +0,0 @@
|
|||||||
"""Tenant-isolated AIOS V2 control-plane domain service.
|
|
||||||
|
|
||||||
This module deliberately contains no OpenClaw or Hermes implementation details.
|
|
||||||
It owns authoritative tenant, agent, task-lifecycle, idempotency and audit state.
|
|
||||||
Runtime adapters consume approved tasks through a separate boundary.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import copy
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import threading
|
|
||||||
import uuid
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from enum import Enum
|
|
||||||
from typing import Any, Callable, Iterable, Mapping
|
|
||||||
|
|
||||||
|
|
||||||
def _utc_now() -> str:
|
|
||||||
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
|
|
||||||
|
|
||||||
|
|
||||||
def _new_id(prefix: str) -> str:
|
|
||||||
return f"{prefix}_{uuid.uuid4().hex}"
|
|
||||||
|
|
||||||
|
|
||||||
def _canonical_digest(value: Mapping[str, Any]) -> str:
|
|
||||||
encoded = json.dumps(
|
|
||||||
value,
|
|
||||||
ensure_ascii=False,
|
|
||||||
sort_keys=True,
|
|
||||||
separators=(",", ":"),
|
|
||||||
default=str,
|
|
||||||
).encode("utf-8")
|
|
||||||
return hashlib.sha256(encoded).hexdigest()
|
|
||||||
|
|
||||||
|
|
||||||
class ControlPlaneError(RuntimeError):
|
|
||||||
"""Base error returned by the control-plane domain."""
|
|
||||||
|
|
||||||
|
|
||||||
class Unauthorized(ControlPlaneError):
|
|
||||||
"""The caller is not authenticated or lacks the required role."""
|
|
||||||
|
|
||||||
|
|
||||||
class TenantIsolationError(ControlPlaneError):
|
|
||||||
"""A request attempted to cross a tenant boundary."""
|
|
||||||
|
|
||||||
|
|
||||||
class NotFound(ControlPlaneError):
|
|
||||||
"""A tenant-scoped resource does not exist or is not visible."""
|
|
||||||
|
|
||||||
|
|
||||||
class InvalidTransition(ControlPlaneError):
|
|
||||||
"""A task state transition violates the lifecycle contract."""
|
|
||||||
|
|
||||||
|
|
||||||
class IdempotencyConflict(ControlPlaneError):
|
|
||||||
"""An idempotency key was reused for a different command."""
|
|
||||||
|
|
||||||
|
|
||||||
class RevisionConflict(ControlPlaneError):
|
|
||||||
"""Optimistic task revision does not match current state."""
|
|
||||||
|
|
||||||
|
|
||||||
class PolicyDenied(ControlPlaneError):
|
|
||||||
"""The policy gate denied a control-plane operation."""
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class RequestContext:
|
|
||||||
"""Authenticated request identity propagated across all AIOS layers."""
|
|
||||||
|
|
||||||
tenant_id: str
|
|
||||||
principal_id: str
|
|
||||||
trace_id: str
|
|
||||||
roles: frozenset[str] = field(default_factory=frozenset)
|
|
||||||
session_id: str | None = None
|
|
||||||
policy_version: str = "v1"
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
for name in ("tenant_id", "principal_id", "trace_id"):
|
|
||||||
if not str(getattr(self, name, "")).strip():
|
|
||||||
raise ValueError(f"{name} is required")
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def create(
|
|
||||||
cls,
|
|
||||||
*,
|
|
||||||
tenant_id: str,
|
|
||||||
principal_id: str,
|
|
||||||
roles: Iterable[str] = (),
|
|
||||||
trace_id: str | None = None,
|
|
||||||
session_id: str | None = None,
|
|
||||||
policy_version: str = "v1",
|
|
||||||
) -> "RequestContext":
|
|
||||||
return cls(
|
|
||||||
tenant_id=tenant_id.strip(),
|
|
||||||
principal_id=principal_id.strip(),
|
|
||||||
trace_id=trace_id or _new_id("trace"),
|
|
||||||
roles=frozenset(str(role).strip() for role in roles if str(role).strip()),
|
|
||||||
session_id=session_id,
|
|
||||||
policy_version=policy_version,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class TenantRecord:
|
|
||||||
tenant_id: str
|
|
||||||
display_name: str
|
|
||||||
status: str = "active"
|
|
||||||
metadata: Mapping[str, Any] = field(default_factory=dict)
|
|
||||||
created_at: str = field(default_factory=_utc_now)
|
|
||||||
updated_at: str = field(default_factory=_utc_now)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class AgentRecord:
|
|
||||||
agent_id: str
|
|
||||||
tenant_id: str
|
|
||||||
display_name: str
|
|
||||||
runtime: str
|
|
||||||
capabilities: tuple[str, ...] = field(default_factory=tuple)
|
|
||||||
enabled: bool = True
|
|
||||||
metadata: Mapping[str, Any] = field(default_factory=dict)
|
|
||||||
created_at: str = field(default_factory=_utc_now)
|
|
||||||
updated_at: str = field(default_factory=_utc_now)
|
|
||||||
|
|
||||||
|
|
||||||
class TaskState(str, Enum):
|
|
||||||
CREATED = "CREATED"
|
|
||||||
APPROVED = "APPROVED"
|
|
||||||
RUNNING = "RUNNING"
|
|
||||||
COMPLETED = "COMPLETED"
|
|
||||||
FAILED = "FAILED"
|
|
||||||
CANCELLED = "CANCELLED"
|
|
||||||
|
|
||||||
|
|
||||||
TERMINAL_TASK_STATES = frozenset(
|
|
||||||
{TaskState.COMPLETED, TaskState.FAILED, TaskState.CANCELLED}
|
|
||||||
)
|
|
||||||
|
|
||||||
ALLOWED_TASK_TRANSITIONS: Mapping[TaskState, frozenset[TaskState]] = {
|
|
||||||
TaskState.CREATED: frozenset({TaskState.APPROVED, TaskState.CANCELLED}),
|
|
||||||
TaskState.APPROVED: frozenset({TaskState.RUNNING, TaskState.CANCELLED}),
|
|
||||||
TaskState.RUNNING: frozenset(
|
|
||||||
{TaskState.COMPLETED, TaskState.FAILED, TaskState.CANCELLED}
|
|
||||||
),
|
|
||||||
TaskState.COMPLETED: frozenset(),
|
|
||||||
TaskState.FAILED: frozenset(),
|
|
||||||
TaskState.CANCELLED: frozenset(),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class TaskRecord:
|
|
||||||
task_id: str
|
|
||||||
tenant_id: str
|
|
||||||
agent_id: str
|
|
||||||
action: str
|
|
||||||
payload: Mapping[str, Any]
|
|
||||||
state: TaskState = TaskState.CREATED
|
|
||||||
revision: int = 1
|
|
||||||
idempotency_key: str | None = None
|
|
||||||
request_digest: str = ""
|
|
||||||
result: Mapping[str, Any] | None = None
|
|
||||||
error: Mapping[str, Any] | None = None
|
|
||||||
created_by: str = ""
|
|
||||||
approved_by: str | None = None
|
|
||||||
created_at: str = field(default_factory=_utc_now)
|
|
||||||
updated_at: str = field(default_factory=_utc_now)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class AuditEvent:
|
|
||||||
event_id: str
|
|
||||||
tenant_id: str
|
|
||||||
principal_id: str
|
|
||||||
trace_id: str
|
|
||||||
action: str
|
|
||||||
resource_type: str
|
|
||||||
resource_id: str
|
|
||||||
outcome: str
|
|
||||||
policy_version: str
|
|
||||||
details: Mapping[str, Any] = field(default_factory=dict)
|
|
||||||
created_at: str = field(default_factory=_utc_now)
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class PolicyDecision:
|
|
||||||
allowed: bool
|
|
||||||
reason: str
|
|
||||||
policy_version: str
|
|
||||||
|
|
||||||
|
|
||||||
PolicyEvaluator = Callable[[RequestContext, str, Mapping[str, Any]], PolicyDecision]
|
|
||||||
|
|
||||||
|
|
||||||
class InMemoryControlPlaneStore:
|
|
||||||
"""Thread-safe reference store used by tests and the first gateway slice.
|
|
||||||
|
|
||||||
Production backends must preserve the same tenant-scoped interface and add
|
|
||||||
database row-level security, migrations, backup and HA evidence.
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self) -> None:
|
|
||||||
self.lock = threading.RLock()
|
|
||||||
self.tenants: dict[str, TenantRecord] = {}
|
|
||||||
self.agents: dict[str, AgentRecord] = {}
|
|
||||||
self.tasks: dict[str, TaskRecord] = {}
|
|
||||||
self.idempotency: dict[tuple[str, str], tuple[str, str]] = {}
|
|
||||||
self.audit_events: list[AuditEvent] = []
|
|
||||||
|
|
||||||
|
|
||||||
class ControlPlaneService:
|
|
||||||
"""Authoritative tenant, agent and task lifecycle service."""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
store: InMemoryControlPlaneStore | None = None,
|
|
||||||
*,
|
|
||||||
policy_evaluator: PolicyEvaluator | None = None,
|
|
||||||
) -> None:
|
|
||||||
self.store = store or InMemoryControlPlaneStore()
|
|
||||||
self.policy_evaluator = policy_evaluator or self._default_policy
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _default_policy(
|
|
||||||
context: RequestContext,
|
|
||||||
action: str,
|
|
||||||
resource: Mapping[str, Any],
|
|
||||||
) -> PolicyDecision:
|
|
||||||
del resource
|
|
||||||
role_requirements = {
|
|
||||||
"tenant.register": {"platform_admin"},
|
|
||||||
"tenant.read": {"tenant_admin", "operator", "auditor", "platform_admin"},
|
|
||||||
"agent.register": {"tenant_admin", "operator", "platform_admin"},
|
|
||||||
"agent.read": {"tenant_admin", "operator", "auditor", "platform_admin"},
|
|
||||||
"task.create": {"tenant_admin", "operator", "platform_admin"},
|
|
||||||
"task.read": {"tenant_admin", "operator", "auditor", "platform_admin"},
|
|
||||||
"task.approve": {"tenant_admin", "approver", "platform_admin"},
|
|
||||||
"task.start": {"runtime", "operator", "platform_admin"},
|
|
||||||
"task.complete": {"runtime", "operator", "platform_admin"},
|
|
||||||
"task.fail": {"runtime", "operator", "platform_admin"},
|
|
||||||
"task.cancel": {"tenant_admin", "operator", "approver", "platform_admin"},
|
|
||||||
"audit.read": {"tenant_admin", "auditor", "platform_admin"},
|
|
||||||
}
|
|
||||||
required = role_requirements.get(action, {"platform_admin"})
|
|
||||||
allowed = bool(context.roles.intersection(required))
|
|
||||||
return PolicyDecision(
|
|
||||||
allowed=allowed,
|
|
||||||
reason="role_allowed" if allowed else "required_role_missing",
|
|
||||||
policy_version=context.policy_version,
|
|
||||||
)
|
|
||||||
|
|
||||||
def _require_context(self, context: RequestContext) -> None:
|
|
||||||
if not context.principal_id or not context.tenant_id or not context.trace_id:
|
|
||||||
raise Unauthorized("authenticated tenant context is required")
|
|
||||||
|
|
||||||
def _authorize(
|
|
||||||
self,
|
|
||||||
context: RequestContext,
|
|
||||||
action: str,
|
|
||||||
resource: Mapping[str, Any],
|
|
||||||
) -> PolicyDecision:
|
|
||||||
self._require_context(context)
|
|
||||||
decision = self.policy_evaluator(context, action, resource)
|
|
||||||
if not decision.allowed:
|
|
||||||
self._audit(
|
|
||||||
context,
|
|
||||||
action=action,
|
|
||||||
resource_type=str(resource.get("resource_type", "policy")),
|
|
||||||
resource_id=str(resource.get("resource_id", "unknown")),
|
|
||||||
outcome="denied",
|
|
||||||
details={"reason": decision.reason},
|
|
||||||
)
|
|
||||||
raise PolicyDenied(f"{action} denied: {decision.reason}")
|
|
||||||
return decision
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _assert_same_tenant(context: RequestContext, tenant_id: str) -> None:
|
|
||||||
if context.tenant_id != tenant_id and "platform_admin" not in context.roles:
|
|
||||||
raise TenantIsolationError("cross-tenant access denied")
|
|
||||||
|
|
||||||
def _audit(
|
|
||||||
self,
|
|
||||||
context: RequestContext,
|
|
||||||
*,
|
|
||||||
action: str,
|
|
||||||
resource_type: str,
|
|
||||||
resource_id: str,
|
|
||||||
outcome: str,
|
|
||||||
details: Mapping[str, Any] | None = None,
|
|
||||||
) -> AuditEvent:
|
|
||||||
event = AuditEvent(
|
|
||||||
event_id=_new_id("audit"),
|
|
||||||
tenant_id=context.tenant_id,
|
|
||||||
principal_id=context.principal_id,
|
|
||||||
trace_id=context.trace_id,
|
|
||||||
action=action,
|
|
||||||
resource_type=resource_type,
|
|
||||||
resource_id=resource_id,
|
|
||||||
outcome=outcome,
|
|
||||||
policy_version=context.policy_version,
|
|
||||||
details=copy.deepcopy(dict(details or {})),
|
|
||||||
)
|
|
||||||
self.store.audit_events.append(event)
|
|
||||||
return event
|
|
||||||
|
|
||||||
def register_tenant(
|
|
||||||
self,
|
|
||||||
context: RequestContext,
|
|
||||||
*,
|
|
||||||
tenant_id: str,
|
|
||||||
display_name: str,
|
|
||||||
metadata: Mapping[str, Any] | None = None,
|
|
||||||
) -> TenantRecord:
|
|
||||||
tenant_id = tenant_id.strip()
|
|
||||||
if not tenant_id or not display_name.strip():
|
|
||||||
raise ValueError("tenant_id and display_name are required")
|
|
||||||
self._authorize(
|
|
||||||
context,
|
|
||||||
"tenant.register",
|
|
||||||
{"resource_type": "tenant", "resource_id": tenant_id},
|
|
||||||
)
|
|
||||||
self._assert_same_tenant(context, tenant_id)
|
|
||||||
with self.store.lock:
|
|
||||||
if tenant_id in self.store.tenants:
|
|
||||||
raise ControlPlaneError(f"tenant already exists: {tenant_id}")
|
|
||||||
tenant = TenantRecord(
|
|
||||||
tenant_id=tenant_id,
|
|
||||||
display_name=display_name.strip(),
|
|
||||||
metadata=copy.deepcopy(dict(metadata or {})),
|
|
||||||
)
|
|
||||||
self.store.tenants[tenant_id] = tenant
|
|
||||||
self._audit(
|
|
||||||
context,
|
|
||||||
action="tenant.register",
|
|
||||||
resource_type="tenant",
|
|
||||||
resource_id=tenant_id,
|
|
||||||
outcome="allowed",
|
|
||||||
)
|
|
||||||
return tenant
|
|
||||||
|
|
||||||
def get_tenant(self, context: RequestContext, tenant_id: str | None = None) -> TenantRecord:
|
|
||||||
tenant_id = tenant_id or context.tenant_id
|
|
||||||
self._authorize(
|
|
||||||
context,
|
|
||||||
"tenant.read",
|
|
||||||
{"resource_type": "tenant", "resource_id": tenant_id},
|
|
||||||
)
|
|
||||||
self._assert_same_tenant(context, tenant_id)
|
|
||||||
with self.store.lock:
|
|
||||||
tenant = self.store.tenants.get(tenant_id)
|
|
||||||
if tenant is None:
|
|
||||||
raise NotFound("tenant not found")
|
|
||||||
return tenant
|
|
||||||
|
|
||||||
def register_agent(
|
|
||||||
self,
|
|
||||||
context: RequestContext,
|
|
||||||
*,
|
|
||||||
agent_id: str,
|
|
||||||
display_name: str,
|
|
||||||
runtime: str,
|
|
||||||
capabilities: Iterable[str] = (),
|
|
||||||
metadata: Mapping[str, Any] | None = None,
|
|
||||||
) -> AgentRecord:
|
|
||||||
self._authorize(
|
|
||||||
context,
|
|
||||||
"agent.register",
|
|
||||||
{"resource_type": "agent", "resource_id": agent_id},
|
|
||||||
)
|
|
||||||
tenant = self.get_tenant(context)
|
|
||||||
if tenant.status != "active":
|
|
||||||
raise ControlPlaneError("tenant is not active")
|
|
||||||
normalized_capabilities = tuple(
|
|
||||||
sorted({str(item).strip() for item in capabilities if str(item).strip()})
|
|
||||||
)
|
|
||||||
if not agent_id.strip() or not display_name.strip() or not runtime.strip():
|
|
||||||
raise ValueError("agent_id, display_name and runtime are required")
|
|
||||||
with self.store.lock:
|
|
||||||
existing = self.store.agents.get(agent_id)
|
|
||||||
if existing is not None and existing.tenant_id != context.tenant_id:
|
|
||||||
raise TenantIsolationError("agent id belongs to another tenant")
|
|
||||||
if existing is not None:
|
|
||||||
raise ControlPlaneError(f"agent already exists: {agent_id}")
|
|
||||||
agent = AgentRecord(
|
|
||||||
agent_id=agent_id.strip(),
|
|
||||||
tenant_id=context.tenant_id,
|
|
||||||
display_name=display_name.strip(),
|
|
||||||
runtime=runtime.strip(),
|
|
||||||
capabilities=normalized_capabilities,
|
|
||||||
metadata=copy.deepcopy(dict(metadata or {})),
|
|
||||||
)
|
|
||||||
self.store.agents[agent.agent_id] = agent
|
|
||||||
self._audit(
|
|
||||||
context,
|
|
||||||
action="agent.register",
|
|
||||||
resource_type="agent",
|
|
||||||
resource_id=agent.agent_id,
|
|
||||||
outcome="allowed",
|
|
||||||
details={"runtime": agent.runtime},
|
|
||||||
)
|
|
||||||
return agent
|
|
||||||
|
|
||||||
def get_agent(self, context: RequestContext, agent_id: str) -> AgentRecord:
|
|
||||||
self._authorize(
|
|
||||||
context,
|
|
||||||
"agent.read",
|
|
||||||
{"resource_type": "agent", "resource_id": agent_id},
|
|
||||||
)
|
|
||||||
with self.store.lock:
|
|
||||||
agent = self.store.agents.get(agent_id)
|
|
||||||
if agent is None:
|
|
||||||
raise NotFound("agent not found")
|
|
||||||
self._assert_same_tenant(context, agent.tenant_id)
|
|
||||||
return agent
|
|
||||||
|
|
||||||
def list_agents(self, context: RequestContext) -> list[AgentRecord]:
|
|
||||||
self._authorize(
|
|
||||||
context,
|
|
||||||
"agent.read",
|
|
||||||
{"resource_type": "agent_collection", "resource_id": context.tenant_id},
|
|
||||||
)
|
|
||||||
with self.store.lock:
|
|
||||||
return [
|
|
||||||
agent
|
|
||||||
for agent in self.store.agents.values()
|
|
||||||
if agent.tenant_id == context.tenant_id
|
|
||||||
]
|
|
||||||
|
|
||||||
def create_task(
|
|
||||||
self,
|
|
||||||
context: RequestContext,
|
|
||||||
*,
|
|
||||||
agent_id: str,
|
|
||||||
action: str,
|
|
||||||
payload: Mapping[str, Any] | None = None,
|
|
||||||
idempotency_key: str | None = None,
|
|
||||||
) -> TaskRecord:
|
|
||||||
self._authorize(
|
|
||||||
context,
|
|
||||||
"task.create",
|
|
||||||
{"resource_type": "task", "resource_id": idempotency_key or "new"},
|
|
||||||
)
|
|
||||||
agent = self.get_agent(context, agent_id)
|
|
||||||
if not agent.enabled:
|
|
||||||
raise ControlPlaneError("agent is disabled")
|
|
||||||
if action not in agent.capabilities and "*" not in agent.capabilities:
|
|
||||||
raise PolicyDenied(f"agent does not advertise capability: {action}")
|
|
||||||
normalized_payload = copy.deepcopy(dict(payload or {}))
|
|
||||||
request_digest = _canonical_digest(
|
|
||||||
{"agent_id": agent_id, "action": action, "payload": normalized_payload}
|
|
||||||
)
|
|
||||||
normalized_key = idempotency_key.strip() if idempotency_key else None
|
|
||||||
with self.store.lock:
|
|
||||||
if normalized_key:
|
|
||||||
idempotency_ref = (context.tenant_id, normalized_key)
|
|
||||||
existing = self.store.idempotency.get(idempotency_ref)
|
|
||||||
if existing is not None:
|
|
||||||
existing_task_id, existing_digest = existing
|
|
||||||
if existing_digest != request_digest:
|
|
||||||
raise IdempotencyConflict(
|
|
||||||
"idempotency key was reused with a different request"
|
|
||||||
)
|
|
||||||
return self.store.tasks[existing_task_id]
|
|
||||||
task = TaskRecord(
|
|
||||||
task_id=_new_id("task"),
|
|
||||||
tenant_id=context.tenant_id,
|
|
||||||
agent_id=agent_id,
|
|
||||||
action=action,
|
|
||||||
payload=normalized_payload,
|
|
||||||
idempotency_key=normalized_key,
|
|
||||||
request_digest=request_digest,
|
|
||||||
created_by=context.principal_id,
|
|
||||||
)
|
|
||||||
self.store.tasks[task.task_id] = task
|
|
||||||
if normalized_key:
|
|
||||||
self.store.idempotency[(context.tenant_id, normalized_key)] = (
|
|
||||||
task.task_id,
|
|
||||||
request_digest,
|
|
||||||
)
|
|
||||||
self._audit(
|
|
||||||
context,
|
|
||||||
action="task.create",
|
|
||||||
resource_type="task",
|
|
||||||
resource_id=task.task_id,
|
|
||||||
outcome="allowed",
|
|
||||||
details={"agent_id": agent_id, "action": action},
|
|
||||||
)
|
|
||||||
return task
|
|
||||||
|
|
||||||
def get_task(self, context: RequestContext, task_id: str) -> TaskRecord:
|
|
||||||
self._authorize(
|
|
||||||
context,
|
|
||||||
"task.read",
|
|
||||||
{"resource_type": "task", "resource_id": task_id},
|
|
||||||
)
|
|
||||||
with self.store.lock:
|
|
||||||
task = self.store.tasks.get(task_id)
|
|
||||||
if task is None:
|
|
||||||
raise NotFound("task not found")
|
|
||||||
self._assert_same_tenant(context, task.tenant_id)
|
|
||||||
return task
|
|
||||||
|
|
||||||
def list_tasks(
|
|
||||||
self,
|
|
||||||
context: RequestContext,
|
|
||||||
*,
|
|
||||||
states: Iterable[TaskState | str] | None = None,
|
|
||||||
) -> list[TaskRecord]:
|
|
||||||
self._authorize(
|
|
||||||
context,
|
|
||||||
"task.read",
|
|
||||||
{"resource_type": "task_collection", "resource_id": context.tenant_id},
|
|
||||||
)
|
|
||||||
state_filter = (
|
|
||||||
{TaskState(item) for item in states}
|
|
||||||
if states is not None
|
|
||||||
else None
|
|
||||||
)
|
|
||||||
with self.store.lock:
|
|
||||||
return [
|
|
||||||
task
|
|
||||||
for task in self.store.tasks.values()
|
|
||||||
if task.tenant_id == context.tenant_id
|
|
||||||
and (state_filter is None or task.state in state_filter)
|
|
||||||
]
|
|
||||||
|
|
||||||
def transition_task(
|
|
||||||
self,
|
|
||||||
context: RequestContext,
|
|
||||||
*,
|
|
||||||
task_id: str,
|
|
||||||
target_state: TaskState | str,
|
|
||||||
expected_revision: int | None = None,
|
|
||||||
result: Mapping[str, Any] | None = None,
|
|
||||||
error: Mapping[str, Any] | None = None,
|
|
||||||
) -> TaskRecord:
|
|
||||||
target = TaskState(target_state)
|
|
||||||
action_name = {
|
|
||||||
TaskState.APPROVED: "task.approve",
|
|
||||||
TaskState.RUNNING: "task.start",
|
|
||||||
TaskState.COMPLETED: "task.complete",
|
|
||||||
TaskState.FAILED: "task.fail",
|
|
||||||
TaskState.CANCELLED: "task.cancel",
|
|
||||||
}.get(target)
|
|
||||||
if action_name is None:
|
|
||||||
raise InvalidTransition(f"cannot transition to {target.value}")
|
|
||||||
self._authorize(
|
|
||||||
context,
|
|
||||||
action_name,
|
|
||||||
{"resource_type": "task", "resource_id": task_id},
|
|
||||||
)
|
|
||||||
with self.store.lock:
|
|
||||||
current = self.store.tasks.get(task_id)
|
|
||||||
if current is None:
|
|
||||||
raise NotFound("task not found")
|
|
||||||
self._assert_same_tenant(context, current.tenant_id)
|
|
||||||
if expected_revision is not None and current.revision != expected_revision:
|
|
||||||
raise RevisionConflict(
|
|
||||||
f"expected revision {expected_revision}, current {current.revision}"
|
|
||||||
)
|
|
||||||
if target not in ALLOWED_TASK_TRANSITIONS[current.state]:
|
|
||||||
raise InvalidTransition(
|
|
||||||
f"invalid transition {current.state.value} -> {target.value}"
|
|
||||||
)
|
|
||||||
if target == TaskState.COMPLETED and result is None:
|
|
||||||
raise ValueError("completed task requires result")
|
|
||||||
if target == TaskState.FAILED and error is None:
|
|
||||||
raise ValueError("failed task requires error")
|
|
||||||
updated = TaskRecord(
|
|
||||||
task_id=current.task_id,
|
|
||||||
tenant_id=current.tenant_id,
|
|
||||||
agent_id=current.agent_id,
|
|
||||||
action=current.action,
|
|
||||||
payload=current.payload,
|
|
||||||
state=target,
|
|
||||||
revision=current.revision + 1,
|
|
||||||
idempotency_key=current.idempotency_key,
|
|
||||||
request_digest=current.request_digest,
|
|
||||||
result=copy.deepcopy(dict(result)) if result is not None else current.result,
|
|
||||||
error=copy.deepcopy(dict(error)) if error is not None else current.error,
|
|
||||||
created_by=current.created_by,
|
|
||||||
approved_by=(
|
|
||||||
context.principal_id
|
|
||||||
if target == TaskState.APPROVED
|
|
||||||
else current.approved_by
|
|
||||||
),
|
|
||||||
created_at=current.created_at,
|
|
||||||
updated_at=_utc_now(),
|
|
||||||
)
|
|
||||||
self.store.tasks[task_id] = updated
|
|
||||||
self._audit(
|
|
||||||
context,
|
|
||||||
action=action_name,
|
|
||||||
resource_type="task",
|
|
||||||
resource_id=task_id,
|
|
||||||
outcome="allowed",
|
|
||||||
details={
|
|
||||||
"from": current.state.value,
|
|
||||||
"to": target.value,
|
|
||||||
"revision": updated.revision,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
return updated
|
|
||||||
|
|
||||||
def list_audit_events(
|
|
||||||
self,
|
|
||||||
context: RequestContext,
|
|
||||||
*,
|
|
||||||
resource_id: str | None = None,
|
|
||||||
) -> list[AuditEvent]:
|
|
||||||
self._authorize(
|
|
||||||
context,
|
|
||||||
"audit.read",
|
|
||||||
{"resource_type": "audit", "resource_id": resource_id or context.tenant_id},
|
|
||||||
)
|
|
||||||
with self.store.lock:
|
|
||||||
return [
|
|
||||||
event
|
|
||||||
for event in self.store.audit_events
|
|
||||||
if event.tenant_id == context.tenant_id
|
|
||||||
and (resource_id is None or event.resource_id == resource_id)
|
|
||||||
]
|
|
||||||
@@ -1,195 +0,0 @@
|
|||||||
"""Git-backed agent execution evidence for AIOS V2.
|
|
||||||
|
|
||||||
Evidence is tenant-scoped, append-friendly, and contains no runtime credentials.
|
|
||||||
It provides a deterministic source for claim, heartbeat, progress, result and
|
|
||||||
verification state across Codex, OpenClaw and Hermes executors.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import threading
|
|
||||||
from dataclasses import asdict, dataclass, field, replace
|
|
||||||
from datetime import datetime, timezone
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any, Mapping
|
|
||||||
|
|
||||||
|
|
||||||
def utc_now() -> str:
|
|
||||||
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
|
|
||||||
|
|
||||||
|
|
||||||
def _required(value: str, label: str) -> str:
|
|
||||||
normalized = str(value or "").strip()
|
|
||||||
if not normalized:
|
|
||||||
raise ValueError(f"{label} is required")
|
|
||||||
return normalized
|
|
||||||
|
|
||||||
|
|
||||||
def _safe(value: Mapping[str, Any] | None) -> dict[str, Any]:
|
|
||||||
return json.loads(json.dumps(dict(value or {}), ensure_ascii=False, default=str))
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class ExecutionEvidence:
|
|
||||||
execution_id: str
|
|
||||||
tenant_id: str
|
|
||||||
task_id: str
|
|
||||||
agent_id: str
|
|
||||||
executor: str
|
|
||||||
branch: str
|
|
||||||
status: str = "claimed"
|
|
||||||
progress_completed: int = 0
|
|
||||||
progress_total: int = 1
|
|
||||||
current_step: str = "claim"
|
|
||||||
last_commit: str | None = None
|
|
||||||
test_status: str = "not_run"
|
|
||||||
test_summary: Mapping[str, Any] = field(default_factory=dict)
|
|
||||||
result_summary: Mapping[str, Any] = field(default_factory=dict)
|
|
||||||
claimed_at: str = field(default_factory=utc_now)
|
|
||||||
heartbeat_at: str = field(default_factory=utc_now)
|
|
||||||
completed_at: str | None = None
|
|
||||||
|
|
||||||
def __post_init__(self) -> None:
|
|
||||||
for name in (
|
|
||||||
"execution_id",
|
|
||||||
"tenant_id",
|
|
||||||
"task_id",
|
|
||||||
"agent_id",
|
|
||||||
"executor",
|
|
||||||
"branch",
|
|
||||||
"status",
|
|
||||||
"current_step",
|
|
||||||
"claimed_at",
|
|
||||||
"heartbeat_at",
|
|
||||||
):
|
|
||||||
object.__setattr__(self, name, _required(getattr(self, name), name))
|
|
||||||
if self.progress_completed < 0 or self.progress_total < 1:
|
|
||||||
raise ValueError("invalid progress values")
|
|
||||||
if self.progress_completed > self.progress_total:
|
|
||||||
raise ValueError("progress_completed cannot exceed progress_total")
|
|
||||||
object.__setattr__(self, "test_summary", _safe(self.test_summary))
|
|
||||||
object.__setattr__(self, "result_summary", _safe(self.result_summary))
|
|
||||||
|
|
||||||
|
|
||||||
class ExecutionTracker:
|
|
||||||
"""Persist one JSON evidence record per tenant and execution."""
|
|
||||||
|
|
||||||
def __init__(self, root: str | Path) -> None:
|
|
||||||
self.root = Path(root).resolve()
|
|
||||||
self.root.mkdir(parents=True, exist_ok=True)
|
|
||||||
self._lock = threading.RLock()
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _component(value: str) -> str:
|
|
||||||
normalized = _required(value, "path component")
|
|
||||||
if normalized in {".", ".."} or "/" in normalized or "\\" in normalized:
|
|
||||||
raise ValueError("invalid evidence path component")
|
|
||||||
return normalized
|
|
||||||
|
|
||||||
def _path(self, tenant_id: str, execution_id: str) -> Path:
|
|
||||||
tenant = self._component(tenant_id)
|
|
||||||
execution = self._component(execution_id)
|
|
||||||
return self.root / tenant / f"{execution}.json"
|
|
||||||
|
|
||||||
def claim(self, evidence: ExecutionEvidence) -> ExecutionEvidence:
|
|
||||||
path = self._path(evidence.tenant_id, evidence.execution_id)
|
|
||||||
with self._lock:
|
|
||||||
if path.exists():
|
|
||||||
current = self._read(path)
|
|
||||||
if current.task_id != evidence.task_id or current.agent_id != evidence.agent_id:
|
|
||||||
raise RuntimeError("execution_id already belongs to another task")
|
|
||||||
return current
|
|
||||||
self._write(path, evidence)
|
|
||||||
return evidence
|
|
||||||
|
|
||||||
def heartbeat(
|
|
||||||
self,
|
|
||||||
*,
|
|
||||||
tenant_id: str,
|
|
||||||
execution_id: str,
|
|
||||||
status: str | None = None,
|
|
||||||
current_step: str | None = None,
|
|
||||||
progress_completed: int | None = None,
|
|
||||||
progress_total: int | None = None,
|
|
||||||
last_commit: str | None = None,
|
|
||||||
) -> ExecutionEvidence:
|
|
||||||
path = self._path(tenant_id, execution_id)
|
|
||||||
with self._lock:
|
|
||||||
current = self._read(path)
|
|
||||||
updated = replace(
|
|
||||||
current,
|
|
||||||
status=status or current.status,
|
|
||||||
current_step=current_step or current.current_step,
|
|
||||||
progress_completed=(
|
|
||||||
progress_completed
|
|
||||||
if progress_completed is not None
|
|
||||||
else current.progress_completed
|
|
||||||
),
|
|
||||||
progress_total=(
|
|
||||||
progress_total if progress_total is not None else current.progress_total
|
|
||||||
),
|
|
||||||
last_commit=last_commit or current.last_commit,
|
|
||||||
heartbeat_at=utc_now(),
|
|
||||||
)
|
|
||||||
self._write(path, updated)
|
|
||||||
return updated
|
|
||||||
|
|
||||||
def complete(
|
|
||||||
self,
|
|
||||||
*,
|
|
||||||
tenant_id: str,
|
|
||||||
execution_id: str,
|
|
||||||
status: str,
|
|
||||||
test_status: str,
|
|
||||||
test_summary: Mapping[str, Any] | None = None,
|
|
||||||
result_summary: Mapping[str, Any] | None = None,
|
|
||||||
last_commit: str | None = None,
|
|
||||||
) -> ExecutionEvidence:
|
|
||||||
path = self._path(tenant_id, execution_id)
|
|
||||||
with self._lock:
|
|
||||||
current = self._read(path)
|
|
||||||
updated = replace(
|
|
||||||
current,
|
|
||||||
status=_required(status, "status"),
|
|
||||||
current_step="complete",
|
|
||||||
progress_completed=current.progress_total,
|
|
||||||
last_commit=last_commit or current.last_commit,
|
|
||||||
test_status=_required(test_status, "test_status"),
|
|
||||||
test_summary=_safe(test_summary),
|
|
||||||
result_summary=_safe(result_summary),
|
|
||||||
heartbeat_at=utc_now(),
|
|
||||||
completed_at=utc_now(),
|
|
||||||
)
|
|
||||||
self._write(path, updated)
|
|
||||||
return updated
|
|
||||||
|
|
||||||
def get(self, *, tenant_id: str, execution_id: str) -> ExecutionEvidence:
|
|
||||||
return self._read(self._path(tenant_id, execution_id))
|
|
||||||
|
|
||||||
def list(self, *, tenant_id: str) -> list[ExecutionEvidence]:
|
|
||||||
tenant_dir = self.root / self._component(tenant_id)
|
|
||||||
if not tenant_dir.exists():
|
|
||||||
return []
|
|
||||||
records = [self._read(path) for path in tenant_dir.glob("*.json")]
|
|
||||||
return sorted(records, key=lambda item: (item.heartbeat_at, item.execution_id), reverse=True)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _read(path: Path) -> ExecutionEvidence:
|
|
||||||
try:
|
|
||||||
value = json.loads(path.read_text(encoding="utf-8"))
|
|
||||||
except FileNotFoundError as exc:
|
|
||||||
raise KeyError("execution evidence not found") from exc
|
|
||||||
if not isinstance(value, Mapping):
|
|
||||||
raise RuntimeError("invalid execution evidence file")
|
|
||||||
return ExecutionEvidence(**dict(value))
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _write(path: Path, evidence: ExecutionEvidence) -> None:
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
temporary = path.with_suffix(f".tmp-{os.getpid()}")
|
|
||||||
temporary.write_text(
|
|
||||||
json.dumps(asdict(evidence), ensure_ascii=False, sort_keys=True, indent=2) + "\n",
|
|
||||||
encoding="utf-8",
|
|
||||||
)
|
|
||||||
os.replace(temporary, path)
|
|
||||||
@@ -1,346 +0,0 @@
|
|||||||
"""HTTP/JSON gateway for the AIOS V2 control plane.
|
|
||||||
|
|
||||||
The gateway is intentionally runtime-neutral: browsers and external clients
|
|
||||||
call this API, while OpenClaw and Hermes remain behind authenticated adapters.
|
|
||||||
Identity headers are accepted only from a trusted reverse proxy in v1; direct
|
|
||||||
public exposure is prohibited until signed identity middleware is enabled.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import dataclasses
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import urllib.parse
|
|
||||||
from enum import Enum
|
|
||||||
from http import HTTPStatus
|
|
||||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
|
||||||
from typing import Any, Mapping
|
|
||||||
|
|
||||||
from .control_plane import (
|
|
||||||
ControlPlaneError,
|
|
||||||
ControlPlaneService,
|
|
||||||
IdempotencyConflict,
|
|
||||||
InvalidTransition,
|
|
||||||
NotFound,
|
|
||||||
PolicyDenied,
|
|
||||||
RequestContext,
|
|
||||||
RevisionConflict,
|
|
||||||
TaskState,
|
|
||||||
TenantIsolationError,
|
|
||||||
Unauthorized,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
TENANT_HEADER = "X-AIOS-Tenant-ID"
|
|
||||||
PRINCIPAL_HEADER = "X-AIOS-Principal-ID"
|
|
||||||
ROLES_HEADER = "X-AIOS-Roles"
|
|
||||||
TRACE_HEADER = "X-AIOS-Trace-ID"
|
|
||||||
SESSION_HEADER = "X-AIOS-Session-ID"
|
|
||||||
POLICY_HEADER = "X-AIOS-Policy-Version"
|
|
||||||
AUTHENTICATED_HEADER = "X-AIOS-Authenticated"
|
|
||||||
|
|
||||||
_RESOURCE_ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,191}$")
|
|
||||||
|
|
||||||
|
|
||||||
def _json_value(value: Any) -> Any:
|
|
||||||
if dataclasses.is_dataclass(value):
|
|
||||||
return {
|
|
||||||
field.name: _json_value(getattr(value, field.name))
|
|
||||||
for field in dataclasses.fields(value)
|
|
||||||
}
|
|
||||||
if isinstance(value, Enum):
|
|
||||||
return value.value
|
|
||||||
if isinstance(value, Mapping):
|
|
||||||
return {str(key): _json_value(item) for key, item in value.items()}
|
|
||||||
if isinstance(value, (list, tuple, set, frozenset)):
|
|
||||||
return [_json_value(item) for item in value]
|
|
||||||
return value
|
|
||||||
|
|
||||||
|
|
||||||
def _require_mapping(value: Any, *, label: str = "JSON body") -> Mapping[str, Any]:
|
|
||||||
if not isinstance(value, Mapping):
|
|
||||||
raise ValueError(f"{label} must be an object")
|
|
||||||
return value
|
|
||||||
|
|
||||||
|
|
||||||
def _require_resource_id(value: Any, *, label: str) -> str:
|
|
||||||
normalized = str(value or "").strip()
|
|
||||||
if not _RESOURCE_ID_RE.fullmatch(normalized):
|
|
||||||
raise ValueError(f"invalid {label}")
|
|
||||||
return normalized
|
|
||||||
|
|
||||||
|
|
||||||
class GatewayApplication:
|
|
||||||
"""Pure request router that can be tested without a listening socket."""
|
|
||||||
|
|
||||||
def __init__(self, service: ControlPlaneService | None = None) -> None:
|
|
||||||
self.service = service or ControlPlaneService()
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def context_from_headers(headers: Mapping[str, str]) -> RequestContext:
|
|
||||||
authenticated = str(headers.get(AUTHENTICATED_HEADER, "")).strip().lower()
|
|
||||||
if authenticated not in {"1", "true", "yes"}:
|
|
||||||
raise Unauthorized("trusted authentication assertion is required")
|
|
||||||
roles = [
|
|
||||||
role.strip()
|
|
||||||
for role in str(headers.get(ROLES_HEADER, "")).split(",")
|
|
||||||
if role.strip()
|
|
||||||
]
|
|
||||||
return RequestContext.create(
|
|
||||||
tenant_id=str(headers.get(TENANT_HEADER, "")),
|
|
||||||
principal_id=str(headers.get(PRINCIPAL_HEADER, "")),
|
|
||||||
trace_id=str(headers.get(TRACE_HEADER, "")).strip() or None,
|
|
||||||
session_id=str(headers.get(SESSION_HEADER, "")).strip() or None,
|
|
||||||
policy_version=str(headers.get(POLICY_HEADER, "v1")).strip() or "v1",
|
|
||||||
roles=roles,
|
|
||||||
)
|
|
||||||
|
|
||||||
def handle(
|
|
||||||
self,
|
|
||||||
method: str,
|
|
||||||
raw_path: str,
|
|
||||||
headers: Mapping[str, str],
|
|
||||||
body: Any = None,
|
|
||||||
) -> tuple[int, Mapping[str, Any]]:
|
|
||||||
parsed = urllib.parse.urlparse(raw_path)
|
|
||||||
path = parsed.path.rstrip("/") or "/"
|
|
||||||
query = urllib.parse.parse_qs(parsed.query, keep_blank_values=False)
|
|
||||||
method = method.upper()
|
|
||||||
|
|
||||||
if method == "GET" and path in {"/healthz", "/api/v2/health"}:
|
|
||||||
return HTTPStatus.OK, {
|
|
||||||
"service": "aios-v2-control-plane",
|
|
||||||
"status": "ok",
|
|
||||||
"store": type(self.service.store).__name__,
|
|
||||||
}
|
|
||||||
|
|
||||||
context = self.context_from_headers(headers)
|
|
||||||
payload = _require_mapping(body or {})
|
|
||||||
|
|
||||||
if method == "POST" and path == "/api/v2/tenants":
|
|
||||||
tenant = self.service.register_tenant(
|
|
||||||
context,
|
|
||||||
tenant_id=_require_resource_id(payload.get("tenant_id"), label="tenant_id"),
|
|
||||||
display_name=str(payload.get("display_name", "")),
|
|
||||||
metadata=_require_mapping(payload.get("metadata", {}), label="metadata"),
|
|
||||||
)
|
|
||||||
return HTTPStatus.CREATED, {"tenant": _json_value(tenant)}
|
|
||||||
|
|
||||||
if method == "GET" and path == "/api/v2/tenants/current":
|
|
||||||
tenant = self.service.get_tenant(context)
|
|
||||||
return HTTPStatus.OK, {"tenant": _json_value(tenant)}
|
|
||||||
|
|
||||||
if method == "POST" and path == "/api/v2/agents":
|
|
||||||
agent = self.service.register_agent(
|
|
||||||
context,
|
|
||||||
agent_id=_require_resource_id(payload.get("agent_id"), label="agent_id"),
|
|
||||||
display_name=str(payload.get("display_name", "")),
|
|
||||||
runtime=str(payload.get("runtime", "")),
|
|
||||||
capabilities=payload.get("capabilities", ()),
|
|
||||||
metadata=_require_mapping(payload.get("metadata", {}), label="metadata"),
|
|
||||||
)
|
|
||||||
return HTTPStatus.CREATED, {"agent": _json_value(agent)}
|
|
||||||
|
|
||||||
if method == "GET" and path == "/api/v2/agents":
|
|
||||||
return HTTPStatus.OK, {
|
|
||||||
"agents": _json_value(self.service.list_agents(context))
|
|
||||||
}
|
|
||||||
|
|
||||||
agent_match = re.fullmatch(r"/api/v2/agents/([^/]+)", path)
|
|
||||||
if method == "GET" and agent_match:
|
|
||||||
agent_id = _require_resource_id(
|
|
||||||
urllib.parse.unquote(agent_match.group(1)), label="agent_id"
|
|
||||||
)
|
|
||||||
return HTTPStatus.OK, {
|
|
||||||
"agent": _json_value(self.service.get_agent(context, agent_id))
|
|
||||||
}
|
|
||||||
|
|
||||||
if method == "POST" and path == "/api/v2/tasks":
|
|
||||||
task = self.service.create_task(
|
|
||||||
context,
|
|
||||||
agent_id=_require_resource_id(payload.get("agent_id"), label="agent_id"),
|
|
||||||
action=str(payload.get("action", "")).strip(),
|
|
||||||
payload=_require_mapping(payload.get("payload", {}), label="payload"),
|
|
||||||
idempotency_key=(
|
|
||||||
str(payload["idempotency_key"]).strip()
|
|
||||||
if payload.get("idempotency_key") is not None
|
|
||||||
else None
|
|
||||||
),
|
|
||||||
)
|
|
||||||
return HTTPStatus.CREATED, {"task": _json_value(task)}
|
|
||||||
|
|
||||||
if method == "GET" and path == "/api/v2/tasks":
|
|
||||||
requested_states = query.get("state")
|
|
||||||
tasks = self.service.list_tasks(context, states=requested_states)
|
|
||||||
return HTTPStatus.OK, {"tasks": _json_value(tasks)}
|
|
||||||
|
|
||||||
task_match = re.fullmatch(r"/api/v2/tasks/([^/]+)", path)
|
|
||||||
if method == "GET" and task_match:
|
|
||||||
task_id = _require_resource_id(
|
|
||||||
urllib.parse.unquote(task_match.group(1)), label="task_id"
|
|
||||||
)
|
|
||||||
return HTTPStatus.OK, {
|
|
||||||
"task": _json_value(self.service.get_task(context, task_id))
|
|
||||||
}
|
|
||||||
|
|
||||||
transition_match = re.fullmatch(
|
|
||||||
r"/api/v2/tasks/([^/]+)/(approve|start|complete|fail|cancel)", path
|
|
||||||
)
|
|
||||||
if method == "POST" and transition_match:
|
|
||||||
task_id = _require_resource_id(
|
|
||||||
urllib.parse.unquote(transition_match.group(1)), label="task_id"
|
|
||||||
)
|
|
||||||
transition = transition_match.group(2)
|
|
||||||
target = {
|
|
||||||
"approve": TaskState.APPROVED,
|
|
||||||
"start": TaskState.RUNNING,
|
|
||||||
"complete": TaskState.COMPLETED,
|
|
||||||
"fail": TaskState.FAILED,
|
|
||||||
"cancel": TaskState.CANCELLED,
|
|
||||||
}[transition]
|
|
||||||
expected_revision = payload.get("expected_revision")
|
|
||||||
if expected_revision is not None:
|
|
||||||
expected_revision = int(expected_revision)
|
|
||||||
task = self.service.transition_task(
|
|
||||||
context,
|
|
||||||
task_id=task_id,
|
|
||||||
target_state=target,
|
|
||||||
expected_revision=expected_revision,
|
|
||||||
result=(
|
|
||||||
_require_mapping(payload.get("result"), label="result")
|
|
||||||
if payload.get("result") is not None
|
|
||||||
else None
|
|
||||||
),
|
|
||||||
error=(
|
|
||||||
_require_mapping(payload.get("error"), label="error")
|
|
||||||
if payload.get("error") is not None
|
|
||||||
else None
|
|
||||||
),
|
|
||||||
)
|
|
||||||
return HTTPStatus.OK, {"task": _json_value(task)}
|
|
||||||
|
|
||||||
if method == "GET" and path == "/api/v2/audit":
|
|
||||||
resource_id = query.get("resource_id", [None])[0]
|
|
||||||
events = self.service.list_audit_events(
|
|
||||||
context,
|
|
||||||
resource_id=(
|
|
||||||
_require_resource_id(resource_id, label="resource_id")
|
|
||||||
if resource_id
|
|
||||||
else None
|
|
||||||
),
|
|
||||||
)
|
|
||||||
return HTTPStatus.OK, {"events": _json_value(events)}
|
|
||||||
|
|
||||||
return HTTPStatus.NOT_FOUND, {
|
|
||||||
"error": "not_found",
|
|
||||||
"message": "route not found",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
_ERROR_STATUS: tuple[tuple[type[BaseException], HTTPStatus], ...] = (
|
|
||||||
(Unauthorized, HTTPStatus.UNAUTHORIZED),
|
|
||||||
(PolicyDenied, HTTPStatus.FORBIDDEN),
|
|
||||||
(TenantIsolationError, HTTPStatus.NOT_FOUND),
|
|
||||||
(NotFound, HTTPStatus.NOT_FOUND),
|
|
||||||
(IdempotencyConflict, HTTPStatus.CONFLICT),
|
|
||||||
(RevisionConflict, HTTPStatus.CONFLICT),
|
|
||||||
(InvalidTransition, HTTPStatus.CONFLICT),
|
|
||||||
(ValueError, HTTPStatus.BAD_REQUEST),
|
|
||||||
(ControlPlaneError, HTTPStatus.BAD_REQUEST),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def error_response(exc: BaseException) -> tuple[int, Mapping[str, Any]]:
|
|
||||||
for error_type, status in _ERROR_STATUS:
|
|
||||||
if isinstance(exc, error_type):
|
|
||||||
return status, {
|
|
||||||
"error": error_type.__name__,
|
|
||||||
"message": str(exc),
|
|
||||||
}
|
|
||||||
return HTTPStatus.INTERNAL_SERVER_ERROR, {
|
|
||||||
"error": "InternalServerError",
|
|
||||||
"message": "internal server error",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
class GatewayRequestHandler(BaseHTTPRequestHandler):
|
|
||||||
server_version = "AIOSV2Gateway/1.0"
|
|
||||||
max_body_bytes = 1_048_576
|
|
||||||
|
|
||||||
@property
|
|
||||||
def application(self) -> GatewayApplication:
|
|
||||||
return self.server.application # type: ignore[attr-defined]
|
|
||||||
|
|
||||||
def _read_body(self) -> Any:
|
|
||||||
length = int(self.headers.get("Content-Length", "0") or "0")
|
|
||||||
if length < 0 or length > self.max_body_bytes:
|
|
||||||
raise ValueError("request body is too large")
|
|
||||||
if length == 0:
|
|
||||||
return {}
|
|
||||||
content_type = self.headers.get("Content-Type", "").split(";", 1)[0].strip()
|
|
||||||
if content_type != "application/json":
|
|
||||||
raise ValueError("Content-Type must be application/json")
|
|
||||||
raw = self.rfile.read(length)
|
|
||||||
return json.loads(raw.decode("utf-8"))
|
|
||||||
|
|
||||||
def _send_json(self, status: int, payload: Mapping[str, Any]) -> None:
|
|
||||||
encoded = json.dumps(
|
|
||||||
_json_value(payload), ensure_ascii=False, separators=(",", ":")
|
|
||||||
).encode("utf-8")
|
|
||||||
self.send_response(int(status))
|
|
||||||
self.send_header("Content-Type", "application/json; charset=utf-8")
|
|
||||||
self.send_header("Content-Length", str(len(encoded)))
|
|
||||||
self.send_header("Cache-Control", "no-store")
|
|
||||||
self.send_header("X-Content-Type-Options", "nosniff")
|
|
||||||
self.end_headers()
|
|
||||||
self.wfile.write(encoded)
|
|
||||||
|
|
||||||
def _dispatch(self, method: str) -> None:
|
|
||||||
try:
|
|
||||||
body = self._read_body() if method in {"POST", "PUT", "PATCH"} else {}
|
|
||||||
status, payload = self.application.handle(
|
|
||||||
method,
|
|
||||||
self.path,
|
|
||||||
{key: value for key, value in self.headers.items()},
|
|
||||||
body,
|
|
||||||
)
|
|
||||||
except BaseException as exc: # handler boundary; sanitized below
|
|
||||||
status, payload = error_response(exc)
|
|
||||||
self._send_json(status, payload)
|
|
||||||
|
|
||||||
def do_GET(self) -> None: # noqa: N802
|
|
||||||
self._dispatch("GET")
|
|
||||||
|
|
||||||
def do_POST(self) -> None: # noqa: N802
|
|
||||||
self._dispatch("POST")
|
|
||||||
|
|
||||||
def log_message(self, format: str, *args: Any) -> None:
|
|
||||||
# Production deployment routes structured access logs through the
|
|
||||||
# reverse proxy; do not log identity headers or request bodies here.
|
|
||||||
super().log_message(format, *args)
|
|
||||||
|
|
||||||
|
|
||||||
class GatewayHTTPServer(ThreadingHTTPServer):
|
|
||||||
daemon_threads = True
|
|
||||||
allow_reuse_address = True
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
server_address: tuple[str, int],
|
|
||||||
application: GatewayApplication,
|
|
||||||
) -> None:
|
|
||||||
self.application = application
|
|
||||||
super().__init__(server_address, GatewayRequestHandler)
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
|
||||||
host = os.getenv("AIOS_V2_GATEWAY_HOST", "127.0.0.1")
|
|
||||||
port = int(os.getenv("AIOS_V2_GATEWAY_PORT", "18920"))
|
|
||||||
server = GatewayHTTPServer((host, port), GatewayApplication())
|
|
||||||
print(f"AIOS V2 control-plane gateway listening on {host}:{port}")
|
|
||||||
server.serve_forever()
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
@@ -1,123 +0,0 @@
|
|||||||
"""Sanitized GitOps runtime status for the AIOS Web control plane."""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import os
|
|
||||||
import subprocess
|
|
||||||
from dataclasses import dataclass, asdict
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any, Mapping
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class GitOpsStatus:
|
|
||||||
available: bool
|
|
||||||
service_status: str
|
|
||||||
workspace_mode: str
|
|
||||||
workspace_branch: str | None
|
|
||||||
workspace_ref: str | None
|
|
||||||
workspace_dirty: bool | None
|
|
||||||
candidate_branch: str
|
|
||||||
release_channel: str
|
|
||||||
release_version: str
|
|
||||||
sync_status: str
|
|
||||||
details: Mapping[str, Any]
|
|
||||||
|
|
||||||
def as_dict(self) -> dict[str, Any]:
|
|
||||||
return asdict(self)
|
|
||||||
|
|
||||||
|
|
||||||
def _run_git(worktree: Path, *args: str) -> tuple[int, str, str]:
|
|
||||||
process = subprocess.run(
|
|
||||||
["git", "-C", str(worktree), *args],
|
|
||||||
text=True,
|
|
||||||
capture_output=True,
|
|
||||||
timeout=5,
|
|
||||||
)
|
|
||||||
return process.returncode, process.stdout.strip(), process.stderr.strip()
|
|
||||||
|
|
||||||
|
|
||||||
def collect_gitops_status() -> GitOpsStatus:
|
|
||||||
"""Return only non-sensitive Git/workspace metadata.
|
|
||||||
|
|
||||||
Remote URLs, environment values, credentials, filesystem contents and
|
|
||||||
command stderr are deliberately excluded from the Web response.
|
|
||||||
"""
|
|
||||||
|
|
||||||
candidate_branch = os.getenv(
|
|
||||||
"AIOS_CANDIDATE_BRANCH",
|
|
||||||
"integration/aios-v2r1-cto-production-candidate-20260720",
|
|
||||||
)
|
|
||||||
release_channel = os.getenv("AIOS_RELEASE_CHANNEL", "candidate")
|
|
||||||
release_version = os.getenv("AIOS_RELEASE_VERSION", "V2R1C0-candidate")
|
|
||||||
sync_status = os.getenv("AIOS_GITOPS_SYNC_STATUS", "unknown")
|
|
||||||
worktree = Path(
|
|
||||||
os.getenv(
|
|
||||||
"AIOS_GITOPS_CONTROLLER_WORKTREE",
|
|
||||||
"/data/aios-gitops/worktrees/controller",
|
|
||||||
)
|
|
||||||
)
|
|
||||||
|
|
||||||
if not worktree.is_dir():
|
|
||||||
return GitOpsStatus(
|
|
||||||
available=False,
|
|
||||||
service_status="unknown",
|
|
||||||
workspace_mode="unavailable",
|
|
||||||
workspace_branch=None,
|
|
||||||
workspace_ref=None,
|
|
||||||
workspace_dirty=None,
|
|
||||||
candidate_branch=candidate_branch,
|
|
||||||
release_channel=release_channel,
|
|
||||||
release_version=release_version,
|
|
||||||
sync_status=sync_status,
|
|
||||||
details={"reason": "controller_worktree_unavailable"},
|
|
||||||
)
|
|
||||||
|
|
||||||
inside_code, inside, _ = _run_git(worktree, "rev-parse", "--is-inside-work-tree")
|
|
||||||
if inside_code != 0 or inside != "true":
|
|
||||||
return GitOpsStatus(
|
|
||||||
available=False,
|
|
||||||
service_status="unknown",
|
|
||||||
workspace_mode="invalid",
|
|
||||||
workspace_branch=None,
|
|
||||||
workspace_ref=None,
|
|
||||||
workspace_dirty=None,
|
|
||||||
candidate_branch=candidate_branch,
|
|
||||||
release_channel=release_channel,
|
|
||||||
release_version=release_version,
|
|
||||||
sync_status=sync_status,
|
|
||||||
details={"reason": "controller_worktree_invalid"},
|
|
||||||
)
|
|
||||||
|
|
||||||
branch_code, branch, _ = _run_git(worktree, "branch", "--show-current")
|
|
||||||
ref_code, ref_value, _ = _run_git(worktree, "rev-parse", "HEAD")
|
|
||||||
dirty_code, dirty_value, _ = _run_git(worktree, "status", "--porcelain=v1")
|
|
||||||
|
|
||||||
service_status = "unknown"
|
|
||||||
try:
|
|
||||||
process = subprocess.run(
|
|
||||||
["systemctl", "is-active", "aios-gitops-model-api"],
|
|
||||||
text=True,
|
|
||||||
capture_output=True,
|
|
||||||
timeout=3,
|
|
||||||
)
|
|
||||||
service_status = process.stdout.strip() or "inactive"
|
|
||||||
except (FileNotFoundError, subprocess.TimeoutExpired):
|
|
||||||
service_status = "unavailable"
|
|
||||||
|
|
||||||
branch_value = branch if branch_code == 0 and branch else None
|
|
||||||
return GitOpsStatus(
|
|
||||||
available=True,
|
|
||||||
service_status=service_status,
|
|
||||||
workspace_mode="branch" if branch_value else "detached",
|
|
||||||
workspace_branch=branch_value,
|
|
||||||
workspace_ref=ref_value if ref_code == 0 else None,
|
|
||||||
workspace_dirty=bool(dirty_value) if dirty_code == 0 else None,
|
|
||||||
candidate_branch=candidate_branch,
|
|
||||||
release_channel=release_channel,
|
|
||||||
release_version=release_version,
|
|
||||||
sync_status=sync_status,
|
|
||||||
details={
|
|
||||||
"controller_clean": dirty_code == 0 and not dirty_value,
|
|
||||||
"controller_detached": branch_code == 0 and not branch,
|
|
||||||
},
|
|
||||||
)
|
|
||||||
@@ -1,264 +0,0 @@
|
|||||||
"""AIOS V2 execution orchestrator.
|
|
||||||
|
|
||||||
The orchestrator is the only component allowed to translate an approved AIOS
|
|
||||||
task into an OpenClaw command or Hermes job. OpenClaw and Hermes remain pure
|
|
||||||
execution runtimes and never own tenant policy or task lifecycle.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from typing import Any, Mapping
|
|
||||||
|
|
||||||
from aios_hermes import HermesAdapter, HermesJob, HermesResult
|
|
||||||
from aios_openclaw import (
|
|
||||||
CommandState,
|
|
||||||
OpenClawAdapter,
|
|
||||||
OpenClawCommand,
|
|
||||||
OpenClawResult,
|
|
||||||
sign_command,
|
|
||||||
)
|
|
||||||
|
|
||||||
from .control_plane import (
|
|
||||||
AgentRecord,
|
|
||||||
ControlPlaneService,
|
|
||||||
RequestContext,
|
|
||||||
TaskRecord,
|
|
||||||
TaskState,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class OrchestrationError(RuntimeError):
|
|
||||||
"""AIOS failed to dispatch or reconcile a runtime execution."""
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class DispatchRecord:
|
|
||||||
tenant_id: str
|
|
||||||
task_id: str
|
|
||||||
agent_id: str
|
|
||||||
runtime: str
|
|
||||||
runtime_execution_id: str
|
|
||||||
state: str
|
|
||||||
trace_id: str
|
|
||||||
|
|
||||||
|
|
||||||
class ExecutionOrchestrator:
|
|
||||||
"""Dispatch approved AIOS tasks to runtime adapters and reconcile results."""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
control_plane: ControlPlaneService,
|
|
||||||
*,
|
|
||||||
openclaw: OpenClawAdapter | None = None,
|
|
||||||
hermes: HermesAdapter | None = None,
|
|
||||||
openclaw_signing_material_resolver=None,
|
|
||||||
openclaw_key_id: str = "default",
|
|
||||||
default_cell_id: str = "default",
|
|
||||||
default_profile_id: str = "default",
|
|
||||||
) -> None:
|
|
||||||
self.control_plane = control_plane
|
|
||||||
self.openclaw = openclaw
|
|
||||||
self.hermes = hermes
|
|
||||||
self.openclaw_signing_material_resolver = openclaw_signing_material_resolver
|
|
||||||
self.openclaw_key_id = openclaw_key_id
|
|
||||||
self.default_cell_id = default_cell_id
|
|
||||||
self.default_profile_id = default_profile_id
|
|
||||||
self._dispatches: dict[tuple[str, str], DispatchRecord] = {}
|
|
||||||
|
|
||||||
def dispatch(
|
|
||||||
self,
|
|
||||||
context: RequestContext,
|
|
||||||
*,
|
|
||||||
task_id: str,
|
|
||||||
timeout_seconds: float | None = None,
|
|
||||||
) -> DispatchRecord:
|
|
||||||
task = self.control_plane.get_task(context, task_id)
|
|
||||||
if task.state != TaskState.APPROVED:
|
|
||||||
raise OrchestrationError("only APPROVED tasks may be dispatched")
|
|
||||||
agent = self.control_plane.get_agent(context, task.agent_id)
|
|
||||||
existing = self._dispatches.get((task.tenant_id, task.task_id))
|
|
||||||
if existing is not None:
|
|
||||||
return existing
|
|
||||||
|
|
||||||
running = self.control_plane.transition_task(
|
|
||||||
context,
|
|
||||||
task_id=task.task_id,
|
|
||||||
target_state=TaskState.RUNNING,
|
|
||||||
expected_revision=task.revision,
|
|
||||||
)
|
|
||||||
|
|
||||||
runtime = agent.runtime.strip().lower()
|
|
||||||
try:
|
|
||||||
if runtime == "openclaw":
|
|
||||||
record = self._dispatch_openclaw(
|
|
||||||
context,
|
|
||||||
agent,
|
|
||||||
running,
|
|
||||||
timeout_seconds=timeout_seconds,
|
|
||||||
)
|
|
||||||
elif runtime == "hermes":
|
|
||||||
record = self._dispatch_hermes(context, agent, running)
|
|
||||||
else:
|
|
||||||
raise OrchestrationError(f"unsupported runtime: {agent.runtime}")
|
|
||||||
except Exception as exc:
|
|
||||||
current = self.control_plane.get_task(context, task.task_id)
|
|
||||||
if current.state == TaskState.RUNNING:
|
|
||||||
self.control_plane.transition_task(
|
|
||||||
context,
|
|
||||||
task_id=task.task_id,
|
|
||||||
target_state=TaskState.FAILED,
|
|
||||||
expected_revision=current.revision,
|
|
||||||
error={"code": "dispatch_failed", "message": str(exc)},
|
|
||||||
)
|
|
||||||
raise
|
|
||||||
|
|
||||||
self._dispatches[(task.tenant_id, task.task_id)] = record
|
|
||||||
return record
|
|
||||||
|
|
||||||
def _dispatch_openclaw(
|
|
||||||
self,
|
|
||||||
context: RequestContext,
|
|
||||||
agent: AgentRecord,
|
|
||||||
task: TaskRecord,
|
|
||||||
*,
|
|
||||||
timeout_seconds: float | None,
|
|
||||||
) -> DispatchRecord:
|
|
||||||
if self.openclaw is None or self.openclaw_signing_material_resolver is None:
|
|
||||||
raise OrchestrationError("OpenClaw runtime is not configured")
|
|
||||||
key_material = self.openclaw_signing_material_resolver(
|
|
||||||
task.tenant_id,
|
|
||||||
self.openclaw_key_id,
|
|
||||||
)
|
|
||||||
command = OpenClawCommand.create(
|
|
||||||
tenant_id=task.tenant_id,
|
|
||||||
task_id=task.task_id,
|
|
||||||
trace_id=context.trace_id,
|
|
||||||
action=task.action,
|
|
||||||
payload=task.payload,
|
|
||||||
idempotency_key=task.idempotency_key or task.task_id,
|
|
||||||
key_id=self.openclaw_key_id,
|
|
||||||
)
|
|
||||||
command = sign_command(command, key_material)
|
|
||||||
result = self.openclaw.execute(command, timeout_seconds=timeout_seconds)
|
|
||||||
self.reconcile_openclaw(context, result)
|
|
||||||
return DispatchRecord(
|
|
||||||
tenant_id=task.tenant_id,
|
|
||||||
task_id=task.task_id,
|
|
||||||
agent_id=agent.agent_id,
|
|
||||||
runtime="openclaw",
|
|
||||||
runtime_execution_id=command.command_id,
|
|
||||||
state=result.state.value,
|
|
||||||
trace_id=context.trace_id,
|
|
||||||
)
|
|
||||||
|
|
||||||
def _dispatch_hermes(
|
|
||||||
self,
|
|
||||||
context: RequestContext,
|
|
||||||
agent: AgentRecord,
|
|
||||||
task: TaskRecord,
|
|
||||||
) -> DispatchRecord:
|
|
||||||
if self.hermes is None:
|
|
||||||
raise OrchestrationError("Hermes runtime is not configured")
|
|
||||||
metadata = dict(agent.metadata)
|
|
||||||
job = HermesJob.create(
|
|
||||||
tenant_id=task.tenant_id,
|
|
||||||
cell_id=str(metadata.get("cell_id") or self.default_cell_id),
|
|
||||||
profile_id=str(metadata.get("profile_id") or self.default_profile_id),
|
|
||||||
task_id=task.task_id,
|
|
||||||
trace_id=context.trace_id,
|
|
||||||
action=task.action,
|
|
||||||
payload=task.payload,
|
|
||||||
idempotency_key=task.idempotency_key or task.task_id,
|
|
||||||
priority=int(metadata.get("priority", 100)),
|
|
||||||
)
|
|
||||||
queued = self.hermes.enqueue(job)
|
|
||||||
return DispatchRecord(
|
|
||||||
tenant_id=task.tenant_id,
|
|
||||||
task_id=task.task_id,
|
|
||||||
agent_id=agent.agent_id,
|
|
||||||
runtime="hermes",
|
|
||||||
runtime_execution_id=queued.job_id,
|
|
||||||
state=queued.state.value,
|
|
||||||
trace_id=context.trace_id,
|
|
||||||
)
|
|
||||||
|
|
||||||
def reconcile_openclaw(
|
|
||||||
self,
|
|
||||||
context: RequestContext,
|
|
||||||
result: OpenClawResult,
|
|
||||||
) -> TaskRecord:
|
|
||||||
task = self.control_plane.get_task(context, result.task_id)
|
|
||||||
if result.tenant_id != task.tenant_id:
|
|
||||||
raise OrchestrationError("OpenClaw result tenant mismatch")
|
|
||||||
if task.state != TaskState.RUNNING:
|
|
||||||
return task
|
|
||||||
if result.state == CommandState.COMPLETED:
|
|
||||||
return self.control_plane.transition_task(
|
|
||||||
context,
|
|
||||||
task_id=task.task_id,
|
|
||||||
target_state=TaskState.COMPLETED,
|
|
||||||
expected_revision=task.revision,
|
|
||||||
result=result.output or {},
|
|
||||||
)
|
|
||||||
if result.state in {
|
|
||||||
CommandState.FAILED,
|
|
||||||
CommandState.TIMED_OUT,
|
|
||||||
CommandState.REJECTED,
|
|
||||||
}:
|
|
||||||
return self.control_plane.transition_task(
|
|
||||||
context,
|
|
||||||
task_id=task.task_id,
|
|
||||||
target_state=TaskState.FAILED,
|
|
||||||
expected_revision=task.revision,
|
|
||||||
error=result.error or {"code": result.state.value.lower()},
|
|
||||||
)
|
|
||||||
if result.state == CommandState.CANCELLED:
|
|
||||||
return self.control_plane.transition_task(
|
|
||||||
context,
|
|
||||||
task_id=task.task_id,
|
|
||||||
target_state=TaskState.CANCELLED,
|
|
||||||
expected_revision=task.revision,
|
|
||||||
)
|
|
||||||
return task
|
|
||||||
|
|
||||||
def reconcile_hermes(
|
|
||||||
self,
|
|
||||||
context: RequestContext,
|
|
||||||
result: HermesResult,
|
|
||||||
) -> TaskRecord:
|
|
||||||
task = self.control_plane.get_task(context, result.task_id)
|
|
||||||
if result.tenant_id != task.tenant_id:
|
|
||||||
raise OrchestrationError("Hermes result tenant mismatch")
|
|
||||||
if task.state != TaskState.RUNNING:
|
|
||||||
return task
|
|
||||||
if result.state.value == "COMPLETED":
|
|
||||||
return self.control_plane.transition_task(
|
|
||||||
context,
|
|
||||||
task_id=task.task_id,
|
|
||||||
target_state=TaskState.COMPLETED,
|
|
||||||
expected_revision=task.revision,
|
|
||||||
result=result.output or {},
|
|
||||||
)
|
|
||||||
if result.state.value == "FAILED":
|
|
||||||
return self.control_plane.transition_task(
|
|
||||||
context,
|
|
||||||
task_id=task.task_id,
|
|
||||||
target_state=TaskState.FAILED,
|
|
||||||
expected_revision=task.revision,
|
|
||||||
error=result.error or {"code": "hermes_failed"},
|
|
||||||
)
|
|
||||||
if result.state.value == "CANCELLED":
|
|
||||||
return self.control_plane.transition_task(
|
|
||||||
context,
|
|
||||||
task_id=task.task_id,
|
|
||||||
target_state=TaskState.CANCELLED,
|
|
||||||
expected_revision=task.revision,
|
|
||||||
)
|
|
||||||
return task
|
|
||||||
|
|
||||||
def dispatch_snapshot(self, *, tenant_id: str) -> Mapping[str, DispatchRecord]:
|
|
||||||
return {
|
|
||||||
task_id: record
|
|
||||||
for (record_tenant, task_id), record in self._dispatches.items()
|
|
||||||
if record_tenant == tenant_id
|
|
||||||
}
|
|
||||||
@@ -1,181 +0,0 @@
|
|||||||
"""Durable single-process persistence for the AIOS V2 control plane.
|
|
||||||
|
|
||||||
The candidate uses an atomic SQLite snapshot so a gateway restart does not lose
|
|
||||||
tenants, agents, task state, idempotency records, or audit events. The domain
|
|
||||||
service remains storage-agnostic; a later HA backend can replace this module
|
|
||||||
without changing API or runtime-adapter contracts.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import sqlite3
|
|
||||||
import threading
|
|
||||||
from dataclasses import asdict
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any, Callable
|
|
||||||
|
|
||||||
from .control_plane import (
|
|
||||||
AgentRecord,
|
|
||||||
AuditEvent,
|
|
||||||
ControlPlaneService,
|
|
||||||
InMemoryControlPlaneStore,
|
|
||||||
TaskRecord,
|
|
||||||
TaskState,
|
|
||||||
TenantRecord,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class SQLiteSnapshotRepository:
|
|
||||||
"""Store one versioned control-plane snapshot in SQLite."""
|
|
||||||
|
|
||||||
SCHEMA_VERSION = 1
|
|
||||||
|
|
||||||
def __init__(self, path: str | Path) -> None:
|
|
||||||
self.path = Path(path).resolve()
|
|
||||||
self.path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
self._lock = threading.RLock()
|
|
||||||
self._initialize()
|
|
||||||
|
|
||||||
def _connect(self) -> sqlite3.Connection:
|
|
||||||
connection = sqlite3.connect(self.path, timeout=30.0)
|
|
||||||
connection.execute("PRAGMA journal_mode=WAL")
|
|
||||||
connection.execute("PRAGMA synchronous=FULL")
|
|
||||||
connection.execute("PRAGMA foreign_keys=ON")
|
|
||||||
return connection
|
|
||||||
|
|
||||||
def _initialize(self) -> None:
|
|
||||||
with self._connect() as connection:
|
|
||||||
connection.execute(
|
|
||||||
"""
|
|
||||||
CREATE TABLE IF NOT EXISTS control_plane_snapshot (
|
|
||||||
singleton_id INTEGER PRIMARY KEY CHECK (singleton_id = 1),
|
|
||||||
schema_version INTEGER NOT NULL,
|
|
||||||
payload_json TEXT NOT NULL,
|
|
||||||
updated_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP
|
|
||||||
)
|
|
||||||
"""
|
|
||||||
)
|
|
||||||
|
|
||||||
def load(self) -> dict[str, Any] | None:
|
|
||||||
with self._lock, self._connect() as connection:
|
|
||||||
row = connection.execute(
|
|
||||||
"SELECT schema_version, payload_json FROM control_plane_snapshot WHERE singleton_id = 1"
|
|
||||||
).fetchone()
|
|
||||||
if row is None:
|
|
||||||
return None
|
|
||||||
schema_version, payload_json = row
|
|
||||||
if int(schema_version) != self.SCHEMA_VERSION:
|
|
||||||
raise RuntimeError(
|
|
||||||
f"unsupported control-plane snapshot version: {schema_version}"
|
|
||||||
)
|
|
||||||
decoded = json.loads(str(payload_json))
|
|
||||||
if not isinstance(decoded, dict):
|
|
||||||
raise RuntimeError("invalid control-plane snapshot")
|
|
||||||
return decoded
|
|
||||||
|
|
||||||
def save(self, payload: dict[str, Any]) -> None:
|
|
||||||
encoded = json.dumps(
|
|
||||||
payload,
|
|
||||||
ensure_ascii=False,
|
|
||||||
sort_keys=True,
|
|
||||||
separators=(",", ":"),
|
|
||||||
default=str,
|
|
||||||
)
|
|
||||||
with self._lock, self._connect() as connection:
|
|
||||||
connection.execute("BEGIN IMMEDIATE")
|
|
||||||
connection.execute(
|
|
||||||
"""
|
|
||||||
INSERT INTO control_plane_snapshot (
|
|
||||||
singleton_id, schema_version, payload_json, updated_at
|
|
||||||
) VALUES (1, ?, ?, CURRENT_TIMESTAMP)
|
|
||||||
ON CONFLICT(singleton_id) DO UPDATE SET
|
|
||||||
schema_version = excluded.schema_version,
|
|
||||||
payload_json = excluded.payload_json,
|
|
||||||
updated_at = CURRENT_TIMESTAMP
|
|
||||||
""",
|
|
||||||
(self.SCHEMA_VERSION, encoded),
|
|
||||||
)
|
|
||||||
connection.commit()
|
|
||||||
|
|
||||||
|
|
||||||
class DurableControlPlaneService(ControlPlaneService):
|
|
||||||
"""Control-plane service that persists after every successful mutation."""
|
|
||||||
|
|
||||||
def __init__(
|
|
||||||
self,
|
|
||||||
repository: SQLiteSnapshotRepository,
|
|
||||||
*,
|
|
||||||
policy_evaluator=None,
|
|
||||||
) -> None:
|
|
||||||
self.repository = repository
|
|
||||||
store = InMemoryControlPlaneStore()
|
|
||||||
self._restore(store, repository.load())
|
|
||||||
super().__init__(store, policy_evaluator=policy_evaluator)
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _restore(
|
|
||||||
store: InMemoryControlPlaneStore,
|
|
||||||
snapshot: dict[str, Any] | None,
|
|
||||||
) -> None:
|
|
||||||
if not snapshot:
|
|
||||||
return
|
|
||||||
for item in snapshot.get("tenants", []):
|
|
||||||
record = TenantRecord(**item)
|
|
||||||
store.tenants[record.tenant_id] = record
|
|
||||||
for item in snapshot.get("agents", []):
|
|
||||||
normalized = dict(item)
|
|
||||||
normalized["capabilities"] = tuple(normalized.get("capabilities", ()))
|
|
||||||
record = AgentRecord(**normalized)
|
|
||||||
store.agents[record.agent_id] = record
|
|
||||||
for item in snapshot.get("tasks", []):
|
|
||||||
normalized = dict(item)
|
|
||||||
normalized["state"] = TaskState(normalized["state"])
|
|
||||||
record = TaskRecord(**normalized)
|
|
||||||
store.tasks[record.task_id] = record
|
|
||||||
store.audit_events.extend(AuditEvent(**item) for item in snapshot.get("audit", []))
|
|
||||||
for item in snapshot.get("idempotency", []):
|
|
||||||
store.idempotency[(item["tenant_id"], item["key"])] = (
|
|
||||||
item["task_id"],
|
|
||||||
item["digest"],
|
|
||||||
)
|
|
||||||
|
|
||||||
def _snapshot(self) -> dict[str, Any]:
|
|
||||||
with self.store.lock:
|
|
||||||
return {
|
|
||||||
"schema_version": self.repository.SCHEMA_VERSION,
|
|
||||||
"tenants": [asdict(item) for item in self.store.tenants.values()],
|
|
||||||
"agents": [asdict(item) for item in self.store.agents.values()],
|
|
||||||
"tasks": [
|
|
||||||
{**asdict(item), "state": item.state.value}
|
|
||||||
for item in self.store.tasks.values()
|
|
||||||
],
|
|
||||||
"audit": [asdict(item) for item in self.store.audit_events],
|
|
||||||
"idempotency": [
|
|
||||||
{
|
|
||||||
"tenant_id": tenant_id,
|
|
||||||
"key": key,
|
|
||||||
"task_id": task_id,
|
|
||||||
"digest": digest,
|
|
||||||
}
|
|
||||||
for (tenant_id, key), (task_id, digest)
|
|
||||||
in self.store.idempotency.items()
|
|
||||||
],
|
|
||||||
}
|
|
||||||
|
|
||||||
def _mutate(self, operation: Callable[[], Any]) -> Any:
|
|
||||||
with self.store.lock:
|
|
||||||
result = operation()
|
|
||||||
self.repository.save(self._snapshot())
|
|
||||||
return result
|
|
||||||
|
|
||||||
def register_tenant(self, *args, **kwargs):
|
|
||||||
return self._mutate(lambda: super(DurableControlPlaneService, self).register_tenant(*args, **kwargs))
|
|
||||||
|
|
||||||
def register_agent(self, *args, **kwargs):
|
|
||||||
return self._mutate(lambda: super(DurableControlPlaneService, self).register_agent(*args, **kwargs))
|
|
||||||
|
|
||||||
def create_task(self, *args, **kwargs):
|
|
||||||
return self._mutate(lambda: super(DurableControlPlaneService, self).create_task(*args, **kwargs))
|
|
||||||
|
|
||||||
def transition_task(self, *args, **kwargs):
|
|
||||||
return self._mutate(lambda: super(DurableControlPlaneService, self).transition_task(*args, **kwargs))
|
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
{
|
||||||
|
"active_dir": "/data/aios-gitops/workspace/agents/commands/active",
|
||||||
|
"completed_dir": "/data/aios-gitops/workspace/agents/commands/completed",
|
||||||
|
"dead_letter_dir": "/data/aios-gitops/workspace/agents/commands/dead-letter",
|
||||||
|
"database_path": "/var/lib/aios-codex-watcher/state.sqlite3",
|
||||||
|
"audit_log_path": "/var/log/aios-codex-watcher/audit.jsonl",
|
||||||
|
"gateway_endpoint": "http://127.0.0.1:18920/v1/orchestrator/events",
|
||||||
|
"gateway_token_file": "/etc/aios-orchestrator-gateway/inbound.token",
|
||||||
|
"contract_commit": "cbe33ecab8e9794ba5f318bea43aeaf447a56fbc",
|
||||||
|
"lease_seconds": 60,
|
||||||
|
"retry_seconds": 10,
|
||||||
|
"max_attempts": 5,
|
||||||
|
"poll_seconds": 5,
|
||||||
|
"executor_path": null,
|
||||||
|
"executor_args": [],
|
||||||
|
"executor_timeout_seconds": 1800
|
||||||
|
}
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
{
|
||||||
|
"host": "127.0.0.1",
|
||||||
|
"port": 18920,
|
||||||
|
"database_path": "/var/lib/aios-orchestrator-gateway/state.sqlite3",
|
||||||
|
"event_inbox_dir": "/var/lib/aios-orchestrator-gateway/inbox/events",
|
||||||
|
"decision_inbox_dir": "/var/lib/aios-orchestrator-gateway/inbox/decisions",
|
||||||
|
"dispatch_outbox_dir": "/var/lib/aios-orchestrator-gateway/outbox",
|
||||||
|
"openclaw_sync_dir": "/var/lib/aios-orchestrator-gateway/openclaw-sync",
|
||||||
|
"command_queue_dir": "/var/lib/aios-orchestrator-gateway/commands",
|
||||||
|
"inbound_token_file": "/etc/aios-orchestrator-gateway/inbound.token",
|
||||||
|
"callback_token_file": "/etc/aios-orchestrator-gateway/callback.token",
|
||||||
|
"contract_commit": "cbe33ecab8e9794ba5f318bea43aeaf447a56fbc",
|
||||||
|
"workspace_agent": {
|
||||||
|
"trigger_id": "agtch_REPLACE_ME",
|
||||||
|
"access_token_file": "/etc/aios-orchestrator-gateway/workspace-agent.token"
|
||||||
|
},
|
||||||
|
"responses_fallback": {
|
||||||
|
"api_key_file": "/etc/aios-orchestrator-gateway/openai-api.key",
|
||||||
|
"model": "REPLACE_WITH_APPROVED_MODEL"
|
||||||
|
},
|
||||||
|
"polling": {
|
||||||
|
"interval_seconds": 10,
|
||||||
|
"summary_interval_seconds": 300
|
||||||
|
},
|
||||||
|
"loop_controls": {
|
||||||
|
"callback_timeout_seconds": 900,
|
||||||
|
"lease_seconds": 60,
|
||||||
|
"retry_base_seconds": 10,
|
||||||
|
"max_retries": 5,
|
||||||
|
"max_same_blocker_retries": 3,
|
||||||
|
"max_hops_per_gate": 20
|
||||||
|
},
|
||||||
|
"openclaw_sync": {
|
||||||
|
"endpoint": null,
|
||||||
|
"token_file": null
|
||||||
|
},
|
||||||
|
"project_key": "v2r1-personal-weixin-production",
|
||||||
|
"git_poll": {
|
||||||
|
"enabled": true,
|
||||||
|
"bare_repo": "/data/aios-gitops/repo.git",
|
||||||
|
"ref_prefixes": [
|
||||||
|
"refs/heads/main",
|
||||||
|
"refs/heads/task/governance/",
|
||||||
|
"refs/heads/patch/",
|
||||||
|
"refs/heads/result/"
|
||||||
|
],
|
||||||
|
"interval_seconds": 30
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -1,23 +0,0 @@
|
|||||||
FROM python:3.12-slim
|
|
||||||
|
|
||||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
|
||||||
PYTHONUNBUFFERED=1 \
|
|
||||||
PYTHONPATH=/opt/aios/app \
|
|
||||||
AIOS_V2_HOST=0.0.0.0 \
|
|
||||||
AIOS_V2_PORT=18920 \
|
|
||||||
AIOS_V2_STATE_DIR=/var/lib/aios-v2
|
|
||||||
|
|
||||||
RUN groupadd --system --gid 12001 aios \
|
|
||||||
&& useradd --system --uid 12001 --gid aios --home-dir /nonexistent --shell /usr/sbin/nologin aios \
|
|
||||||
&& install -d -o aios -g aios -m 0750 /opt/aios/app /var/lib/aios-v2
|
|
||||||
|
|
||||||
WORKDIR /opt/aios/app
|
|
||||||
COPY --chown=aios:aios active-runtime/app/ /opt/aios/app/
|
|
||||||
|
|
||||||
USER aios:aios
|
|
||||||
EXPOSE 18920
|
|
||||||
|
|
||||||
HEALTHCHECK --interval=15s --timeout=5s --start-period=10s --retries=4 \
|
|
||||||
CMD python3 -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:18920/api/v2/health', timeout=3).read()"
|
|
||||||
|
|
||||||
CMD ["python3", "-m", "aios_v2.app"]
|
|
||||||
@@ -1,44 +0,0 @@
|
|||||||
services:
|
|
||||||
aios-v2:
|
|
||||||
build:
|
|
||||||
context: ../../..
|
|
||||||
dockerfile: deploy/aios-v2/docker/Dockerfile
|
|
||||||
image: aios-v2:V2R1C0-candidate
|
|
||||||
init: true
|
|
||||||
restart: unless-stopped
|
|
||||||
ports:
|
|
||||||
- "127.0.0.1:${AIOS_V2_PORT:-18920}:18920"
|
|
||||||
environment:
|
|
||||||
AIOS_V2_HOST: 0.0.0.0
|
|
||||||
AIOS_V2_PORT: 18920
|
|
||||||
AIOS_V2_STATE_DIR: /var/lib/aios-v2
|
|
||||||
AIOS_OPENCLAW_URL: ${AIOS_OPENCLAW_URL:-}
|
|
||||||
AIOS_OPENCLAW_CREDENTIAL_REF: ${AIOS_OPENCLAW_CREDENTIAL_REF:-}
|
|
||||||
AIOS_OPENCLAW_SIGNING_MATERIAL_FILE: ${AIOS_OPENCLAW_SIGNING_MATERIAL_FILE:-}
|
|
||||||
AIOS_RELEASE_CHANNEL: candidate
|
|
||||||
AIOS_RELEASE_VERSION: V2R1C0-candidate
|
|
||||||
volumes:
|
|
||||||
- aios-v2-state:/var/lib/aios-v2
|
|
||||||
# Mount runtime signing material explicitly when OpenClaw is enabled.
|
|
||||||
# The host file must be mode 0600; never store it in this repository.
|
|
||||||
# - /etc/aios-v2/openclaw-signing-material:/run/secrets/openclaw-signing-material:ro
|
|
||||||
read_only: true
|
|
||||||
tmpfs:
|
|
||||||
- /tmp:rw,noexec,nosuid,nodev,size=64m
|
|
||||||
cap_drop:
|
|
||||||
- ALL
|
|
||||||
security_opt:
|
|
||||||
- no-new-privileges:true
|
|
||||||
healthcheck:
|
|
||||||
test:
|
|
||||||
- CMD
|
|
||||||
- python3
|
|
||||||
- -c
|
|
||||||
- "import urllib.request; urllib.request.urlopen('http://127.0.0.1:18920/api/v2/health', timeout=3).read()"
|
|
||||||
interval: 15s
|
|
||||||
timeout: 5s
|
|
||||||
retries: 4
|
|
||||||
start_period: 10s
|
|
||||||
|
|
||||||
volumes:
|
|
||||||
aios-v2-state:
|
|
||||||
-21
@@ -1,21 +0,0 @@
|
|||||||
# AIOS V2 candidate runtime configuration.
|
|
||||||
# Copy to /etc/aios-v2/aios-v2.env and keep ownership root:aios, mode 0640.
|
|
||||||
|
|
||||||
AIOS_V2_HOST=127.0.0.1
|
|
||||||
AIOS_V2_PORT=18920
|
|
||||||
AIOS_V2_STATE_DIR=/var/lib/aios-v2
|
|
||||||
|
|
||||||
# Optional OpenClaw runtime. Leave blank to run with OpenClaw disabled.
|
|
||||||
AIOS_OPENCLAW_URL=
|
|
||||||
AIOS_OPENCLAW_CREDENTIAL_REF=
|
|
||||||
|
|
||||||
# File path only. The referenced file must be mode 0600 or stricter and contain
|
|
||||||
# at least 32 bytes. Never place signing material in this env file or Git.
|
|
||||||
AIOS_OPENCLAW_SIGNING_MATERIAL_FILE=
|
|
||||||
|
|
||||||
# Reverse proxy must authenticate users and inject X-AIOS-* identity headers.
|
|
||||||
AIOS_TRUST_PROXY_HEADERS=true
|
|
||||||
|
|
||||||
# Candidate deployment metadata exposed through health tooling.
|
|
||||||
AIOS_RELEASE_CHANNEL=candidate
|
|
||||||
AIOS_RELEASE_VERSION=V2R1C0-candidate
|
|
||||||
@@ -1,86 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
set -Eeuo pipefail
|
|
||||||
|
|
||||||
BASE_URL="${AIOS_V2_BASE_URL:-http://127.0.0.1:18920}"
|
|
||||||
STATE_DIR="${AIOS_V2_STATE_DIR:-/var/lib/aios-v2}"
|
|
||||||
CONTROLLER_WT="${AIOS_GITOPS_CONTROLLER_WORKTREE:-/data/aios-gitops/worktrees/controller}"
|
|
||||||
SERVICE="${AIOS_V2_SERVICE:-aios-v2}"
|
|
||||||
|
|
||||||
fail() {
|
|
||||||
echo "FAIL: $*" >&2
|
|
||||||
exit 1
|
|
||||||
}
|
|
||||||
|
|
||||||
pass() {
|
|
||||||
echo "PASS: $*"
|
|
||||||
}
|
|
||||||
|
|
||||||
command -v python3 >/dev/null || fail "python3 not installed"
|
|
||||||
|
|
||||||
if command -v systemctl >/dev/null && systemctl list-unit-files "$SERVICE.service" >/dev/null 2>&1; then
|
|
||||||
systemctl is-active --quiet "$SERVICE" || fail "$SERVICE is not active"
|
|
||||||
pass "$SERVICE is active"
|
|
||||||
fi
|
|
||||||
|
|
||||||
python3 - "$BASE_URL" <<'PY'
|
|
||||||
import json
|
|
||||||
import sys
|
|
||||||
import urllib.request
|
|
||||||
|
|
||||||
base_url = sys.argv[1].rstrip("/")
|
|
||||||
with urllib.request.urlopen(f"{base_url}/api/v2/health", timeout=5) as response:
|
|
||||||
payload = json.loads(response.read().decode("utf-8"))
|
|
||||||
if payload.get("status") not in {"ok", "healthy", "ready"}:
|
|
||||||
raise SystemExit(f"unexpected health payload: {payload}")
|
|
||||||
print(f"PASS: HTTP health status={payload.get('status')}")
|
|
||||||
PY
|
|
||||||
|
|
||||||
[[ -d "$STATE_DIR" ]] || fail "state directory missing: $STATE_DIR"
|
|
||||||
[[ -w "$STATE_DIR" ]] || fail "state directory not writable: $STATE_DIR"
|
|
||||||
pass "state directory writable"
|
|
||||||
|
|
||||||
if [[ -e "$STATE_DIR/control-plane.sqlite3" ]]; then
|
|
||||||
python3 - "$STATE_DIR/control-plane.sqlite3" <<'PY'
|
|
||||||
import sqlite3
|
|
||||||
import sys
|
|
||||||
|
|
||||||
path = sys.argv[1]
|
|
||||||
connection = sqlite3.connect(f"file:{path}?mode=ro", uri=True, timeout=5)
|
|
||||||
try:
|
|
||||||
row = connection.execute("PRAGMA quick_check").fetchone()
|
|
||||||
finally:
|
|
||||||
connection.close()
|
|
||||||
if not row or row[0] != "ok":
|
|
||||||
raise SystemExit(f"SQLite quick_check failed: {row}")
|
|
||||||
print("PASS: SQLite quick_check=ok")
|
|
||||||
PY
|
|
||||||
else
|
|
||||||
echo "WARN: control-plane.sqlite3 not created yet"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -d "$CONTROLLER_WT" ]] && git -C "$CONTROLLER_WT" rev-parse --is-inside-work-tree >/dev/null 2>&1; then
|
|
||||||
branch="$(git -C "$CONTROLLER_WT" branch --show-current)"
|
|
||||||
dirty="$(git -C "$CONTROLLER_WT" status --porcelain=v1)"
|
|
||||||
[[ -z "$branch" ]] || fail "GitOps controller attached to branch: $branch"
|
|
||||||
[[ -z "$dirty" ]] || fail "GitOps controller workspace is dirty"
|
|
||||||
pass "GitOps controller detached and clean"
|
|
||||||
fi
|
|
||||||
|
|
||||||
python3 - "$STATE_DIR/executions" <<'PY'
|
|
||||||
import json
|
|
||||||
import pathlib
|
|
||||||
import sys
|
|
||||||
|
|
||||||
root = pathlib.Path(sys.argv[1])
|
|
||||||
if not root.exists():
|
|
||||||
print("WARN: execution evidence directory not created yet")
|
|
||||||
raise SystemExit(0)
|
|
||||||
for path in root.glob("*/*.json"):
|
|
||||||
value = json.loads(path.read_text(encoding="utf-8"))
|
|
||||||
for key in ("execution_id", "tenant_id", "task_id", "executor", "status"):
|
|
||||||
if not value.get(key):
|
|
||||||
raise SystemExit(f"invalid execution evidence {path}: missing {key}")
|
|
||||||
print("PASS: execution evidence files valid")
|
|
||||||
PY
|
|
||||||
|
|
||||||
pass "AIOS V2 candidate healthcheck complete"
|
|
||||||
@@ -1,86 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
set -Eeuo pipefail
|
|
||||||
|
|
||||||
SERVICE="${AIOS_V2_SERVICE:-aios-v2}"
|
|
||||||
RELEASE_ROOT="${AIOS_V2_RELEASE_ROOT:-/opt/aios/releases}"
|
|
||||||
CURRENT_LINK="${AIOS_V2_CURRENT_LINK:-/opt/aios/current}"
|
|
||||||
STATE_DIR="${AIOS_V2_STATE_DIR:-/var/lib/aios-v2}"
|
|
||||||
BACKUP_ROOT="${AIOS_V2_BACKUP_ROOT:-/var/backups/aios-v2}"
|
|
||||||
TARGET_RELEASE="${1:-}"
|
|
||||||
RESTORE_STATE="${AIOS_V2_RESTORE_STATE:-0}"
|
|
||||||
|
|
||||||
fail() {
|
|
||||||
echo "ERROR: $*" >&2
|
|
||||||
exit 1
|
|
||||||
}
|
|
||||||
|
|
||||||
[[ "${EUID}" -eq 0 ]] || fail "rollback must run as root"
|
|
||||||
[[ -n "$TARGET_RELEASE" ]] || fail "usage: $0 <release-name-or-absolute-path>"
|
|
||||||
|
|
||||||
if [[ "$TARGET_RELEASE" = /* ]]; then
|
|
||||||
TARGET_PATH="$(readlink -f "$TARGET_RELEASE")"
|
|
||||||
else
|
|
||||||
[[ "$TARGET_RELEASE" != *"/"* && "$TARGET_RELEASE" != "." && "$TARGET_RELEASE" != ".." ]] \
|
|
||||||
|| fail "invalid release name"
|
|
||||||
TARGET_PATH="$(readlink -f "$RELEASE_ROOT/$TARGET_RELEASE")"
|
|
||||||
fi
|
|
||||||
|
|
||||||
[[ -d "$TARGET_PATH/active-runtime/app" ]] || fail "target release is invalid: $TARGET_PATH"
|
|
||||||
[[ -f "$TARGET_PATH/deploy/aios-v2/healthcheck/check.sh" ]] \
|
|
||||||
|| fail "target release has no healthcheck"
|
|
||||||
|
|
||||||
PREVIOUS_PATH=""
|
|
||||||
if [[ -L "$CURRENT_LINK" ]]; then
|
|
||||||
PREVIOUS_PATH="$(readlink -f "$CURRENT_LINK")"
|
|
||||||
fi
|
|
||||||
|
|
||||||
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
|
|
||||||
BACKUP_DIR="$BACKUP_ROOT/rollback-$STAMP"
|
|
||||||
install -d -m 0750 "$BACKUP_DIR"
|
|
||||||
|
|
||||||
if [[ -d "$STATE_DIR" ]]; then
|
|
||||||
tar --xattrs --acls -C "$(dirname "$STATE_DIR")" \
|
|
||||||
-czf "$BACKUP_DIR/state-before-rollback.tar.gz" \
|
|
||||||
"$(basename "$STATE_DIR")"
|
|
||||||
fi
|
|
||||||
|
|
||||||
printf '%s\n' "$PREVIOUS_PATH" > "$BACKUP_DIR/previous-release.txt"
|
|
||||||
printf '%s\n' "$TARGET_PATH" > "$BACKUP_DIR/target-release.txt"
|
|
||||||
|
|
||||||
systemctl stop "$SERVICE"
|
|
||||||
ln -sfn "$TARGET_PATH" "${CURRENT_LINK}.next"
|
|
||||||
mv -Tf "${CURRENT_LINK}.next" "$CURRENT_LINK"
|
|
||||||
systemctl daemon-reload
|
|
||||||
systemctl start "$SERVICE"
|
|
||||||
|
|
||||||
if AIOS_V2_SERVICE="$SERVICE" \
|
|
||||||
AIOS_V2_STATE_DIR="$STATE_DIR" \
|
|
||||||
"$TARGET_PATH/deploy/aios-v2/healthcheck/check.sh"; then
|
|
||||||
echo "ROLLBACK_COMPLETE target=$TARGET_PATH backup=$BACKUP_DIR"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "Target release failed healthcheck; restoring previous release." >&2
|
|
||||||
systemctl stop "$SERVICE" || true
|
|
||||||
|
|
||||||
if [[ -n "$PREVIOUS_PATH" && -d "$PREVIOUS_PATH" ]]; then
|
|
||||||
ln -sfn "$PREVIOUS_PATH" "${CURRENT_LINK}.previous"
|
|
||||||
mv -Tf "${CURRENT_LINK}.previous" "$CURRENT_LINK"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ "$RESTORE_STATE" == "1" && -f "$BACKUP_DIR/state-before-rollback.tar.gz" ]]; then
|
|
||||||
rm -rf "$STATE_DIR"
|
|
||||||
tar --xattrs --acls -C "$(dirname "$STATE_DIR")" \
|
|
||||||
-xzf "$BACKUP_DIR/state-before-rollback.tar.gz"
|
|
||||||
fi
|
|
||||||
|
|
||||||
systemctl daemon-reload
|
|
||||||
systemctl start "$SERVICE" || true
|
|
||||||
|
|
||||||
if [[ -n "$PREVIOUS_PATH" && -x "$PREVIOUS_PATH/deploy/aios-v2/healthcheck/check.sh" ]]; then
|
|
||||||
AIOS_V2_SERVICE="$SERVICE" \
|
|
||||||
AIOS_V2_STATE_DIR="$STATE_DIR" \
|
|
||||||
"$PREVIOUS_PATH/deploy/aios-v2/healthcheck/check.sh" || true
|
|
||||||
fi
|
|
||||||
|
|
||||||
fail "rollback target failed; previous release restored; evidence=$BACKUP_DIR"
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
[Unit]
|
|
||||||
Description=AIOS V2 Control Plane Candidate
|
|
||||||
After=network-online.target
|
|
||||||
Wants=network-online.target
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Type=simple
|
|
||||||
User=aios
|
|
||||||
Group=aios
|
|
||||||
WorkingDirectory=/opt/aios/current/active-runtime/app
|
|
||||||
EnvironmentFile=-/etc/aios-v2/aios-v2.env
|
|
||||||
Environment=PYTHONUNBUFFERED=1
|
|
||||||
ExecStart=/usr/bin/python3 -m aios_v2.app
|
|
||||||
Restart=on-failure
|
|
||||||
RestartSec=3
|
|
||||||
TimeoutStartSec=30
|
|
||||||
TimeoutStopSec=30
|
|
||||||
KillSignal=SIGTERM
|
|
||||||
NoNewPrivileges=true
|
|
||||||
PrivateTmp=true
|
|
||||||
ProtectSystem=strict
|
|
||||||
ProtectHome=true
|
|
||||||
ProtectKernelTunables=true
|
|
||||||
ProtectKernelModules=true
|
|
||||||
ProtectControlGroups=true
|
|
||||||
RestrictSUIDSGID=true
|
|
||||||
RestrictRealtime=true
|
|
||||||
LockPersonality=true
|
|
||||||
MemoryDenyWriteExecute=true
|
|
||||||
ReadWritePaths=/var/lib/aios-v2
|
|
||||||
StateDirectory=aios-v2
|
|
||||||
StateDirectoryMode=0750
|
|
||||||
UMask=0027
|
|
||||||
LimitNOFILE=65536
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=multi-user.target
|
|
||||||
@@ -1,243 +0,0 @@
|
|||||||
# AIOS V2 用户需求规格
|
|
||||||
|
|
||||||
- 文档版本:`V2R1C0-candidate`
|
|
||||||
- 状态:候选版本需求基线
|
|
||||||
- 适用范围:AIOS Control Plane、Agent Router、OpenClaw Adapter、Hermes Runtime、Web Console、GitOps、审计与部署
|
|
||||||
|
|
||||||
## 1. 背景与目标
|
|
||||||
|
|
||||||
现有系统已经具备 OpenClaw、Hermes、Agent Router、GitOps、局部 WebUI、审计和多租户候选能力。AIOS V2 的目标不是推倒重写,而是将这些分散能力收敛到统一、可审计、可 Web 化、可多租户隔离的控制面。
|
|
||||||
|
|
||||||
AIOS V2 必须成为唯一控制面,负责租户、身份、Agent、策略、任务生命周期、审批、审计、Runtime 目录和执行证据。OpenClaw 与 Hermes 只承担执行职责,不得持有第二套控制面状态。
|
|
||||||
|
|
||||||
## 2. 用户角色
|
|
||||||
|
|
||||||
| 角色 | 主要诉求 |
|
|
||||||
|---|---|
|
|
||||||
| 平台管理员 | 管理租户、运行时、发布、GitOps 状态和全局策略 |
|
|
||||||
| 租户管理员 | 管理本租户 Agent、任务、审批、审计和成员权限 |
|
|
||||||
| 操作员 | 创建任务、发起执行、取消任务、查看运行状态 |
|
|
||||||
| 审批人 | 审批高风险任务,查看策略判定依据 |
|
|
||||||
| 审计员 | 查询本租户完整审计链和执行证据 |
|
|
||||||
| Runtime Worker | 领取已批准任务、心跳、返回执行结果 |
|
|
||||||
| 开发与运维人员 | 通过 GitOps 提交、测试、部署、回滚和修复 Bug |
|
|
||||||
|
|
||||||
## 3. 核心用户需求
|
|
||||||
|
|
||||||
### UR-001 唯一控制面
|
|
||||||
|
|
||||||
所有任务创建、审批、取消、状态变化、策略判定和审计必须由 AIOS Control Plane 统一处理。
|
|
||||||
|
|
||||||
验收:
|
|
||||||
|
|
||||||
- Web Console、API、Codex、OpenClaw 和 Hermes 均不得绕过 AIOS 创建或修改任务生命周期。
|
|
||||||
- OpenClaw/Hermes 的公开接口仅限执行、状态、取消、心跳和结果回传。
|
|
||||||
|
|
||||||
### UR-002 复用现有能力
|
|
||||||
|
|
||||||
系统不得重写已经稳定的 OpenClaw 执行能力、Hermes Runtime、Agent Router 或 GitOps 基础设施。
|
|
||||||
|
|
||||||
验收:
|
|
||||||
|
|
||||||
- 通过 Adapter/Boundary 接入现有能力。
|
|
||||||
- 替换控制逻辑时保留原执行接口和可迁移数据。
|
|
||||||
|
|
||||||
### UR-003 OpenClaw 与 Hermes 解耦
|
|
||||||
|
|
||||||
OpenClaw 与 Hermes 必须从策略、租户、审批和任务生命周期中逐步解耦。
|
|
||||||
|
|
||||||
验收:
|
|
||||||
|
|
||||||
- OpenClaw 仅接收已签名、已批准、带租户上下文的命令。
|
|
||||||
- Hermes 仅接收已批准的租户分区 Job。
|
|
||||||
- 二者均不得直接写 Control Plane 权威状态。
|
|
||||||
|
|
||||||
### UR-004 全面 Web 化
|
|
||||||
|
|
||||||
所有面向用户和运维的能力必须通过 Web Console 或稳定 API 访问。
|
|
||||||
|
|
||||||
Web Console 至少展示:
|
|
||||||
|
|
||||||
- Tenant、Agent、Task、Approval、Runtime、Audit、Execution Evidence、GitOps Status。
|
|
||||||
- 所有写操作由 AIOS API 完成,浏览器不得直接调用 OpenClaw/Hermes。
|
|
||||||
|
|
||||||
### UR-005 多租户隔离
|
|
||||||
|
|
||||||
所有模块必须支持租户隔离,且默认拒绝跨租户访问。
|
|
||||||
|
|
||||||
隔离范围:
|
|
||||||
|
|
||||||
- API 与会话
|
|
||||||
- Tenant/Agent/Task 数据
|
|
||||||
- OpenClaw 命令
|
|
||||||
- Hermes Queue、Cell、Profile、Worker Lease
|
|
||||||
- Memory、Cache、Artifact、SecretRef
|
|
||||||
- Audit、Execution Evidence、日志和指标
|
|
||||||
|
|
||||||
验收:
|
|
||||||
|
|
||||||
- 资源记录的 `tenant_id` 不可为空。
|
|
||||||
- 跨租户访问返回拒绝或“资源不存在”,不得泄露存在性。
|
|
||||||
- 必须有跨租户负向测试。
|
|
||||||
|
|
||||||
### UR-006 权威任务生命周期
|
|
||||||
|
|
||||||
任务必须遵循统一状态机:
|
|
||||||
|
|
||||||
```text
|
|
||||||
CREATED -> APPROVED -> RUNNING -> COMPLETED
|
|
||||||
| | |
|
|
||||||
+----------+----------+-> CANCELLED
|
|
||||||
+-> FAILED
|
|
||||||
```
|
|
||||||
|
|
||||||
验收:
|
|
||||||
|
|
||||||
- 禁止越级变化,例如 `CREATED -> COMPLETED`。
|
|
||||||
- 每次变化包含 revision、principal、trace、时间和审计事件。
|
|
||||||
|
|
||||||
### UR-007 策略与审批
|
|
||||||
|
|
||||||
所有有副作用任务必须经过策略判定;高风险任务必须支持人工审批。
|
|
||||||
|
|
||||||
验收:
|
|
||||||
|
|
||||||
- Policy Decision 包含 allow/deny、reason 和 policy version。
|
|
||||||
- 拒绝事件进入审计。
|
|
||||||
- Runtime 不得自行放宽策略。
|
|
||||||
|
|
||||||
### UR-008 幂等与并发安全
|
|
||||||
|
|
||||||
任务和 Runtime 执行必须支持幂等与乐观并发控制。
|
|
||||||
|
|
||||||
验收:
|
|
||||||
|
|
||||||
- 相同租户与幂等键、相同内容返回同一任务/执行。
|
|
||||||
- 相同幂等键不同内容返回冲突。
|
|
||||||
- revision 或 branch lease 冲突不得静默覆盖。
|
|
||||||
|
|
||||||
### UR-009 可观测与执行证据
|
|
||||||
|
|
||||||
平台必须能够确认 Codex、OpenClaw、Hermes 等执行器是否领取任务、当前进度、最近提交、测试结果和完成状态。
|
|
||||||
|
|
||||||
验收:
|
|
||||||
|
|
||||||
- 存在 claim、heartbeat、progress、result 和 evidence 记录。
|
|
||||||
- Web/API 可按租户查询。
|
|
||||||
- 执行证据不得包含明文密钥。
|
|
||||||
|
|
||||||
### UR-010 GitOps 唯一变更流程
|
|
||||||
|
|
||||||
所有代码、配置、文档和部署资产修改必须进入非 `main` 分支,通过 commit、测试、审查和受控提升。
|
|
||||||
|
|
||||||
验收:
|
|
||||||
|
|
||||||
- Controller worktree 必须 detached 且 clean。
|
|
||||||
- 本地 bare repo 与 Gitea 目标分支 OID 一致。
|
|
||||||
- 不允许直接 push `main`、直接创建生产 Tag 或绕过审批部署。
|
|
||||||
|
|
||||||
### UR-011 持久化与恢复
|
|
||||||
|
|
||||||
租户、Agent、Task、Audit、Idempotency 和 Execution Evidence 必须在服务重启后恢复。
|
|
||||||
|
|
||||||
验收:
|
|
||||||
|
|
||||||
- 候选版支持 SQLite WAL 原子快照。
|
|
||||||
- 有 restart recovery 测试。
|
|
||||||
- 生产版需进一步提供迁移、备份、恢复、HA 和容量证据。
|
|
||||||
|
|
||||||
### UR-012 审计与合规
|
|
||||||
|
|
||||||
所有控制与执行动作必须生成不可混租户的审计事件。
|
|
||||||
|
|
||||||
至少记录:
|
|
||||||
|
|
||||||
- tenant、principal、trace、action、resource、outcome、policy version、timestamp。
|
|
||||||
- OpenClaw/Hermes Runtime 生命周期。
|
|
||||||
- GitOps、部署和回滚动作。
|
|
||||||
|
|
||||||
### UR-013 密钥安全
|
|
||||||
|
|
||||||
Git、日志、Web、任务 payload 和审计记录中不得出现明文 API Key、密码、Token 或私钥。
|
|
||||||
|
|
||||||
验收:
|
|
||||||
|
|
||||||
- Git 只存 `secret_ref`、`credential_ref`、`key_id` 或受保护文件路径。
|
|
||||||
- Secret 文件权限为 `0600` 或更严格。
|
|
||||||
- Secret Scanner 默认拒绝可疑提交。
|
|
||||||
|
|
||||||
### UR-014 部署与回滚
|
|
||||||
|
|
||||||
系统必须提供 systemd、Docker/Compose、健康检查和受保护回滚入口。
|
|
||||||
|
|
||||||
验收:
|
|
||||||
|
|
||||||
- 服务以非 root 权限运行。
|
|
||||||
- 状态目录独立持久化。
|
|
||||||
- 回滚前自动备份状态。
|
|
||||||
- 默认只回滚代码,状态恢复需要显式授权。
|
|
||||||
|
|
||||||
### UR-015 文档与代码同步
|
|
||||||
|
|
||||||
任何影响用户行为、架构边界、API、数据模型、状态机、安全策略、部署或运维方式的变更,必须在同一分支同步更新 `docs/aios-v2/`。
|
|
||||||
|
|
||||||
验收:
|
|
||||||
|
|
||||||
- 架构/功能代码变化但无文档变化时,文档同步门禁失败。
|
|
||||||
- Traceability Matrix 必须指向当前代码和测试。
|
|
||||||
- 新架构决策必须增加或更新 ADR。
|
|
||||||
|
|
||||||
## 4. 非功能需求
|
|
||||||
|
|
||||||
### NFR-001 安全
|
|
||||||
|
|
||||||
- 默认拒绝、最小权限、租户隔离、身份可信链、SecretRef、审计。
|
|
||||||
|
|
||||||
### NFR-002 可靠性
|
|
||||||
|
|
||||||
- 幂等、revision、worker lease、heartbeat、重试边界、原子持久化、健康检查和回滚。
|
|
||||||
|
|
||||||
### NFR-003 可维护性
|
|
||||||
|
|
||||||
- 模块边界稳定,Control Plane 与 Runtime 解耦;代码、文档、测试和部署可追踪。
|
|
||||||
|
|
||||||
### NFR-004 可扩展性
|
|
||||||
|
|
||||||
- Store、Transport、Policy、EventPublisher、Runtime Adapter 均通过接口扩展。
|
|
||||||
|
|
||||||
### NFR-005 可运维性
|
|
||||||
|
|
||||||
- Web/API 状态、systemd/Docker、健康检查、日志、GitOps 状态、候选报告。
|
|
||||||
|
|
||||||
### NFR-006 性能候选指标
|
|
||||||
|
|
||||||
候选版上线前至少验证:
|
|
||||||
|
|
||||||
- API P95、任务调度延迟、队列等待、Worker 心跳、SQLite 锁等待、审计写入时间。
|
|
||||||
- 具体 SLO 在预发布基准测试后固化,禁止无证据承诺。
|
|
||||||
|
|
||||||
## 5. 本候选版本范围
|
|
||||||
|
|
||||||
### 已纳入代码
|
|
||||||
|
|
||||||
- Control Plane、持久化、Gateway、Orchestrator、OpenClaw Adapter、Hermes Boundary、Execution Tracking、Web Console、GitOps Status、部署和回滚。
|
|
||||||
|
|
||||||
### 尚需环境验证
|
|
||||||
|
|
||||||
- 精确测试执行证据。
|
|
||||||
- OpenClaw 真实 Runtime 联调。
|
|
||||||
- 生产数据库/队列替换或容量验证。
|
|
||||||
- 预发布性能、安全、重启和回滚演练。
|
|
||||||
- Owner 对 merge、Tag 和生产部署的批准。
|
|
||||||
|
|
||||||
## 6. 用户验收场景
|
|
||||||
|
|
||||||
1. 平台管理员创建租户并注册 OpenClaw/Hermes Agent。
|
|
||||||
2. 操作员创建任务,审批人批准。
|
|
||||||
3. AIOS Orchestrator 选择 Runtime,并创建签名命令或租户分区 Job。
|
|
||||||
4. Runtime 执行并返回结果。
|
|
||||||
5. AIOS 更新 Task、Audit 和 Execution Evidence。
|
|
||||||
6. 租户用户在 Web Console 查看完整链路。
|
|
||||||
7. 跨租户访问被拒绝。
|
|
||||||
8. 服务重启后状态恢复。
|
|
||||||
9. 候选部署健康检查通过,回滚演练成功。
|
|
||||||
@@ -1,461 +0,0 @@
|
|||||||
# AIOS V2 目标架构
|
|
||||||
|
|
||||||
- 文档版本:`V2R1C0-candidate`
|
|
||||||
- 架构状态:目标架构已实现为候选代码,待预发布验证
|
|
||||||
- 适用范围:Control Plane、Agent Router、OpenClaw、Hermes、Web Console、GitOps、审计、部署与运维
|
|
||||||
|
|
||||||
## 1. 架构目标
|
|
||||||
|
|
||||||
AIOS V2 的核心目标是把分散在 OpenClaw、Hermes、Agent Router、局部 WebUI 和 GitOps 中的控制能力收敛为一个权威控制面,同时保留现有执行能力,避免推倒重写。
|
|
||||||
|
|
||||||
目标状态:
|
|
||||||
|
|
||||||
```text
|
|
||||||
用户 / Web Console / Public API
|
|
||||||
|
|
|
||||||
v
|
|
||||||
AIOS Control Plane
|
|
||||||
Tenant | Identity | Agent | Task | Policy | Approval | Audit
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Execution Orchestrator / Agent Router
|
|
||||||
/ \
|
|
||||||
v v
|
|
||||||
OpenClaw Adapter Hermes Boundary
|
|
||||||
| |
|
|
||||||
v v
|
|
||||||
OpenClaw Runtime Hermes Workers
|
|
||||||
\ /
|
|
||||||
v v
|
|
||||||
Result Reconciliation + Execution Evidence
|
|
||||||
```
|
|
||||||
|
|
||||||
设计结论:
|
|
||||||
|
|
||||||
- AIOS 是唯一控制面。
|
|
||||||
- Agent Router 是调度平面,不是第二控制面。
|
|
||||||
- OpenClaw 是执行适配与 Runtime。
|
|
||||||
- Hermes 是队列、Worker、Profile 与执行 Runtime。
|
|
||||||
- Web Console 只调用 AIOS API。
|
|
||||||
- GitOps 是代码、配置、文档和发布变更的唯一入口。
|
|
||||||
|
|
||||||
## 2. 从现有架构到目标架构
|
|
||||||
|
|
||||||
### 2.1 现有能力
|
|
||||||
|
|
||||||
现有系统已经具备:
|
|
||||||
|
|
||||||
- OpenClaw 执行与 Fleet 能力。
|
|
||||||
- Hermes Worker、Profile、队列和工作区能力。
|
|
||||||
- Agent Router 路由经验和部分调度接口。
|
|
||||||
- GitOps 分支、提交、审批、回滚和审计能力。
|
|
||||||
- Tenant Context、Tenant Router、Memory Scope、SecretRef、Message Ledger 和审计候选实现。
|
|
||||||
- 局部 WebUI、Gateway 和自动任务领取能力。
|
|
||||||
|
|
||||||
### 2.2 现有问题
|
|
||||||
|
|
||||||
- 控制逻辑分散,可能存在多个任务事实源。
|
|
||||||
- OpenClaw/Hermes 内部仍可能保存租户、策略或任务状态。
|
|
||||||
- 候选能力分散在多个分支,主线缺少权威实现。
|
|
||||||
- WebUI 与 API 入口不统一。
|
|
||||||
- 分支名称、代码、测试和部署证据之间缺少映射。
|
|
||||||
- 多租户隔离多为逻辑约束,尚缺全链路生产证据。
|
|
||||||
|
|
||||||
### 2.3 演进原则
|
|
||||||
|
|
||||||
- 先加边界和适配器,再迁移控制权。
|
|
||||||
- 先复用接口,再替换内部实现。
|
|
||||||
- 先在候选分支验证,再受控提升。
|
|
||||||
- 不允许同时存在两套权威任务生命周期。
|
|
||||||
- 不以“大分支存在”代替验收证据。
|
|
||||||
|
|
||||||
## 3. 逻辑分层
|
|
||||||
|
|
||||||
### 3.1 Web 与 API 层
|
|
||||||
|
|
||||||
职责:
|
|
||||||
|
|
||||||
- 提供 Tenant、Agent、Task、Approval、Runtime、Audit、Execution Evidence 和 GitOps 页面/API。
|
|
||||||
- 依赖可信身份代理注入 `X-AIOS-*` 身份头。
|
|
||||||
- 不允许浏览器直接访问 OpenClaw 或 Hermes。
|
|
||||||
- 不允许客户端自行选择或覆盖 `tenant_id`。
|
|
||||||
|
|
||||||
候选实现:
|
|
||||||
|
|
||||||
- `active-runtime/app/aios_v2/app.py`
|
|
||||||
- `active-runtime/app/aios_v2/gateway.py`
|
|
||||||
- `active-runtime/app/aios_v2/console.py`
|
|
||||||
|
|
||||||
### 3.2 AIOS Control Plane
|
|
||||||
|
|
||||||
权威职责:
|
|
||||||
|
|
||||||
- Tenant Registry
|
|
||||||
- Agent Registry
|
|
||||||
- Task Lifecycle
|
|
||||||
- Policy Decision
|
|
||||||
- Approval
|
|
||||||
- Audit Event
|
|
||||||
- Runtime Inventory
|
|
||||||
- Execution Evidence
|
|
||||||
- Deployment/GitOps Status
|
|
||||||
|
|
||||||
不负责:
|
|
||||||
|
|
||||||
- OpenClaw 内部执行细节。
|
|
||||||
- Hermes Worker 内部执行细节。
|
|
||||||
- 直接持有 Runtime 凭据明文。
|
|
||||||
|
|
||||||
候选实现:
|
|
||||||
|
|
||||||
- `active-runtime/app/aios_v2/control_plane.py`
|
|
||||||
- `active-runtime/app/aios_v2/persistence.py`
|
|
||||||
- `active-runtime/app/aios_v2/execution_tracking.py`
|
|
||||||
- `active-runtime/app/aios_v2/gitops_status.py`
|
|
||||||
|
|
||||||
### 3.3 Execution Orchestrator / Agent Router
|
|
||||||
|
|
||||||
职责:
|
|
||||||
|
|
||||||
- 只接受 `APPROVED` 任务。
|
|
||||||
- 根据 Agent Registry 和 Policy Result 选择 Runtime。
|
|
||||||
- 生成 OpenClawCommand 或 HermesJob。
|
|
||||||
- 传播 tenant、task、trace、idempotency 和 policy version。
|
|
||||||
- 将 Runtime Result 回写 AIOS Task。
|
|
||||||
|
|
||||||
不负责:
|
|
||||||
|
|
||||||
- 创建租户。
|
|
||||||
- 修改策略。
|
|
||||||
- 绕过审批。
|
|
||||||
- 保存第二套任务状态机。
|
|
||||||
|
|
||||||
候选实现:
|
|
||||||
|
|
||||||
- `active-runtime/app/aios_v2/orchestrator.py`
|
|
||||||
|
|
||||||
### 3.4 OpenClaw Adapter
|
|
||||||
|
|
||||||
职责:
|
|
||||||
|
|
||||||
- 验证签名命令。
|
|
||||||
- 执行、查询、取消、健康检查和回调。
|
|
||||||
- 支持幂等、超时和租户隔离。
|
|
||||||
- 把 Runtime 生命周期转换为 AIOS 审计事件。
|
|
||||||
|
|
||||||
不负责:
|
|
||||||
|
|
||||||
- Tenant、Policy、Approval、Task Lifecycle。
|
|
||||||
- 向浏览器暴露公网控制接口。
|
|
||||||
|
|
||||||
候选实现:
|
|
||||||
|
|
||||||
- `active-runtime/app/aios_openclaw/contract.py`
|
|
||||||
- `active-runtime/app/aios_openclaw/adapter.py`
|
|
||||||
- `active-runtime/app/aios_openclaw/transport.py`
|
|
||||||
- `active-runtime/app/aios_openclaw/health.py`
|
|
||||||
- `active-runtime/app/aios_openclaw/callback.py`
|
|
||||||
|
|
||||||
### 3.5 Hermes Runtime Boundary
|
|
||||||
|
|
||||||
职责:
|
|
||||||
|
|
||||||
- Tenant/Cell/Profile 分区队列。
|
|
||||||
- Worker claim、start、heartbeat、complete、fail、cancel。
|
|
||||||
- Lease 超时和自动回队列。
|
|
||||||
- Result 回传和 Runtime Health。
|
|
||||||
|
|
||||||
不负责:
|
|
||||||
|
|
||||||
- Policy、Approval、Tenant Registry、权威 Task Lifecycle。
|
|
||||||
|
|
||||||
候选实现:
|
|
||||||
|
|
||||||
- `active-runtime/app/aios_hermes/contract.py`
|
|
||||||
- `active-runtime/app/aios_hermes/adapter.py`
|
|
||||||
- `active-runtime/app/aios_hermes/worker.py`
|
|
||||||
|
|
||||||
### 3.6 Persistence
|
|
||||||
|
|
||||||
候选实现使用 SQLite WAL 原子快照,持久化:
|
|
||||||
|
|
||||||
- Tenant
|
|
||||||
- Agent
|
|
||||||
- Task
|
|
||||||
- Audit
|
|
||||||
- Idempotency
|
|
||||||
- Execution Evidence 文件
|
|
||||||
|
|
||||||
生产演进:
|
|
||||||
|
|
||||||
- Store 接口保持稳定。
|
|
||||||
- 可替换为 PostgreSQL、对象存储或分布式事件存储。
|
|
||||||
- 生产替换必须提供迁移、备份、恢复、HA、容量和租户隔离证据。
|
|
||||||
|
|
||||||
## 4. 多租户架构
|
|
||||||
|
|
||||||
### 4.1 权威租户上下文
|
|
||||||
|
|
||||||
每次请求和执行至少传播:
|
|
||||||
|
|
||||||
```text
|
|
||||||
tenant_id
|
|
||||||
principal_id
|
|
||||||
session_id
|
|
||||||
trace_id
|
|
||||||
task_id
|
|
||||||
policy_version
|
|
||||||
```
|
|
||||||
|
|
||||||
Runtime 还传播:
|
|
||||||
|
|
||||||
```text
|
|
||||||
command_id / job_id
|
|
||||||
cell_id
|
|
||||||
profile_id
|
|
||||||
idempotency_key
|
|
||||||
worker_id
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4.2 隔离层级
|
|
||||||
|
|
||||||
| 层级 | 隔离要求 |
|
|
||||||
|---|---|
|
|
||||||
| HTTP/API | 可信身份头、租户不可由浏览器覆盖 |
|
|
||||||
| Control Plane | 所有资源带 `tenant_id`,跨租户默认拒绝 |
|
|
||||||
| OpenClaw | 签名命令绑定 tenant/task/trace |
|
|
||||||
| Hermes | tenant/cell/profile 三层分区 |
|
|
||||||
| Storage | 候选为显式租户字段;生产需 RLS 或物理隔离策略 |
|
|
||||||
| Evidence/Audit | 仅按当前租户查询 |
|
|
||||||
| Cache/Memory | key 必须包含 tenant namespace |
|
|
||||||
| Secret | 只保存引用,不保存明文 |
|
|
||||||
| Logs/Metrics | 必须包含 tenant/task/trace 标签,并限制查询范围 |
|
|
||||||
|
|
||||||
### 4.3 失败行为
|
|
||||||
|
|
||||||
- 跨租户资源访问返回拒绝或“未找到”。
|
|
||||||
- 不泄露其他租户资源存在性。
|
|
||||||
- 缺少 tenant context 时 fail closed。
|
|
||||||
- Runtime 返回的 tenant/task/trace 不匹配时拒绝回写。
|
|
||||||
|
|
||||||
## 5. 任务与执行数据流
|
|
||||||
|
|
||||||
### 5.1 任务创建与审批
|
|
||||||
|
|
||||||
```text
|
|
||||||
User/API
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Trusted Identity
|
|
||||||
|
|
|
||||||
v
|
|
||||||
AIOS Gateway
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Policy Gate
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Task CREATED
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Approval
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Task APPROVED
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5.2 OpenClaw 执行
|
|
||||||
|
|
||||||
```text
|
|
||||||
APPROVED Task
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Execution Orchestrator
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Signed OpenClawCommand
|
|
||||||
|
|
|
||||||
v
|
|
||||||
OpenClaw Adapter
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Runtime Result
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Task Reconciliation + Audit + Evidence
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5.3 Hermes 执行
|
|
||||||
|
|
||||||
```text
|
|
||||||
APPROVED Task
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Execution Orchestrator
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Tenant/Cell/Profile HermesJob
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Worker Claim + Lease + Heartbeat
|
|
||||||
|
|
|
||||||
v
|
|
||||||
HermesResult
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Task Reconciliation + Audit + Evidence
|
|
||||||
```
|
|
||||||
|
|
||||||
## 6. 安全架构
|
|
||||||
|
|
||||||
### 6.1 身份边界
|
|
||||||
|
|
||||||
- Public API 位于可信反向代理之后。
|
|
||||||
- 反向代理完成登录、会话、MFA 和身份头注入。
|
|
||||||
- 候选应用拒绝未认证身份断言。
|
|
||||||
- 生产不得允许客户端直接伪造 `X-AIOS-*`。
|
|
||||||
|
|
||||||
### 6.2 Runtime 命令完整性
|
|
||||||
|
|
||||||
OpenClawCommand 使用规范化序列化和 HMAC-SHA256:
|
|
||||||
|
|
||||||
- 签名覆盖 tenant、task、trace、action、payload、idempotency 和时间。
|
|
||||||
- 支持 key ID、到期时间和时钟偏差。
|
|
||||||
- Runtime 密钥通过受保护文件或外部 Secret Provider 解析。
|
|
||||||
|
|
||||||
### 6.3 Secret 边界
|
|
||||||
|
|
||||||
Git 允许:
|
|
||||||
|
|
||||||
```text
|
|
||||||
secret_ref
|
|
||||||
credential_ref
|
|
||||||
key_id
|
|
||||||
受保护文件路径
|
|
||||||
```
|
|
||||||
|
|
||||||
Git 禁止:
|
|
||||||
|
|
||||||
```text
|
|
||||||
API Key
|
|
||||||
密码
|
|
||||||
Token
|
|
||||||
私钥
|
|
||||||
签名材料
|
|
||||||
```
|
|
||||||
|
|
||||||
### 6.4 Web 安全
|
|
||||||
|
|
||||||
候选应用设置:
|
|
||||||
|
|
||||||
- no-store
|
|
||||||
- nosniff
|
|
||||||
- DENY frame
|
|
||||||
- no-referrer
|
|
||||||
- CSP
|
|
||||||
- 请求体大小限制
|
|
||||||
|
|
||||||
生产还需验证:
|
|
||||||
|
|
||||||
- CSRF/CORS
|
|
||||||
- Rate Limit
|
|
||||||
- Session Timeout
|
|
||||||
- CSP Nonce/Hash
|
|
||||||
- WAF 和反向代理访问策略
|
|
||||||
|
|
||||||
## 7. 可靠性架构
|
|
||||||
|
|
||||||
### 7.1 幂等
|
|
||||||
|
|
||||||
- Control Plane:`tenant_id + idempotency_key + request_digest`。
|
|
||||||
- OpenClaw:`tenant_id + idempotency_key + signature`。
|
|
||||||
- Hermes:`tenant_id + idempotency_key + job_digest`。
|
|
||||||
|
|
||||||
### 7.2 并发
|
|
||||||
|
|
||||||
- Task 使用 revision。
|
|
||||||
- Hermes 使用 revision + worker lease。
|
|
||||||
- GitOps 使用 detached controller + branch lease。
|
|
||||||
- 持久化使用 SQLite `BEGIN IMMEDIATE` 和原子快照。
|
|
||||||
|
|
||||||
### 7.3 恢复
|
|
||||||
|
|
||||||
- Controller restart:detached guard。
|
|
||||||
- AIOS restart:SQLite restore。
|
|
||||||
- Worker crash:lease 到期后 Job 回到 QUEUED。
|
|
||||||
- Release failure:健康检查失败自动恢复 previous release。
|
|
||||||
|
|
||||||
## 8. 可观测架构
|
|
||||||
|
|
||||||
统一关联键:
|
|
||||||
|
|
||||||
```text
|
|
||||||
tenant_id
|
|
||||||
principal_id
|
|
||||||
trace_id
|
|
||||||
task_id
|
|
||||||
agent_id
|
|
||||||
command_id / job_id
|
|
||||||
worker_id
|
|
||||||
commit
|
|
||||||
```
|
|
||||||
|
|
||||||
Web Console 展示:
|
|
||||||
|
|
||||||
- Tasks
|
|
||||||
- Agents
|
|
||||||
- Runtime Health
|
|
||||||
- Audit
|
|
||||||
- Execution Evidence
|
|
||||||
- GitOps Status
|
|
||||||
|
|
||||||
生产还需接入:
|
|
||||||
|
|
||||||
- 结构化日志
|
|
||||||
- Metrics
|
|
||||||
- Distributed Tracing
|
|
||||||
- Alerting
|
|
||||||
- SLO Dashboard
|
|
||||||
|
|
||||||
## 9. 部署架构
|
|
||||||
|
|
||||||
支持两种候选部署:
|
|
||||||
|
|
||||||
### systemd
|
|
||||||
|
|
||||||
- 非 root `aios` 用户。
|
|
||||||
- `ProtectSystem=strict`、`NoNewPrivileges` 等安全选项。
|
|
||||||
- 状态目录 `/var/lib/aios-v2`。
|
|
||||||
|
|
||||||
### Docker/Compose
|
|
||||||
|
|
||||||
- 非 root UID/GID。
|
|
||||||
- Read-only root filesystem。
|
|
||||||
- Drop all capabilities。
|
|
||||||
- State volume。
|
|
||||||
- HTTP healthcheck。
|
|
||||||
|
|
||||||
## 10. 架构约束
|
|
||||||
|
|
||||||
以下行为视为架构违规:
|
|
||||||
|
|
||||||
1. OpenClaw/Hermes 直接创建或批准 AIOS Task。
|
|
||||||
2. Web Console 直接调用 Runtime。
|
|
||||||
3. 新模块不携带 `tenant_id`。
|
|
||||||
4. 新增第二套任务状态机或租户事实源。
|
|
||||||
5. 在 Git 中提交明文凭据。
|
|
||||||
6. 绕过 GitOps 修改生产代码/配置。
|
|
||||||
7. 架构或功能变化未同步更新 `docs/aios-v2/`。
|
|
||||||
8. 未通过测试和预发布门禁直接提升候选版本。
|
|
||||||
|
|
||||||
## 11. 目标架构完成判定
|
|
||||||
|
|
||||||
目标架构只有在以下证据齐全时才视为完成:
|
|
||||||
|
|
||||||
- 文档与代码映射完整。
|
|
||||||
- Candidate tests 对精确 commit 执行通过。
|
|
||||||
- OpenClaw/Hermes 实际 Runtime 联调通过。
|
|
||||||
- 多租户负向测试通过。
|
|
||||||
- 重启恢复和回滚演练通过。
|
|
||||||
- GitOps 本地/Gitea OID 一致。
|
|
||||||
- Owner 批准 merge、Tag 和生产部署。
|
|
||||||
@@ -1,313 +0,0 @@
|
|||||||
# AIOS V2 迁移与交付策略
|
|
||||||
|
|
||||||
- 文档版本:`V2R1C0-candidate`
|
|
||||||
- 交付原则:增量迁移、能力复用、控制权收敛、GitOps 唯一变更入口
|
|
||||||
|
|
||||||
## 1. 迁移目标
|
|
||||||
|
|
||||||
AIOS V2 不通过一次性重写替换现有系统,而是在保留 OpenClaw、Hermes、Agent Router、GitOps 和既有运行能力的前提下,将租户、策略、任务、审批、审计和发布治理收敛到 AIOS Control Plane。
|
|
||||||
|
|
||||||
迁移完成的标志不是“新代码存在”,而是:
|
|
||||||
|
|
||||||
- AIOS 成为唯一权威控制面;
|
|
||||||
- OpenClaw/Hermes 不再保存第二套控制状态;
|
|
||||||
- 所有控制功能 Web/API 化;
|
|
||||||
- 所有模块通过租户隔离、测试和预发布门禁;
|
|
||||||
- 本地 Git、Gitea、测试、部署和回滚证据可追踪。
|
|
||||||
|
|
||||||
## 2. 为什么采用增量迁移
|
|
||||||
|
|
||||||
### 2.1 避免能力回退
|
|
||||||
|
|
||||||
OpenClaw、Hermes 和 Agent Router 已经包含大量运行经验、边界处理和执行能力。整体重写会扩大风险面,并导致业务能力、操作流程和历史数据回退。
|
|
||||||
|
|
||||||
### 2.2 保持可回滚
|
|
||||||
|
|
||||||
每个迁移步骤必须:
|
|
||||||
|
|
||||||
- 独立分支;
|
|
||||||
- 小范围提交;
|
|
||||||
- 独立测试;
|
|
||||||
- 可单独回滚;
|
|
||||||
- 不要求立即关闭旧执行路径。
|
|
||||||
|
|
||||||
### 2.3 先收敛控制权,再优化内部实现
|
|
||||||
|
|
||||||
迁移优先级:
|
|
||||||
|
|
||||||
```text
|
|
||||||
唯一入口
|
|
||||||
-> 权威租户/任务/策略
|
|
||||||
-> Runtime Adapter
|
|
||||||
-> 数据层生产化
|
|
||||||
-> Web 与可观测
|
|
||||||
-> 性能与 HA
|
|
||||||
```
|
|
||||||
|
|
||||||
## 3. 分阶段迁移路线
|
|
||||||
|
|
||||||
### Phase 0:GitOps 与架构事实源
|
|
||||||
|
|
||||||
目标:建立可靠变更链和权威设计基线。
|
|
||||||
|
|
||||||
已完成候选能力:
|
|
||||||
|
|
||||||
- Controller detached guard;
|
|
||||||
- 避免 fetch 到已检出本地分支;
|
|
||||||
- Gitea SSH 镜像;
|
|
||||||
- `docs/aios-v2/` 架构事实源;
|
|
||||||
- 候选分支、部署和回滚资产。
|
|
||||||
|
|
||||||
退出条件:
|
|
||||||
|
|
||||||
- Controller detached 且 clean;
|
|
||||||
- 本地/Gitea OID 一致;
|
|
||||||
- 架构与需求文档纳入版本。
|
|
||||||
|
|
||||||
### Phase 1:Control Plane 权威化
|
|
||||||
|
|
||||||
目标:将 Tenant、Agent、Task、Policy、Approval 和 Audit 收敛到 AIOS。
|
|
||||||
|
|
||||||
候选实现:
|
|
||||||
|
|
||||||
- `aios_v2/control_plane.py`
|
|
||||||
- `aios_v2/persistence.py`
|
|
||||||
- `aios_v2/gateway.py`
|
|
||||||
- `aios_v2/app.py`
|
|
||||||
|
|
||||||
退出条件:
|
|
||||||
|
|
||||||
- 任务状态机统一;
|
|
||||||
- 幂等、revision、审计和跨租户拒绝通过测试;
|
|
||||||
- 服务重启后权威状态恢复。
|
|
||||||
|
|
||||||
### Phase 2:OpenClaw 解耦
|
|
||||||
|
|
||||||
目标:OpenClaw 从控制逻辑转为执行适配器。
|
|
||||||
|
|
||||||
步骤:
|
|
||||||
|
|
||||||
1. 定义签名命令合同;
|
|
||||||
2. 引入 Adapter;
|
|
||||||
3. 将直接调用迁移到 AIOS Orchestrator;
|
|
||||||
4. Runtime 只返回执行结果和健康状态;
|
|
||||||
5. 关闭或网络隔离旧公共控制入口。
|
|
||||||
|
|
||||||
退出条件:
|
|
||||||
|
|
||||||
- 未批准任务无法生成有效命令;
|
|
||||||
- 命令绑定 tenant/task/trace/idempotency;
|
|
||||||
- 旧控制接口冻结并记录替代路径。
|
|
||||||
|
|
||||||
### Phase 3:Hermes 解耦
|
|
||||||
|
|
||||||
目标:Hermes 只承担租户分区队列和 Worker Runtime。
|
|
||||||
|
|
||||||
步骤:
|
|
||||||
|
|
||||||
1. 定义 HermesJob/HermesResult;
|
|
||||||
2. 强制 tenant/cell/profile 分区;
|
|
||||||
3. 引入 claim/lease/heartbeat;
|
|
||||||
4. 将结果统一回写 AIOS;
|
|
||||||
5. 移除 Hermes 内部策略和任务事实源。
|
|
||||||
|
|
||||||
退出条件:
|
|
||||||
|
|
||||||
- Worker 无权创建 AIOS Task;
|
|
||||||
- 租约过期后 Job 可恢复;
|
|
||||||
- 跨租户 claim/read/result 被拒绝。
|
|
||||||
|
|
||||||
### Phase 4:Web 与运维统一
|
|
||||||
|
|
||||||
目标:所有用户与运维功能通过 AIOS Web/API。
|
|
||||||
|
|
||||||
范围:
|
|
||||||
|
|
||||||
- Tenant、Agent、Task、Runtime、Audit、Execution Evidence、GitOps;
|
|
||||||
- systemd/Docker/Compose;
|
|
||||||
- healthcheck、rollback、release report。
|
|
||||||
|
|
||||||
退出条件:
|
|
||||||
|
|
||||||
- 浏览器不直接调用 Runtime;
|
|
||||||
- 生产身份代理和权限模型通过验证;
|
|
||||||
- 部署、重启和回滚演练通过。
|
|
||||||
|
|
||||||
### Phase 5:生产数据层与 HA
|
|
||||||
|
|
||||||
目标:将候选 SQLite/内存运行状态替换或强化为生产级后端。
|
|
||||||
|
|
||||||
要求:
|
|
||||||
|
|
||||||
- 数据迁移;
|
|
||||||
- RLS 或物理隔离;
|
|
||||||
- 备份恢复;
|
|
||||||
- HA 与容量;
|
|
||||||
- 多实例并发;
|
|
||||||
- 灾难恢复目标。
|
|
||||||
|
|
||||||
该阶段不得改变上层合同,避免 Runtime 和 Web 再次耦合存储实现。
|
|
||||||
|
|
||||||
## 4. 分支与提交策略
|
|
||||||
|
|
||||||
### 4.1 分支类型
|
|
||||||
|
|
||||||
| 类型 | 用途 |
|
|
||||||
|---|---|
|
|
||||||
| `task/*` | 可执行任务合同与主体实现 |
|
|
||||||
| `patch/*` | 独立修复或窄范围增强 |
|
|
||||||
| `integration/*` | 多模块候选收敛 |
|
|
||||||
| `result/*` | 自动执行结果与证据 |
|
|
||||||
| `release/*` | 受控发布候选 |
|
|
||||||
|
|
||||||
### 4.2 规则
|
|
||||||
|
|
||||||
- 禁止直接修改 `main`;
|
|
||||||
- 一个提交应可独立审阅和回滚;
|
|
||||||
- 大文件和大范围变更应拆分;
|
|
||||||
- 所有行为变化必须同步文档和测试;
|
|
||||||
- 生产 Tag、部署和回滚需要 Owner 批准。
|
|
||||||
|
|
||||||
## 5. GitOps 交付流程
|
|
||||||
|
|
||||||
```text
|
|
||||||
需求/架构更新
|
|
||||||
|
|
|
||||||
v
|
|
||||||
任务分支 + 文档 + 测试
|
|
||||||
|
|
|
||||||
v
|
|
||||||
代码提交 + Secret Gate
|
|
||||||
|
|
|
||||||
v
|
|
||||||
本地 bare repo 同步
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Gitea 镜像与 OID 校验
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Candidate 集成
|
|
||||||
|
|
|
||||||
v
|
|
||||||
自动测试与预发布
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Owner 审批
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Merge / Tag / Production Deploy
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5.1 同步状态
|
|
||||||
|
|
||||||
GitOps 返回必须区分:
|
|
||||||
|
|
||||||
- local sync;
|
|
||||||
- Gitea sync;
|
|
||||||
- branch OID;
|
|
||||||
- workspace mode;
|
|
||||||
- dirty state。
|
|
||||||
|
|
||||||
仅返回 `common_git_dir synced` 不足以证明 Web 远端可见。
|
|
||||||
|
|
||||||
## 6. 发布门禁
|
|
||||||
|
|
||||||
### Gate A:设计完整性
|
|
||||||
|
|
||||||
- 用户需求、目标架构、模块契约、ADR 和追踪矩阵同步;
|
|
||||||
- 无冲突标记、断链或过期路径。
|
|
||||||
|
|
||||||
### Gate B:代码质量
|
|
||||||
|
|
||||||
- 包可导入;
|
|
||||||
- 静态编译通过;
|
|
||||||
- Secret Scanner 通过;
|
|
||||||
- 无已知越权旁路。
|
|
||||||
|
|
||||||
### Gate C:测试
|
|
||||||
|
|
||||||
- 单元、集成、多租户负向、重启恢复、Runtime 边界测试通过;
|
|
||||||
- 记录 commit、命令、exit code、pass/fail 和 artifact digest。
|
|
||||||
|
|
||||||
### Gate D:预发布
|
|
||||||
|
|
||||||
- systemd 或 Compose 启动成功;
|
|
||||||
- 健康检查通过;
|
|
||||||
- OpenClaw/Hermes 真实联调通过;
|
|
||||||
- 备份、重启和回滚演练通过;
|
|
||||||
- 日志中无敏感信息。
|
|
||||||
|
|
||||||
### Gate E:生产批准
|
|
||||||
|
|
||||||
- Owner 批准 merge、Tag 和生产部署;
|
|
||||||
- 发布窗口、回滚负责人和观察指标明确;
|
|
||||||
- 生产变更通过 GitOps 记录。
|
|
||||||
|
|
||||||
## 7. 回滚策略
|
|
||||||
|
|
||||||
### 7.1 代码回滚
|
|
||||||
|
|
||||||
- 使用 `/opt/aios/current` 原子 symlink 切换;
|
|
||||||
- 回滚前备份状态;
|
|
||||||
- 健康检查失败自动恢复 previous release。
|
|
||||||
|
|
||||||
### 7.2 状态回滚
|
|
||||||
|
|
||||||
默认不自动回滚状态,避免覆盖候选运行期间新增的 Task、Audit 和 Evidence。
|
|
||||||
|
|
||||||
状态恢复只有在:
|
|
||||||
|
|
||||||
- 明确确认 schema 不兼容;
|
|
||||||
- 数据备份可验证;
|
|
||||||
- Owner 显式授权;
|
|
||||||
|
|
||||||
时启用。
|
|
||||||
|
|
||||||
### 7.3 Runtime 回滚
|
|
||||||
|
|
||||||
OpenClaw/Hermes Adapter 采用稳定合同。若候选执行层失败,可恢复旧 Runtime 版本,但不得恢复旧控制权旁路。
|
|
||||||
|
|
||||||
## 8. Bug 修复流程
|
|
||||||
|
|
||||||
生产或预发布 Bug:
|
|
||||||
|
|
||||||
1. 创建 `patch/bugfix/*`;
|
|
||||||
2. 添加可复现测试;
|
|
||||||
3. 修复代码;
|
|
||||||
4. 更新相关需求/架构/契约/ADR;
|
|
||||||
5. 运行候选验收;
|
|
||||||
6. 进入新的 release candidate;
|
|
||||||
7. Owner 批准部署。
|
|
||||||
|
|
||||||
禁止在生产目录直接编辑文件后不回写 Git。
|
|
||||||
|
|
||||||
## 9. 文档同步规则
|
|
||||||
|
|
||||||
以下变化必须更新文档:
|
|
||||||
|
|
||||||
- 用户行为或页面;
|
|
||||||
- API 路径、Header、请求/响应;
|
|
||||||
- 数据模型、状态机、策略;
|
|
||||||
- OpenClaw/Hermes/Router 边界;
|
|
||||||
- 多租户与安全模型;
|
|
||||||
- 部署、健康检查、回滚;
|
|
||||||
- 测试或发布门禁。
|
|
||||||
|
|
||||||
对应关系:
|
|
||||||
|
|
||||||
- 需求变化:更新 `01-USER-REQUIREMENTS.md`;
|
|
||||||
- 架构变化:更新 `02-TARGET-ARCHITECTURE.md` 和 ADR;
|
|
||||||
- 迁移/发布变化:更新本文件;
|
|
||||||
- 合同变化:更新 `04-MODULE-CONTRACTS.md`;
|
|
||||||
- 代码/测试路径变化:更新 `05-TRACEABILITY-MATRIX.md`。
|
|
||||||
|
|
||||||
## 10. 当前候选后续步骤
|
|
||||||
|
|
||||||
1. 完成文档同步门禁和 Acceptance Runner;
|
|
||||||
2. 在精确 Candidate commit 执行测试;
|
|
||||||
3. 推送 Gitea 并校验 OID;
|
|
||||||
4. 部署隔离预发布环境;
|
|
||||||
5. 联调真实 OpenClaw/Hermes;
|
|
||||||
6. 执行重启、性能、安全和回滚演练;
|
|
||||||
7. 生成最终 evidence;
|
|
||||||
8. 请求 Owner 的 merge/Tag/production approval。
|
|
||||||
@@ -1,502 +0,0 @@
|
|||||||
# AIOS V2 模块契约
|
|
||||||
|
|
||||||
- 文档版本:`V2R1C0-candidate`
|
|
||||||
- 状态:候选实现契约基线
|
|
||||||
- 原则:Control Plane 权威、Runtime 执行、全链路租户隔离、所有状态可审计
|
|
||||||
|
|
||||||
## 1. 通用上下文契约
|
|
||||||
|
|
||||||
所有控制与执行请求必须携带以下上下文:
|
|
||||||
|
|
||||||
| 字段 | 必填 | 说明 |
|
|
||||||
|---|---:|---|
|
|
||||||
| `tenant_id` | 是 | 权威租户标识 |
|
|
||||||
| `principal_id` | 控制请求是 | 已认证主体 |
|
|
||||||
| `trace_id` | 是 | 全链路追踪 |
|
|
||||||
| `task_id` | 执行阶段是 | AIOS 权威任务 |
|
|
||||||
| `session_id` | Web/交互请求建议 | 用户会话 |
|
|
||||||
| `policy_version` | 控制请求是 | 策略版本 |
|
|
||||||
| `idempotency_key` | 创建/执行是 | 幂等键 |
|
|
||||||
|
|
||||||
规则:
|
|
||||||
|
|
||||||
- 缺少必填字段时 fail closed。
|
|
||||||
- 客户端不得覆盖由可信代理或 AIOS 生成的 `tenant_id`。
|
|
||||||
- Runtime 返回的 tenant/task/trace 必须与请求完全匹配。
|
|
||||||
- 所有持久化键、缓存键和队列分区必须包含租户命名空间。
|
|
||||||
|
|
||||||
## 2. Tenant Contract
|
|
||||||
|
|
||||||
### 2.1 TenantRecord
|
|
||||||
|
|
||||||
```text
|
|
||||||
tenant_id
|
|
||||||
display_name
|
|
||||||
status
|
|
||||||
metadata
|
|
||||||
created_at
|
|
||||||
updated_at
|
|
||||||
```
|
|
||||||
|
|
||||||
约束:
|
|
||||||
|
|
||||||
- `tenant_id` 不可为空且不可变。
|
|
||||||
- `status` 至少支持 `active` 和受控停用状态。
|
|
||||||
- Tenant 只由 AIOS Control Plane 创建和维护。
|
|
||||||
- Runtime 不得创建租户。
|
|
||||||
|
|
||||||
实现:`active-runtime/app/aios_v2/control_plane.py`
|
|
||||||
|
|
||||||
## 3. Agent Contract
|
|
||||||
|
|
||||||
### 3.1 AgentRecord
|
|
||||||
|
|
||||||
```text
|
|
||||||
agent_id
|
|
||||||
tenant_id
|
|
||||||
display_name
|
|
||||||
runtime
|
|
||||||
capabilities
|
|
||||||
enabled
|
|
||||||
metadata
|
|
||||||
created_at
|
|
||||||
updated_at
|
|
||||||
```
|
|
||||||
|
|
||||||
约束:
|
|
||||||
|
|
||||||
- Agent 属于唯一租户。
|
|
||||||
- `runtime` 由 Registry 决定,可取 `openclaw`、`hermes` 或后续注册 Runtime。
|
|
||||||
- `capabilities` 是允许调度的动作集合。
|
|
||||||
- `metadata` 可存 `cell_id`、`profile_id`、priority 等非敏感引用。
|
|
||||||
- 不得在 Agent metadata 中保存明文凭据。
|
|
||||||
|
|
||||||
## 4. Task Contract
|
|
||||||
|
|
||||||
### 4.1 TaskRecord
|
|
||||||
|
|
||||||
```text
|
|
||||||
task_id
|
|
||||||
tenant_id
|
|
||||||
agent_id
|
|
||||||
action
|
|
||||||
payload
|
|
||||||
state
|
|
||||||
revision
|
|
||||||
idempotency_key
|
|
||||||
request_digest
|
|
||||||
result
|
|
||||||
error
|
|
||||||
created_by
|
|
||||||
approved_by
|
|
||||||
created_at
|
|
||||||
updated_at
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4.2 状态机
|
|
||||||
|
|
||||||
```text
|
|
||||||
CREATED -> APPROVED -> RUNNING -> COMPLETED
|
|
||||||
| | |
|
|
||||||
+----------+----------+-> CANCELLED
|
|
||||||
+-> FAILED
|
|
||||||
```
|
|
||||||
|
|
||||||
允许转换:
|
|
||||||
|
|
||||||
| 当前 | 允许目标 |
|
|
||||||
|---|---|
|
|
||||||
| CREATED | APPROVED, CANCELLED |
|
|
||||||
| APPROVED | RUNNING, CANCELLED |
|
|
||||||
| RUNNING | COMPLETED, FAILED, CANCELLED |
|
|
||||||
| COMPLETED | 无 |
|
|
||||||
| FAILED | 无 |
|
|
||||||
| CANCELLED | 无 |
|
|
||||||
|
|
||||||
约束:
|
|
||||||
|
|
||||||
- 每次转换必须校验 `revision`。
|
|
||||||
- COMPLETED 必须带 `result`。
|
|
||||||
- FAILED 必须带 `error`。
|
|
||||||
- 每次转换生成 AuditEvent。
|
|
||||||
- Runtime 不得直接写 Task Store,只能通过 Orchestrator reconcile。
|
|
||||||
|
|
||||||
## 5. Policy Contract
|
|
||||||
|
|
||||||
### 5.1 PolicyDecision
|
|
||||||
|
|
||||||
```text
|
|
||||||
allowed
|
|
||||||
reason
|
|
||||||
policy_version
|
|
||||||
```
|
|
||||||
|
|
||||||
约束:
|
|
||||||
|
|
||||||
- Policy 默认拒绝。
|
|
||||||
- Runtime 不得覆盖 PolicyDecision。
|
|
||||||
- 拒绝必须审计。
|
|
||||||
- 高风险动作必须经过 Approval。
|
|
||||||
|
|
||||||
## 6. Audit Contract
|
|
||||||
|
|
||||||
### 6.1 AuditEvent
|
|
||||||
|
|
||||||
```text
|
|
||||||
event_id
|
|
||||||
tenant_id
|
|
||||||
principal_id
|
|
||||||
trace_id
|
|
||||||
action
|
|
||||||
resource_type
|
|
||||||
resource_id
|
|
||||||
outcome
|
|
||||||
policy_version
|
|
||||||
details
|
|
||||||
created_at
|
|
||||||
```
|
|
||||||
|
|
||||||
约束:
|
|
||||||
|
|
||||||
- 按租户查询。
|
|
||||||
- `details` 不得包含明文凭据。
|
|
||||||
- OpenClaw/Hermes 生命周期事件必须映射到统一 AuditEvent。
|
|
||||||
- 生产版需增加不可变外部固化和完整性验证。
|
|
||||||
|
|
||||||
## 7. OpenClaw Contract
|
|
||||||
|
|
||||||
### 7.1 OpenClawCommand
|
|
||||||
|
|
||||||
```text
|
|
||||||
contract_version
|
|
||||||
command_id
|
|
||||||
tenant_id
|
|
||||||
task_id
|
|
||||||
trace_id
|
|
||||||
action
|
|
||||||
payload
|
|
||||||
idempotency_key
|
|
||||||
created_at
|
|
||||||
expires_at
|
|
||||||
key_id
|
|
||||||
signature
|
|
||||||
```
|
|
||||||
|
|
||||||
签名覆盖:
|
|
||||||
|
|
||||||
- command_id
|
|
||||||
- tenant_id
|
|
||||||
- task_id
|
|
||||||
- trace_id
|
|
||||||
- action
|
|
||||||
- payload
|
|
||||||
- idempotency_key
|
|
||||||
- created_at / expires_at
|
|
||||||
- contract_version / key_id
|
|
||||||
|
|
||||||
约束:
|
|
||||||
|
|
||||||
- 使用规范化 JSON 和 HMAC-SHA256。
|
|
||||||
- 只接受已批准任务生成的命令。
|
|
||||||
- 过期、未来时间超差、签名错误命令必须拒绝。
|
|
||||||
- `key_id` 只引用运行时材料,不包含材料本身。
|
|
||||||
|
|
||||||
### 7.2 OpenClawResult
|
|
||||||
|
|
||||||
```text
|
|
||||||
command_id
|
|
||||||
tenant_id
|
|
||||||
task_id
|
|
||||||
trace_id
|
|
||||||
state
|
|
||||||
output
|
|
||||||
error
|
|
||||||
started_at
|
|
||||||
finished_at
|
|
||||||
runtime_id
|
|
||||||
attempts
|
|
||||||
```
|
|
||||||
|
|
||||||
状态:
|
|
||||||
|
|
||||||
```text
|
|
||||||
ACCEPTED
|
|
||||||
RUNNING
|
|
||||||
COMPLETED
|
|
||||||
FAILED
|
|
||||||
CANCELLED
|
|
||||||
TIMED_OUT
|
|
||||||
REJECTED
|
|
||||||
```
|
|
||||||
|
|
||||||
约束:
|
|
||||||
|
|
||||||
- Adapter 对外只接受终态结果用于 reconcile。
|
|
||||||
- 身份字段不匹配时拒绝回写。
|
|
||||||
- 结果通过 Orchestrator 更新 AIOS Task。
|
|
||||||
|
|
||||||
### 7.3 RuntimeHealth
|
|
||||||
|
|
||||||
```text
|
|
||||||
runtime
|
|
||||||
runtime_id
|
|
||||||
status
|
|
||||||
version
|
|
||||||
heartbeat_at
|
|
||||||
active_tasks
|
|
||||||
workspace_isolated
|
|
||||||
credential_ref_configured
|
|
||||||
details
|
|
||||||
```
|
|
||||||
|
|
||||||
健康策略:
|
|
||||||
|
|
||||||
- heartbeat 不得超时。
|
|
||||||
- workspace isolation 与 credential reference 必须符合策略。
|
|
||||||
- 不安全 Runtime 应 quarantine,不得继续接收新任务。
|
|
||||||
|
|
||||||
实现:`active-runtime/app/aios_openclaw/`
|
|
||||||
|
|
||||||
## 8. Hermes Contract
|
|
||||||
|
|
||||||
### 8.1 HermesJob
|
|
||||||
|
|
||||||
```text
|
|
||||||
job_id
|
|
||||||
tenant_id
|
|
||||||
cell_id
|
|
||||||
profile_id
|
|
||||||
task_id
|
|
||||||
trace_id
|
|
||||||
action
|
|
||||||
payload
|
|
||||||
idempotency_key
|
|
||||||
priority
|
|
||||||
state
|
|
||||||
revision
|
|
||||||
assigned_worker_id
|
|
||||||
lease_expires_at
|
|
||||||
created_at
|
|
||||||
updated_at
|
|
||||||
```
|
|
||||||
|
|
||||||
状态机:
|
|
||||||
|
|
||||||
```text
|
|
||||||
QUEUED -> CLAIMED -> RUNNING -> COMPLETED
|
|
||||||
| | |
|
|
||||||
+---------+----------+-> CANCELLED
|
|
||||||
+-> FAILED
|
|
||||||
```
|
|
||||||
|
|
||||||
约束:
|
|
||||||
|
|
||||||
- `tenant_id + cell_id + profile_id` 形成执行分区。
|
|
||||||
- claim 只能领取同分区 Job。
|
|
||||||
- Worker ID 不得跨分区复用。
|
|
||||||
- Lease 到期 Job 回到 QUEUED。
|
|
||||||
- revision 冲突不得覆盖。
|
|
||||||
|
|
||||||
### 8.2 HermesResult
|
|
||||||
|
|
||||||
```text
|
|
||||||
job_id
|
|
||||||
tenant_id
|
|
||||||
task_id
|
|
||||||
trace_id
|
|
||||||
state
|
|
||||||
worker_id
|
|
||||||
output
|
|
||||||
error
|
|
||||||
started_at
|
|
||||||
finished_at
|
|
||||||
```
|
|
||||||
|
|
||||||
约束:
|
|
||||||
|
|
||||||
- COMPLETED 必须有 output。
|
|
||||||
- FAILED 必须有 error。
|
|
||||||
- 只允许 AIOS Orchestrator reconcile。
|
|
||||||
|
|
||||||
### 8.3 HermesHealth
|
|
||||||
|
|
||||||
```text
|
|
||||||
runtime_id
|
|
||||||
status
|
|
||||||
version
|
|
||||||
heartbeat_at
|
|
||||||
active_workers
|
|
||||||
active_jobs
|
|
||||||
queued_jobs
|
|
||||||
tenant_partitions
|
|
||||||
profile_isolation_enforced
|
|
||||||
details
|
|
||||||
```
|
|
||||||
|
|
||||||
实现:`active-runtime/app/aios_hermes/`
|
|
||||||
|
|
||||||
## 9. Execution Evidence Contract
|
|
||||||
|
|
||||||
### 9.1 ExecutionEvidence
|
|
||||||
|
|
||||||
```text
|
|
||||||
execution_id
|
|
||||||
tenant_id
|
|
||||||
task_id
|
|
||||||
agent_id
|
|
||||||
executor
|
|
||||||
branch
|
|
||||||
status
|
|
||||||
progress_completed
|
|
||||||
progress_total
|
|
||||||
current_step
|
|
||||||
last_commit
|
|
||||||
test_status
|
|
||||||
test_summary
|
|
||||||
result_summary
|
|
||||||
claimed_at
|
|
||||||
heartbeat_at
|
|
||||||
completed_at
|
|
||||||
```
|
|
||||||
|
|
||||||
状态建议:
|
|
||||||
|
|
||||||
```text
|
|
||||||
claimed
|
|
||||||
running
|
|
||||||
blocked
|
|
||||||
complete
|
|
||||||
failed
|
|
||||||
cancelled
|
|
||||||
```
|
|
||||||
|
|
||||||
约束:
|
|
||||||
|
|
||||||
- 按租户存储和查询。
|
|
||||||
- 每个阻塞必须记录原因、责任方和解除条件。
|
|
||||||
- 每次结果必须记录 branch、commit、测试命令和状态。
|
|
||||||
- 不得包含 Secret 或完整敏感 payload。
|
|
||||||
|
|
||||||
## 10. GitOps Status Contract
|
|
||||||
|
|
||||||
```text
|
|
||||||
available
|
|
||||||
service_status
|
|
||||||
workspace_mode
|
|
||||||
workspace_branch
|
|
||||||
workspace_ref
|
|
||||||
workspace_dirty
|
|
||||||
candidate_branch
|
|
||||||
release_channel
|
|
||||||
release_version
|
|
||||||
sync_status
|
|
||||||
details
|
|
||||||
```
|
|
||||||
|
|
||||||
约束:
|
|
||||||
|
|
||||||
- Web 响应不得包含 remote URL、凭据或 stderr。
|
|
||||||
- Controller 必须 `detached + clean`。
|
|
||||||
- `sync_status` 必须区分 local 和 Gitea 镜像状态。
|
|
||||||
|
|
||||||
## 11. Web/API Contract
|
|
||||||
|
|
||||||
### 11.1 身份头
|
|
||||||
|
|
||||||
可信代理注入:
|
|
||||||
|
|
||||||
```text
|
|
||||||
X-AIOS-Authenticated
|
|
||||||
X-AIOS-Tenant-ID
|
|
||||||
X-AIOS-Principal-ID
|
|
||||||
X-AIOS-Roles
|
|
||||||
X-AIOS-Trace-ID
|
|
||||||
X-AIOS-Session-ID
|
|
||||||
X-AIOS-Policy-Version
|
|
||||||
```
|
|
||||||
|
|
||||||
生产约束:
|
|
||||||
|
|
||||||
- 代理必须删除客户端同名头后重新注入。
|
|
||||||
- 应用不得直接暴露公网端口。
|
|
||||||
|
|
||||||
### 11.2 核心 API
|
|
||||||
|
|
||||||
```text
|
|
||||||
GET /api/v2/health
|
|
||||||
POST /api/v2/tenants
|
|
||||||
GET /api/v2/tenants/current
|
|
||||||
POST /api/v2/agents
|
|
||||||
GET /api/v2/agents
|
|
||||||
POST /api/v2/tasks
|
|
||||||
GET /api/v2/tasks
|
|
||||||
GET /api/v2/tasks/{id}
|
|
||||||
POST /api/v2/tasks/{id}/approve
|
|
||||||
POST /api/v2/tasks/{id}/start
|
|
||||||
POST /api/v2/tasks/{id}/complete
|
|
||||||
POST /api/v2/tasks/{id}/fail
|
|
||||||
POST /api/v2/tasks/{id}/cancel
|
|
||||||
POST /api/v2/tasks/{id}/dispatch
|
|
||||||
GET /api/v2/audit
|
|
||||||
GET /api/v2/runtime
|
|
||||||
GET /api/v2/executions
|
|
||||||
GET /api/v2/gitops
|
|
||||||
POST /api/v2/hermes/results/reconcile
|
|
||||||
```
|
|
||||||
|
|
||||||
## 12. Persistence Contract
|
|
||||||
|
|
||||||
候选版:
|
|
||||||
|
|
||||||
- SQLite WAL
|
|
||||||
- `BEGIN IMMEDIATE`
|
|
||||||
- 原子 snapshot
|
|
||||||
- schema version
|
|
||||||
- restart restore
|
|
||||||
|
|
||||||
持久化范围:
|
|
||||||
|
|
||||||
- tenants
|
|
||||||
- agents
|
|
||||||
- tasks
|
|
||||||
- audit
|
|
||||||
- idempotency
|
|
||||||
|
|
||||||
生产替换要求:
|
|
||||||
|
|
||||||
- 不改变 Domain/API Contract。
|
|
||||||
- 提供 migration、backup、restore、HA、RLS/隔离和容量测试。
|
|
||||||
|
|
||||||
## 13. Deployment Contract
|
|
||||||
|
|
||||||
### systemd
|
|
||||||
|
|
||||||
- 非 root `aios` 用户。
|
|
||||||
- 状态目录 `/var/lib/aios-v2`。
|
|
||||||
- 最小权限和文件系统保护。
|
|
||||||
|
|
||||||
### Docker
|
|
||||||
|
|
||||||
- 非 root UID/GID。
|
|
||||||
- read-only root filesystem。
|
|
||||||
- drop all capabilities。
|
|
||||||
- 独立状态 volume。
|
|
||||||
|
|
||||||
### Rollback
|
|
||||||
|
|
||||||
- 回滚前备份状态。
|
|
||||||
- 默认只回滚代码。
|
|
||||||
- 状态恢复需显式授权。
|
|
||||||
- 回滚后必须执行 healthcheck。
|
|
||||||
|
|
||||||
## 14. 契约变更规则
|
|
||||||
|
|
||||||
任何以下变化必须在同一提交或同一分支更新本文档:
|
|
||||||
|
|
||||||
- 字段、状态、Header、API、错误语义;
|
|
||||||
- 租户或安全约束;
|
|
||||||
- OpenClaw/Hermes 边界;
|
|
||||||
- Persistence 或 Deployment Contract;
|
|
||||||
- Evidence/Audit 格式。
|
|
||||||
|
|
||||||
不兼容变化必须新增 ADR 和版本号,不得静默修改。
|
|
||||||
@@ -1,113 +0,0 @@
|
|||||||
# AIOS V2 需求—架构—代码—测试—部署追踪矩阵
|
|
||||||
|
|
||||||
- 文档版本:`V2R1C0-candidate`
|
|
||||||
- 目的:确保每项用户需求都能追踪到架构设计、代码实现、自动测试、部署资产和验收命令
|
|
||||||
|
|
||||||
## 1. 追踪规则
|
|
||||||
|
|
||||||
每条需求至少具备:
|
|
||||||
|
|
||||||
1. 需求编号;
|
|
||||||
2. 架构章节;
|
|
||||||
3. 实现路径;
|
|
||||||
4. 测试路径;
|
|
||||||
5. 部署/运维资产;
|
|
||||||
6. 验收状态。
|
|
||||||
|
|
||||||
状态定义:
|
|
||||||
|
|
||||||
- `Implemented`:代码已进入 Candidate 分支;
|
|
||||||
- `Test-defined`:测试代码已进入 Candidate 分支;
|
|
||||||
- `Test-verified`:精确 commit 的测试已运行并保存证据;
|
|
||||||
- `Pre-production verified`:预发布部署、健康检查和回滚演练已通过;
|
|
||||||
- `Production released`:Owner 批准并完成 merge/tag/deploy。
|
|
||||||
|
|
||||||
## 2. 核心追踪矩阵
|
|
||||||
|
|
||||||
| 需求 | 设计依据 | 代码实现 | 测试 | 部署/运维 | 当前状态 |
|
|
||||||
|---|---|---|---|---|---|
|
|
||||||
| UR-001 AIOS 唯一控制面 | `02-TARGET-ARCHITECTURE.md` §3.2/§3.3;ADR-001 | `aios_v2/control_plane.py`、`aios_v2/orchestrator.py`、`aios_v2/app.py` | `tests/aios_v2_candidate/test_candidate_flow.py` | `deploy/aios-v2/systemd/aios-v2.service` | Implemented / Test-defined |
|
|
||||||
| UR-002 复用现有能力 | `02-TARGET-ARCHITECTURE.md` §2;`03-MIGRATION-AND-DELIVERY.md` §2 | `aios_openclaw/`、`aios_hermes/` | OpenClaw/Hermes flow tests | Candidate branch + adapter boundaries | Implemented / Test-defined |
|
|
||||||
| UR-003 OpenClaw/Hermes 解耦 | `02-TARGET-ARCHITECTURE.md` §3.4/§3.5;ADR-002 | `aios_openclaw/adapter.py`、`aios_hermes/adapter.py` | runtime boundary tests in candidate flow suite | systemd/Docker runtime configuration | Implemented / Test-defined |
|
|
||||||
| UR-004 全面 Web 化 | `02-TARGET-ARCHITECTURE.md` §3.1/§8 | `aios_v2/console.py`、`aios_v2/gateway.py`、`aios_v2/app.py` | API/console smoke test pending | Docker/systemd + reverse proxy requirement | Implemented / Test pending |
|
|
||||||
| UR-005 多租户隔离 | `02-TARGET-ARCHITECTURE.md` §4;ADR-004 | `control_plane.py`、`orchestrator.py`、`aios_openclaw/*`、`aios_hermes/*`、`execution_tracking.py` | `test_cross_tenant_task_read_is_denied` | tenant-scoped state/evidence directories | Implemented / Test-defined |
|
|
||||||
| UR-006 权威任务生命周期 | `04-MODULE-CONTRACTS.md` §4 | `control_plane.py` | OpenClaw/Hermes full-flow tests | persisted state + audit | Implemented / Test-defined |
|
|
||||||
| UR-007 策略与审批 | `04-MODULE-CONTRACTS.md` §5 | `control_plane.py`、`gateway.py` | role/policy negative tests需补充 | reverse proxy identity + audit | Implemented / Partial test |
|
|
||||||
| UR-008 幂等与并发安全 | `02-TARGET-ARCHITECTURE.md` §7 | `control_plane.py`、`aios_openclaw/adapter.py`、`aios_hermes/adapter.py`、`worker.py` | flow + restart tests;并发专项测试需补充 | GitOps detached guard + SQLite WAL | Implemented / Partial test |
|
|
||||||
| UR-009 执行证据 | `04-MODULE-CONTRACTS.md` §9 | `execution_tracking.py` | `test_execution_tracking_persists_claim_heartbeat_and_result` | `/var/lib/aios-v2/executions` | Implemented / Test-defined |
|
|
||||||
| UR-010 GitOps 唯一变更流程 | `03-MIGRATION-AND-DELIVERY.md` §4/§5;ADR-003 | GitOps Runtime、`gitops_status.py` | healthcheck 的 detached/clean 检查 | `deploy/aios-v2/healthcheck/check.sh` | Implemented / Runtime verified partially |
|
|
||||||
| UR-011 持久化与恢复 | `02-TARGET-ARCHITECTURE.md` §3.6 | `persistence.py` | `test_durable_service_recovers_state_after_restart` | state dir + backup/rollback | Implemented / Test-defined |
|
|
||||||
| UR-012 审计与合规 | `04-MODULE-CONTRACTS.md` §6 | `control_plane.py`、`aios_openclaw/callback.py`、`app.py` | flow test + audit assertions需补充 | Console audit view | Implemented / Partial test |
|
|
||||||
| UR-013 密钥安全 | `02-TARGET-ARCHITECTURE.md` §6;ADR-006 | `FileSigningMaterialResolver`、credential refs、Secret Gate | permission/secret-scan test需补充 | env template + external protected file | Implemented / Partial test |
|
|
||||||
| UR-014 部署与回滚 | `03-MIGRATION-AND-DELIVERY.md` §7 | deployment scripts/assets | acceptance runner + healthcheck | `deploy/aios-v2/` | Implemented / Pre-prod pending |
|
|
||||||
| UR-015 文档与代码同步 | `03-MIGRATION-AND-DELIVERY.md` §9;ADR-005 | `tools/aios_v2_docs_code_check.py` | checker self-test / acceptance runner | CI/Gitea Actions 可调用 | Implemented after checker commit |
|
|
||||||
|
|
||||||
## 3. API 追踪
|
|
||||||
|
|
||||||
| API | 需求 | 实现 | 测试状态 |
|
|
||||||
|---|---|---|---|
|
|
||||||
| `GET /api/v2/health` | UR-004, UR-014 | `gateway.py` | healthcheck 定义完成 |
|
|
||||||
| `POST /api/v2/tenants` | UR-001, UR-005 | `gateway.py` / `control_plane.py` | 需增加 HTTP 集成测试 |
|
|
||||||
| `POST /api/v2/agents` | UR-001, UR-005 | `gateway.py` / `control_plane.py` | Domain flow 已覆盖 |
|
|
||||||
| `POST /api/v2/tasks` | UR-006, UR-008 | `gateway.py` / `control_plane.py` | Domain flow 已覆盖 |
|
|
||||||
| `POST /api/v2/tasks/{id}/approve` | UR-006, UR-007 | `gateway.py` | Domain flow 已覆盖 |
|
|
||||||
| `POST /api/v2/tasks/{id}/dispatch` | UR-001, UR-003 | `app.py` / `orchestrator.py` | OpenClaw/Hermes flow 已覆盖 |
|
|
||||||
| `GET /api/v2/runtime` | UR-004, UR-009 | `app.py` | HTTP smoke test待执行 |
|
|
||||||
| `GET /api/v2/executions` | UR-009 | `app.py` / `execution_tracking.py` | Evidence persistence test已定义 |
|
|
||||||
| `GET /api/v2/gitops` | UR-010 | `app.py` / `gitops_status.py` | Runtime/healthcheck验证待执行 |
|
|
||||||
| `POST /api/v2/hermes/results/reconcile` | UR-003, UR-006 | `app.py` / `orchestrator.py` | Hermes flow已覆盖 Domain 路径 |
|
|
||||||
|
|
||||||
## 4. 运行时契约追踪
|
|
||||||
|
|
||||||
| 契约 | 设计 | 代码 | 测试 |
|
|
||||||
|---|---|---|---|
|
|
||||||
| OpenClawCommand | `04-MODULE-CONTRACTS.md` §7.1 | `aios_openclaw/contract.py` | OpenClaw full-flow;签名负向测试需补充 |
|
|
||||||
| OpenClawResult | §7.2 | `aios_openclaw/contract.py` | OpenClaw full-flow |
|
|
||||||
| RuntimeHealth | §7.3 | `aios_openclaw/contract.py`、`health.py` | health policy测试需补充 |
|
|
||||||
| HermesJob | §8.1 | `aios_hermes/contract.py` | Hermes full-flow |
|
|
||||||
| HermesResult | §8.2 | `aios_hermes/contract.py` | Hermes full-flow |
|
|
||||||
| HermesHealth | §8.3 | `aios_hermes/contract.py`、`adapter.py` | healthcheck/worker test待执行 |
|
|
||||||
| ExecutionEvidence | §9 | `execution_tracking.py` | evidence persistence test |
|
|
||||||
| GitOpsStatus | §10 | `gitops_status.py` | healthcheck detached/clean检查 |
|
|
||||||
|
|
||||||
## 5. 部署追踪
|
|
||||||
|
|
||||||
| 能力 | 文件 | 验收命令 |
|
|
||||||
|---|---|---|
|
|
||||||
| systemd | `deploy/aios-v2/systemd/aios-v2.service` | `systemctl status aios-v2` |
|
|
||||||
| 环境模板 | `deploy/aios-v2/env/aios-v2.env.example` | `systemd-analyze verify` + 文件权限检查 |
|
|
||||||
| Docker | `deploy/aios-v2/docker/Dockerfile` | `docker build` |
|
|
||||||
| Compose | `deploy/aios-v2/docker/compose.yaml` | `docker compose config` / `up` |
|
|
||||||
| 健康检查 | `deploy/aios-v2/healthcheck/check.sh` | 直接执行 |
|
|
||||||
| 回滚 | `deploy/aios-v2/rollback/rollback.sh` | 预发布回滚演练 |
|
|
||||||
| 全量验收 | `deploy/aios-v2/acceptance/run-all.sh` | 直接执行 |
|
|
||||||
|
|
||||||
## 6. 文档覆盖追踪
|
|
||||||
|
|
||||||
| 文档 | 覆盖内容 | 代码/变更触发条件 |
|
|
||||||
|---|---|---|
|
|
||||||
| `README.md` | 事实源索引、状态语言、实现地图 | 新模块/目录/版本 |
|
|
||||||
| `01-USER-REQUIREMENTS.md` | 用户需求、NFR、验收场景 | 用户行为、功能范围、验收变化 |
|
|
||||||
| `02-TARGET-ARCHITECTURE.md` | 分层、边界、数据流、安全、租户 | 架构边界、数据流、运行时变化 |
|
|
||||||
| `03-MIGRATION-AND-DELIVERY.md` | 迁移、GitOps、Gate、回滚 | 发布流程、迁移阶段、门禁变化 |
|
|
||||||
| `04-MODULE-CONTRACTS.md` | 字段、API、状态机、错误语义 | Contract/API/模型变化 |
|
|
||||||
| `05-TRACEABILITY-MATRIX.md` | 需求到代码测试部署映射 | 代码、测试、部署路径变化 |
|
|
||||||
| `adr/*` | 关键不可逆或跨模块决策 | 新架构决策或旧决策替换 |
|
|
||||||
|
|
||||||
## 7. 当前未完成验证项
|
|
||||||
|
|
||||||
以下项目代码或测试定义已存在,但尚不能标记为 `Test-verified` 或 `Pre-production verified`:
|
|
||||||
|
|
||||||
1. 精确 Candidate commit 的测试执行结果;
|
|
||||||
2. HTTP API 与 Web Console 冒烟测试;
|
|
||||||
3. 真实 OpenClaw Runtime 签名、取消、超时和健康联调;
|
|
||||||
4. Hermes 长任务心跳、Worker 崩溃和 Lease 恢复演练;
|
|
||||||
5. 多实例 SQLite/并发容量边界;
|
|
||||||
6. systemd、Docker、Compose、restart 和 rollback 演练;
|
|
||||||
7. local/Gitea Candidate OID 一致证明;
|
|
||||||
8. 生产身份代理、防伪 Header、CSRF/CORS/Rate Limit 验证;
|
|
||||||
9. 文档—代码同步检查器在 CI/Acceptance 中执行通过。
|
|
||||||
|
|
||||||
## 8. 更新要求
|
|
||||||
|
|
||||||
任何代码、API、部署或测试路径变化,必须同步更新本矩阵。验收脚本会检查本文档引用的关键路径存在,并检查架构/功能代码变化是否同时包含 `docs/aios-v2/` 变化。
|
|
||||||
@@ -1,98 +0,0 @@
|
|||||||
# AIOS V2 Architecture and Delivery Index
|
|
||||||
|
|
||||||
- Version line: `V2R1C0-candidate`
|
|
||||||
- Candidate branch: `integration/aios-v2r1-cto-production-candidate-20260720`
|
|
||||||
- Architecture owner: AIOS V2 CTO
|
|
||||||
- Status: implementation-complete candidate; runtime and production verification pending
|
|
||||||
|
|
||||||
## 1. Purpose
|
|
||||||
|
|
||||||
This directory is the single source of truth for AIOS V2 requirements, target architecture, module boundaries, migration strategy, release gates and architectural decisions.
|
|
||||||
|
|
||||||
The design is based on the following non-negotiable principles:
|
|
||||||
|
|
||||||
1. Do not rewrite the existing platform.
|
|
||||||
2. Reuse OpenClaw, Hermes and Agent Router capabilities.
|
|
||||||
3. Decouple OpenClaw and Hermes from control-plane ownership.
|
|
||||||
4. AIOS is the only control plane.
|
|
||||||
5. All user-facing and operational capabilities are Web/API accessible.
|
|
||||||
6. Every module enforces multi-tenant isolation.
|
|
||||||
7. Every change flows through Git branches, commits, tests, review and controlled promotion.
|
|
||||||
|
|
||||||
## 2. Document map
|
|
||||||
|
|
||||||
| Document | Purpose |
|
|
||||||
|---|---|
|
|
||||||
| [01-USER-REQUIREMENTS.md](01-USER-REQUIREMENTS.md) | Product goals, user requirements, non-functional requirements and acceptance criteria |
|
|
||||||
| [02-TARGET-ARCHITECTURE.md](02-TARGET-ARCHITECTURE.md) | Target logical architecture, control boundaries, data flow, security and tenancy model |
|
|
||||||
| [03-MIGRATION-AND-DELIVERY.md](03-MIGRATION-AND-DELIVERY.md) | Incremental migration plan, reuse strategy, GitOps workflow and release gates |
|
|
||||||
| [04-MODULE-CONTRACTS.md](04-MODULE-CONTRACTS.md) | Tenant context, task lifecycle, OpenClaw, Hermes, audit and execution-evidence contracts |
|
|
||||||
| [05-TRACEABILITY-MATRIX.md](05-TRACEABILITY-MATRIX.md) | Requirement-to-design-to-code-to-test mapping |
|
|
||||||
| [adr/README.md](adr/README.md) | Architecture decision records |
|
|
||||||
|
|
||||||
## 3. Current implementation map
|
|
||||||
|
|
||||||
| Architecture capability | Candidate implementation |
|
|
||||||
|---|---|
|
|
||||||
| Tenant, agent, task, policy and audit control plane | `active-runtime/app/aios_v2/control_plane.py` |
|
|
||||||
| Durable control-plane state | `active-runtime/app/aios_v2/persistence.py` |
|
|
||||||
| Web/API gateway | `active-runtime/app/aios_v2/gateway.py` |
|
|
||||||
| Unified application entrypoint | `active-runtime/app/aios_v2/app.py` |
|
|
||||||
| Runtime dispatch and reconciliation | `active-runtime/app/aios_v2/orchestrator.py` |
|
|
||||||
| Agent execution evidence | `active-runtime/app/aios_v2/execution_tracking.py` |
|
|
||||||
| GitOps runtime visibility | `active-runtime/app/aios_v2/gitops_status.py` |
|
|
||||||
| Web Console | `active-runtime/app/aios_v2/console.py` |
|
|
||||||
| OpenClaw execution boundary | `active-runtime/app/aios_openclaw/` |
|
|
||||||
| Hermes runtime boundary | `active-runtime/app/aios_hermes/` |
|
|
||||||
| Candidate tests | `tests/aios_v2_candidate/` |
|
|
||||||
| Deployment, health and rollback | `deploy/aios-v2/` |
|
|
||||||
|
|
||||||
## 4. Status vocabulary
|
|
||||||
|
|
||||||
The project separates code availability from release readiness:
|
|
||||||
|
|
||||||
- **Designed**: documented boundary and acceptance criteria exist.
|
|
||||||
- **Implemented**: code is committed on the candidate branch.
|
|
||||||
- **Test-defined**: automated tests exist in Git.
|
|
||||||
- **Test-verified**: tests were executed for the exact commit and evidence was recorded.
|
|
||||||
- **Pre-production verified**: deployed in an isolated environment with health, restart and rollback evidence.
|
|
||||||
- **Production released**: owner-approved merge/tag/deployment completed.
|
|
||||||
|
|
||||||
A branch or file existing does not, by itself, mean a capability is production-ready.
|
|
||||||
|
|
||||||
## 5. Design summary
|
|
||||||
|
|
||||||
```text
|
|
||||||
Browser / Web Console / Public API
|
|
||||||
|
|
|
||||||
v
|
|
||||||
AIOS Control Plane
|
|
||||||
Tenant | Agent | Task | Policy | Audit
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Execution Orchestrator / Router
|
|
||||||
/ \
|
|
||||||
v v
|
|
||||||
OpenClaw Adapter Hermes Boundary
|
|
||||||
| |
|
|
||||||
v v
|
|
||||||
OpenClaw Runtime Hermes Workers
|
|
||||||
\
|
|
||||||
v
|
|
||||||
Result Reconciliation + Evidence
|
|
||||||
```
|
|
||||||
|
|
||||||
The control plane owns authoritative identity, policy, approval, task lifecycle and audit state. OpenClaw and Hermes receive only tenant-scoped, approved execution work and return runtime results.
|
|
||||||
|
|
||||||
## 6. Release rule
|
|
||||||
|
|
||||||
No candidate may be promoted until:
|
|
||||||
|
|
||||||
- requirement-to-code traceability is complete;
|
|
||||||
- multi-tenant negative tests pass;
|
|
||||||
- controller workspace is detached and clean;
|
|
||||||
- local Git and Gitea branch OIDs match;
|
|
||||||
- persistence restart recovery passes;
|
|
||||||
- healthcheck and rollback pass in pre-production;
|
|
||||||
- no plaintext secret enters Git;
|
|
||||||
- owner approval is recorded for merge, tag and production deployment.
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# ADR-001: AIOS 作为唯一控制面
|
|
||||||
|
|
||||||
- Status: Accepted
|
|
||||||
- Date: 2026-07-20
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
现有系统包含多个执行和调度能力,若每个 Runtime 保存自己的任务、租户和策略状态,会形成多个事实源。
|
|
||||||
|
|
||||||
## Decision
|
|
||||||
|
|
||||||
AIOS Control Plane 是唯一权威控制面,负责:
|
|
||||||
|
|
||||||
- Tenant
|
|
||||||
- Agent
|
|
||||||
- Task
|
|
||||||
- Policy
|
|
||||||
- Approval
|
|
||||||
- Audit
|
|
||||||
- Execution Evidence
|
|
||||||
|
|
||||||
OpenClaw、Hermes、Agent Router 只能执行或路由,不拥有第二套控制状态。
|
|
||||||
|
|
||||||
## Consequences
|
|
||||||
|
|
||||||
Positive:
|
|
||||||
|
|
||||||
- 状态唯一;
|
|
||||||
- 审计完整;
|
|
||||||
- Web/API 统一。
|
|
||||||
|
|
||||||
Negative:
|
|
||||||
|
|
||||||
- Runtime 接入需要 Adapter。
|
|
||||||
- 迁移需要阶段性边界收敛。
|
|
||||||
|
|
||||||
## Compliance
|
|
||||||
|
|
||||||
任何新 Runtime 必须通过 AIOS Orchestrator 接入。
|
|
||||||
@@ -1,46 +0,0 @@
|
|||||||
# ADR-002: OpenClaw/Hermes 与控制面解耦
|
|
||||||
|
|
||||||
- Status: Accepted
|
|
||||||
- Date: 2026-07-20
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
OpenClaw 和 Hermes 已具备执行能力,但历史上可能承担部分控制职责。
|
|
||||||
|
|
||||||
## Decision
|
|
||||||
|
|
||||||
引入边界层:
|
|
||||||
|
|
||||||
```text
|
|
||||||
AIOS Control Plane
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Adapter / Boundary
|
|
||||||
|
|
|
||||||
v
|
|
||||||
Runtime
|
|
||||||
```
|
|
||||||
|
|
||||||
OpenClaw:
|
|
||||||
|
|
||||||
- 执行签名命令;
|
|
||||||
- 返回结果;
|
|
||||||
- 提供健康状态。
|
|
||||||
|
|
||||||
Hermes:
|
|
||||||
|
|
||||||
- 队列;
|
|
||||||
- Worker;
|
|
||||||
- Lease;
|
|
||||||
- Heartbeat;
|
|
||||||
- 执行结果。
|
|
||||||
|
|
||||||
## Consequences
|
|
||||||
|
|
||||||
- Runtime 可独立演进。
|
|
||||||
- 控制逻辑不会分裂。
|
|
||||||
- 需要维护稳定 Contract。
|
|
||||||
|
|
||||||
## Migration
|
|
||||||
|
|
||||||
先接 Adapter,再迁移旧调用入口,最后关闭控制旁路。
|
|
||||||
@@ -1,44 +0,0 @@
|
|||||||
# ADR-003: 所有变更必须通过 GitOps
|
|
||||||
|
|
||||||
- Status: Accepted
|
|
||||||
- Date: 2026-07-20
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
直接在生产目录、长期 worktree 或主分支上修改代码,会造成不可追踪、不可回滚和分支冲突。
|
|
||||||
|
|
||||||
## Decision
|
|
||||||
|
|
||||||
所有代码、配置、文档、测试、部署和回滚资产必须通过以下流程:
|
|
||||||
|
|
||||||
```text
|
|
||||||
non-main branch -> commit -> local sync -> Gitea sync -> test -> review -> controlled promotion
|
|
||||||
```
|
|
||||||
|
|
||||||
Controller worktree 必须长期保持 detached 且 clean。每个写任务使用独立工作区或受控写入流程;相同分支写入必须串行。
|
|
||||||
|
|
||||||
## Required evidence
|
|
||||||
|
|
||||||
每个任务必须记录:
|
|
||||||
|
|
||||||
- branch
|
|
||||||
- commit
|
|
||||||
- changed files
|
|
||||||
- test command and result
|
|
||||||
- blocker and resolution
|
|
||||||
- local/Gitea sync status
|
|
||||||
- rollback note
|
|
||||||
|
|
||||||
## Prohibited
|
|
||||||
|
|
||||||
- 直接 push `main`
|
|
||||||
- 未审批创建生产 Tag
|
|
||||||
- 在生产目录手工修改后不回写 Git
|
|
||||||
- fetch 到已被 worktree 检出的本地 branch ref
|
|
||||||
- 在 Git 中提交明文密钥
|
|
||||||
|
|
||||||
## Consequences
|
|
||||||
|
|
||||||
优点:可审计、可回滚、可并行治理。
|
|
||||||
|
|
||||||
成本:紧急修复也必须形成 patch branch 和证据。
|
|
||||||
@@ -1,43 +0,0 @@
|
|||||||
# ADR-005: 文档与代码同步门禁
|
|
||||||
|
|
||||||
- Status: Accepted
|
|
||||||
- Date: 2026-07-20
|
|
||||||
|
|
||||||
## Context
|
|
||||||
|
|
||||||
架构、功能和部署变化如果不同步文档,会导致版本无法审计,后续 Agent 无法判断真实设计。
|
|
||||||
|
|
||||||
## Decision
|
|
||||||
|
|
||||||
所有影响以下内容的变更必须同步 `docs/aios-v2/`:
|
|
||||||
|
|
||||||
- 用户行为;
|
|
||||||
- API;
|
|
||||||
- 数据模型;
|
|
||||||
- 状态机;
|
|
||||||
- Runtime Boundary;
|
|
||||||
- 安全策略;
|
|
||||||
- 部署流程;
|
|
||||||
- 测试门禁。
|
|
||||||
|
|
||||||
## Gate
|
|
||||||
|
|
||||||
验收必须检查:
|
|
||||||
|
|
||||||
1. 文档引用代码路径存在;
|
|
||||||
2. API 文档有实现映射;
|
|
||||||
3. Contract 字段有模型映射;
|
|
||||||
4. Traceability Matrix 已更新;
|
|
||||||
5. ADR 与架构变化一致。
|
|
||||||
|
|
||||||
## Consequences
|
|
||||||
|
|
||||||
优点:
|
|
||||||
|
|
||||||
- 架构长期可维护;
|
|
||||||
- Agent 执行有明确上下文;
|
|
||||||
- Bug 修复有历史依据。
|
|
||||||
|
|
||||||
成本:
|
|
||||||
|
|
||||||
- 小功能也需要更新对应文档。
|
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
# AIOS V2 Architecture Decision Records
|
|
||||||
|
|
||||||
本目录记录影响多个模块、长期演进、兼容性、安全或交付流程的关键架构决策。
|
|
||||||
|
|
||||||
## ADR 列表
|
|
||||||
|
|
||||||
| ADR | 决策 |
|
|
||||||
|---|---|
|
|
||||||
| [ADR-001](ADR-001-single-control-plane.md) | AIOS 作为唯一控制面 |
|
|
||||||
| [ADR-002](ADR-002-openclaw-hermes-decoupling.md) | OpenClaw/Hermes 与控制面解耦 |
|
|
||||||
| [ADR-003](ADR-003-gitops-only-change-flow.md) | 所有变更必须通过 GitOps |
|
|
||||||
| [ADR-004](ADR-004-multi-tenant-isolation.md) | 多租户隔离默认拒绝 |
|
|
||||||
| [ADR-005](ADR-005-document-code-sync-gate.md) | 文档与代码同步门禁 |
|
|
||||||
| [ADR-006](ADR-006-candidate-persistence-strategy.md) | 候选持久化采用可替换 SQLite Snapshot |
|
|
||||||
|
|
||||||
## ADR 状态
|
|
||||||
|
|
||||||
- `Proposed`:提出但未采纳。
|
|
||||||
- `Accepted`:当前有效决策。
|
|
||||||
- `Superseded`:被新 ADR 替代。
|
|
||||||
- `Deprecated`:保留历史但禁止新实现采用。
|
|
||||||
|
|
||||||
## 何时必须新增或更新 ADR
|
|
||||||
|
|
||||||
以下变化必须新增或更新 ADR:
|
|
||||||
|
|
||||||
1. 控制面与 Runtime 权限边界变化;
|
|
||||||
2. 多租户隔离模型变化;
|
|
||||||
3. API、数据模型或状态机出现不兼容变化;
|
|
||||||
4. 持久化、消息、密钥、审计或部署方案发生跨模块变化;
|
|
||||||
5. GitOps、发布、回滚或文档门禁变化;
|
|
||||||
6. 替换 OpenClaw、Hermes、Agent Router 或核心基础设施。
|
|
||||||
|
|
||||||
ADR 变化必须同步更新:
|
|
||||||
|
|
||||||
- `02-TARGET-ARCHITECTURE.md`
|
|
||||||
- `04-MODULE-CONTRACTS.md`
|
|
||||||
- `05-TRACEABILITY-MATRIX.md`
|
|
||||||
- 对应代码、测试和部署资产
|
|
||||||
@@ -1,47 +0,0 @@
|
|||||||
# TASK-AIOS-CONTROL-PLANE-GATEWAY-V1
|
|
||||||
|
|
||||||
## Owner
|
|
||||||
Codex
|
|
||||||
|
|
||||||
## Goal
|
|
||||||
建立 AIOS V2 唯一控制面 Gateway v1。
|
|
||||||
|
|
||||||
## Architecture Boundary
|
|
||||||
|
|
||||||
AIOS Control Plane:
|
|
||||||
- Tenant Registry
|
|
||||||
- Agent Registry
|
|
||||||
- Task Lifecycle
|
|
||||||
- Policy Gate
|
|
||||||
- Approval
|
|
||||||
- Audit Event
|
|
||||||
|
|
||||||
不允许:
|
|
||||||
- OpenClaw 直接管理任务生命周期
|
|
||||||
- Hermes 直接暴露控制 API
|
|
||||||
- Web UI 绕过 AIOS API
|
|
||||||
|
|
||||||
## Deliverables
|
|
||||||
|
|
||||||
1. Control Plane API skeleton
|
|
||||||
2. Tenant context middleware
|
|
||||||
3. Task state machine
|
|
||||||
4. Agent registration contract
|
|
||||||
5. Audit event interface
|
|
||||||
|
|
||||||
## Acceptance
|
|
||||||
|
|
||||||
- 所有 API 携带 tenant_id
|
|
||||||
- 未授权跨租户访问拒绝
|
|
||||||
- Task create/update/cancel 可审计
|
|
||||||
- GitOps commit 可追踪
|
|
||||||
- pytest + integration test 通过
|
|
||||||
|
|
||||||
## Migration
|
|
||||||
|
|
||||||
复用:
|
|
||||||
- Agent Router
|
|
||||||
- OpenClaw adapter
|
|
||||||
- Hermes runtime
|
|
||||||
|
|
||||||
不重写执行层。
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
{
|
|
||||||
"type": "branch",
|
|
||||||
"branch": "integration/aios-v2r1-cto-production-candidate-20260720",
|
|
||||||
"base": "task/codex/AIOS-V2-CONTROL-PLANE-GATEWAY-V1-20260720",
|
|
||||||
"created_by": "chatgpt-gitops-bot",
|
|
||||||
"created_at": "2026-07-20T01:47:01Z",
|
|
||||||
"direct_main_push": false,
|
|
||||||
"owner_approval_required": true
|
|
||||||
}
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
{
|
|
||||||
"type": "branch",
|
|
||||||
"branch": "task/codex/AIOS-V2-CONTROL-PLANE-GATEWAY-V1-20260720",
|
|
||||||
"base": "main",
|
|
||||||
"created_by": "chatgpt-gitops-bot",
|
|
||||||
"created_at": "2026-07-19T17:06:04Z",
|
|
||||||
"direct_main_push": false,
|
|
||||||
"owner_approval_required": true
|
|
||||||
}
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
{
|
|
||||||
"type": "branch",
|
|
||||||
"branch": "task/openclaw/AIOS-V2R1-PRODUCTION-DEPLOY-20260720",
|
|
||||||
"base": "integration/aios-v2r1-cto-production-candidate-20260720",
|
|
||||||
"created_by": "chatgpt-gitops-bot",
|
|
||||||
"created_at": "2026-07-20T03:49:09Z",
|
|
||||||
"direct_main_push": false,
|
|
||||||
"owner_approval_required": true
|
|
||||||
}
|
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$id": "https://aios.local/schemas/codex-command.schema.json",
|
||||||
|
"title": "AIOS Codex command",
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["schema_version", "command_id", "project_key", "work_unit_id", "contract_commit", "state_version", "execution"],
|
||||||
|
"properties": {
|
||||||
|
"schema_version": {"const": 1},
|
||||||
|
"command_id": {"type": "string", "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$"},
|
||||||
|
"project_key": {"type": "string", "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$"},
|
||||||
|
"work_unit_id": {"type": "string", "pattern": "^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$"},
|
||||||
|
"contract_commit": {"type": "string", "pattern": "^[0-9a-f]{40}$"},
|
||||||
|
"state_version": {"type": "integer", "minimum": 0},
|
||||||
|
"execution": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["kind", "input"],
|
||||||
|
"properties": {
|
||||||
|
"kind": {"enum": ["acceptance_probe", "codex"]},
|
||||||
|
"input": {"type": "string", "maxLength": 65536}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,145 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"title": "AIOS Orchestrator Decision",
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"schema_version",
|
||||||
|
"project_key",
|
||||||
|
"contract_commit",
|
||||||
|
"event_id",
|
||||||
|
"expected_state_version",
|
||||||
|
"verdict",
|
||||||
|
"state_updates",
|
||||||
|
"next_work_units",
|
||||||
|
"requires_human_input",
|
||||||
|
"stop_reason"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"schema_version": {
|
||||||
|
"type": "integer",
|
||||||
|
"const": 1
|
||||||
|
},
|
||||||
|
"project_key": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"contract_commit": {
|
||||||
|
"type": "string",
|
||||||
|
"pattern": "^[0-9a-f]{40}$"
|
||||||
|
},
|
||||||
|
"event_id": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"expected_state_version": {
|
||||||
|
"type": "integer",
|
||||||
|
"minimum": 0
|
||||||
|
},
|
||||||
|
"verdict": {
|
||||||
|
"type": "string",
|
||||||
|
"enum": [
|
||||||
|
"BLOCKED",
|
||||||
|
"COMPLETE",
|
||||||
|
"DISPATCH",
|
||||||
|
"NOOP"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"state_updates": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"key",
|
||||||
|
"value"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"key": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"value": {
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"next_work_units": {
|
||||||
|
"type": "array",
|
||||||
|
"maxItems": 32,
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"id",
|
||||||
|
"track_id",
|
||||||
|
"gate_id",
|
||||||
|
"primary_agent",
|
||||||
|
"fallback_agent",
|
||||||
|
"clean_base_commit",
|
||||||
|
"allowed_paths",
|
||||||
|
"required_checks",
|
||||||
|
"forbidden_operations",
|
||||||
|
"instruction"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"id": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"track_id": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"gate_id": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"primary_agent": {
|
||||||
|
"type": "string",
|
||||||
|
"enum": [
|
||||||
|
"openclaw",
|
||||||
|
"codex"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"fallback_agent": {
|
||||||
|
"type": "string",
|
||||||
|
"enum": [
|
||||||
|
"openclaw",
|
||||||
|
"codex"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"clean_base_commit": {
|
||||||
|
"type": "string",
|
||||||
|
"pattern": "^[0-9a-f]{40}$"
|
||||||
|
},
|
||||||
|
"allowed_paths": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"required_checks": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"forbidden_operations": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"instruction": {
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"requires_human_input": {
|
||||||
|
"type": "boolean"
|
||||||
|
},
|
||||||
|
"stop_reason": {
|
||||||
|
"type": [
|
||||||
|
"string",
|
||||||
|
"null"
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"title": "AIOS Orchestrator Event",
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": [
|
||||||
|
"schema_version",
|
||||||
|
"project_key",
|
||||||
|
"contract_commit",
|
||||||
|
"event_id",
|
||||||
|
"source_agent",
|
||||||
|
"event_type",
|
||||||
|
"work_unit_id",
|
||||||
|
"status",
|
||||||
|
"state_version",
|
||||||
|
"implementation_commit",
|
||||||
|
"result_commit",
|
||||||
|
"blocker",
|
||||||
|
"evidence_refs",
|
||||||
|
"payload"
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"schema_version": {
|
||||||
|
"const": 1
|
||||||
|
},
|
||||||
|
"project_key": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1,
|
||||||
|
"maxLength": 128
|
||||||
|
},
|
||||||
|
"contract_commit": {
|
||||||
|
"type": "string",
|
||||||
|
"pattern": "^[0-9a-f]{40}$"
|
||||||
|
},
|
||||||
|
"event_id": {
|
||||||
|
"type": "string",
|
||||||
|
"minLength": 1,
|
||||||
|
"maxLength": 128
|
||||||
|
},
|
||||||
|
"source_agent": {
|
||||||
|
"enum": [
|
||||||
|
"openclaw",
|
||||||
|
"codex",
|
||||||
|
"gitea",
|
||||||
|
"system",
|
||||||
|
"workspace-agent"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"event_type": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"work_unit_id": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"status": {
|
||||||
|
"enum": [
|
||||||
|
"READY",
|
||||||
|
"RUNNING",
|
||||||
|
"PASS",
|
||||||
|
"BLOCKED",
|
||||||
|
"FAIL",
|
||||||
|
"TIMEOUT",
|
||||||
|
"NOT_EXECUTED"
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"state_version": {
|
||||||
|
"type": "integer",
|
||||||
|
"minimum": 0
|
||||||
|
},
|
||||||
|
"implementation_commit": {
|
||||||
|
"type": [
|
||||||
|
"string",
|
||||||
|
"null"
|
||||||
|
],
|
||||||
|
"pattern": "^[0-9a-f]{40}$"
|
||||||
|
},
|
||||||
|
"result_commit": {
|
||||||
|
"type": [
|
||||||
|
"string",
|
||||||
|
"null"
|
||||||
|
],
|
||||||
|
"pattern": "^[0-9a-f]{40}$"
|
||||||
|
},
|
||||||
|
"blocker": {
|
||||||
|
"type": [
|
||||||
|
"string",
|
||||||
|
"null"
|
||||||
|
],
|
||||||
|
"maxLength": 512
|
||||||
|
},
|
||||||
|
"evidence_refs": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "string"
|
||||||
|
},
|
||||||
|
"maxItems": 64
|
||||||
|
},
|
||||||
|
"payload": {
|
||||||
|
"type": "object"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Executable
+63
@@ -0,0 +1,63 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Seed exactly one orchestration event when the project has no durable state."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(REPO_ROOT))
|
||||||
|
|
||||||
|
from services.aios_orchestrator_gateway.config import Config
|
||||||
|
from services.aios_orchestrator_gateway.models import Event
|
||||||
|
from services.aios_orchestrator_gateway.securefs import atomic_write_bytes_shared
|
||||||
|
from services.aios_orchestrator_gateway.store import Store
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("--config", type=Path, required=True)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
config = Config.from_file(args.config)
|
||||||
|
store = Store(config.database_path)
|
||||||
|
if store.project_state(config.project_key) is not None:
|
||||||
|
print("ORCHESTRATION_BOOTSTRAP=NOOP reason=STATE_EXISTS")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
event = Event.from_dict(
|
||||||
|
{
|
||||||
|
"schema_version": 1,
|
||||||
|
"project_key": config.project_key,
|
||||||
|
"contract_commit": config.contract_commit,
|
||||||
|
"event_id": "bootstrap-" + config.contract_commit[:24],
|
||||||
|
"source_agent": "system",
|
||||||
|
"event_type": "orchestration_bootstrap",
|
||||||
|
"work_unit_id": "ORCHESTRATION-BOOTSTRAP",
|
||||||
|
"status": "READY",
|
||||||
|
"state_version": 0,
|
||||||
|
"implementation_commit": None,
|
||||||
|
"result_commit": None,
|
||||||
|
"blocker": None,
|
||||||
|
"evidence_refs": [],
|
||||||
|
"payload": {
|
||||||
|
"reason": "initial_gateway_install",
|
||||||
|
"polling_enabled": True,
|
||||||
|
"openclaw_sync_required": True,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
path = config.event_inbox_dir / "pending" / f"{event.event_id}.json"
|
||||||
|
atomic_write_bytes_shared(
|
||||||
|
path,
|
||||||
|
(event.canonical_json() + "\n").encode(),
|
||||||
|
mode=0o640,
|
||||||
|
group_writable_directory=True,
|
||||||
|
)
|
||||||
|
print(f"ORCHESTRATION_BOOTSTRAP=PASS event_id={event.event_id}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
Executable
+64
@@ -0,0 +1,64 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
[[ ${EUID} -eq 0 ]] || { echo "root required" >&2; exit 1; }
|
||||||
|
ROOT=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd -P)
|
||||||
|
CONFIG=${1:-/etc/aios-codex-watcher/config.json}
|
||||||
|
[[ "$CONFIG" = /* ]] || { echo "config path must be absolute" >&2; exit 1; }
|
||||||
|
[[ -f "$CONFIG" && ! -L "$CONFIG" ]] || { echo "pre-provisioned regular config required" >&2; exit 1; }
|
||||||
|
|
||||||
|
for path in \
|
||||||
|
services/aios_codex_watcher/__init__.py \
|
||||||
|
services/aios_codex_watcher/__main__.py \
|
||||||
|
services/aios_codex_watcher/watcher.py \
|
||||||
|
systemd/aios-codex-watcher.service \
|
||||||
|
systemd/aios-codex-watcher.timer \
|
||||||
|
systemd/aios-codex-watcher.sysusers \
|
||||||
|
systemd/aios-codex-watcher.tmpfiles; do
|
||||||
|
[[ -f "$ROOT/$path" && ! -L "$ROOT/$path" ]] || { echo "missing or unsafe: $path" >&2; exit 1; }
|
||||||
|
done
|
||||||
|
|
||||||
|
digest=$(sha256sum \
|
||||||
|
"$ROOT/services/aios_codex_watcher/__init__.py" \
|
||||||
|
"$ROOT/services/aios_codex_watcher/__main__.py" \
|
||||||
|
"$ROOT/services/aios_codex_watcher/watcher.py" | sha256sum | awk '{print $1}')
|
||||||
|
release="/opt/aios-codex-watcher/releases/$digest"
|
||||||
|
install -d -o root -g root -m 0755 "$release/services/aios_codex_watcher" "$release/bin"
|
||||||
|
install -o root -g root -m 0644 "$ROOT/services/aios_codex_watcher/"*.py "$release/services/aios_codex_watcher/"
|
||||||
|
|
||||||
|
launcher="$release/bin/aios-codex-watcher"
|
||||||
|
printf '%s\n' '#!/usr/bin/env bash' 'set -euo pipefail' \
|
||||||
|
"export PYTHONPATH='$release'" \
|
||||||
|
"exec /usr/bin/python3 -m services.aios_codex_watcher \"\$@\"" >"$launcher"
|
||||||
|
chown root:root "$launcher"
|
||||||
|
chmod 0755 "$launcher"
|
||||||
|
|
||||||
|
install -d -o root -g root -m 0755 /opt/aios-codex-watcher /etc/aios-codex-watcher
|
||||||
|
ln -sfn "releases/$digest" /opt/aios-codex-watcher/current.new
|
||||||
|
mv -Tf /opt/aios-codex-watcher/current.new /opt/aios-codex-watcher/current
|
||||||
|
install -o root -g root -m 0644 "$ROOT/systemd/aios-codex-watcher.service" /etc/systemd/system/
|
||||||
|
install -o root -g root -m 0644 "$ROOT/systemd/aios-codex-watcher.timer" /etc/systemd/system/
|
||||||
|
install -o root -g root -m 0644 "$ROOT/systemd/aios-codex-watcher.sysusers" /usr/lib/sysusers.d/
|
||||||
|
install -o root -g root -m 0644 "$ROOT/systemd/aios-codex-watcher.tmpfiles" /usr/lib/tmpfiles.d/
|
||||||
|
systemd-sysusers /usr/lib/sysusers.d/aios-codex-watcher.sysusers
|
||||||
|
systemd-tmpfiles --create /usr/lib/tmpfiles.d/aios-codex-watcher.tmpfiles
|
||||||
|
commands=/data/aios-gitops/workspace/agents/commands
|
||||||
|
for directory in "$commands/active" "$commands/completed"; do
|
||||||
|
[[ -d "$directory" && ! -L "$directory" ]] || { echo "missing or unsafe command directory: $directory" >&2; exit 1; }
|
||||||
|
[[ $(stat -c '%U:%G' "$directory") == aios-orchestrator:aios-orchestrator ]] || { echo "unexpected command directory owner: $directory" >&2; exit 1; }
|
||||||
|
mode=$(stat -c '%a' "$directory")
|
||||||
|
(( (8#$mode & 2) == 0 )) || { echo "world-writable command directory: $directory" >&2; exit 1; }
|
||||||
|
done
|
||||||
|
if [[ -e "$commands/dead-letter" || -L "$commands/dead-letter" ]]; then
|
||||||
|
[[ -d "$commands/dead-letter" && ! -L "$commands/dead-letter" ]] || { echo "unsafe dead-letter directory" >&2; exit 1; }
|
||||||
|
else
|
||||||
|
install -d -o aios-orchestrator -g aios-orchestrator -m 0750 "$commands/dead-letter"
|
||||||
|
fi
|
||||||
|
[[ $(stat -c '%U:%G' "$commands/dead-letter") == aios-orchestrator:aios-orchestrator ]] || { echo "unexpected dead-letter owner" >&2; exit 1; }
|
||||||
|
mode=$(stat -c '%a' "$commands/dead-letter")
|
||||||
|
(( (8#$mode & 2) == 0 )) || { echo "world-writable dead-letter directory" >&2; exit 1; }
|
||||||
|
printf '{"release_sha256":"%s","installed_at":"%s"}\n' "$digest" "$(date -u +%FT%TZ)" >"$release/receipt.json"
|
||||||
|
chmod 0644 "$release/receipt.json"
|
||||||
|
systemctl daemon-reload
|
||||||
|
systemctl enable --now aios-codex-watcher.timer
|
||||||
|
printf 'installed_sha256=%s\n' "$digest"
|
||||||
Executable
+239
@@ -0,0 +1,239 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
umask 0077
|
||||||
|
|
||||||
|
SOURCE_ROOT="${SOURCE_ROOT:-$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)}"
|
||||||
|
INSTALL_ROOT="${INSTALL_ROOT:-/opt/aios-orchestrator-gateway}"
|
||||||
|
CONFIG_ROOT="${CONFIG_ROOT:-/etc/aios-orchestrator-gateway}"
|
||||||
|
STATE_ROOT="${STATE_ROOT:-/var/lib/aios-orchestrator-gateway}"
|
||||||
|
UNIT_ROOT="${UNIT_ROOT:-/etc/systemd/system}"
|
||||||
|
SYSUSERS_ROOT="${SYSUSERS_ROOT:-/etc/sysusers.d}"
|
||||||
|
TMPFILES_ROOT="${TMPFILES_ROOT:-/etc/tmpfiles.d}"
|
||||||
|
DRY_RUN=0
|
||||||
|
|
||||||
|
if [[ "${1:-}" == "--dry-run" ]]; then
|
||||||
|
DRY_RUN=1
|
||||||
|
fi
|
||||||
|
|
||||||
|
fail() {
|
||||||
|
printf 'AIOS_ORCHESTRATOR_INSTALL=BLOCKED reason=%s\n' "$1" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
[[ "$(id -u)" -eq 0 ]] || fail OWNER_INSTALL_REQUIRED
|
||||||
|
|
||||||
|
for cmd in git python3 systemd-analyze systemctl systemd-sysusers systemd-tmpfiles sha256sum find sort xargs; do
|
||||||
|
command -v "$cmd" >/dev/null 2>&1 || fail "MISSING_COMMAND_${cmd}"
|
||||||
|
done
|
||||||
|
|
||||||
|
if [[ -f "$SOURCE_ROOT/SHA256SUMS" ]]; then
|
||||||
|
(cd "$SOURCE_ROOT" && sha256sum --check SHA256SUMS)
|
||||||
|
fi
|
||||||
|
|
||||||
|
for path in \
|
||||||
|
"$SOURCE_ROOT/services/aios_orchestrator_gateway" \
|
||||||
|
"$SOURCE_ROOT/systemd/aios-orchestrator-gateway.service" \
|
||||||
|
"$SOURCE_ROOT/systemd/aios-orchestrator-gateway-poll.service" \
|
||||||
|
"$SOURCE_ROOT/systemd/aios-orchestrator-gateway-poll.timer" \
|
||||||
|
"$SOURCE_ROOT/systemd/aios-orchestrator-gateway.sysusers" \
|
||||||
|
"$SOURCE_ROOT/systemd/aios-orchestrator-gateway.tmpfiles" \
|
||||||
|
"$SOURCE_ROOT/config/aios-orchestrator-gateway.example.json"; do
|
||||||
|
[[ -e "$path" ]] || fail "MISSING_SOURCE_${path}"
|
||||||
|
done
|
||||||
|
|
||||||
|
PYCACHE="/tmp/aios-orchestrator-install-pycache-$$"
|
||||||
|
rm -rf "$PYCACHE"
|
||||||
|
PYTHONPYCACHEPREFIX="$PYCACHE" python3 -m compileall -q "$SOURCE_ROOT/services" "$SOURCE_ROOT/scripts"
|
||||||
|
PYTHONDONTWRITEBYTECODE=1 PYTHONPATH="$SOURCE_ROOT" \
|
||||||
|
python3 -m unittest -v tests.test_aios_orchestrator_gateway
|
||||||
|
rm -rf "$PYCACHE"
|
||||||
|
for script in "$SOURCE_ROOT"/scripts/*.sh; do bash -n "$script"; done
|
||||||
|
python3 -m json.tool "$SOURCE_ROOT/config/aios-orchestrator-gateway.example.json" >/dev/null
|
||||||
|
systemd-analyze verify \
|
||||||
|
"$SOURCE_ROOT/systemd/aios-orchestrator-gateway.service" \
|
||||||
|
"$SOURCE_ROOT/systemd/aios-orchestrator-gateway-poll.service" \
|
||||||
|
"$SOURCE_ROOT/systemd/aios-orchestrator-gateway-poll.timer"
|
||||||
|
|
||||||
|
SOURCE_FILES=(
|
||||||
|
README-AIOS-ORCHESTRATOR-GATEWAY.md
|
||||||
|
config/aios-orchestrator-gateway.example.json
|
||||||
|
schemas/decision.schema.json
|
||||||
|
schemas/event.schema.json
|
||||||
|
scripts/bootstrap_orchestration_loop.py
|
||||||
|
scripts/install_aios_orchestrator_gateway.sh
|
||||||
|
scripts/report_orchestrator_event.py
|
||||||
|
scripts/rollback_aios_orchestrator_gateway.sh
|
||||||
|
scripts/verify_aios_orchestrator_gateway.sh
|
||||||
|
systemd/aios-orchestrator-gateway-poll.service
|
||||||
|
systemd/aios-orchestrator-gateway-poll.timer
|
||||||
|
systemd/aios-orchestrator-gateway.service
|
||||||
|
systemd/aios-orchestrator-gateway.sysusers
|
||||||
|
systemd/aios-orchestrator-gateway.tmpfiles
|
||||||
|
tests/test_aios_orchestrator_gateway.py
|
||||||
|
tests/test_aios_orchestrator_gateway_rollback.sh
|
||||||
|
workspace-agent/CENTRAL_AGENT_INSTRUCTIONS.md
|
||||||
|
workspace-agent/orchestrator-decision-action.openapi.yaml
|
||||||
|
)
|
||||||
|
while IFS= read -r path; do SOURCE_FILES+=("$path"); done < <(
|
||||||
|
cd "$SOURCE_ROOT"
|
||||||
|
find services/aios_orchestrator_gateway -type f ! -path '*/__pycache__/*' ! -name '*.pyc' -print | sort
|
||||||
|
)
|
||||||
|
for path in "${SOURCE_FILES[@]}"; do
|
||||||
|
[[ -f "$SOURCE_ROOT/$path" && ! -L "$SOURCE_ROOT/$path" ]] || fail "INVALID_SOURCE_FILE_${path}"
|
||||||
|
done
|
||||||
|
|
||||||
|
SOURCE_DIGEST="$(
|
||||||
|
cd "$SOURCE_ROOT"
|
||||||
|
printf '%s\0' "${SOURCE_FILES[@]}" | sort -z | xargs -0 sha256sum \
|
||||||
|
| sha256sum \
|
||||||
|
| awk '{print $1}'
|
||||||
|
)"
|
||||||
|
[[ "$SOURCE_DIGEST" =~ ^[0-9a-f]{64}$ ]] || fail SOURCE_DIGEST_INVALID
|
||||||
|
RELEASE_DIR="$INSTALL_ROOT/releases/$SOURCE_DIGEST"
|
||||||
|
|
||||||
|
if [[ "$DRY_RUN" -eq 1 ]]; then
|
||||||
|
printf 'AIOS_ORCHESTRATOR_INSTALL_DRY_RUN=PASS digest=%s release=%s\n' \
|
||||||
|
"$SOURCE_DIGEST" "$RELEASE_DIR"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
install -d -m 0755 "$INSTALL_ROOT/releases"
|
||||||
|
install -d -m 0700 "$CONFIG_ROOT"
|
||||||
|
install -d -m 0700 "$STATE_ROOT"
|
||||||
|
|
||||||
|
systemd-sysusers "$SOURCE_ROOT/systemd/aios-orchestrator-gateway.sysusers"
|
||||||
|
systemd-tmpfiles --create "$SOURCE_ROOT/systemd/aios-orchestrator-gateway.tmpfiles"
|
||||||
|
|
||||||
|
if [[ ! -d "$RELEASE_DIR" ]]; then
|
||||||
|
TMP_RELEASE="$INSTALL_ROOT/releases/.tmp-$SOURCE_DIGEST-$$"
|
||||||
|
install -d -m 0755 "$TMP_RELEASE"
|
||||||
|
for path in "${SOURCE_FILES[@]}"; do
|
||||||
|
install -d -m 0755 "$TMP_RELEASE/$(dirname "$path")"
|
||||||
|
cp -a "$SOURCE_ROOT/$path" "$TMP_RELEASE/$path"
|
||||||
|
done
|
||||||
|
printf '%s\n' "$SOURCE_DIGEST" > "$TMP_RELEASE/SOURCE_DIGEST"
|
||||||
|
chmod -R go-w "$TMP_RELEASE"
|
||||||
|
mv "$TMP_RELEASE" "$RELEASE_DIR"
|
||||||
|
fi
|
||||||
|
|
||||||
|
PREVIOUS_TARGET=""
|
||||||
|
if [[ -L "$INSTALL_ROOT/current" ]]; then
|
||||||
|
PREVIOUS_TARGET="$(readlink -f "$INSTALL_ROOT/current")"
|
||||||
|
fi
|
||||||
|
|
||||||
|
ln -s "$RELEASE_DIR" "$INSTALL_ROOT/.current-new-$$"
|
||||||
|
mv -Tf "$INSTALL_ROOT/.current-new-$$" "$INSTALL_ROOT/current"
|
||||||
|
|
||||||
|
install -m 0644 "$SOURCE_ROOT/systemd/aios-orchestrator-gateway.service" \
|
||||||
|
"$UNIT_ROOT/aios-orchestrator-gateway.service"
|
||||||
|
install -m 0644 "$SOURCE_ROOT/systemd/aios-orchestrator-gateway-poll.service" \
|
||||||
|
"$UNIT_ROOT/aios-orchestrator-gateway-poll.service"
|
||||||
|
install -m 0644 "$SOURCE_ROOT/systemd/aios-orchestrator-gateway-poll.timer" \
|
||||||
|
"$UNIT_ROOT/aios-orchestrator-gateway-poll.timer"
|
||||||
|
install -m 0644 "$SOURCE_ROOT/systemd/aios-orchestrator-gateway.sysusers" \
|
||||||
|
"$SYSUSERS_ROOT/aios-orchestrator-gateway.conf"
|
||||||
|
install -m 0644 "$SOURCE_ROOT/systemd/aios-orchestrator-gateway.tmpfiles" \
|
||||||
|
"$TMPFILES_ROOT/aios-orchestrator-gateway.conf"
|
||||||
|
|
||||||
|
if [[ ! -f "$CONFIG_ROOT/inbound.token" ]]; then
|
||||||
|
python3 - <<'PY' > "$CONFIG_ROOT/inbound.token"
|
||||||
|
import secrets
|
||||||
|
print(secrets.token_urlsafe(48))
|
||||||
|
PY
|
||||||
|
fi
|
||||||
|
[[ ! -L "$CONFIG_ROOT/inbound.token" ]] || fail INBOUND_TOKEN_SYMLINK
|
||||||
|
chown root:aios-orchestrator "$CONFIG_ROOT/inbound.token"
|
||||||
|
chmod 0640 "$CONFIG_ROOT/inbound.token"
|
||||||
|
|
||||||
|
if [[ ! -f "$CONFIG_ROOT/callback.token" ]]; then
|
||||||
|
python3 - <<'PY' > "$CONFIG_ROOT/callback.token"
|
||||||
|
import secrets
|
||||||
|
print(secrets.token_urlsafe(48))
|
||||||
|
PY
|
||||||
|
fi
|
||||||
|
[[ ! -L "$CONFIG_ROOT/callback.token" ]] || fail CALLBACK_TOKEN_SYMLINK
|
||||||
|
chown root:aios-orchestrator "$CONFIG_ROOT/callback.token"
|
||||||
|
chmod 0640 "$CONFIG_ROOT/callback.token"
|
||||||
|
|
||||||
|
if [[ ! -f "$CONFIG_ROOT/config.json" ]]; then
|
||||||
|
install -m 0600 \
|
||||||
|
"$SOURCE_ROOT/config/aios-orchestrator-gateway.example.json" \
|
||||||
|
"$CONFIG_ROOT/config.json.example"
|
||||||
|
printf 'AIOS_ORCHESTRATOR_INSTALL=BLOCKED reason=CONFIGURATION_REQUIRED digest=%s\n' \
|
||||||
|
"$SOURCE_DIGEST"
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
chown root:aios-orchestrator "$CONFIG_ROOT/config.json"
|
||||||
|
chmod 0640 "$CONFIG_ROOT/config.json"
|
||||||
|
|
||||||
|
python3 -m json.tool "$CONFIG_ROOT/config.json" >/dev/null
|
||||||
|
AIOS_ORCHESTRATOR_CONFIG_OWNER_UID=0 \
|
||||||
|
AIOS_ORCHESTRATOR_SECRET_OWNER_UID=0 \
|
||||||
|
PYTHONPATH="$INSTALL_ROOT/current" \
|
||||||
|
python3 - "$CONFIG_ROOT/config.json" <<'PY'
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from services.aios_orchestrator_gateway.config import Config
|
||||||
|
cfg = Config.from_file(Path(sys.argv[1]))
|
||||||
|
cfg.inbound_token()
|
||||||
|
cfg.callback_token()
|
||||||
|
if cfg.workspace_trigger_id:
|
||||||
|
cfg.workspace_token()
|
||||||
|
if cfg.responses_api_key_file:
|
||||||
|
cfg.responses_api_key()
|
||||||
|
if cfg.openclaw_sync_endpoint:
|
||||||
|
cfg.openclaw_sync_token()
|
||||||
|
print("CONFIG_VALIDATE=PASS")
|
||||||
|
PY
|
||||||
|
|
||||||
|
python3 - "$STATE_ROOT/install-receipt.json" "$SOURCE_DIGEST" "$RELEASE_DIR" "$PREVIOUS_TARGET" <<'PY'
|
||||||
|
import json, os, sys, tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
path = Path(sys.argv[1])
|
||||||
|
data = {
|
||||||
|
"schema_version": 1,
|
||||||
|
"source_digest": sys.argv[2],
|
||||||
|
"current_release": sys.argv[3],
|
||||||
|
"previous_release": sys.argv[4] or None,
|
||||||
|
"last_rollback": None,
|
||||||
|
}
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
fd, temp = tempfile.mkstemp(prefix=".install-receipt-", dir=path.parent)
|
||||||
|
try:
|
||||||
|
with os.fdopen(fd, "w") as handle:
|
||||||
|
json.dump(data, handle, indent=2, sort_keys=True)
|
||||||
|
handle.write("\n")
|
||||||
|
handle.flush()
|
||||||
|
os.fsync(handle.fileno())
|
||||||
|
os.chmod(temp, 0o600)
|
||||||
|
os.replace(temp, path)
|
||||||
|
directory = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY)
|
||||||
|
try:
|
||||||
|
os.fsync(directory)
|
||||||
|
finally:
|
||||||
|
os.close(directory)
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
os.unlink(temp)
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
PY
|
||||||
|
chown aios-orchestrator:aios-orchestrator "$STATE_ROOT/install-receipt.json"
|
||||||
|
chmod 0600 "$STATE_ROOT/install-receipt.json"
|
||||||
|
|
||||||
|
systemctl daemon-reload
|
||||||
|
systemctl enable --now aios-orchestrator-gateway.service
|
||||||
|
systemctl enable --now aios-orchestrator-gateway-poll.timer
|
||||||
|
systemctl is-active --quiet aios-orchestrator-gateway.service
|
||||||
|
systemctl is-active --quiet aios-orchestrator-gateway-poll.timer
|
||||||
|
|
||||||
|
AIOS_ORCHESTRATOR_CONFIG_OWNER_UID=0 \
|
||||||
|
AIOS_ORCHESTRATOR_SECRET_OWNER_UID=0 \
|
||||||
|
PYTHONPATH="$INSTALL_ROOT/current" \
|
||||||
|
python3 "$INSTALL_ROOT/current/scripts/bootstrap_orchestration_loop.py" \
|
||||||
|
--config "$CONFIG_ROOT/config.json"
|
||||||
|
systemctl start aios-orchestrator-gateway-poll.service
|
||||||
|
|
||||||
|
printf 'AIOS_ORCHESTRATOR_INSTALL=PASS digest=%s previous=%s\n' \
|
||||||
|
"$SOURCE_DIGEST" "${PREVIOUS_TARGET:-NONE}"
|
||||||
Executable
+130
@@ -0,0 +1,130 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Submit an OpenClaw/Codex task lifecycle event over HTTP or a polled inbox."""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
import sys
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(REPO_ROOT))
|
||||||
|
|
||||||
|
from services.aios_orchestrator_gateway.models import Event
|
||||||
|
from services.aios_orchestrator_gateway.securefs import atomic_write_bytes_shared, read_secure_json, read_secure_text
|
||||||
|
|
||||||
|
|
||||||
|
def parser() -> argparse.ArgumentParser:
|
||||||
|
p = argparse.ArgumentParser()
|
||||||
|
p.add_argument("--project-key", required=True)
|
||||||
|
p.add_argument("--contract-commit", required=True)
|
||||||
|
p.add_argument("--event-id", required=True)
|
||||||
|
p.add_argument("--source-agent", required=True, choices=["openclaw", "codex", "gitea", "system"])
|
||||||
|
p.add_argument("--event-type", required=True)
|
||||||
|
p.add_argument("--work-unit-id", required=True)
|
||||||
|
p.add_argument(
|
||||||
|
"--status",
|
||||||
|
required=True,
|
||||||
|
choices=["READY", "RUNNING", "PASS", "BLOCKED", "FAIL", "TIMEOUT", "NOT_EXECUTED"],
|
||||||
|
)
|
||||||
|
p.add_argument("--state-version", type=int, required=True)
|
||||||
|
p.add_argument("--implementation-commit")
|
||||||
|
p.add_argument("--result-commit")
|
||||||
|
p.add_argument("--blocker")
|
||||||
|
p.add_argument("--evidence-ref", action="append", default=[])
|
||||||
|
p.add_argument("--payload-file", type=Path)
|
||||||
|
p.add_argument("--endpoint")
|
||||||
|
p.add_argument("--token-file", type=Path)
|
||||||
|
p.add_argument("--inbox-dir", type=Path)
|
||||||
|
p.add_argument("--timeout-seconds", type=int, default=15)
|
||||||
|
return p
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
args = parser().parse_args()
|
||||||
|
if not args.endpoint and not args.inbox_dir:
|
||||||
|
raise SystemExit("either --endpoint or --inbox-dir is required")
|
||||||
|
if args.endpoint and not args.token_file:
|
||||||
|
raise SystemExit("--token-file is required with --endpoint")
|
||||||
|
payload = {}
|
||||||
|
if args.payload_file:
|
||||||
|
if not args.payload_file.is_absolute():
|
||||||
|
raise SystemExit("--payload-file must be absolute")
|
||||||
|
payload = read_secure_json(
|
||||||
|
args.payload_file, expected_uid=os.getuid(), max_bytes=262_144
|
||||||
|
)
|
||||||
|
if not isinstance(payload, dict):
|
||||||
|
raise SystemExit("payload root must be an object")
|
||||||
|
|
||||||
|
event = Event.from_dict(
|
||||||
|
{
|
||||||
|
"schema_version": 1,
|
||||||
|
"project_key": args.project_key,
|
||||||
|
"contract_commit": args.contract_commit,
|
||||||
|
"event_id": args.event_id,
|
||||||
|
"source_agent": args.source_agent,
|
||||||
|
"event_type": args.event_type,
|
||||||
|
"work_unit_id": args.work_unit_id,
|
||||||
|
"status": args.status,
|
||||||
|
"state_version": args.state_version,
|
||||||
|
"implementation_commit": args.implementation_commit,
|
||||||
|
"result_commit": args.result_commit,
|
||||||
|
"blocker": args.blocker,
|
||||||
|
"evidence_refs": args.evidence_ref,
|
||||||
|
"payload": payload,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
encoded = event.canonical_json().encode()
|
||||||
|
http_error: Exception | None = None
|
||||||
|
|
||||||
|
if args.endpoint:
|
||||||
|
token = read_secure_text(
|
||||||
|
args.token_file,
|
||||||
|
expected_uid=None,
|
||||||
|
max_bytes=64_000,
|
||||||
|
)
|
||||||
|
request = urllib.request.Request(
|
||||||
|
args.endpoint.rstrip("/") + "/v1/orchestrator/events",
|
||||||
|
data=encoded,
|
||||||
|
method="POST",
|
||||||
|
headers={
|
||||||
|
"Authorization": f"Bearer {token}",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
"Idempotency-Key": event.event_id,
|
||||||
|
"User-Agent": "aios-agent-event-reporter/0.2",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(request, timeout=args.timeout_seconds) as response:
|
||||||
|
response.read(65_536)
|
||||||
|
if response.status not in {200, 202}:
|
||||||
|
raise RuntimeError(f"unexpected HTTP {response.status}")
|
||||||
|
print(f"EVENT_SUBMIT=PASS event_id={event.event_id} channel=http")
|
||||||
|
return 0
|
||||||
|
except (urllib.error.URLError, urllib.error.HTTPError, TimeoutError, OSError) as exc:
|
||||||
|
http_error = exc
|
||||||
|
|
||||||
|
if args.inbox_dir:
|
||||||
|
pending = args.inbox_dir / "pending"
|
||||||
|
path = pending / f"{event.event_id}.json"
|
||||||
|
atomic_write_bytes_shared(
|
||||||
|
path,
|
||||||
|
(json.dumps(event.to_dict(), indent=2, sort_keys=True) + "\n").encode(),
|
||||||
|
mode=0o640,
|
||||||
|
group_writable_directory=True,
|
||||||
|
)
|
||||||
|
print(f"EVENT_SUBMIT=PASS event_id={event.event_id} channel=file_inbox")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
print(
|
||||||
|
f"EVENT_SUBMIT=FAIL event_id={event.event_id} reason={type(http_error).__name__}",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
+102
@@ -0,0 +1,102 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
umask 0077
|
||||||
|
|
||||||
|
INSTALL_ROOT="${INSTALL_ROOT:-/opt/aios-orchestrator-gateway}"
|
||||||
|
STATE_ROOT="${STATE_ROOT:-/var/lib/aios-orchestrator-gateway}"
|
||||||
|
RECEIPT="$STATE_ROOT/install-receipt.json"
|
||||||
|
|
||||||
|
fail() {
|
||||||
|
printf 'AIOS_ORCHESTRATOR_ROLLBACK=BLOCKED reason=%s\n' "$1" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
|
||||||
|
[[ "$(id -u)" -eq 0 ]] || fail OWNER_REQUIRED
|
||||||
|
[[ -f "$RECEIPT" ]] || fail RECEIPT_MISSING
|
||||||
|
[[ ! -L "$RECEIPT" ]] || fail RECEIPT_SYMLINK
|
||||||
|
[[ "$(stat -c '%u:%a' "$RECEIPT")" == "0:600" ]] || fail RECEIPT_PERMISSION
|
||||||
|
|
||||||
|
readarray -t VALUES < <(
|
||||||
|
python3 - "$RECEIPT" "$INSTALL_ROOT/releases" "$INSTALL_ROOT/current" <<'PY'
|
||||||
|
import json, os, stat, sys
|
||||||
|
from pathlib import Path
|
||||||
|
receipt = Path(sys.argv[1])
|
||||||
|
releases = Path(sys.argv[2]).resolve()
|
||||||
|
current_link = Path(sys.argv[3])
|
||||||
|
data = json.loads(receipt.read_text())
|
||||||
|
previous = data.get("previous_release")
|
||||||
|
current = data.get("current_release")
|
||||||
|
if not previous or not current:
|
||||||
|
raise SystemExit(2)
|
||||||
|
previous_path = Path(previous).resolve()
|
||||||
|
current_path = Path(current).resolve()
|
||||||
|
if previous_path.parent != releases or current_path.parent != releases:
|
||||||
|
raise SystemExit(2)
|
||||||
|
if not stat.S_ISDIR(os.lstat(previous_path).st_mode) or not stat.S_ISDIR(os.lstat(current_path).st_mode):
|
||||||
|
raise SystemExit(2)
|
||||||
|
for release in (previous_path, current_path):
|
||||||
|
if len(release.name) != 64 or any(ch not in "0123456789abcdef" for ch in release.name):
|
||||||
|
raise SystemExit(2)
|
||||||
|
digest_file = release / "SOURCE_DIGEST"
|
||||||
|
if digest_file.is_symlink() or not digest_file.is_file():
|
||||||
|
raise SystemExit(2)
|
||||||
|
if digest_file.read_text(encoding="utf-8").strip() != release.name:
|
||||||
|
raise SystemExit(2)
|
||||||
|
last = data.get("last_rollback")
|
||||||
|
if isinstance(last, dict) and last.get("to") == current and current_link.is_symlink():
|
||||||
|
if current_link.resolve() == current_path:
|
||||||
|
print("ALREADY_ROLLED_BACK")
|
||||||
|
raise SystemExit(0)
|
||||||
|
print(previous_path)
|
||||||
|
print(current_path)
|
||||||
|
PY
|
||||||
|
) || fail RECEIPT_INVALID
|
||||||
|
|
||||||
|
if [[ "${VALUES[0]:-}" == "ALREADY_ROLLED_BACK" ]]; then
|
||||||
|
echo "AIOS_ORCHESTRATOR_ROLLBACK=PASS result=NOOP_ALREADY_ROLLED_BACK"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
[[ "${#VALUES[@]}" -eq 2 ]] || fail RECEIPT_INVALID
|
||||||
|
PREVIOUS="${VALUES[0]}"
|
||||||
|
CURRENT="${VALUES[1]}"
|
||||||
|
systemctl stop aios-orchestrator-gateway-poll.timer || true
|
||||||
|
systemctl stop aios-orchestrator-gateway-poll.service || true
|
||||||
|
systemctl stop aios-orchestrator-gateway.service || true
|
||||||
|
ln -s "$PREVIOUS" "$INSTALL_ROOT/.current-rollback-$$"
|
||||||
|
mv -Tf "$INSTALL_ROOT/.current-rollback-$$" "$INSTALL_ROOT/current"
|
||||||
|
systemctl start aios-orchestrator-gateway.service
|
||||||
|
systemctl is-active --quiet aios-orchestrator-gateway.service
|
||||||
|
systemctl start aios-orchestrator-gateway-poll.timer
|
||||||
|
systemctl is-active --quiet aios-orchestrator-gateway-poll.timer
|
||||||
|
|
||||||
|
python3 - "$RECEIPT" "$PREVIOUS" "$CURRENT" <<'PY'
|
||||||
|
import json, os, sys, tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
path = Path(sys.argv[1])
|
||||||
|
data = json.loads(path.read_text())
|
||||||
|
data["current_release"] = sys.argv[2]
|
||||||
|
data["previous_release"] = sys.argv[3]
|
||||||
|
data["last_rollback"] = {"from": sys.argv[3], "to": sys.argv[2]}
|
||||||
|
fd, temp = tempfile.mkstemp(prefix=".rollback-receipt-", dir=path.parent)
|
||||||
|
try:
|
||||||
|
with os.fdopen(fd, "w") as handle:
|
||||||
|
json.dump(data, handle, indent=2, sort_keys=True)
|
||||||
|
handle.write("\n")
|
||||||
|
handle.flush()
|
||||||
|
os.fsync(handle.fileno())
|
||||||
|
os.chmod(temp, 0o600)
|
||||||
|
os.chown(temp, 0, 0)
|
||||||
|
os.replace(temp, path)
|
||||||
|
directory = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY)
|
||||||
|
try:
|
||||||
|
os.fsync(directory)
|
||||||
|
finally:
|
||||||
|
os.close(directory)
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
os.unlink(temp)
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
PY
|
||||||
|
|
||||||
|
printf 'AIOS_ORCHESTRATOR_ROLLBACK=PASS current=%s previous=%s\n' "$PREVIOUS" "$CURRENT"
|
||||||
Executable
+32
@@ -0,0 +1,32 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
|
||||||
|
CONFIG="${1:-/etc/aios-orchestrator-gateway/config.json}"
|
||||||
|
PORT="${PORT:-18920}"
|
||||||
|
|
||||||
|
python3 -m json.tool "$CONFIG" >/dev/null
|
||||||
|
systemd-analyze verify \
|
||||||
|
/etc/systemd/system/aios-orchestrator-gateway.service \
|
||||||
|
/etc/systemd/system/aios-orchestrator-gateway-poll.service \
|
||||||
|
/etc/systemd/system/aios-orchestrator-gateway-poll.timer
|
||||||
|
systemctl is-active --quiet aios-orchestrator-gateway.service
|
||||||
|
systemctl is-active --quiet aios-orchestrator-gateway-poll.timer
|
||||||
|
|
||||||
|
health="$(curl --fail --silent --show-error "http://127.0.0.1:${PORT}/healthz")"
|
||||||
|
python3 -c '
|
||||||
|
import json, sys
|
||||||
|
v=json.load(sys.stdin)
|
||||||
|
assert v["service"]=="aios-orchestrator-gateway"
|
||||||
|
assert v["status"]=="active"
|
||||||
|
assert v["polling"] is True
|
||||||
|
assert len(v["contract_commit"])==40
|
||||||
|
' <<<"$health"
|
||||||
|
|
||||||
|
systemctl start aios-orchestrator-gateway-poll.service
|
||||||
|
if journalctl -u aios-orchestrator-gateway.service -n 200 --no-pager \
|
||||||
|
| grep -Eqi 'Bearer |token=|api[_-]?key|secret='; then
|
||||||
|
echo "AIOS_ORCHESTRATOR_VERIFY=BLOCKED reason=SECRET_PATTERN_IN_LOG" >&2
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "AIOS_ORCHESTRATOR_VERIFY=PASS"
|
||||||
Executable
+652
@@ -0,0 +1,652 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Hardened WeChat outbound backend for AIOS.
|
||||||
|
|
||||||
|
Only the dedicated UCC adapter is a supported send backend. OpenClaw channel
|
||||||
|
discovery is retained for diagnostics, but generic OpenClaw chat or agent
|
||||||
|
delivery is never promoted to WeChat delivery.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import selectors
|
||||||
|
import signal
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
import urllib.error
|
||||||
|
import urllib.parse
|
||||||
|
import urllib.request
|
||||||
|
from typing import Any, Dict, List, Optional, Tuple
|
||||||
|
|
||||||
|
|
||||||
|
OPENCLAW_BIN = os.environ.get("AIOS_OPENCLAW_BIN", "openclaw")
|
||||||
|
WECHAT_ACCOUNT_ID = os.environ.get("AIOS_WECHAT_ACCOUNT_ID", "")
|
||||||
|
WECHAT_CHANNEL = os.environ.get("AIOS_WECHAT_CHANNEL", "")
|
||||||
|
SEND_BACKEND = os.environ.get("AIOS_WECHAT_SEND_BACKEND", "")
|
||||||
|
UCC_ADAPTER_URL = os.environ.get("AIOS_UCC_WECHAT_ADAPTER_URL", "http://127.0.0.1:18800")
|
||||||
|
UCC_ADAPTER_TOKEN = os.environ.get("AIOS_UCC_WECHAT_ADAPTER_TOKEN") or os.environ.get(
|
||||||
|
"AIOS_WECHAT_QR_ADAPTER_TOKEN", ""
|
||||||
|
)
|
||||||
|
UCC_SEND_PATH = os.environ.get("AIOS_UCC_WECHAT_SEND_PATH", "/api/internal/wechat/send")
|
||||||
|
_allowed_origins = os.environ.get("AIOS_UCC_WECHAT_ALLOWED_ORIGINS", "").strip()
|
||||||
|
UCC_ADAPTER_ALLOWED_ORIGINS = tuple(
|
||||||
|
origin.strip() for origin in _allowed_origins.split(",") if origin.strip()
|
||||||
|
) or (
|
||||||
|
"http://127.0.0.1:18800",
|
||||||
|
"http://[::1]:18800",
|
||||||
|
)
|
||||||
|
|
||||||
|
MAX_TARGET_BYTES = 512
|
||||||
|
MAX_CONTENT_BYTES = 32 * 1024
|
||||||
|
MAX_IDENTIFIER_BYTES = 512
|
||||||
|
MAX_METADATA_BYTES = 8 * 1024
|
||||||
|
MAX_METADATA_DEPTH = 32
|
||||||
|
MAX_METADATA_NODES = 1024
|
||||||
|
MAX_RESPONSE_BYTES = 64 * 1024
|
||||||
|
MAX_ENDPOINT_BYTES = 256
|
||||||
|
_DEFAULT_TIMEOUT_SECONDS = 10
|
||||||
|
_MAX_TIMEOUT_SECONDS = 30
|
||||||
|
_LEGACY_SEND_PATH = "/internal/channels/wechat/send"
|
||||||
|
_SAFE_PATH = re.compile(r"^/[A-Za-z0-9._~!$&'()*+,;=:@%/-]+$")
|
||||||
|
_MAX_PATH_DECODE_ROUNDS = 4
|
||||||
|
_OPENCLAW_TIMEOUT_RETURN_CODE = 124
|
||||||
|
_OPENCLAW_OUTPUT_LIMIT_RETURN_CODE = 125
|
||||||
|
_OPENCLAW_SPAWN_ERROR_RETURN_CODE = 126
|
||||||
|
|
||||||
|
|
||||||
|
def _bounded_timeout(value: str) -> int:
|
||||||
|
try:
|
||||||
|
parsed = int(value)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
return _DEFAULT_TIMEOUT_SECONDS
|
||||||
|
return min(max(parsed, 1), _MAX_TIMEOUT_SECONDS)
|
||||||
|
|
||||||
|
|
||||||
|
REQUEST_TIMEOUT_SECONDS = _bounded_timeout(os.environ.get("AIOS_UCC_WECHAT_TIMEOUT_SECONDS", "10"))
|
||||||
|
|
||||||
|
|
||||||
|
class BackendResult:
|
||||||
|
def __init__(self, ok: bool, backend: str, status: str, data: Dict[str, Any]) -> None:
|
||||||
|
self.ok = ok
|
||||||
|
self.backend = backend
|
||||||
|
self.status = status
|
||||||
|
self.data = data
|
||||||
|
|
||||||
|
def to_dict(self) -> Dict[str, Any]:
|
||||||
|
return {"ok": self.ok, "backend": self.backend, "status": self.status, **self.data}
|
||||||
|
|
||||||
|
|
||||||
|
class _BackendProtocolError(Exception):
|
||||||
|
"""Internal marker whose details must never cross the module boundary."""
|
||||||
|
|
||||||
|
|
||||||
|
def _openclaw_base_args() -> List[str]:
|
||||||
|
"""Return the OpenClaw 2026.6.11-compatible command prefix.
|
||||||
|
|
||||||
|
Gateway/auth global flags are not supported by the installed CLI and would
|
||||||
|
also expose credentials through the process argument vector.
|
||||||
|
"""
|
||||||
|
|
||||||
|
return [OPENCLAW_BIN]
|
||||||
|
|
||||||
|
|
||||||
|
def _terminate_process_tree(process: Any) -> None:
|
||||||
|
if os.name == "posix":
|
||||||
|
try:
|
||||||
|
os.killpg(process.pid, signal.SIGTERM)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
process.wait(timeout=0.5)
|
||||||
|
except (OSError, subprocess.TimeoutExpired):
|
||||||
|
pass
|
||||||
|
time.sleep(0.05)
|
||||||
|
try:
|
||||||
|
os.killpg(process.pid, signal.SIGKILL)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
process.wait(timeout=0.5)
|
||||||
|
except (OSError, subprocess.TimeoutExpired):
|
||||||
|
pass
|
||||||
|
return
|
||||||
|
if process.poll() is not None:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
process.terminate()
|
||||||
|
process.wait(timeout=1)
|
||||||
|
return
|
||||||
|
except (OSError, subprocess.TimeoutExpired):
|
||||||
|
pass
|
||||||
|
try:
|
||||||
|
process.kill()
|
||||||
|
process.wait(timeout=1)
|
||||||
|
except (OSError, subprocess.TimeoutExpired):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _run_openclaw(
|
||||||
|
args: List[str], timeout: int = 15
|
||||||
|
) -> subprocess.CompletedProcess[str]:
|
||||||
|
bounded_timeout = min(max(int(timeout), 1), _MAX_TIMEOUT_SECONDS)
|
||||||
|
popen_options: Dict[str, Any] = {
|
||||||
|
"stdout": subprocess.PIPE,
|
||||||
|
"stderr": subprocess.PIPE,
|
||||||
|
}
|
||||||
|
if os.name == "posix":
|
||||||
|
popen_options["start_new_session"] = True
|
||||||
|
elif hasattr(subprocess, "CREATE_NEW_PROCESS_GROUP"):
|
||||||
|
popen_options["creationflags"] = subprocess.CREATE_NEW_PROCESS_GROUP
|
||||||
|
try:
|
||||||
|
process = subprocess.Popen(args, **popen_options)
|
||||||
|
except (OSError, ValueError):
|
||||||
|
return subprocess.CompletedProcess(args, _OPENCLAW_SPAWN_ERROR_RETURN_CODE, "", "")
|
||||||
|
|
||||||
|
selector = selectors.DefaultSelector()
|
||||||
|
streams = {"stdout": process.stdout, "stderr": process.stderr}
|
||||||
|
buffers = {"stdout": bytearray(), "stderr": bytearray()}
|
||||||
|
try:
|
||||||
|
for name, stream in streams.items():
|
||||||
|
if stream is not None:
|
||||||
|
selector.register(stream, selectors.EVENT_READ, name)
|
||||||
|
except (OSError, ValueError):
|
||||||
|
selector.close()
|
||||||
|
_terminate_process_tree(process)
|
||||||
|
for stream in streams.values():
|
||||||
|
if stream is not None:
|
||||||
|
stream.close()
|
||||||
|
return subprocess.CompletedProcess(args, _OPENCLAW_SPAWN_ERROR_RETURN_CODE, "", "")
|
||||||
|
|
||||||
|
deadline = time.monotonic() + bounded_timeout
|
||||||
|
total_bytes = 0
|
||||||
|
return_code: Optional[int] = None
|
||||||
|
try:
|
||||||
|
while selector.get_map():
|
||||||
|
remaining = deadline - time.monotonic()
|
||||||
|
if remaining <= 0:
|
||||||
|
return_code = _OPENCLAW_TIMEOUT_RETURN_CODE
|
||||||
|
break
|
||||||
|
events = selector.select(min(remaining, 0.1))
|
||||||
|
for key, _mask in events:
|
||||||
|
allowance = MAX_RESPONSE_BYTES - total_bytes
|
||||||
|
try:
|
||||||
|
chunk = os.read(key.fileobj.fileno(), min(8192, allowance + 1))
|
||||||
|
except OSError:
|
||||||
|
return_code = _OPENCLAW_SPAWN_ERROR_RETURN_CODE
|
||||||
|
break
|
||||||
|
if not chunk:
|
||||||
|
selector.unregister(key.fileobj)
|
||||||
|
key.fileobj.close()
|
||||||
|
continue
|
||||||
|
if len(chunk) > allowance:
|
||||||
|
return_code = _OPENCLAW_OUTPUT_LIMIT_RETURN_CODE
|
||||||
|
break
|
||||||
|
buffers[key.data].extend(chunk)
|
||||||
|
total_bytes += len(chunk)
|
||||||
|
if return_code is not None:
|
||||||
|
break
|
||||||
|
finally:
|
||||||
|
selector.close()
|
||||||
|
|
||||||
|
if return_code is not None:
|
||||||
|
_terminate_process_tree(process)
|
||||||
|
for stream in streams.values():
|
||||||
|
if stream is not None and not stream.closed:
|
||||||
|
stream.close()
|
||||||
|
return subprocess.CompletedProcess(args, return_code, "", "")
|
||||||
|
try:
|
||||||
|
completed_return_code = process.wait(timeout=1)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
_terminate_process_tree(process)
|
||||||
|
return subprocess.CompletedProcess(args, _OPENCLAW_TIMEOUT_RETURN_CODE, "", "")
|
||||||
|
stdout = buffers["stdout"].decode("utf-8", errors="replace")
|
||||||
|
stderr = buffers["stderr"].decode("utf-8", errors="replace")
|
||||||
|
return subprocess.CompletedProcess(args, completed_return_code, stdout, stderr)
|
||||||
|
|
||||||
|
|
||||||
|
def _json_dict(text: str) -> Dict[str, Any]:
|
||||||
|
try:
|
||||||
|
value = json.loads(text or "{}")
|
||||||
|
except (RecursionError, TypeError, ValueError, json.JSONDecodeError):
|
||||||
|
return {}
|
||||||
|
return value if isinstance(value, dict) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def discover_openclaw_wechat_channel() -> Optional[str]:
|
||||||
|
"""Discover an explicitly typed WeChat/Weixin channel for diagnostics."""
|
||||||
|
|
||||||
|
if WECHAT_CHANNEL:
|
||||||
|
return WECHAT_CHANNEL
|
||||||
|
try:
|
||||||
|
proc = _run_openclaw(_openclaw_base_args() + ["channels", "list", "--json"], timeout=15)
|
||||||
|
except (OSError, subprocess.SubprocessError, ValueError):
|
||||||
|
return None
|
||||||
|
if proc.returncode != 0:
|
||||||
|
return None
|
||||||
|
data = _json_dict(proc.stdout[:MAX_RESPONSE_BYTES])
|
||||||
|
candidates: List[str] = []
|
||||||
|
|
||||||
|
def walk(value: Any) -> None:
|
||||||
|
if isinstance(value, dict):
|
||||||
|
name = value.get("name") or value.get("channel") or value.get("id")
|
||||||
|
kind = value.get("type") or value.get("kind")
|
||||||
|
if isinstance(name, str) and isinstance(kind, str):
|
||||||
|
label = f"{name} {kind}".lower()
|
||||||
|
if "wechat" in label or "weixin" in label:
|
||||||
|
candidates.append(name)
|
||||||
|
for child in value.values():
|
||||||
|
walk(child)
|
||||||
|
elif isinstance(value, list):
|
||||||
|
for child in value:
|
||||||
|
walk(child)
|
||||||
|
|
||||||
|
walk(data)
|
||||||
|
return candidates[0] if candidates else None
|
||||||
|
|
||||||
|
|
||||||
|
def _auth_headers() -> Dict[str, str]:
|
||||||
|
headers = {"Content-Type": "application/json", "Accept": "application/json"}
|
||||||
|
if UCC_ADAPTER_TOKEN:
|
||||||
|
headers["Authorization"] = "Bearer " + UCC_ADAPTER_TOKEN
|
||||||
|
return headers
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_path(value: Any) -> Optional[str]:
|
||||||
|
if not isinstance(value, str):
|
||||||
|
return None
|
||||||
|
path = value if value.startswith("/") else "/" + value
|
||||||
|
try:
|
||||||
|
encoded_path = path.encode("utf-8")
|
||||||
|
except UnicodeEncodeError:
|
||||||
|
return None
|
||||||
|
if len(encoded_path) > MAX_ENDPOINT_BYTES:
|
||||||
|
return None
|
||||||
|
if not _SAFE_PATH.fullmatch(path):
|
||||||
|
return None
|
||||||
|
decoded = path
|
||||||
|
for _ in range(_MAX_PATH_DECODE_ROUNDS):
|
||||||
|
try:
|
||||||
|
next_decoded = urllib.parse.unquote_to_bytes(decoded).decode("utf-8")
|
||||||
|
except UnicodeDecodeError:
|
||||||
|
return None
|
||||||
|
if next_decoded == decoded:
|
||||||
|
break
|
||||||
|
decoded = next_decoded
|
||||||
|
if "%" in decoded or "\\" in decoded or "\x00" in decoded:
|
||||||
|
return None
|
||||||
|
if not _SAFE_PATH.fullmatch(decoded):
|
||||||
|
return None
|
||||||
|
if any(segment in {".", ".."} for segment in decoded.split("/")):
|
||||||
|
return None
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def _unique_paths(paths: List[str]) -> List[str]:
|
||||||
|
out: List[str] = []
|
||||||
|
for candidate in paths:
|
||||||
|
path = _safe_path(candidate)
|
||||||
|
if path and path not in out:
|
||||||
|
out.append(path)
|
||||||
|
if len(out) == 2:
|
||||||
|
break
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _ucc_send_paths() -> List[str]:
|
||||||
|
canonical = _safe_path(UCC_SEND_PATH) or "/api/internal/wechat/send"
|
||||||
|
return _unique_paths([canonical, _LEGACY_SEND_PATH])
|
||||||
|
|
||||||
|
|
||||||
|
def _canonical_origin(value: Any) -> Optional[str]:
|
||||||
|
if not isinstance(value, str) or not value or len(value) > 2048:
|
||||||
|
return None
|
||||||
|
if "\\" in value or any(ord(character) < 32 for character in value):
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
parsed = urllib.parse.urlsplit(value)
|
||||||
|
port = parsed.port
|
||||||
|
except (UnicodeError, ValueError):
|
||||||
|
return None
|
||||||
|
scheme = parsed.scheme.lower()
|
||||||
|
if scheme not in {"http", "https"} or not parsed.hostname:
|
||||||
|
return None
|
||||||
|
if parsed.username or parsed.password or parsed.query or parsed.fragment or parsed.path not in {"", "/"}:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
host = parsed.hostname.rstrip(".").encode("idna").decode("ascii").lower()
|
||||||
|
except UnicodeError:
|
||||||
|
return None
|
||||||
|
if not host:
|
||||||
|
return None
|
||||||
|
if port is None:
|
||||||
|
port = 80 if scheme == "http" else 443
|
||||||
|
rendered_host = f"[{host}]" if ":" in host else host
|
||||||
|
return f"{scheme}://{rendered_host}:{port}"
|
||||||
|
|
||||||
|
|
||||||
|
def _adapter_base_url() -> Optional[str]:
|
||||||
|
origin = _canonical_origin(UCC_ADAPTER_URL)
|
||||||
|
allowed = {
|
||||||
|
canonical
|
||||||
|
for canonical in (_canonical_origin(value) for value in UCC_ADAPTER_ALLOWED_ORIGINS)
|
||||||
|
if canonical
|
||||||
|
}
|
||||||
|
return origin if origin and origin in allowed else None
|
||||||
|
|
||||||
|
|
||||||
|
class _NoRedirectHandler(urllib.request.HTTPRedirectHandler):
|
||||||
|
def redirect_request(
|
||||||
|
self,
|
||||||
|
req: Any,
|
||||||
|
fp: Any,
|
||||||
|
code: int,
|
||||||
|
msg: str,
|
||||||
|
headers: Any,
|
||||||
|
newurl: str,
|
||||||
|
) -> None:
|
||||||
|
del req, fp, code, msg, headers, newurl
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _open_no_redirect(request: urllib.request.Request, timeout: int) -> Any:
|
||||||
|
opener = urllib.request.build_opener(_NoRedirectHandler())
|
||||||
|
return opener.open(request, timeout=timeout)
|
||||||
|
|
||||||
|
|
||||||
|
def ucc_adapter_alive() -> bool:
|
||||||
|
base_url = _adapter_base_url()
|
||||||
|
if not base_url:
|
||||||
|
return False
|
||||||
|
request = urllib.request.Request(
|
||||||
|
base_url + "/health", headers=_auth_headers(), method="GET"
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with _open_no_redirect(request, timeout=min(5, REQUEST_TIMEOUT_SECONDS)) as response:
|
||||||
|
return 200 <= int(response.status) < 300
|
||||||
|
except (OSError, ValueError, urllib.error.URLError):
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def select_backend() -> Dict[str, Any]:
|
||||||
|
explicit = SEND_BACKEND.strip().lower()
|
||||||
|
if explicit:
|
||||||
|
if explicit == "ucc-adapter":
|
||||||
|
return {"backend": "ucc-adapter", "reason": "explicit"}
|
||||||
|
return {"backend": "unavailable", "reason": "unsupported_backend"}
|
||||||
|
if ucc_adapter_alive():
|
||||||
|
return {"backend": "ucc-adapter", "reason": "ucc_adapter_alive"}
|
||||||
|
return {"backend": "unavailable", "reason": "ucc_adapter_unavailable"}
|
||||||
|
|
||||||
|
|
||||||
|
def _post_ucc(path: str, payload: Dict[str, Any]) -> Tuple[int, Dict[str, Any]]:
|
||||||
|
base_url = _adapter_base_url()
|
||||||
|
safe_path = _safe_path(path)
|
||||||
|
if not base_url or not safe_path:
|
||||||
|
raise _BackendProtocolError("invalid adapter endpoint")
|
||||||
|
try:
|
||||||
|
body = json.dumps(payload, ensure_ascii=False, separators=(",", ":")).encode("utf-8")
|
||||||
|
except (TypeError, ValueError) as exc:
|
||||||
|
raise _BackendProtocolError("invalid request payload") from exc
|
||||||
|
request = urllib.request.Request(
|
||||||
|
base_url + safe_path,
|
||||||
|
data=body,
|
||||||
|
headers=_auth_headers(),
|
||||||
|
method="POST",
|
||||||
|
)
|
||||||
|
with _open_no_redirect(request, timeout=REQUEST_TIMEOUT_SECONDS) as response:
|
||||||
|
raw = response.read(MAX_RESPONSE_BYTES + 1)
|
||||||
|
if len(raw) > MAX_RESPONSE_BYTES:
|
||||||
|
raise _BackendProtocolError("adapter response too large")
|
||||||
|
try:
|
||||||
|
text = raw.decode("utf-8")
|
||||||
|
except UnicodeDecodeError as exc:
|
||||||
|
raise _BackendProtocolError("adapter response encoding invalid") from exc
|
||||||
|
data = _json_dict(text)
|
||||||
|
if not data:
|
||||||
|
raise _BackendProtocolError("adapter response invalid")
|
||||||
|
return int(response.status), data
|
||||||
|
|
||||||
|
|
||||||
|
def _ucc_response_sent(data: Dict[str, Any]) -> bool:
|
||||||
|
return isinstance(data, dict) and data.get("ok") is True and data.get("status") == "sent"
|
||||||
|
|
||||||
|
|
||||||
|
def _ucc_result_is_ok(data: Dict[str, Any], dry_run: bool = False) -> bool:
|
||||||
|
expected_status = "dry_run" if dry_run else "sent"
|
||||||
|
return isinstance(data, dict) and data.get("ok") is True and data.get("status") == expected_status
|
||||||
|
|
||||||
|
|
||||||
|
def _utf8_size(value: str) -> Optional[int]:
|
||||||
|
try:
|
||||||
|
return len(value.encode("utf-8"))
|
||||||
|
except UnicodeEncodeError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _valid_identifier(value: Any, required: bool = False) -> bool:
|
||||||
|
if not isinstance(value, str):
|
||||||
|
return False
|
||||||
|
if required and not value.strip():
|
||||||
|
return False
|
||||||
|
if not value:
|
||||||
|
return not required
|
||||||
|
if len(value) > MAX_IDENTIFIER_BYTES:
|
||||||
|
return False
|
||||||
|
size = _utf8_size(value)
|
||||||
|
return size is not None and size <= MAX_IDENTIFIER_BYTES and not any(
|
||||||
|
ord(character) < 32 or ord(character) == 127 for character in value
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_metadata(metadata: Any) -> Optional[str]:
|
||||||
|
if not isinstance(metadata, dict):
|
||||||
|
return "invalid_input"
|
||||||
|
nodes = 0
|
||||||
|
stack: List[Tuple[Any, int]] = [(metadata, 0)]
|
||||||
|
while stack:
|
||||||
|
value, depth = stack.pop()
|
||||||
|
nodes += 1
|
||||||
|
if depth > MAX_METADATA_DEPTH or nodes > MAX_METADATA_NODES:
|
||||||
|
return "input_too_large"
|
||||||
|
if isinstance(value, dict):
|
||||||
|
if len(value) + nodes + len(stack) > MAX_METADATA_NODES:
|
||||||
|
return "input_too_large"
|
||||||
|
for key, child in value.items():
|
||||||
|
if (
|
||||||
|
not isinstance(key, str)
|
||||||
|
or len(key) > MAX_METADATA_BYTES
|
||||||
|
or _utf8_size(key) is None
|
||||||
|
):
|
||||||
|
return "invalid_input"
|
||||||
|
stack.append((child, depth + 1))
|
||||||
|
elif isinstance(value, list):
|
||||||
|
if len(value) + nodes + len(stack) > MAX_METADATA_NODES:
|
||||||
|
return "input_too_large"
|
||||||
|
stack.extend((child, depth + 1) for child in value)
|
||||||
|
elif isinstance(value, str):
|
||||||
|
if len(value) > MAX_METADATA_BYTES:
|
||||||
|
return "input_too_large"
|
||||||
|
if _utf8_size(value) is None:
|
||||||
|
return "invalid_input"
|
||||||
|
elif value is not None and not isinstance(value, (bool, int, float)):
|
||||||
|
return "invalid_input"
|
||||||
|
try:
|
||||||
|
encoded = json.dumps(
|
||||||
|
metadata,
|
||||||
|
ensure_ascii=False,
|
||||||
|
allow_nan=False,
|
||||||
|
separators=(",", ":"),
|
||||||
|
).encode("utf-8")
|
||||||
|
except (RecursionError, TypeError, UnicodeEncodeError, ValueError):
|
||||||
|
return "invalid_input"
|
||||||
|
return "input_too_large" if len(encoded) > MAX_METADATA_BYTES else None
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_payload(
|
||||||
|
target: Any,
|
||||||
|
content: Any,
|
||||||
|
account_id: Any,
|
||||||
|
message_id: Any,
|
||||||
|
metadata: Any,
|
||||||
|
dry_run: Any,
|
||||||
|
) -> Optional[str]:
|
||||||
|
if not isinstance(target, str) or not isinstance(content, str):
|
||||||
|
return "invalid_input"
|
||||||
|
if not target.strip() or not content.strip() or not isinstance(dry_run, bool):
|
||||||
|
return "invalid_input"
|
||||||
|
if any(ord(character) < 32 or ord(character) == 127 for character in target):
|
||||||
|
return "invalid_input"
|
||||||
|
if "\x00" in content or not _valid_identifier(account_id) or not _valid_identifier(message_id):
|
||||||
|
return "invalid_input"
|
||||||
|
if len(target) > MAX_TARGET_BYTES or len(content) > MAX_CONTENT_BYTES:
|
||||||
|
return "input_too_large"
|
||||||
|
target_size = _utf8_size(target)
|
||||||
|
content_size = _utf8_size(content)
|
||||||
|
if target_size is None or content_size is None:
|
||||||
|
return "invalid_input"
|
||||||
|
if target_size > MAX_TARGET_BYTES or content_size > MAX_CONTENT_BYTES:
|
||||||
|
return "input_too_large"
|
||||||
|
return _validate_metadata(metadata)
|
||||||
|
|
||||||
|
|
||||||
|
def _safe_provider_message_id(value: Any) -> Optional[str]:
|
||||||
|
if not isinstance(value, str) or not value:
|
||||||
|
return None
|
||||||
|
if len(value) > MAX_IDENTIFIER_BYTES:
|
||||||
|
return None
|
||||||
|
size = _utf8_size(value)
|
||||||
|
if size is None or size > MAX_IDENTIFIER_BYTES:
|
||||||
|
return None
|
||||||
|
if any(ord(character) < 32 or ord(character) == 127 for character in value):
|
||||||
|
return None
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def send_ucc_adapter(
|
||||||
|
target: str,
|
||||||
|
content: str,
|
||||||
|
account_id: str = "",
|
||||||
|
message_id: str = "",
|
||||||
|
metadata: Optional[Dict[str, Any]] = None,
|
||||||
|
dry_run: bool = False,
|
||||||
|
) -> BackendResult:
|
||||||
|
effective_account = account_id or WECHAT_ACCOUNT_ID
|
||||||
|
effective_metadata: Any = {} if metadata is None else metadata
|
||||||
|
error_code = _validate_payload(
|
||||||
|
target,
|
||||||
|
content,
|
||||||
|
effective_account,
|
||||||
|
message_id,
|
||||||
|
effective_metadata,
|
||||||
|
dry_run,
|
||||||
|
)
|
||||||
|
if error_code:
|
||||||
|
return BackendResult(False, "ucc-adapter", "failed", {"code": error_code})
|
||||||
|
|
||||||
|
payload = {
|
||||||
|
"target": target,
|
||||||
|
"content": content,
|
||||||
|
"account_id": effective_account,
|
||||||
|
"message_id": message_id,
|
||||||
|
"metadata": effective_metadata,
|
||||||
|
"dry_run": dry_run,
|
||||||
|
}
|
||||||
|
last_endpoint: Optional[str] = None
|
||||||
|
for path in _ucc_send_paths():
|
||||||
|
last_endpoint = path
|
||||||
|
try:
|
||||||
|
http_status, data = _post_ucc(path, payload)
|
||||||
|
except urllib.error.HTTPError as exc:
|
||||||
|
if exc.code in {404, 405}:
|
||||||
|
continue
|
||||||
|
return BackendResult(
|
||||||
|
False,
|
||||||
|
"ucc-adapter",
|
||||||
|
"failed",
|
||||||
|
{"code": "ucc_adapter_http_error", "http_status": int(exc.code), "endpoint": path},
|
||||||
|
)
|
||||||
|
except (OSError, ValueError, urllib.error.URLError, _BackendProtocolError):
|
||||||
|
return BackendResult(
|
||||||
|
False,
|
||||||
|
"ucc-adapter",
|
||||||
|
"failed",
|
||||||
|
{"code": "ucc_adapter_unavailable", "endpoint": path},
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
# Fail closed without returning exception text, type, or chained cause.
|
||||||
|
return BackendResult(
|
||||||
|
False,
|
||||||
|
"ucc-adapter",
|
||||||
|
"failed",
|
||||||
|
{"code": "ucc_adapter_unavailable", "endpoint": path},
|
||||||
|
)
|
||||||
|
|
||||||
|
if not (200 <= http_status < 300):
|
||||||
|
return BackendResult(
|
||||||
|
False,
|
||||||
|
"ucc-adapter",
|
||||||
|
"failed",
|
||||||
|
{"code": "ucc_adapter_http_error", "http_status": http_status, "endpoint": path},
|
||||||
|
)
|
||||||
|
if not _ucc_result_is_ok(data, dry_run):
|
||||||
|
return BackendResult(
|
||||||
|
False,
|
||||||
|
"ucc-adapter",
|
||||||
|
"failed",
|
||||||
|
{"code": "ucc_adapter_rejected", "http_status": http_status, "endpoint": path},
|
||||||
|
)
|
||||||
|
confirmed_status = "dry_run" if dry_run else "sent"
|
||||||
|
result_data: Dict[str, Any] = {
|
||||||
|
"code": confirmed_status,
|
||||||
|
"http_status": http_status,
|
||||||
|
"endpoint": path,
|
||||||
|
}
|
||||||
|
provider_message_id = _safe_provider_message_id(data.get("provider_message_id"))
|
||||||
|
if provider_message_id:
|
||||||
|
result_data["provider_message_id"] = provider_message_id
|
||||||
|
return BackendResult(True, "ucc-adapter", confirmed_status, result_data)
|
||||||
|
|
||||||
|
data: Dict[str, Any] = {"code": "ucc_adapter_unavailable"}
|
||||||
|
if last_endpoint:
|
||||||
|
data["endpoint"] = last_endpoint
|
||||||
|
return BackendResult(False, "ucc-adapter", "failed", data)
|
||||||
|
|
||||||
|
|
||||||
|
def send_openclaw_channel(
|
||||||
|
target: str,
|
||||||
|
content: str,
|
||||||
|
account_id: str = "",
|
||||||
|
message_id: str = "",
|
||||||
|
metadata: Optional[Dict[str, Any]] = None,
|
||||||
|
dry_run: bool = False,
|
||||||
|
) -> BackendResult:
|
||||||
|
del target, content, account_id, message_id, metadata, dry_run
|
||||||
|
return BackendResult(
|
||||||
|
False,
|
||||||
|
"openclaw-channel",
|
||||||
|
"failed",
|
||||||
|
{"code": "unsupported_backend"},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def send_message(
|
||||||
|
target: str,
|
||||||
|
content: str,
|
||||||
|
account_id: str = "",
|
||||||
|
message_id: str = "",
|
||||||
|
metadata: Optional[Dict[str, Any]] = None,
|
||||||
|
dry_run: bool = False,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
selected = select_backend()
|
||||||
|
if selected.get("backend") == "ucc-adapter":
|
||||||
|
return send_ucc_adapter(
|
||||||
|
target, content, account_id, message_id, metadata, dry_run
|
||||||
|
).to_dict()
|
||||||
|
code = "unsupported_backend" if selected.get("reason") == "unsupported_backend" else "wechat_outbound_backend_unavailable"
|
||||||
|
return BackendResult(False, "unavailable", "failed", {"code": code}).to_dict()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
print(json.dumps(select_backend(), ensure_ascii=False, separators=(",", ":")))
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
"""AIOS services namespace."""
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
"""Reliable, fail-closed Codex command watcher."""
|
||||||
|
|
||||||
|
from .watcher import CommandWatcher, Config, load_config
|
||||||
|
|
||||||
|
__all__ = ["CommandWatcher", "Config", "load_config"]
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import sys
|
||||||
|
|
||||||
|
from .watcher import CommandWatcher, load_config
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description="Claim and execute Codex work units")
|
||||||
|
parser.add_argument("--config", required=True)
|
||||||
|
parser.add_argument("--once", action="store_true", help="process one scan and exit")
|
||||||
|
args = parser.parse_args()
|
||||||
|
watcher = CommandWatcher(load_config(args.config))
|
||||||
|
return watcher.run_once() if args.once else watcher.run_forever()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,480 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
import re
|
||||||
|
import sqlite3
|
||||||
|
import stat
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
from typing import Any, Callable
|
||||||
|
import urllib.error
|
||||||
|
import urllib.parse
|
||||||
|
import urllib.request
|
||||||
|
import uuid
|
||||||
|
|
||||||
|
|
||||||
|
OID_RE = re.compile(r"^[0-9a-f]{40}$")
|
||||||
|
ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$")
|
||||||
|
TERMINAL = {"COMPLETED", "DEAD_LETTER", "REJECTED"}
|
||||||
|
|
||||||
|
|
||||||
|
class WatcherError(RuntimeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Config:
|
||||||
|
active_dir: Path
|
||||||
|
completed_dir: Path
|
||||||
|
dead_letter_dir: Path
|
||||||
|
database_path: Path
|
||||||
|
audit_log_path: Path
|
||||||
|
gateway_endpoint: str
|
||||||
|
gateway_token_file: Path
|
||||||
|
contract_commit: str
|
||||||
|
lease_seconds: int
|
||||||
|
retry_seconds: int
|
||||||
|
max_attempts: int
|
||||||
|
poll_seconds: float
|
||||||
|
executor_path: Path | None
|
||||||
|
executor_args: tuple[str, ...]
|
||||||
|
executor_timeout_seconds: int
|
||||||
|
|
||||||
|
|
||||||
|
def _absolute(value: Any, key: str) -> Path:
|
||||||
|
if not isinstance(value, str) or not value or not Path(value).is_absolute():
|
||||||
|
raise WatcherError(f"invalid_{key}")
|
||||||
|
return Path(value)
|
||||||
|
|
||||||
|
|
||||||
|
def _secure_regular(path: Path, *, writable_mask: int = 0o022) -> os.stat_result:
|
||||||
|
info = os.lstat(path)
|
||||||
|
if stat.S_ISLNK(info.st_mode) or not stat.S_ISREG(info.st_mode):
|
||||||
|
raise WatcherError(f"unsafe_file:{path}")
|
||||||
|
if stat.S_IMODE(info.st_mode) & writable_mask:
|
||||||
|
raise WatcherError(f"unsafe_mode:{path}")
|
||||||
|
return info
|
||||||
|
|
||||||
|
|
||||||
|
def _read_secure_text(
|
||||||
|
path: Path,
|
||||||
|
*,
|
||||||
|
writable_mask: int = 0o022,
|
||||||
|
max_bytes: int,
|
||||||
|
allowed_owners: set[int] | None = None,
|
||||||
|
) -> str:
|
||||||
|
flags = os.O_RDONLY | getattr(os, "O_NOFOLLOW", 0)
|
||||||
|
fd = os.open(path, flags)
|
||||||
|
try:
|
||||||
|
info = os.fstat(fd)
|
||||||
|
if not stat.S_ISREG(info.st_mode) or stat.S_IMODE(info.st_mode) & writable_mask:
|
||||||
|
raise WatcherError(f"unsafe_file:{path}")
|
||||||
|
if allowed_owners is not None and info.st_uid not in allowed_owners:
|
||||||
|
raise WatcherError(f"owner_mismatch:{path}")
|
||||||
|
if info.st_size > max_bytes:
|
||||||
|
raise WatcherError(f"file_too_large:{path}")
|
||||||
|
chunks: list[bytes] = []
|
||||||
|
total = 0
|
||||||
|
while True:
|
||||||
|
chunk = os.read(fd, min(65_536, max_bytes + 1 - total))
|
||||||
|
if not chunk:
|
||||||
|
break
|
||||||
|
chunks.append(chunk)
|
||||||
|
total += len(chunk)
|
||||||
|
if total > max_bytes:
|
||||||
|
raise WatcherError(f"file_too_large:{path}")
|
||||||
|
return b"".join(chunks).decode("utf-8")
|
||||||
|
finally:
|
||||||
|
os.close(fd)
|
||||||
|
|
||||||
|
|
||||||
|
def load_config(path_value: str) -> Config:
|
||||||
|
path = _absolute(path_value, "config_path")
|
||||||
|
raw = json.loads(_read_secure_text(path, max_bytes=131_072, allowed_owners={0, os.geteuid()}))
|
||||||
|
expected = {
|
||||||
|
"active_dir", "completed_dir", "dead_letter_dir", "database_path",
|
||||||
|
"audit_log_path", "gateway_endpoint", "gateway_token_file",
|
||||||
|
"contract_commit", "lease_seconds", "retry_seconds", "max_attempts",
|
||||||
|
"poll_seconds", "executor_path", "executor_args",
|
||||||
|
"executor_timeout_seconds",
|
||||||
|
}
|
||||||
|
if not isinstance(raw, dict) or set(raw) != expected:
|
||||||
|
raise WatcherError("invalid_config_fields")
|
||||||
|
contract = raw["contract_commit"]
|
||||||
|
if not isinstance(contract, str) or not OID_RE.fullmatch(contract):
|
||||||
|
raise WatcherError("invalid_contract_commit")
|
||||||
|
endpoint = raw["gateway_endpoint"]
|
||||||
|
if not isinstance(endpoint, str) or not endpoint.startswith("http://127.0.0.1:") or not endpoint.endswith("/v1/orchestrator/events"):
|
||||||
|
raise WatcherError("gateway_endpoint_must_be_loopback")
|
||||||
|
executor_raw = raw["executor_path"]
|
||||||
|
executor = None if executor_raw is None else _absolute(executor_raw, "executor_path")
|
||||||
|
executor_args = raw["executor_args"]
|
||||||
|
if not isinstance(executor_args, list) or any(not isinstance(x, str) or not x for x in executor_args):
|
||||||
|
raise WatcherError("invalid_executor_args")
|
||||||
|
ints = ("lease_seconds", "retry_seconds", "max_attempts", "executor_timeout_seconds")
|
||||||
|
if any(not isinstance(raw[x], int) or raw[x] <= 0 for x in ints):
|
||||||
|
raise WatcherError("invalid_positive_integer")
|
||||||
|
if not isinstance(raw["poll_seconds"], (int, float)) or raw["poll_seconds"] <= 0:
|
||||||
|
raise WatcherError("invalid_poll_seconds")
|
||||||
|
return Config(
|
||||||
|
active_dir=_absolute(raw["active_dir"], "active_dir"),
|
||||||
|
completed_dir=_absolute(raw["completed_dir"], "completed_dir"),
|
||||||
|
dead_letter_dir=_absolute(raw["dead_letter_dir"], "dead_letter_dir"),
|
||||||
|
database_path=_absolute(raw["database_path"], "database_path"),
|
||||||
|
audit_log_path=_absolute(raw["audit_log_path"], "audit_log_path"),
|
||||||
|
gateway_endpoint=endpoint,
|
||||||
|
gateway_token_file=_absolute(raw["gateway_token_file"], "gateway_token_file"),
|
||||||
|
contract_commit=contract,
|
||||||
|
lease_seconds=raw["lease_seconds"], retry_seconds=raw["retry_seconds"],
|
||||||
|
max_attempts=raw["max_attempts"], poll_seconds=float(raw["poll_seconds"]),
|
||||||
|
executor_path=executor, executor_args=tuple(executor_args),
|
||||||
|
executor_timeout_seconds=raw["executor_timeout_seconds"],
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def canonical_digest(value: dict[str, Any]) -> str:
|
||||||
|
return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(",", ":")).encode()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
def validate_command(value: Any, expected_contract: str) -> dict[str, Any]:
|
||||||
|
expected = {
|
||||||
|
"schema_version", "command_id", "project_key", "work_unit_id",
|
||||||
|
"contract_commit", "state_version", "execution",
|
||||||
|
}
|
||||||
|
if not isinstance(value, dict) or set(value) != expected or value.get("schema_version") != 1:
|
||||||
|
raise WatcherError("invalid_command_fields")
|
||||||
|
for key in ("command_id", "project_key", "work_unit_id"):
|
||||||
|
if not isinstance(value[key], str) or not ID_RE.fullmatch(value[key]):
|
||||||
|
raise WatcherError(f"invalid_{key}")
|
||||||
|
if value["contract_commit"] != expected_contract:
|
||||||
|
raise WatcherError("contract_commit_mismatch")
|
||||||
|
if not isinstance(value["state_version"], int) or value["state_version"] < 0:
|
||||||
|
raise WatcherError("invalid_state_version")
|
||||||
|
execution = value["execution"]
|
||||||
|
if not isinstance(execution, dict) or set(execution) != {"kind", "input"}:
|
||||||
|
raise WatcherError("invalid_execution")
|
||||||
|
if execution["kind"] not in {"acceptance_probe", "codex"}:
|
||||||
|
raise WatcherError("invalid_execution_kind")
|
||||||
|
if not isinstance(execution["input"], str) or len(execution["input"].encode()) > 65_536:
|
||||||
|
raise WatcherError("invalid_execution_input")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
class CommandWatcher:
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
config: Config,
|
||||||
|
*,
|
||||||
|
clock: Callable[[], float] = time.time,
|
||||||
|
event_poster: Callable[[dict[str, Any]], None] | None = None,
|
||||||
|
executor: Callable[[dict[str, Any]], dict[str, Any]] | None = None,
|
||||||
|
worker_id: str | None = None,
|
||||||
|
) -> None:
|
||||||
|
self.config = config
|
||||||
|
self.clock = clock
|
||||||
|
self.worker_id = worker_id or f"{os.uname().nodename}:{os.getpid()}:{uuid.uuid4().hex[:8]}"
|
||||||
|
self._uses_gateway = event_poster is None
|
||||||
|
self.event_poster = event_poster or self._post_event
|
||||||
|
self.executor = executor or self._execute
|
||||||
|
self._prepare()
|
||||||
|
|
||||||
|
def _prepare(self) -> None:
|
||||||
|
for directory in (self.config.active_dir, self.config.completed_dir, self.config.dead_letter_dir):
|
||||||
|
info = os.lstat(directory)
|
||||||
|
if stat.S_ISLNK(info.st_mode) or not stat.S_ISDIR(info.st_mode) or stat.S_IMODE(info.st_mode) & 0o002:
|
||||||
|
raise WatcherError(f"unsafe_directory:{directory}")
|
||||||
|
self.config.database_path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||||
|
self.config.audit_log_path.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||||
|
with self._db() as db:
|
||||||
|
db.execute("PRAGMA journal_mode=WAL")
|
||||||
|
db.execute("""
|
||||||
|
CREATE TABLE IF NOT EXISTS commands (
|
||||||
|
command_id TEXT PRIMARY KEY, work_unit_id TEXT NOT NULL UNIQUE,
|
||||||
|
payload_sha256 TEXT NOT NULL,
|
||||||
|
source_path TEXT NOT NULL, status TEXT NOT NULL,
|
||||||
|
phase TEXT NOT NULL, attempts INTEGER NOT NULL DEFAULT 0,
|
||||||
|
lease_owner TEXT, lease_until REAL, next_attempt REAL NOT NULL DEFAULT 0,
|
||||||
|
result_json TEXT, start_event_json TEXT, end_event_json TEXT,
|
||||||
|
last_error TEXT, updated_at REAL NOT NULL
|
||||||
|
)
|
||||||
|
""")
|
||||||
|
|
||||||
|
def _db(self) -> sqlite3.Connection:
|
||||||
|
db = sqlite3.connect(self.config.database_path, timeout=5, isolation_level=None)
|
||||||
|
db.row_factory = sqlite3.Row
|
||||||
|
return db
|
||||||
|
|
||||||
|
def _audit(self, marker: str, command_id: str, **fields: Any) -> None:
|
||||||
|
safe = {k: v for k, v in fields.items() if k not in {"input", "token", "secret"}}
|
||||||
|
record = {"time": int(self.clock()), "marker": marker, "command_id": command_id, **safe}
|
||||||
|
fd = os.open(self.config.audit_log_path, os.O_APPEND | os.O_CREAT | os.O_WRONLY | os.O_NOFOLLOW, 0o600)
|
||||||
|
try:
|
||||||
|
os.write(fd, (json.dumps(record, sort_keys=True, separators=(",", ":")) + "\n").encode())
|
||||||
|
finally:
|
||||||
|
os.close(fd)
|
||||||
|
|
||||||
|
def _read_command(self, path: Path) -> tuple[dict[str, Any], str]:
|
||||||
|
raw = _read_secure_text(path, max_bytes=131_072, allowed_owners={0, os.geteuid()})
|
||||||
|
value = validate_command(json.loads(raw), self.config.contract_commit)
|
||||||
|
return value, canonical_digest(value)
|
||||||
|
|
||||||
|
def _register(self, command: dict[str, Any], digest: str, path: Path) -> str:
|
||||||
|
now = self.clock()
|
||||||
|
with self._db() as db:
|
||||||
|
db.execute("BEGIN IMMEDIATE")
|
||||||
|
row = db.execute("SELECT payload_sha256,status FROM commands WHERE command_id=?", (command["command_id"],)).fetchone()
|
||||||
|
if row:
|
||||||
|
db.execute("COMMIT")
|
||||||
|
if row["payload_sha256"] != digest:
|
||||||
|
raise WatcherError("duplicate_command_payload_conflict")
|
||||||
|
return "duplicate_terminal" if row["status"] in TERMINAL else "existing"
|
||||||
|
prior_work_unit = db.execute(
|
||||||
|
"SELECT command_id FROM commands WHERE work_unit_id=?",
|
||||||
|
(command["work_unit_id"],),
|
||||||
|
).fetchone()
|
||||||
|
if prior_work_unit:
|
||||||
|
db.execute("COMMIT")
|
||||||
|
raise WatcherError("duplicate_work_unit")
|
||||||
|
db.execute(
|
||||||
|
"INSERT INTO commands(command_id,work_unit_id,payload_sha256,source_path,status,phase,updated_at) VALUES(?,?,?,?,?,?,?)",
|
||||||
|
(command["command_id"], command["work_unit_id"], digest, str(path), "PENDING", "START", now),
|
||||||
|
)
|
||||||
|
db.execute("COMMIT")
|
||||||
|
return "new"
|
||||||
|
|
||||||
|
def _claim(self, command_id: str) -> sqlite3.Row | None:
|
||||||
|
now = self.clock()
|
||||||
|
with self._db() as db:
|
||||||
|
db.execute("BEGIN IMMEDIATE")
|
||||||
|
row = db.execute("SELECT * FROM commands WHERE command_id=?", (command_id,)).fetchone()
|
||||||
|
if not row or row["status"] in TERMINAL or row["next_attempt"] > now:
|
||||||
|
db.execute("COMMIT")
|
||||||
|
return None
|
||||||
|
if row["lease_until"] is not None and row["lease_until"] > now and row["lease_owner"] != self.worker_id:
|
||||||
|
db.execute("COMMIT")
|
||||||
|
return None
|
||||||
|
attempts = row["attempts"] + 1
|
||||||
|
db.execute(
|
||||||
|
"UPDATE commands SET status='CLAIMED',attempts=?,lease_owner=?,lease_until=?,updated_at=? WHERE command_id=?",
|
||||||
|
(attempts, self.worker_id, now + self.config.lease_seconds, now, command_id),
|
||||||
|
)
|
||||||
|
row = db.execute("SELECT * FROM commands WHERE command_id=?", (command_id,)).fetchone()
|
||||||
|
db.execute("COMMIT")
|
||||||
|
assert row is not None
|
||||||
|
self._audit("CODEX_AUTO_CLAIM", command_id, attempt=row["attempts"], worker=self.worker_id)
|
||||||
|
return row
|
||||||
|
|
||||||
|
def _update(self, command_id: str, *, status: str, phase: str, result: dict[str, Any] | None = None, error: str | None = None) -> None:
|
||||||
|
with self._db() as db:
|
||||||
|
release = status in TERMINAL
|
||||||
|
db.execute(
|
||||||
|
"UPDATE commands SET status=?,phase=?,result_json=COALESCE(?,result_json),last_error=?,lease_owner=CASE WHEN ? THEN NULL ELSE lease_owner END,lease_until=CASE WHEN ? THEN NULL ELSE lease_until END,updated_at=? WHERE command_id=?",
|
||||||
|
(status, phase, json.dumps(result, sort_keys=True) if result is not None else None, error, release, release, self.clock(), command_id),
|
||||||
|
)
|
||||||
|
|
||||||
|
def _retry_or_dead(self, command_id: str, path: Path, error: Exception) -> None:
|
||||||
|
message = f"{type(error).__name__}:{str(error)[:240]}"
|
||||||
|
with self._db() as db:
|
||||||
|
row = db.execute("SELECT attempts,phase FROM commands WHERE command_id=?", (command_id,)).fetchone()
|
||||||
|
attempts = row["attempts"] if row else self.config.max_attempts
|
||||||
|
if str(error) == "gateway_stale_state_version" and row:
|
||||||
|
column = "start_event_json" if row["phase"] == "START" else "end_event_json"
|
||||||
|
db.execute(f"UPDATE commands SET {column}=NULL WHERE command_id=?", (command_id,))
|
||||||
|
if attempts >= self.config.max_attempts:
|
||||||
|
db.execute("UPDATE commands SET status='DEAD_LETTER',last_error=?,lease_owner=NULL,lease_until=NULL,updated_at=? WHERE command_id=?", (message, self.clock(), command_id))
|
||||||
|
self._move(path, self.config.dead_letter_dir)
|
||||||
|
self._audit("DEAD_LETTER", command_id, error=message)
|
||||||
|
else:
|
||||||
|
db.execute("UPDATE commands SET status='RETRY',last_error=?,next_attempt=?,lease_owner=NULL,lease_until=NULL,updated_at=? WHERE command_id=?", (message, self.clock() + self.config.retry_seconds, self.clock(), command_id))
|
||||||
|
self._audit("RETRY_SCHEDULED", command_id, attempt=attempts, error=message)
|
||||||
|
|
||||||
|
def _event(self, command: dict[str, Any], event_type: str, status: str, suffix: str, result: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"schema_version": 1, "project_key": command["project_key"],
|
||||||
|
"contract_commit": command["contract_commit"],
|
||||||
|
"event_id": f"{command['command_id']}:{suffix}", "source_agent": "codex",
|
||||||
|
"event_type": event_type, "work_unit_id": command["work_unit_id"],
|
||||||
|
"status": status, "state_version": command["state_version"],
|
||||||
|
"implementation_commit": None, "result_commit": None, "blocker": None,
|
||||||
|
"evidence_refs": [], "payload": {"command_id": command["command_id"], "result": result or {}},
|
||||||
|
}
|
||||||
|
|
||||||
|
def _gateway_token(self) -> str:
|
||||||
|
token = _read_secure_text(
|
||||||
|
self.config.gateway_token_file,
|
||||||
|
writable_mask=0o027,
|
||||||
|
max_bytes=16_384,
|
||||||
|
allowed_owners={0, os.geteuid()},
|
||||||
|
).strip()
|
||||||
|
if not token or "\n" in token or "\r" in token:
|
||||||
|
raise WatcherError("invalid_gateway_token")
|
||||||
|
return token
|
||||||
|
|
||||||
|
def _gateway_state_version(self, project_key: str, fallback: int) -> int:
|
||||||
|
state_url = self.config.gateway_endpoint.removesuffix("/events") + "/state/" + urllib.parse.quote(project_key, safe="")
|
||||||
|
request = urllib.request.Request(
|
||||||
|
state_url,
|
||||||
|
headers={"Authorization": f"Bearer {self._gateway_token()}"},
|
||||||
|
method="GET",
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(request, timeout=15) as response:
|
||||||
|
payload = json.loads(response.read(262_145))
|
||||||
|
version = payload.get("state_version")
|
||||||
|
if not isinstance(version, int) or version < 0:
|
||||||
|
raise WatcherError("invalid_gateway_state")
|
||||||
|
return version
|
||||||
|
except urllib.error.HTTPError as exc:
|
||||||
|
if exc.code == 404:
|
||||||
|
return fallback
|
||||||
|
raise WatcherError(f"gateway_state_http_{exc.code}") from exc
|
||||||
|
except urllib.error.URLError as exc:
|
||||||
|
raise WatcherError("gateway_state_unavailable") from exc
|
||||||
|
|
||||||
|
def _durable_event(self, command: dict[str, Any], column: str, event: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
with self._db() as db:
|
||||||
|
row = db.execute(f"SELECT {column} FROM commands WHERE command_id=?", (command["command_id"],)).fetchone()
|
||||||
|
if row and row[column]:
|
||||||
|
return json.loads(row[column])
|
||||||
|
if self._uses_gateway:
|
||||||
|
event["state_version"] = self._gateway_state_version(command["project_key"], command["state_version"])
|
||||||
|
encoded = json.dumps(event, sort_keys=True, separators=(",", ":"))
|
||||||
|
with self._db() as db:
|
||||||
|
db.execute(f"UPDATE commands SET {column}=? WHERE command_id=? AND {column} IS NULL", (encoded, command["command_id"]))
|
||||||
|
saved = db.execute(f"SELECT {column} FROM commands WHERE command_id=?", (command["command_id"],)).fetchone()
|
||||||
|
if not saved or not saved[column]:
|
||||||
|
raise WatcherError("event_persistence_failed")
|
||||||
|
return json.loads(saved[column])
|
||||||
|
|
||||||
|
def _post_event(self, event: dict[str, Any]) -> None:
|
||||||
|
request = urllib.request.Request(
|
||||||
|
self.config.gateway_endpoint,
|
||||||
|
data=json.dumps(event, separators=(",", ":")).encode(),
|
||||||
|
headers={"Authorization": f"Bearer {self._gateway_token()}", "Content-Type": "application/json"},
|
||||||
|
method="POST",
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(request, timeout=15) as response:
|
||||||
|
if response.status not in {200, 201, 202}:
|
||||||
|
raise WatcherError(f"gateway_http_{response.status}")
|
||||||
|
except urllib.error.HTTPError as exc:
|
||||||
|
body = exc.read(16_384)
|
||||||
|
try:
|
||||||
|
reason = json.loads(body).get("error")
|
||||||
|
except (json.JSONDecodeError, UnicodeDecodeError, AttributeError):
|
||||||
|
reason = None
|
||||||
|
if exc.code == 409 and reason == "stale_state_version":
|
||||||
|
raise WatcherError("gateway_stale_state_version") from exc
|
||||||
|
raise WatcherError(f"gateway_http_{exc.code}") from exc
|
||||||
|
except urllib.error.URLError as exc:
|
||||||
|
raise WatcherError("gateway_unavailable") from exc
|
||||||
|
|
||||||
|
def _execute(self, command: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
execution = command["execution"]
|
||||||
|
if execution["kind"] == "acceptance_probe":
|
||||||
|
return {"kind": "acceptance_probe", "input_sha256": hashlib.sha256(execution["input"].encode()).hexdigest(), "exit_code": 0}
|
||||||
|
assert self.config.executor_path is not None
|
||||||
|
proc = subprocess.run(
|
||||||
|
[str(self.config.executor_path), *self.config.executor_args],
|
||||||
|
input=execution["input"], text=True, stdout=subprocess.DEVNULL,
|
||||||
|
stderr=subprocess.DEVNULL, timeout=self.config.executor_timeout_seconds,
|
||||||
|
check=False, env={"PATH": "/usr/bin:/bin", "LANG": "C.UTF-8"},
|
||||||
|
)
|
||||||
|
if proc.returncode != 0:
|
||||||
|
raise WatcherError(f"codex_executor_failed:{proc.returncode}")
|
||||||
|
return {"kind": "codex", "exit_code": 0}
|
||||||
|
|
||||||
|
def _preflight_execution(self, command: dict[str, Any]) -> None:
|
||||||
|
if command["execution"]["kind"] == "acceptance_probe":
|
||||||
|
return
|
||||||
|
if self.config.executor_path is None:
|
||||||
|
raise WatcherError("codex_executor_unavailable")
|
||||||
|
info = _secure_regular(self.config.executor_path)
|
||||||
|
if not info.st_mode & stat.S_IXUSR:
|
||||||
|
raise WatcherError("codex_executor_not_executable")
|
||||||
|
|
||||||
|
def _move(self, source: Path, target_dir: Path) -> None:
|
||||||
|
if not source.exists() and not source.is_symlink():
|
||||||
|
return
|
||||||
|
target = target_dir / source.name
|
||||||
|
if target.exists() or target.is_symlink():
|
||||||
|
target = target_dir / f"{source.stem}-{uuid.uuid4().hex[:8]}{source.suffix}"
|
||||||
|
os.replace(source, target)
|
||||||
|
|
||||||
|
def _reject_unregistered(self, path: Path, error: Exception) -> None:
|
||||||
|
command_id = path.stem[:128] if ID_RE.fullmatch(path.stem[:128]) else f"invalid-{uuid.uuid4().hex[:12]}"
|
||||||
|
self._audit("COMMAND_REJECTED", command_id, error=f"{type(error).__name__}:{str(error)[:240]}")
|
||||||
|
self._move(path, self.config.dead_letter_dir)
|
||||||
|
|
||||||
|
def process_path(self, path: Path) -> bool:
|
||||||
|
try:
|
||||||
|
command, digest = self._read_command(path)
|
||||||
|
registration = self._register(command, digest, path)
|
||||||
|
if registration == "duplicate_terminal":
|
||||||
|
self._move(path, self.config.completed_dir)
|
||||||
|
return False
|
||||||
|
row = self._claim(command["command_id"])
|
||||||
|
if row is None:
|
||||||
|
return False
|
||||||
|
phase = row["phase"]
|
||||||
|
result = json.loads(row["result_json"]) if row["result_json"] else None
|
||||||
|
if phase == "START":
|
||||||
|
start_event = self._durable_event(
|
||||||
|
command,
|
||||||
|
"start_event_json",
|
||||||
|
self._event(command, "work_unit_started", "RUNNING", "started"),
|
||||||
|
)
|
||||||
|
self.event_poster(start_event)
|
||||||
|
self._audit("TASK_STARTED", command["command_id"])
|
||||||
|
self._update(command["command_id"], status="PENDING", phase="EXECUTE")
|
||||||
|
phase = "EXECUTE"
|
||||||
|
if phase == "EXECUTE":
|
||||||
|
self._preflight_execution(command)
|
||||||
|
self._update(command["command_id"], status="CLAIMED", phase="EXECUTING")
|
||||||
|
phase = "EXECUTING"
|
||||||
|
result = self.executor(command)
|
||||||
|
self._audit("TASK_EXECUTED", command["command_id"], result=result)
|
||||||
|
self._update(command["command_id"], status="PENDING", phase="END", result=result)
|
||||||
|
phase = "END"
|
||||||
|
elif phase == "EXECUTING":
|
||||||
|
raise WatcherError("ambiguous_execution_not_repeated")
|
||||||
|
if phase == "END":
|
||||||
|
assert result is not None
|
||||||
|
end_event = self._durable_event(
|
||||||
|
command,
|
||||||
|
"end_event_json",
|
||||||
|
self._event(command, "work_unit_ended", "PASS", "ended", result),
|
||||||
|
)
|
||||||
|
self.event_poster(end_event)
|
||||||
|
self._audit("TASK_ENDED", command["command_id"])
|
||||||
|
self._audit("EVENT_POSTED", command["command_id"], event="work_unit_ended")
|
||||||
|
self._update(command["command_id"], status="COMPLETED", phase="DONE", result=result)
|
||||||
|
self._move(path, self.config.completed_dir)
|
||||||
|
return True
|
||||||
|
except Exception as exc:
|
||||||
|
try:
|
||||||
|
if isinstance(exc, WatcherError) and str(exc) in {"duplicate_command_payload_conflict", "duplicate_work_unit"}:
|
||||||
|
self._reject_unregistered(path, exc)
|
||||||
|
elif "command" in locals():
|
||||||
|
self._retry_or_dead(command["command_id"], path, exc)
|
||||||
|
else:
|
||||||
|
self._reject_unregistered(path, exc)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return False
|
||||||
|
|
||||||
|
def run_once(self) -> int:
|
||||||
|
failures = 0
|
||||||
|
for path in sorted(self.config.active_dir.glob("*.json")):
|
||||||
|
if not self.process_path(path):
|
||||||
|
failures += 1
|
||||||
|
return 1 if failures else 0
|
||||||
|
|
||||||
|
def run_forever(self) -> int:
|
||||||
|
while True:
|
||||||
|
self.run_once()
|
||||||
|
time.sleep(self.config.poll_seconds)
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
"""Reliable central orchestration gateway for AIOS."""
|
||||||
|
|
||||||
|
__version__ = "0.2.0"
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import logging
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from .config import Config
|
||||||
|
from .service import Runtime, serve
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description="AIOS Orchestrator Gateway")
|
||||||
|
parser.add_argument("--config", type=Path, required=True)
|
||||||
|
parser.add_argument("--log-level", default="INFO")
|
||||||
|
parser.add_argument(
|
||||||
|
"--poll-once",
|
||||||
|
action="store_true",
|
||||||
|
help="Run one inbox/model/outbox/OpenClaw synchronization polling cycle.",
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
logging.basicConfig(
|
||||||
|
level=getattr(logging, args.log_level.upper(), logging.INFO),
|
||||||
|
format="%(asctime)s %(levelname)s %(name)s %(message)s",
|
||||||
|
)
|
||||||
|
if args.poll_once:
|
||||||
|
runtime = Runtime(Config.from_file(args.config))
|
||||||
|
runtime.poll_once()
|
||||||
|
return 0
|
||||||
|
serve(args.config)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,178 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .errors import TriggerError
|
||||||
|
from .models import DECISION_SCHEMA, Decision, Event
|
||||||
|
|
||||||
|
|
||||||
|
class WorkspaceAgentClient:
|
||||||
|
"""Asynchronous Workspace Agent trigger.
|
||||||
|
|
||||||
|
The official endpoint returns only 202 and no run ID or retrievable output.
|
||||||
|
Completion therefore arrives through the gateway callback endpoint.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, trigger_id: str, token: str, timeout_seconds: int = 30):
|
||||||
|
self.trigger_id = trigger_id
|
||||||
|
self.token = token
|
||||||
|
self.timeout_seconds = timeout_seconds
|
||||||
|
|
||||||
|
def trigger(self, event: Event) -> None:
|
||||||
|
url = f"https://api.chatgpt.com/v1/workspace_agents/{self.trigger_id}/trigger"
|
||||||
|
prompt = (
|
||||||
|
"Process this AIOS orchestration event under the committed contract. "
|
||||||
|
"When complete, call submit_orchestration_decision with strict JSON. "
|
||||||
|
"Do not fabricate tests, Git state, secrets, evidence, or deployment facts.\n\n"
|
||||||
|
+ event.canonical_json()
|
||||||
|
)
|
||||||
|
body = json.dumps(
|
||||||
|
{
|
||||||
|
"conversation_key": event.project_key,
|
||||||
|
"input": prompt,
|
||||||
|
}
|
||||||
|
).encode()
|
||||||
|
request = urllib.request.Request(
|
||||||
|
url,
|
||||||
|
data=body,
|
||||||
|
method="POST",
|
||||||
|
headers={
|
||||||
|
"Authorization": f"Bearer {self.token}",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
"Idempotency-Key": event.event_id,
|
||||||
|
"User-Agent": "aios-orchestrator-gateway/0.2",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(request, timeout=self.timeout_seconds) as response:
|
||||||
|
response.read(1024)
|
||||||
|
if response.status != 202:
|
||||||
|
raise TriggerError(f"workspace_agent_http_{response.status}")
|
||||||
|
except urllib.error.HTTPError as exc:
|
||||||
|
raise TriggerError(f"workspace_agent_http_{exc.code}") from exc
|
||||||
|
except (urllib.error.URLError, TimeoutError, OSError) as exc:
|
||||||
|
raise TriggerError("workspace_agent_trigger_failed") from exc
|
||||||
|
|
||||||
|
|
||||||
|
class ResponsesClient:
|
||||||
|
"""Background Responses API client with polling and strict output validation."""
|
||||||
|
|
||||||
|
def __init__(self, api_key: str, model: str, timeout_seconds: int = 60):
|
||||||
|
self.api_key = api_key
|
||||||
|
self.model = model
|
||||||
|
self.timeout_seconds = timeout_seconds
|
||||||
|
|
||||||
|
def start(self, event: Event) -> str:
|
||||||
|
body = {
|
||||||
|
"model": self.model,
|
||||||
|
"background": True,
|
||||||
|
"instructions": (
|
||||||
|
"You are the AIOS central orchestration brain. Apply the exact "
|
||||||
|
"contract_commit in the event. Return only a strict decision. "
|
||||||
|
"Never fabricate evidence, secrets, Git facts, tests, approvals, "
|
||||||
|
"or deployment state. Keep external blockers fail-closed."
|
||||||
|
),
|
||||||
|
"input": event.canonical_json(),
|
||||||
|
"text": {
|
||||||
|
"format": {
|
||||||
|
"type": "json_schema",
|
||||||
|
"name": "aios_orchestrator_decision",
|
||||||
|
"strict": True,
|
||||||
|
"schema": DECISION_SCHEMA,
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"metadata": {
|
||||||
|
"event_id": event.event_id,
|
||||||
|
"project_key": event.project_key,
|
||||||
|
"contract_commit": event.contract_commit,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
payload = self._request(
|
||||||
|
"POST",
|
||||||
|
"https://api.openai.com/v1/responses",
|
||||||
|
body,
|
||||||
|
idempotency_key=event.event_id,
|
||||||
|
)
|
||||||
|
response_id = payload.get("id")
|
||||||
|
if not isinstance(response_id, str) or not response_id.startswith("resp_"):
|
||||||
|
raise TriggerError("responses_missing_id")
|
||||||
|
return response_id
|
||||||
|
|
||||||
|
def poll(self, response_id: str) -> tuple[str, Decision | None]:
|
||||||
|
payload = self._request(
|
||||||
|
"GET",
|
||||||
|
f"https://api.openai.com/v1/responses/{response_id}",
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
status = payload.get("status")
|
||||||
|
if status in {"queued", "in_progress"}:
|
||||||
|
return str(status), None
|
||||||
|
if status == "completed":
|
||||||
|
text = _extract_output_text(payload)
|
||||||
|
try:
|
||||||
|
return "completed", Decision.from_dict(json.loads(text))
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise TriggerError("responses_invalid_json") from exc
|
||||||
|
if status in {"failed", "cancelled", "incomplete"}:
|
||||||
|
raise TriggerError(f"responses_{status}")
|
||||||
|
raise TriggerError("responses_unknown_status")
|
||||||
|
|
||||||
|
def _request(
|
||||||
|
self,
|
||||||
|
method: str,
|
||||||
|
url: str,
|
||||||
|
body: dict[str, Any] | None,
|
||||||
|
*,
|
||||||
|
idempotency_key: str | None = None,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
encoded = json.dumps(body).encode() if body is not None else None
|
||||||
|
headers = {
|
||||||
|
"Authorization": f"Bearer {self.api_key}",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
"User-Agent": "aios-orchestrator-gateway/0.2",
|
||||||
|
}
|
||||||
|
if idempotency_key:
|
||||||
|
headers["Idempotency-Key"] = idempotency_key
|
||||||
|
request = urllib.request.Request(
|
||||||
|
url,
|
||||||
|
data=encoded,
|
||||||
|
method=method,
|
||||||
|
headers=headers,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(request, timeout=self.timeout_seconds) as response:
|
||||||
|
raw = response.read(4_194_304)
|
||||||
|
if response.status not in {200, 201}:
|
||||||
|
raise TriggerError(f"responses_http_{response.status}")
|
||||||
|
except urllib.error.HTTPError as exc:
|
||||||
|
raise TriggerError(f"responses_http_{exc.code}") from exc
|
||||||
|
except (urllib.error.URLError, TimeoutError, OSError) as exc:
|
||||||
|
raise TriggerError("responses_request_failed") from exc
|
||||||
|
try:
|
||||||
|
payload = json.loads(raw)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise TriggerError("responses_invalid_payload") from exc
|
||||||
|
if not isinstance(payload, dict):
|
||||||
|
raise TriggerError("responses_payload_not_object")
|
||||||
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def _extract_output_text(payload: dict[str, Any]) -> str:
|
||||||
|
output = payload.get("output")
|
||||||
|
if not isinstance(output, list):
|
||||||
|
raise TriggerError("responses_missing_output")
|
||||||
|
for item in output:
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
continue
|
||||||
|
content = item.get("content")
|
||||||
|
if not isinstance(content, list):
|
||||||
|
continue
|
||||||
|
for part in content:
|
||||||
|
if isinstance(part, dict) and part.get("type") == "output_text":
|
||||||
|
text = part.get("text")
|
||||||
|
if isinstance(text, str):
|
||||||
|
return text
|
||||||
|
raise TriggerError("responses_missing_output_text")
|
||||||
@@ -0,0 +1,203 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .errors import ConfigurationError
|
||||||
|
from .securefs import read_secure_json, read_secure_text
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Config:
|
||||||
|
host: str
|
||||||
|
port: int
|
||||||
|
project_key: str
|
||||||
|
database_path: Path
|
||||||
|
event_inbox_dir: Path
|
||||||
|
decision_inbox_dir: Path
|
||||||
|
dispatch_outbox_dir: Path
|
||||||
|
openclaw_sync_dir: Path
|
||||||
|
command_queue_dir: Path
|
||||||
|
inbound_token_file: Path
|
||||||
|
callback_token_file: Path
|
||||||
|
contract_commit: str
|
||||||
|
workspace_trigger_id: str | None
|
||||||
|
workspace_token_file: Path | None
|
||||||
|
responses_api_key_file: Path | None
|
||||||
|
responses_model: str | None
|
||||||
|
poll_interval_seconds: int
|
||||||
|
summary_interval_seconds: int
|
||||||
|
callback_timeout_seconds: int
|
||||||
|
lease_seconds: int
|
||||||
|
retry_base_seconds: int
|
||||||
|
max_retries: int
|
||||||
|
max_same_blocker_retries: int
|
||||||
|
max_hops_per_gate: int
|
||||||
|
openclaw_sync_endpoint: str | None
|
||||||
|
openclaw_sync_token_file: Path | None
|
||||||
|
git_poll_enabled: bool
|
||||||
|
git_poll_bare_repo: Path | None
|
||||||
|
git_poll_ref_prefixes: tuple[str, ...]
|
||||||
|
git_poll_interval_seconds: int
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_file(cls, path: Path) -> "Config":
|
||||||
|
owner_uid = int(os.environ.get("AIOS_ORCHESTRATOR_CONFIG_OWNER_UID", "0"))
|
||||||
|
raw = read_secure_json(path, expected_uid=owner_uid)
|
||||||
|
expected = {
|
||||||
|
"host", "port", "project_key", "database_path", "event_inbox_dir",
|
||||||
|
"decision_inbox_dir", "dispatch_outbox_dir", "openclaw_sync_dir",
|
||||||
|
"command_queue_dir", "inbound_token_file", "callback_token_file",
|
||||||
|
"contract_commit", "workspace_agent", "responses_fallback",
|
||||||
|
"polling", "loop_controls", "openclaw_sync", "git_poll",
|
||||||
|
}
|
||||||
|
if set(raw) != expected:
|
||||||
|
raise ConfigurationError("invalid_config_fields")
|
||||||
|
|
||||||
|
def path_value(key: str) -> Path:
|
||||||
|
value = raw.get(key)
|
||||||
|
if not isinstance(value, str):
|
||||||
|
raise ConfigurationError(f"invalid_{key}")
|
||||||
|
result = Path(value)
|
||||||
|
if not result.is_absolute():
|
||||||
|
raise ConfigurationError(f"path_not_absolute:{key}")
|
||||||
|
return result
|
||||||
|
|
||||||
|
def integer(container: dict[str, Any], key: str, lo: int, hi: int) -> int:
|
||||||
|
value = container.get(key)
|
||||||
|
if not isinstance(value, int) or not lo <= value <= hi:
|
||||||
|
raise ConfigurationError(f"invalid_{key}")
|
||||||
|
return value
|
||||||
|
|
||||||
|
host = raw.get("host")
|
||||||
|
port = raw.get("port")
|
||||||
|
project_key = raw.get("project_key")
|
||||||
|
contract = raw.get("contract_commit")
|
||||||
|
if not isinstance(host, str) or not host:
|
||||||
|
raise ConfigurationError("invalid_host")
|
||||||
|
if not isinstance(project_key, str) or not project_key:
|
||||||
|
raise ConfigurationError("invalid_project_key")
|
||||||
|
if not isinstance(port, int) or not 1 <= port <= 65535:
|
||||||
|
raise ConfigurationError("invalid_port")
|
||||||
|
if not isinstance(contract, str) or len(contract) != 40:
|
||||||
|
raise ConfigurationError("invalid_contract_commit")
|
||||||
|
|
||||||
|
workspace = raw.get("workspace_agent")
|
||||||
|
responses = raw.get("responses_fallback")
|
||||||
|
polling = raw.get("polling")
|
||||||
|
loop = raw.get("loop_controls")
|
||||||
|
sync = raw.get("openclaw_sync")
|
||||||
|
git_poll = raw.get("git_poll")
|
||||||
|
if not all(isinstance(x, dict) for x in (workspace, responses, polling, loop, sync, git_poll)):
|
||||||
|
raise ConfigurationError("invalid_nested_config")
|
||||||
|
|
||||||
|
trigger_id = workspace.get("trigger_id")
|
||||||
|
workspace_token = workspace.get("access_token_file")
|
||||||
|
if trigger_id is not None:
|
||||||
|
if not isinstance(trigger_id, str) or not trigger_id.startswith("agtch_"):
|
||||||
|
raise ConfigurationError("invalid_workspace_trigger_id")
|
||||||
|
if not isinstance(workspace_token, str) or not Path(workspace_token).is_absolute():
|
||||||
|
raise ConfigurationError("invalid_workspace_token_file")
|
||||||
|
workspace_token_file = Path(workspace_token)
|
||||||
|
else:
|
||||||
|
workspace_token_file = None
|
||||||
|
|
||||||
|
responses_key = responses.get("api_key_file")
|
||||||
|
responses_model = responses.get("model")
|
||||||
|
if responses_key is not None:
|
||||||
|
if not isinstance(responses_key, str) or not Path(responses_key).is_absolute():
|
||||||
|
raise ConfigurationError("invalid_responses_key_file")
|
||||||
|
if not isinstance(responses_model, str) or not responses_model:
|
||||||
|
raise ConfigurationError("invalid_responses_model")
|
||||||
|
responses_key_file = Path(responses_key)
|
||||||
|
else:
|
||||||
|
responses_key_file = None
|
||||||
|
responses_model = None
|
||||||
|
|
||||||
|
if trigger_id is None and responses_key_file is None:
|
||||||
|
raise ConfigurationError("no_central_model_channel")
|
||||||
|
|
||||||
|
endpoint = sync.get("endpoint")
|
||||||
|
token_file = sync.get("token_file")
|
||||||
|
if endpoint is not None and (not isinstance(endpoint, str) or not endpoint.startswith(("http://", "https://"))):
|
||||||
|
raise ConfigurationError("invalid_openclaw_sync_endpoint")
|
||||||
|
if endpoint is not None:
|
||||||
|
if not isinstance(token_file, str) or not Path(token_file).is_absolute():
|
||||||
|
raise ConfigurationError("invalid_openclaw_sync_token_file")
|
||||||
|
sync_token_file = Path(token_file)
|
||||||
|
else:
|
||||||
|
sync_token_file = None
|
||||||
|
|
||||||
|
git_enabled = git_poll.get("enabled")
|
||||||
|
if not isinstance(git_enabled, bool):
|
||||||
|
raise ConfigurationError("invalid_git_poll_enabled")
|
||||||
|
git_bare_repo_raw = git_poll.get("bare_repo")
|
||||||
|
git_prefixes_raw = git_poll.get("ref_prefixes")
|
||||||
|
if git_enabled:
|
||||||
|
if not isinstance(git_bare_repo_raw, str) or not Path(git_bare_repo_raw).is_absolute():
|
||||||
|
raise ConfigurationError("invalid_git_poll_bare_repo")
|
||||||
|
if (
|
||||||
|
not isinstance(git_prefixes_raw, list)
|
||||||
|
or not git_prefixes_raw
|
||||||
|
or any(not isinstance(item, str) or not item.startswith("refs/") for item in git_prefixes_raw)
|
||||||
|
):
|
||||||
|
raise ConfigurationError("invalid_git_poll_ref_prefixes")
|
||||||
|
git_bare_repo = Path(git_bare_repo_raw)
|
||||||
|
git_prefixes = tuple(git_prefixes_raw)
|
||||||
|
else:
|
||||||
|
git_bare_repo = None
|
||||||
|
git_prefixes = ()
|
||||||
|
|
||||||
|
return cls(
|
||||||
|
host=host,
|
||||||
|
port=port,
|
||||||
|
project_key=project_key,
|
||||||
|
database_path=path_value("database_path"),
|
||||||
|
event_inbox_dir=path_value("event_inbox_dir"),
|
||||||
|
decision_inbox_dir=path_value("decision_inbox_dir"),
|
||||||
|
dispatch_outbox_dir=path_value("dispatch_outbox_dir"),
|
||||||
|
openclaw_sync_dir=path_value("openclaw_sync_dir"),
|
||||||
|
command_queue_dir=path_value("command_queue_dir"),
|
||||||
|
inbound_token_file=path_value("inbound_token_file"),
|
||||||
|
callback_token_file=path_value("callback_token_file"),
|
||||||
|
contract_commit=contract,
|
||||||
|
workspace_trigger_id=trigger_id,
|
||||||
|
workspace_token_file=workspace_token_file,
|
||||||
|
responses_api_key_file=responses_key_file,
|
||||||
|
responses_model=responses_model,
|
||||||
|
poll_interval_seconds=integer(polling, "interval_seconds", 1, 300),
|
||||||
|
summary_interval_seconds=integer(polling, "summary_interval_seconds", 10, 86400),
|
||||||
|
callback_timeout_seconds=integer(loop, "callback_timeout_seconds", 30, 86400),
|
||||||
|
lease_seconds=integer(loop, "lease_seconds", 10, 3600),
|
||||||
|
retry_base_seconds=integer(loop, "retry_base_seconds", 1, 3600),
|
||||||
|
max_retries=integer(loop, "max_retries", 0, 20),
|
||||||
|
max_same_blocker_retries=integer(loop, "max_same_blocker_retries", 0, 20),
|
||||||
|
max_hops_per_gate=integer(loop, "max_hops_per_gate", 1, 100),
|
||||||
|
openclaw_sync_endpoint=endpoint,
|
||||||
|
openclaw_sync_token_file=sync_token_file,
|
||||||
|
git_poll_enabled=git_enabled,
|
||||||
|
git_poll_bare_repo=git_bare_repo,
|
||||||
|
git_poll_ref_prefixes=git_prefixes,
|
||||||
|
git_poll_interval_seconds=integer(git_poll, "interval_seconds", 5, 3600),
|
||||||
|
)
|
||||||
|
|
||||||
|
def _secret(self, path: Path) -> str:
|
||||||
|
owner_uid = int(os.environ.get("AIOS_ORCHESTRATOR_SECRET_OWNER_UID", "0"))
|
||||||
|
return read_secure_text(path, expected_uid=owner_uid, max_bytes=64_000)
|
||||||
|
|
||||||
|
def inbound_token(self) -> str:
|
||||||
|
return self._secret(self.inbound_token_file)
|
||||||
|
|
||||||
|
def callback_token(self) -> str:
|
||||||
|
return self._secret(self.callback_token_file)
|
||||||
|
|
||||||
|
def workspace_token(self) -> str | None:
|
||||||
|
return self._secret(self.workspace_token_file) if self.workspace_token_file else None
|
||||||
|
|
||||||
|
def responses_api_key(self) -> str | None:
|
||||||
|
return self._secret(self.responses_api_key_file) if self.responses_api_key_file else None
|
||||||
|
|
||||||
|
def openclaw_sync_token(self) -> str | None:
|
||||||
|
return self._secret(self.openclaw_sync_token_file) if self.openclaw_sync_token_file else None
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import re
|
||||||
|
import os
|
||||||
|
import stat
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .errors import DispatchError
|
||||||
|
from .securefs import (
|
||||||
|
atomic_write_bytes,
|
||||||
|
atomic_write_bytes_shared,
|
||||||
|
ensure_secure_directory,
|
||||||
|
ensure_shared_directory,
|
||||||
|
read_secure_bytes,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
SAFE_ID = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$")
|
||||||
|
|
||||||
|
|
||||||
|
class DispatchSink:
|
||||||
|
"""Reliable file-based dispatch adapter for existing OpenClaw/Codex watchers."""
|
||||||
|
|
||||||
|
def __init__(self, outbox_dir: Path, command_queue_dir: Path, contract_commit: str):
|
||||||
|
self.outbox_dir = Path(outbox_dir)
|
||||||
|
self.command_queue_dir = Path(command_queue_dir)
|
||||||
|
self.contract_commit = contract_commit
|
||||||
|
ensure_secure_directory(self.outbox_dir)
|
||||||
|
ensure_shared_directory(self.command_queue_dir)
|
||||||
|
|
||||||
|
def dispatch(self, record: dict[str, Any]) -> tuple[Path, Path]:
|
||||||
|
work_unit_id = record.get("work_unit_id")
|
||||||
|
event_id = record.get("event_id")
|
||||||
|
agent = record.get("assigned_agent")
|
||||||
|
payload_json = record.get("payload_json")
|
||||||
|
if (
|
||||||
|
not isinstance(work_unit_id, str)
|
||||||
|
or not SAFE_ID.fullmatch(work_unit_id)
|
||||||
|
or not isinstance(event_id, str)
|
||||||
|
or not SAFE_ID.fullmatch(event_id)
|
||||||
|
or agent not in {"openclaw", "codex"}
|
||||||
|
or not isinstance(payload_json, str)
|
||||||
|
):
|
||||||
|
raise DispatchError("invalid_dispatch_record")
|
||||||
|
try:
|
||||||
|
work_unit = json.loads(payload_json)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise DispatchError("invalid_dispatch_payload") from exc
|
||||||
|
|
||||||
|
envelope = {
|
||||||
|
"schema_version": 1,
|
||||||
|
"contract_commit": self.contract_commit,
|
||||||
|
"event_id": event_id,
|
||||||
|
"assigned_agent": agent,
|
||||||
|
"work_unit": work_unit,
|
||||||
|
"callback": {
|
||||||
|
"event_endpoint": "/v1/orchestrator/events",
|
||||||
|
"start_event_type": "work_unit_started",
|
||||||
|
"end_event_type": "work_unit_completed",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
outbox_path = self.outbox_dir / "dispatched" / f"{work_unit_id}.json"
|
||||||
|
queue_path = self.command_queue_dir / agent / "pending" / f"{work_unit_id}.json"
|
||||||
|
ensure_secure_directory(outbox_path.parent)
|
||||||
|
ensure_shared_directory(queue_path.parent)
|
||||||
|
|
||||||
|
# Idempotent immutable payload: existing identical file is accepted.
|
||||||
|
self._write_idempotent_json(outbox_path, envelope, shared=False)
|
||||||
|
self._write_idempotent_json(queue_path, envelope, shared=True)
|
||||||
|
return outbox_path, queue_path
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _write_idempotent_json(path: Path, value: dict[str, Any], *, shared: bool) -> None:
|
||||||
|
payload = (json.dumps(value, indent=2, sort_keys=True) + "\n").encode()
|
||||||
|
try:
|
||||||
|
st = os.lstat(path)
|
||||||
|
except FileNotFoundError:
|
||||||
|
if shared:
|
||||||
|
atomic_write_bytes_shared(path, payload, mode=0o640)
|
||||||
|
else:
|
||||||
|
atomic_write_bytes(path, payload, mode=0o600)
|
||||||
|
return
|
||||||
|
if not stat.S_ISREG(st.st_mode):
|
||||||
|
raise DispatchError(f"dispatch_target_not_regular:{path.name}")
|
||||||
|
existing = read_secure_bytes(path, expected_uid=None, max_bytes=1_048_576)
|
||||||
|
if existing != payload:
|
||||||
|
raise DispatchError(f"dispatch_idempotency_conflict:{path.name}")
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
class OrchestratorError(Exception):
|
||||||
|
"""Base orchestration error."""
|
||||||
|
|
||||||
|
|
||||||
|
class ConfigurationError(OrchestratorError):
|
||||||
|
"""Unsafe or invalid configuration."""
|
||||||
|
|
||||||
|
|
||||||
|
class ValidationError(OrchestratorError):
|
||||||
|
"""Invalid event or decision."""
|
||||||
|
|
||||||
|
|
||||||
|
class AuthenticationError(OrchestratorError):
|
||||||
|
"""Authentication failed."""
|
||||||
|
|
||||||
|
|
||||||
|
class ConflictError(OrchestratorError):
|
||||||
|
"""Idempotency, lease, or optimistic concurrency conflict."""
|
||||||
|
|
||||||
|
|
||||||
|
class TriggerError(OrchestratorError):
|
||||||
|
"""Central model trigger failed."""
|
||||||
|
|
||||||
|
|
||||||
|
class DispatchError(OrchestratorError):
|
||||||
|
"""Dispatch or synchronization output failed."""
|
||||||
@@ -0,0 +1,346 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .errors import ValidationError
|
||||||
|
|
||||||
|
|
||||||
|
OID_RE = re.compile(r"^[0-9a-f]{40}$")
|
||||||
|
ID_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._:-]{0,127}$")
|
||||||
|
AGENTS = {"openclaw", "codex", "gitea", "system", "workspace-agent"}
|
||||||
|
EVENT_STATUSES = {"READY", "RUNNING", "PASS", "BLOCKED", "FAIL", "TIMEOUT", "NOT_EXECUTED"}
|
||||||
|
VERDICTS = {"DISPATCH", "BLOCKED", "COMPLETE", "NOOP"}
|
||||||
|
EXECUTORS = {"openclaw", "codex"}
|
||||||
|
|
||||||
|
|
||||||
|
def req_str(value: dict[str, Any], key: str, pattern: re.Pattern[str] | None = None) -> str:
|
||||||
|
item = value.get(key)
|
||||||
|
if not isinstance(item, str) or not item:
|
||||||
|
raise ValidationError(f"invalid_{key}")
|
||||||
|
if pattern is not None and not pattern.fullmatch(item):
|
||||||
|
raise ValidationError(f"invalid_{key}")
|
||||||
|
return item
|
||||||
|
|
||||||
|
|
||||||
|
def opt_oid(item: Any, key: str) -> str | None:
|
||||||
|
if item is None:
|
||||||
|
return None
|
||||||
|
if not isinstance(item, str) or not OID_RE.fullmatch(item):
|
||||||
|
raise ValidationError(f"invalid_{key}")
|
||||||
|
return item
|
||||||
|
|
||||||
|
|
||||||
|
def string_list(value: dict[str, Any], key: str, limit: int) -> tuple[str, ...]:
|
||||||
|
raw = value.get(key)
|
||||||
|
if not isinstance(raw, list) or len(raw) > limit:
|
||||||
|
raise ValidationError(f"invalid_{key}")
|
||||||
|
result: list[str] = []
|
||||||
|
for item in raw:
|
||||||
|
if not isinstance(item, str) or not item or len(item) > 1024:
|
||||||
|
raise ValidationError(f"invalid_{key}")
|
||||||
|
result.append(item)
|
||||||
|
return tuple(result)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Event:
|
||||||
|
schema_version: int
|
||||||
|
project_key: str
|
||||||
|
contract_commit: str
|
||||||
|
event_id: str
|
||||||
|
source_agent: str
|
||||||
|
event_type: str
|
||||||
|
work_unit_id: str
|
||||||
|
status: str
|
||||||
|
state_version: int
|
||||||
|
implementation_commit: str | None = None
|
||||||
|
result_commit: str | None = None
|
||||||
|
blocker: str | None = None
|
||||||
|
evidence_refs: tuple[str, ...] = ()
|
||||||
|
payload: dict[str, Any] = field(default_factory=dict)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_dict(cls, value: dict[str, Any]) -> "Event":
|
||||||
|
allowed = {
|
||||||
|
"schema_version", "project_key", "contract_commit", "event_id",
|
||||||
|
"source_agent", "event_type", "work_unit_id", "status",
|
||||||
|
"state_version", "implementation_commit", "result_commit",
|
||||||
|
"blocker", "evidence_refs", "payload",
|
||||||
|
}
|
||||||
|
if not isinstance(value, dict) or set(value) - allowed:
|
||||||
|
raise ValidationError("invalid_event_fields")
|
||||||
|
if value.get("schema_version") != 1:
|
||||||
|
raise ValidationError("unsupported_event_schema")
|
||||||
|
source = req_str(value, "source_agent")
|
||||||
|
status = req_str(value, "status")
|
||||||
|
if source not in AGENTS:
|
||||||
|
raise ValidationError("invalid_source_agent")
|
||||||
|
if status not in EVENT_STATUSES:
|
||||||
|
raise ValidationError("invalid_event_status")
|
||||||
|
version = value.get("state_version")
|
||||||
|
if not isinstance(version, int) or version < 0:
|
||||||
|
raise ValidationError("invalid_state_version")
|
||||||
|
blocker = value.get("blocker")
|
||||||
|
if blocker is not None and (not isinstance(blocker, str) or len(blocker) > 512):
|
||||||
|
raise ValidationError("invalid_blocker")
|
||||||
|
payload = value.get("payload", {})
|
||||||
|
if not isinstance(payload, dict):
|
||||||
|
raise ValidationError("invalid_payload")
|
||||||
|
if len(json.dumps(payload, sort_keys=True).encode()) > 262_144:
|
||||||
|
raise ValidationError("payload_too_large")
|
||||||
|
return cls(
|
||||||
|
schema_version=1,
|
||||||
|
project_key=req_str(value, "project_key", ID_RE),
|
||||||
|
contract_commit=req_str(value, "contract_commit", OID_RE),
|
||||||
|
event_id=req_str(value, "event_id", ID_RE),
|
||||||
|
source_agent=source,
|
||||||
|
event_type=req_str(value, "event_type", ID_RE),
|
||||||
|
work_unit_id=req_str(value, "work_unit_id", ID_RE),
|
||||||
|
status=status,
|
||||||
|
state_version=version,
|
||||||
|
implementation_commit=opt_oid(value.get("implementation_commit"), "implementation_commit"),
|
||||||
|
result_commit=opt_oid(value.get("result_commit"), "result_commit"),
|
||||||
|
blocker=blocker,
|
||||||
|
evidence_refs=string_list(value, "evidence_refs", 64),
|
||||||
|
payload=payload,
|
||||||
|
)
|
||||||
|
|
||||||
|
def to_dict(self) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"schema_version": 1,
|
||||||
|
"project_key": self.project_key,
|
||||||
|
"contract_commit": self.contract_commit,
|
||||||
|
"event_id": self.event_id,
|
||||||
|
"source_agent": self.source_agent,
|
||||||
|
"event_type": self.event_type,
|
||||||
|
"work_unit_id": self.work_unit_id,
|
||||||
|
"status": self.status,
|
||||||
|
"state_version": self.state_version,
|
||||||
|
"implementation_commit": self.implementation_commit,
|
||||||
|
"result_commit": self.result_commit,
|
||||||
|
"blocker": self.blocker,
|
||||||
|
"evidence_refs": list(self.evidence_refs),
|
||||||
|
"payload": self.payload,
|
||||||
|
}
|
||||||
|
|
||||||
|
def canonical_json(self) -> str:
|
||||||
|
return json.dumps(self.to_dict(), separators=(",", ":"), sort_keys=True)
|
||||||
|
|
||||||
|
def digest(self) -> str:
|
||||||
|
return hashlib.sha256(self.canonical_json().encode()).hexdigest()
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class WorkUnit:
|
||||||
|
id: str
|
||||||
|
track_id: str
|
||||||
|
gate_id: str
|
||||||
|
primary_agent: str
|
||||||
|
fallback_agent: str
|
||||||
|
clean_base_commit: str
|
||||||
|
allowed_paths: tuple[str, ...]
|
||||||
|
required_checks: tuple[str, ...]
|
||||||
|
forbidden_operations: tuple[str, ...]
|
||||||
|
instruction: str
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_dict(cls, value: dict[str, Any]) -> "WorkUnit":
|
||||||
|
expected = {
|
||||||
|
"id", "track_id", "gate_id", "primary_agent", "fallback_agent",
|
||||||
|
"clean_base_commit", "allowed_paths", "required_checks",
|
||||||
|
"forbidden_operations", "instruction",
|
||||||
|
}
|
||||||
|
if not isinstance(value, dict) or set(value) != expected:
|
||||||
|
raise ValidationError("invalid_work_unit_fields")
|
||||||
|
primary = req_str(value, "primary_agent")
|
||||||
|
fallback = req_str(value, "fallback_agent")
|
||||||
|
if primary not in EXECUTORS or fallback not in EXECUTORS or primary == fallback:
|
||||||
|
raise ValidationError("invalid_agent_assignment")
|
||||||
|
instruction = req_str(value, "instruction")
|
||||||
|
if len(instruction.encode()) > 65_536:
|
||||||
|
raise ValidationError("instruction_too_large")
|
||||||
|
return cls(
|
||||||
|
id=req_str(value, "id", ID_RE),
|
||||||
|
track_id=req_str(value, "track_id", ID_RE),
|
||||||
|
gate_id=req_str(value, "gate_id", ID_RE),
|
||||||
|
primary_agent=primary,
|
||||||
|
fallback_agent=fallback,
|
||||||
|
clean_base_commit=req_str(value, "clean_base_commit", OID_RE),
|
||||||
|
allowed_paths=string_list(value, "allowed_paths", 128),
|
||||||
|
required_checks=string_list(value, "required_checks", 128),
|
||||||
|
forbidden_operations=string_list(value, "forbidden_operations", 64),
|
||||||
|
instruction=instruction,
|
||||||
|
)
|
||||||
|
|
||||||
|
def to_dict(self) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"id": self.id,
|
||||||
|
"track_id": self.track_id,
|
||||||
|
"gate_id": self.gate_id,
|
||||||
|
"primary_agent": self.primary_agent,
|
||||||
|
"fallback_agent": self.fallback_agent,
|
||||||
|
"clean_base_commit": self.clean_base_commit,
|
||||||
|
"allowed_paths": list(self.allowed_paths),
|
||||||
|
"required_checks": list(self.required_checks),
|
||||||
|
"forbidden_operations": list(self.forbidden_operations),
|
||||||
|
"instruction": self.instruction,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Decision:
|
||||||
|
schema_version: int
|
||||||
|
project_key: str
|
||||||
|
contract_commit: str
|
||||||
|
event_id: str
|
||||||
|
expected_state_version: int
|
||||||
|
verdict: str
|
||||||
|
state_updates: dict[str, str]
|
||||||
|
next_work_units: tuple[WorkUnit, ...]
|
||||||
|
requires_human_input: bool
|
||||||
|
stop_reason: str | None
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_dict(cls, value: dict[str, Any]) -> "Decision":
|
||||||
|
expected = {
|
||||||
|
"schema_version", "project_key", "contract_commit", "event_id",
|
||||||
|
"expected_state_version", "verdict", "state_updates",
|
||||||
|
"next_work_units", "requires_human_input", "stop_reason",
|
||||||
|
}
|
||||||
|
if not isinstance(value, dict) or set(value) != expected:
|
||||||
|
raise ValidationError("invalid_decision_fields")
|
||||||
|
if value.get("schema_version") != 1:
|
||||||
|
raise ValidationError("unsupported_decision_schema")
|
||||||
|
version = value.get("expected_state_version")
|
||||||
|
if not isinstance(version, int) or version < 0:
|
||||||
|
raise ValidationError("invalid_expected_state_version")
|
||||||
|
verdict = req_str(value, "verdict")
|
||||||
|
if verdict not in VERDICTS:
|
||||||
|
raise ValidationError("invalid_verdict")
|
||||||
|
updates = value.get("state_updates")
|
||||||
|
normalized: dict[str, str] = {}
|
||||||
|
if isinstance(updates, dict):
|
||||||
|
update_items = [{"key": key, "value": item} for key, item in updates.items()]
|
||||||
|
elif isinstance(updates, list):
|
||||||
|
update_items = updates
|
||||||
|
else:
|
||||||
|
raise ValidationError("invalid_state_updates")
|
||||||
|
if len(update_items) > 128:
|
||||||
|
raise ValidationError("invalid_state_updates")
|
||||||
|
for update in update_items:
|
||||||
|
if not isinstance(update, dict) or set(update) != {"key", "value"}:
|
||||||
|
raise ValidationError("invalid_state_update")
|
||||||
|
key = update.get("key")
|
||||||
|
item = update.get("value")
|
||||||
|
if not isinstance(key, str) or not ID_RE.fullmatch(key):
|
||||||
|
raise ValidationError("invalid_state_update_key")
|
||||||
|
if not isinstance(item, str) or not item or len(item) > 256:
|
||||||
|
raise ValidationError("invalid_state_update_value")
|
||||||
|
if key in normalized:
|
||||||
|
raise ValidationError("duplicate_state_update_key")
|
||||||
|
normalized[key] = item
|
||||||
|
raw_units = value.get("next_work_units")
|
||||||
|
if not isinstance(raw_units, list) or len(raw_units) > 32:
|
||||||
|
raise ValidationError("invalid_next_work_units")
|
||||||
|
units = tuple(WorkUnit.from_dict(item) for item in raw_units)
|
||||||
|
human = value.get("requires_human_input")
|
||||||
|
if not isinstance(human, bool):
|
||||||
|
raise ValidationError("invalid_requires_human_input")
|
||||||
|
stop = value.get("stop_reason")
|
||||||
|
if stop is not None and (not isinstance(stop, str) or len(stop) > 512):
|
||||||
|
raise ValidationError("invalid_stop_reason")
|
||||||
|
if verdict == "DISPATCH" and not units:
|
||||||
|
raise ValidationError("dispatch_without_work_units")
|
||||||
|
if verdict != "DISPATCH" and units:
|
||||||
|
raise ValidationError("work_units_without_dispatch")
|
||||||
|
return cls(
|
||||||
|
schema_version=1,
|
||||||
|
project_key=req_str(value, "project_key", ID_RE),
|
||||||
|
contract_commit=req_str(value, "contract_commit", OID_RE),
|
||||||
|
event_id=req_str(value, "event_id", ID_RE),
|
||||||
|
expected_state_version=version,
|
||||||
|
verdict=verdict,
|
||||||
|
state_updates=normalized,
|
||||||
|
next_work_units=units,
|
||||||
|
requires_human_input=human,
|
||||||
|
stop_reason=stop,
|
||||||
|
)
|
||||||
|
|
||||||
|
def to_dict(self) -> dict[str, Any]:
|
||||||
|
return {
|
||||||
|
"schema_version": 1,
|
||||||
|
"project_key": self.project_key,
|
||||||
|
"contract_commit": self.contract_commit,
|
||||||
|
"event_id": self.event_id,
|
||||||
|
"expected_state_version": self.expected_state_version,
|
||||||
|
"verdict": self.verdict,
|
||||||
|
"state_updates": [
|
||||||
|
{"key": key, "value": value}
|
||||||
|
for key, value in sorted(self.state_updates.items())
|
||||||
|
],
|
||||||
|
"next_work_units": [u.to_dict() for u in self.next_work_units],
|
||||||
|
"requires_human_input": self.requires_human_input,
|
||||||
|
"stop_reason": self.stop_reason,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
DECISION_SCHEMA: dict[str, Any] = {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": False,
|
||||||
|
"required": [
|
||||||
|
"schema_version", "project_key", "contract_commit", "event_id",
|
||||||
|
"expected_state_version", "verdict", "state_updates",
|
||||||
|
"next_work_units", "requires_human_input", "stop_reason",
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"schema_version": {"type": "integer", "const": 1},
|
||||||
|
"project_key": {"type": "string"},
|
||||||
|
"contract_commit": {"type": "string", "pattern": "^[0-9a-f]{40}$"},
|
||||||
|
"event_id": {"type": "string"},
|
||||||
|
"expected_state_version": {"type": "integer", "minimum": 0},
|
||||||
|
"verdict": {"type": "string", "enum": sorted(VERDICTS)},
|
||||||
|
"state_updates": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": False,
|
||||||
|
"required": ["key", "value"],
|
||||||
|
"properties": {
|
||||||
|
"key": {"type": "string"},
|
||||||
|
"value": {"type": "string"}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"next_work_units": {
|
||||||
|
"type": "array",
|
||||||
|
"maxItems": 32,
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": False,
|
||||||
|
"required": [
|
||||||
|
"id", "track_id", "gate_id", "primary_agent",
|
||||||
|
"fallback_agent", "clean_base_commit", "allowed_paths",
|
||||||
|
"required_checks", "forbidden_operations", "instruction",
|
||||||
|
],
|
||||||
|
"properties": {
|
||||||
|
"id": {"type": "string"},
|
||||||
|
"track_id": {"type": "string"},
|
||||||
|
"gate_id": {"type": "string"},
|
||||||
|
"primary_agent": {"type": "string", "enum": ["openclaw", "codex"]},
|
||||||
|
"fallback_agent": {"type": "string", "enum": ["openclaw", "codex"]},
|
||||||
|
"clean_base_commit": {"type": "string", "pattern": "^[0-9a-f]{40}$"},
|
||||||
|
"allowed_paths": {"type": "array", "items": {"type": "string"}},
|
||||||
|
"required_checks": {"type": "array", "items": {"type": "string"}},
|
||||||
|
"forbidden_operations": {"type": "array", "items": {"type": "string"}},
|
||||||
|
"instruction": {"type": "string"},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"requires_human_input": {"type": "boolean"},
|
||||||
|
"stop_reason": {"type": ["string", "null"]},
|
||||||
|
},
|
||||||
|
}
|
||||||
@@ -0,0 +1,229 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
import stat
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .errors import ConfigurationError
|
||||||
|
|
||||||
|
|
||||||
|
_NOFOLLOW = getattr(os, "O_NOFOLLOW", 0)
|
||||||
|
|
||||||
|
|
||||||
|
def reject_symlink_chain(path: Path) -> None:
|
||||||
|
"""Reject symlinks in every existing component of an absolute path."""
|
||||||
|
path = Path(path)
|
||||||
|
if not path.is_absolute():
|
||||||
|
raise ConfigurationError(f"path_not_absolute:{path}")
|
||||||
|
current = Path(path.anchor)
|
||||||
|
for part in path.parts[1:]:
|
||||||
|
current = current / part
|
||||||
|
try:
|
||||||
|
st = os.lstat(current)
|
||||||
|
except FileNotFoundError:
|
||||||
|
continue
|
||||||
|
if stat.S_ISLNK(st.st_mode):
|
||||||
|
raise ConfigurationError(f"symlink_rejected:{current}")
|
||||||
|
|
||||||
|
|
||||||
|
def verify_secure_file(
|
||||||
|
path: Path,
|
||||||
|
*,
|
||||||
|
expected_uid: int | None,
|
||||||
|
allowed_write_bits: int = stat.S_IWUSR,
|
||||||
|
require_executable: bool = False,
|
||||||
|
) -> os.stat_result:
|
||||||
|
path = Path(path)
|
||||||
|
reject_symlink_chain(path)
|
||||||
|
before = os.lstat(path)
|
||||||
|
if not stat.S_ISREG(before.st_mode):
|
||||||
|
raise ConfigurationError(f"not_regular_file:{path}")
|
||||||
|
if expected_uid is not None and before.st_uid != expected_uid:
|
||||||
|
raise ConfigurationError(f"unexpected_owner:{path}")
|
||||||
|
write_bits = before.st_mode & (stat.S_IWUSR | stat.S_IWGRP | stat.S_IWOTH)
|
||||||
|
if write_bits & ~allowed_write_bits:
|
||||||
|
raise ConfigurationError(f"permissive_mode:{path}")
|
||||||
|
if require_executable and not (before.st_mode & stat.S_IXUSR):
|
||||||
|
raise ConfigurationError(f"not_executable:{path}")
|
||||||
|
fd = os.open(path, os.O_RDONLY | _NOFOLLOW)
|
||||||
|
try:
|
||||||
|
after = os.fstat(fd)
|
||||||
|
if (before.st_dev, before.st_ino) != (after.st_dev, after.st_ino):
|
||||||
|
raise ConfigurationError(f"inode_changed:{path}")
|
||||||
|
finally:
|
||||||
|
os.close(fd)
|
||||||
|
return before
|
||||||
|
|
||||||
|
|
||||||
|
def read_secure_bytes(
|
||||||
|
path: Path,
|
||||||
|
*,
|
||||||
|
expected_uid: int | None,
|
||||||
|
max_bytes: int = 1_048_576,
|
||||||
|
) -> bytes:
|
||||||
|
path = Path(path)
|
||||||
|
before = verify_secure_file(path, expected_uid=expected_uid)
|
||||||
|
if before.st_size > max_bytes:
|
||||||
|
raise ConfigurationError(f"file_too_large:{path}")
|
||||||
|
fd = os.open(path, os.O_RDONLY | _NOFOLLOW)
|
||||||
|
try:
|
||||||
|
data = bytearray()
|
||||||
|
while True:
|
||||||
|
chunk = os.read(fd, min(65536, max_bytes + 1 - len(data)))
|
||||||
|
if not chunk:
|
||||||
|
break
|
||||||
|
data.extend(chunk)
|
||||||
|
if len(data) > max_bytes:
|
||||||
|
raise ConfigurationError(f"file_too_large:{path}")
|
||||||
|
return bytes(data)
|
||||||
|
finally:
|
||||||
|
os.close(fd)
|
||||||
|
|
||||||
|
|
||||||
|
def read_secure_text(path: Path, *, expected_uid: int | None, max_bytes: int = 1_048_576) -> str:
|
||||||
|
try:
|
||||||
|
return read_secure_bytes(path, expected_uid=expected_uid, max_bytes=max_bytes).decode("utf-8").strip()
|
||||||
|
except UnicodeDecodeError as exc:
|
||||||
|
raise ConfigurationError(f"not_utf8:{path}") from exc
|
||||||
|
|
||||||
|
|
||||||
|
def read_secure_json(path: Path, *, expected_uid: int | None, max_bytes: int = 1_048_576) -> dict[str, Any]:
|
||||||
|
try:
|
||||||
|
value = json.loads(read_secure_text(path, expected_uid=expected_uid, max_bytes=max_bytes))
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise ConfigurationError(f"invalid_json:{path}") from exc
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
raise ConfigurationError(f"json_root_not_object:{path}")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def ensure_secure_directory(path: Path, *, mode: int = 0o700) -> None:
|
||||||
|
path = Path(path)
|
||||||
|
if not path.is_absolute():
|
||||||
|
raise ConfigurationError(f"path_not_absolute:{path}")
|
||||||
|
reject_symlink_chain(path.parent)
|
||||||
|
path.mkdir(parents=True, exist_ok=True, mode=mode)
|
||||||
|
reject_symlink_chain(path)
|
||||||
|
st = os.lstat(path)
|
||||||
|
if not stat.S_ISDIR(st.st_mode):
|
||||||
|
raise ConfigurationError(f"not_directory:{path}")
|
||||||
|
if st.st_mode & (stat.S_IWGRP | stat.S_IWOTH):
|
||||||
|
raise ConfigurationError(f"permissive_directory:{path}")
|
||||||
|
|
||||||
|
|
||||||
|
def atomic_write_bytes(path: Path, payload: bytes, *, mode: int = 0o600) -> None:
|
||||||
|
"""Write through a verified directory fd and atomically replace the target."""
|
||||||
|
path = Path(path)
|
||||||
|
ensure_secure_directory(path.parent)
|
||||||
|
parent_before = os.lstat(path.parent)
|
||||||
|
parent_fd = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY | _NOFOLLOW)
|
||||||
|
temp_name = f".{path.name}.tmp-{os.getpid()}-{os.urandom(6).hex()}"
|
||||||
|
try:
|
||||||
|
parent_after = os.fstat(parent_fd)
|
||||||
|
if (parent_before.st_dev, parent_before.st_ino) != (
|
||||||
|
parent_after.st_dev,
|
||||||
|
parent_after.st_ino,
|
||||||
|
):
|
||||||
|
raise ConfigurationError(f"directory_changed:{path.parent}")
|
||||||
|
fd = os.open(
|
||||||
|
temp_name,
|
||||||
|
os.O_WRONLY | os.O_CREAT | os.O_EXCL | _NOFOLLOW,
|
||||||
|
mode,
|
||||||
|
dir_fd=parent_fd,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
view = memoryview(payload)
|
||||||
|
while view:
|
||||||
|
written = os.write(fd, view)
|
||||||
|
view = view[written:]
|
||||||
|
os.fsync(fd)
|
||||||
|
finally:
|
||||||
|
os.close(fd)
|
||||||
|
os.replace(
|
||||||
|
temp_name,
|
||||||
|
path.name,
|
||||||
|
src_dir_fd=parent_fd,
|
||||||
|
dst_dir_fd=parent_fd,
|
||||||
|
)
|
||||||
|
os.fsync(parent_fd)
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
os.unlink(temp_name, dir_fd=parent_fd)
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
os.close(parent_fd)
|
||||||
|
|
||||||
|
|
||||||
|
def atomic_write_json(path: Path, value: Any, *, mode: int = 0o600) -> None:
|
||||||
|
payload = (json.dumps(value, indent=2, sort_keys=True) + "\n").encode("utf-8")
|
||||||
|
atomic_write_bytes(path, payload, mode=mode)
|
||||||
|
|
||||||
|
def ensure_shared_directory(path: Path, *, group_writable: bool = False) -> None:
|
||||||
|
"""Verify/create a setgid queue directory shared with a consumer group."""
|
||||||
|
path = Path(path)
|
||||||
|
if not path.is_absolute():
|
||||||
|
raise ConfigurationError(f"path_not_absolute:{path}")
|
||||||
|
reject_symlink_chain(path.parent)
|
||||||
|
mode = 0o2770 if group_writable else 0o2750
|
||||||
|
try:
|
||||||
|
path.mkdir(parents=True, mode=mode)
|
||||||
|
except FileExistsError:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
os.chmod(path, mode)
|
||||||
|
reject_symlink_chain(path)
|
||||||
|
st = os.lstat(path)
|
||||||
|
if not stat.S_ISDIR(st.st_mode):
|
||||||
|
raise ConfigurationError(f"not_directory:{path}")
|
||||||
|
observed = stat.S_IMODE(st.st_mode)
|
||||||
|
required = stat.S_ISGID | stat.S_IRGRP | stat.S_IXGRP
|
||||||
|
if group_writable:
|
||||||
|
required |= stat.S_IWGRP
|
||||||
|
if observed & 0o007 or observed & required != required:
|
||||||
|
raise ConfigurationError(f"invalid_shared_directory_mode:{path}")
|
||||||
|
|
||||||
|
def atomic_write_bytes_shared(
|
||||||
|
path: Path,
|
||||||
|
payload: bytes,
|
||||||
|
*,
|
||||||
|
mode: int = 0o640,
|
||||||
|
group_writable_directory: bool = False,
|
||||||
|
) -> None:
|
||||||
|
"""Atomic write into a setgid/group-readable shared queue directory."""
|
||||||
|
path = Path(path)
|
||||||
|
ensure_shared_directory(path.parent, group_writable=group_writable_directory)
|
||||||
|
parent_before = os.lstat(path.parent)
|
||||||
|
parent_fd = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY | _NOFOLLOW)
|
||||||
|
temp_name = f".{path.name}.tmp-{os.getpid()}-{os.urandom(6).hex()}"
|
||||||
|
try:
|
||||||
|
parent_after = os.fstat(parent_fd)
|
||||||
|
if (parent_before.st_dev, parent_before.st_ino) != (
|
||||||
|
parent_after.st_dev,
|
||||||
|
parent_after.st_ino,
|
||||||
|
):
|
||||||
|
raise ConfigurationError(f"directory_changed:{path.parent}")
|
||||||
|
fd = os.open(
|
||||||
|
temp_name,
|
||||||
|
os.O_WRONLY | os.O_CREAT | os.O_EXCL | _NOFOLLOW,
|
||||||
|
mode,
|
||||||
|
dir_fd=parent_fd,
|
||||||
|
)
|
||||||
|
os.fchmod(fd, mode)
|
||||||
|
try:
|
||||||
|
view = memoryview(payload)
|
||||||
|
while view:
|
||||||
|
written = os.write(fd, view)
|
||||||
|
view = view[written:]
|
||||||
|
os.fsync(fd)
|
||||||
|
finally:
|
||||||
|
os.close(fd)
|
||||||
|
os.replace(temp_name, path.name, src_dir_fd=parent_fd, dst_dir_fd=parent_fd)
|
||||||
|
os.fsync(parent_fd)
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
os.unlink(temp_name, dir_fd=parent_fd)
|
||||||
|
except FileNotFoundError:
|
||||||
|
pass
|
||||||
|
os.close(parent_fd)
|
||||||
@@ -0,0 +1,642 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import hmac
|
||||||
|
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
import socket
|
||||||
|
import subprocess
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from typing import Any
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
from .clients import ResponsesClient, WorkspaceAgentClient
|
||||||
|
from .config import Config
|
||||||
|
from .dispatch import DispatchSink
|
||||||
|
from .errors import (
|
||||||
|
AuthenticationError,
|
||||||
|
ConflictError,
|
||||||
|
DispatchError,
|
||||||
|
OrchestratorError,
|
||||||
|
TriggerError,
|
||||||
|
ValidationError,
|
||||||
|
)
|
||||||
|
from .models import Decision, Event
|
||||||
|
from .securefs import ensure_secure_directory, ensure_shared_directory, read_secure_json
|
||||||
|
from .store import Store
|
||||||
|
from .sync import OpenClawSynchronizer, lifecycle_record
|
||||||
|
|
||||||
|
|
||||||
|
LOG = logging.getLogger("aios-orchestrator-gateway")
|
||||||
|
|
||||||
|
|
||||||
|
class Runtime:
|
||||||
|
def __init__(self, config: Config):
|
||||||
|
self.config = config
|
||||||
|
self.store = Store(config.database_path)
|
||||||
|
self.dispatch = DispatchSink(
|
||||||
|
config.dispatch_outbox_dir,
|
||||||
|
config.command_queue_dir,
|
||||||
|
config.contract_commit,
|
||||||
|
)
|
||||||
|
self.workspace = None
|
||||||
|
workspace_token = config.workspace_token()
|
||||||
|
if config.workspace_trigger_id and workspace_token:
|
||||||
|
self.workspace = WorkspaceAgentClient(
|
||||||
|
config.workspace_trigger_id,
|
||||||
|
workspace_token,
|
||||||
|
)
|
||||||
|
self.responses = None
|
||||||
|
responses_key = config.responses_api_key()
|
||||||
|
if responses_key and config.responses_model:
|
||||||
|
self.responses = ResponsesClient(
|
||||||
|
responses_key,
|
||||||
|
config.responses_model,
|
||||||
|
)
|
||||||
|
self.synchronizer = OpenClawSynchronizer(
|
||||||
|
config.openclaw_sync_dir,
|
||||||
|
endpoint=config.openclaw_sync_endpoint,
|
||||||
|
bearer_token=config.openclaw_sync_token(),
|
||||||
|
)
|
||||||
|
self.worker_id = f"{socket.gethostname()}-{os.getpid()}-{threading.get_native_id()}"
|
||||||
|
self.stop_event = threading.Event()
|
||||||
|
self.worker = threading.Thread(target=self._loop, daemon=True)
|
||||||
|
self._next_git_poll = 0.0
|
||||||
|
for directory in (
|
||||||
|
config.event_inbox_dir,
|
||||||
|
config.decision_inbox_dir,
|
||||||
|
config.dispatch_outbox_dir,
|
||||||
|
config.openclaw_sync_dir,
|
||||||
|
):
|
||||||
|
ensure_secure_directory(directory)
|
||||||
|
ensure_shared_directory(config.command_queue_dir)
|
||||||
|
|
||||||
|
def start(self) -> None:
|
||||||
|
self.worker.start()
|
||||||
|
|
||||||
|
def stop(self) -> None:
|
||||||
|
self.stop_event.set()
|
||||||
|
if self.worker.is_alive():
|
||||||
|
self.worker.join(timeout=15)
|
||||||
|
self.store.close()
|
||||||
|
|
||||||
|
def submit_event(self, payload: dict[str, Any]) -> tuple[bool, dict[str, Any]]:
|
||||||
|
event = Event.from_dict(payload)
|
||||||
|
if event.contract_commit != self.config.contract_commit:
|
||||||
|
raise ConflictError("contract_version_mismatch")
|
||||||
|
created, stored = self.store.register_event(event)
|
||||||
|
if created:
|
||||||
|
self._enqueue_event_lifecycle(event)
|
||||||
|
return created, stored
|
||||||
|
|
||||||
|
def submit_decision(self, payload: dict[str, Any]) -> int:
|
||||||
|
decision = Decision.from_dict(payload)
|
||||||
|
if decision.contract_commit != self.config.contract_commit:
|
||||||
|
raise ConflictError("contract_version_mismatch")
|
||||||
|
version = self.store.apply_decision(
|
||||||
|
decision,
|
||||||
|
max_hops_per_gate=self.config.max_hops_per_gate,
|
||||||
|
max_same_blocker_retries=self.config.max_same_blocker_retries,
|
||||||
|
)
|
||||||
|
self._enqueue_decision_lifecycle(decision, version)
|
||||||
|
return version
|
||||||
|
|
||||||
|
def _loop(self) -> None:
|
||||||
|
next_summary = 0.0
|
||||||
|
while not self.stop_event.wait(self.config.poll_interval_seconds):
|
||||||
|
try:
|
||||||
|
self.poll_once()
|
||||||
|
if time.monotonic() >= next_summary:
|
||||||
|
self.enqueue_periodic_summaries()
|
||||||
|
next_summary = time.monotonic() + self.config.summary_interval_seconds
|
||||||
|
except Exception:
|
||||||
|
LOG.exception("poll_cycle_failed")
|
||||||
|
|
||||||
|
def poll_once(self) -> None:
|
||||||
|
"""One deterministic reliability cycle.
|
||||||
|
|
||||||
|
This is deliberately callable from a systemd timer or health supervisor
|
||||||
|
in addition to the internal loop.
|
||||||
|
"""
|
||||||
|
self.ingest_file_inboxes()
|
||||||
|
self.poll_git_refs()
|
||||||
|
self.process_new_events()
|
||||||
|
self.process_workspace_timeouts()
|
||||||
|
self.poll_background_responses()
|
||||||
|
self.dispatch_work_units()
|
||||||
|
self.synchronize_openclaw()
|
||||||
|
|
||||||
|
def ingest_file_inboxes(self) -> None:
|
||||||
|
self._ingest_directory(self.config.event_inbox_dir, kind="event")
|
||||||
|
self._ingest_directory(self.config.decision_inbox_dir, kind="decision")
|
||||||
|
|
||||||
|
def _ingest_directory(self, directory: Path, *, kind: str) -> None:
|
||||||
|
pending = directory / "pending"
|
||||||
|
processed = directory / "processed"
|
||||||
|
rejected = directory / "rejected"
|
||||||
|
ensure_shared_directory(pending, group_writable=True)
|
||||||
|
ensure_secure_directory(processed)
|
||||||
|
ensure_secure_directory(rejected)
|
||||||
|
for path in sorted(pending.glob("*.json"))[:128]:
|
||||||
|
try:
|
||||||
|
payload = read_secure_json(path, expected_uid=None)
|
||||||
|
if kind == "event":
|
||||||
|
self.submit_event(payload)
|
||||||
|
else:
|
||||||
|
self.submit_decision(payload)
|
||||||
|
target = processed / path.name
|
||||||
|
except Exception as exc:
|
||||||
|
LOG.warning("file_inbox_rejected path=%s error=%s", path, type(exc).__name__)
|
||||||
|
target = rejected / path.name
|
||||||
|
if target.exists():
|
||||||
|
# Same file name is an idempotency replay. Keep original audit copy.
|
||||||
|
path.unlink()
|
||||||
|
else:
|
||||||
|
path.replace(target)
|
||||||
|
|
||||||
|
def poll_git_refs(self) -> None:
|
||||||
|
"""Poll authoritative local bare refs so missed webhooks cannot stall the loop."""
|
||||||
|
if not self.config.git_poll_enabled or self.config.git_poll_bare_repo is None:
|
||||||
|
return
|
||||||
|
now = time.monotonic()
|
||||||
|
if now < self._next_git_poll:
|
||||||
|
return
|
||||||
|
self._next_git_poll = now + self.config.git_poll_interval_seconds
|
||||||
|
command = [
|
||||||
|
"git",
|
||||||
|
f"--git-dir={self.config.git_poll_bare_repo}",
|
||||||
|
"for-each-ref",
|
||||||
|
"--format=%(refname) %(objectname)",
|
||||||
|
*self.config.git_poll_ref_prefixes,
|
||||||
|
]
|
||||||
|
try:
|
||||||
|
result = subprocess.run(
|
||||||
|
command,
|
||||||
|
stdout=subprocess.PIPE,
|
||||||
|
stderr=subprocess.PIPE,
|
||||||
|
text=True,
|
||||||
|
check=False,
|
||||||
|
timeout=30,
|
||||||
|
env={"PATH": "/usr/bin:/bin", "LANG": "C", "LC_ALL": "C"},
|
||||||
|
)
|
||||||
|
except (OSError, subprocess.SubprocessError):
|
||||||
|
LOG.exception("git_ref_poll_failed")
|
||||||
|
return
|
||||||
|
if result.returncode != 0:
|
||||||
|
LOG.warning("git_ref_poll_rc=%s", result.returncode)
|
||||||
|
return
|
||||||
|
refs: dict[str, str] = {}
|
||||||
|
for line in result.stdout.splitlines():
|
||||||
|
refname, sep, oid = line.partition(" ")
|
||||||
|
if not sep or not refname.startswith("refs/") or len(oid) != 40:
|
||||||
|
continue
|
||||||
|
if any(ch not in "0123456789abcdef" for ch in oid):
|
||||||
|
continue
|
||||||
|
refs[refname] = oid
|
||||||
|
initialized, changes = self.store.observe_refs(refs)
|
||||||
|
if not initialized or not changes:
|
||||||
|
return
|
||||||
|
payload_changes = [
|
||||||
|
{"ref": refname, "old_oid": old_oid, "new_oid": new_oid}
|
||||||
|
for refname, old_oid, new_oid in sorted(changes)
|
||||||
|
]
|
||||||
|
canonical = json.dumps(payload_changes, separators=(",", ":"), sort_keys=True)
|
||||||
|
event_id = "gitpoll-" + hashlib.sha256(canonical.encode()).hexdigest()[:40]
|
||||||
|
state = self.store.project_state(self.config.project_key)
|
||||||
|
state_version = 0 if state is None else int(state["state_version"])
|
||||||
|
event = Event.from_dict(
|
||||||
|
{
|
||||||
|
"schema_version": 1,
|
||||||
|
"project_key": self.config.project_key,
|
||||||
|
"contract_commit": self.config.contract_commit,
|
||||||
|
"event_id": event_id,
|
||||||
|
"source_agent": "gitea",
|
||||||
|
"event_type": "git_refs_changed",
|
||||||
|
"work_unit_id": "GIT-REF-POLL",
|
||||||
|
"status": "READY",
|
||||||
|
"state_version": state_version,
|
||||||
|
"implementation_commit": None,
|
||||||
|
"result_commit": None,
|
||||||
|
"blocker": None,
|
||||||
|
"evidence_refs": [],
|
||||||
|
"payload": {"changes": payload_changes},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
self.submit_event(event.to_dict())
|
||||||
|
except ConflictError:
|
||||||
|
LOG.info("git_ref_event_already_registered event_id=%s", event_id)
|
||||||
|
|
||||||
|
def process_new_events(self) -> None:
|
||||||
|
rows = self.store.claim_events(
|
||||||
|
owner=self.worker_id,
|
||||||
|
statuses=("RECEIVED", "TRIGGER_FAILED"),
|
||||||
|
limit=32,
|
||||||
|
lease_seconds=self.config.lease_seconds,
|
||||||
|
)
|
||||||
|
for row in rows:
|
||||||
|
event = Event.from_dict(json.loads(row["event_json"]))
|
||||||
|
attempts = int(row["attempts"])
|
||||||
|
try:
|
||||||
|
if self.workspace is not None:
|
||||||
|
self.workspace.trigger(event)
|
||||||
|
self.store.update_event_trigger(
|
||||||
|
event.event_id,
|
||||||
|
status="AWAITING_DECISION",
|
||||||
|
channel="workspace_agent",
|
||||||
|
remote_id=None,
|
||||||
|
)
|
||||||
|
elif self.responses is not None:
|
||||||
|
response_id = self.responses.start(event)
|
||||||
|
self.store.update_event_trigger(
|
||||||
|
event.event_id,
|
||||||
|
status="POLLING_RESPONSE",
|
||||||
|
channel="responses_api",
|
||||||
|
remote_id=response_id,
|
||||||
|
retry_after=self.config.poll_interval_seconds,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
raise TriggerError("no_model_channel")
|
||||||
|
except TriggerError as exc:
|
||||||
|
self._retry_or_dead_letter(event, attempts, str(exc))
|
||||||
|
|
||||||
|
def process_workspace_timeouts(self) -> None:
|
||||||
|
cutoff = int(time.time()) - self.config.callback_timeout_seconds
|
||||||
|
for row in self.store.claim_workspace_timeouts(
|
||||||
|
owner=self.worker_id,
|
||||||
|
older_than=cutoff,
|
||||||
|
limit=32,
|
||||||
|
lease_seconds=self.config.lease_seconds,
|
||||||
|
):
|
||||||
|
event = Event.from_dict(json.loads(row["event_json"]))
|
||||||
|
if self.responses is None:
|
||||||
|
self._retry_or_dead_letter(
|
||||||
|
event,
|
||||||
|
int(row["attempts"]),
|
||||||
|
"workspace_callback_timeout",
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
response_id = self.responses.start(event)
|
||||||
|
self.store.update_event_trigger(
|
||||||
|
event.event_id,
|
||||||
|
status="POLLING_RESPONSE",
|
||||||
|
channel="responses_api",
|
||||||
|
remote_id=response_id,
|
||||||
|
retry_after=self.config.poll_interval_seconds,
|
||||||
|
increment_attempt=True,
|
||||||
|
)
|
||||||
|
except TriggerError as exc:
|
||||||
|
self._retry_or_dead_letter(event, int(row["attempts"]), str(exc))
|
||||||
|
|
||||||
|
def poll_background_responses(self) -> None:
|
||||||
|
if self.responses is None:
|
||||||
|
return
|
||||||
|
for row in self.store.claim_response_polls(
|
||||||
|
owner=self.worker_id,
|
||||||
|
limit=32,
|
||||||
|
lease_seconds=self.config.lease_seconds,
|
||||||
|
):
|
||||||
|
event = Event.from_dict(json.loads(row["event_json"]))
|
||||||
|
response_id = row.get("remote_id")
|
||||||
|
if not isinstance(response_id, str):
|
||||||
|
self._retry_or_dead_letter(event, int(row["attempts"]), "missing_response_id")
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
status, decision = self.responses.poll(response_id)
|
||||||
|
if status in {"queued", "in_progress"}:
|
||||||
|
attempts = int(row["attempts"])
|
||||||
|
if attempts + 1 > self.config.max_retries:
|
||||||
|
self._retry_or_dead_letter(
|
||||||
|
event, attempts, f"response_{status}_retry_limit"
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
self.store.update_event_trigger(
|
||||||
|
event.event_id,
|
||||||
|
status="POLLING_RESPONSE",
|
||||||
|
channel="responses_api",
|
||||||
|
remote_id=response_id,
|
||||||
|
retry_after=self.config.poll_interval_seconds,
|
||||||
|
increment_attempt=True,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
if decision is None:
|
||||||
|
raise TriggerError("completed_without_decision")
|
||||||
|
self.submit_decision(decision.to_dict())
|
||||||
|
except OrchestratorError as exc:
|
||||||
|
self._retry_or_dead_letter(event, int(row["attempts"]), str(exc))
|
||||||
|
|
||||||
|
def dispatch_work_units(self) -> None:
|
||||||
|
rows = self.store.claim_outbox(
|
||||||
|
owner=self.worker_id,
|
||||||
|
limit=64,
|
||||||
|
lease_seconds=self.config.lease_seconds,
|
||||||
|
)
|
||||||
|
for row in rows:
|
||||||
|
try:
|
||||||
|
outbox_path, queue_path = self.dispatch.dispatch(row)
|
||||||
|
self.store.mark_outbox(row["work_unit_id"], status="DISPATCHED")
|
||||||
|
payload = {
|
||||||
|
"assigned_agent": row["assigned_agent"],
|
||||||
|
"event_id": row["event_id"],
|
||||||
|
"dispatch_outbox": str(outbox_path),
|
||||||
|
"command_queue": str(queue_path),
|
||||||
|
}
|
||||||
|
record = lifecycle_record(
|
||||||
|
contract_commit=self.config.contract_commit,
|
||||||
|
project_key=self._project_for_event(row["event_id"]),
|
||||||
|
kind="TASK_DISPATCHED",
|
||||||
|
work_unit_id=row["work_unit_id"],
|
||||||
|
payload=payload,
|
||||||
|
)
|
||||||
|
self.store.enqueue_lifecycle(
|
||||||
|
sync_id=record["sync_id"],
|
||||||
|
project_key=record["project_key"],
|
||||||
|
work_unit_id=row["work_unit_id"],
|
||||||
|
kind=record["kind"],
|
||||||
|
payload=record,
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
attempts = int(row["attempts"])
|
||||||
|
status = "DEAD_LETTER" if attempts >= self.config.max_retries else "RETRY"
|
||||||
|
self.store.mark_outbox(
|
||||||
|
row["work_unit_id"],
|
||||||
|
status=status,
|
||||||
|
retry_after=self._backoff(attempts),
|
||||||
|
increment_attempt=True,
|
||||||
|
)
|
||||||
|
LOG.exception("dispatch_failed work_unit=%s", row["work_unit_id"])
|
||||||
|
|
||||||
|
def synchronize_openclaw(self) -> None:
|
||||||
|
rows = self.store.claim_lifecycle(
|
||||||
|
owner=self.worker_id,
|
||||||
|
limit=64,
|
||||||
|
lease_seconds=self.config.lease_seconds,
|
||||||
|
)
|
||||||
|
for row in rows:
|
||||||
|
try:
|
||||||
|
payload = json.loads(row["payload_json"])
|
||||||
|
self.synchronizer.emit(payload)
|
||||||
|
self.store.mark_lifecycle(row["sync_id"], status="SENT")
|
||||||
|
except Exception:
|
||||||
|
attempts = int(row["attempts"])
|
||||||
|
status = "DEAD_LETTER" if attempts >= self.config.max_retries else "RETRY"
|
||||||
|
self.store.mark_lifecycle(
|
||||||
|
row["sync_id"],
|
||||||
|
status=status,
|
||||||
|
retry_after=self._backoff(attempts),
|
||||||
|
increment_attempt=True,
|
||||||
|
)
|
||||||
|
LOG.exception("openclaw_sync_failed sync_id=%s", row["sync_id"])
|
||||||
|
|
||||||
|
def enqueue_periodic_summaries(self) -> None:
|
||||||
|
for state in self.store.all_project_states():
|
||||||
|
project_key = state["project_key"]
|
||||||
|
if not self.store.summary_due(project_key, self.config.summary_interval_seconds):
|
||||||
|
continue
|
||||||
|
summary = self.store.project_summary(project_key)
|
||||||
|
if summary is None:
|
||||||
|
continue
|
||||||
|
digest = hashlib.sha256(
|
||||||
|
json.dumps(summary, separators=(",", ":"), sort_keys=True).encode()
|
||||||
|
).hexdigest()
|
||||||
|
record = lifecycle_record(
|
||||||
|
contract_commit=self.config.contract_commit,
|
||||||
|
project_key=project_key,
|
||||||
|
kind="PROJECT_SUMMARY",
|
||||||
|
work_unit_id=None,
|
||||||
|
payload=summary,
|
||||||
|
)
|
||||||
|
self.store.enqueue_lifecycle(
|
||||||
|
sync_id=record["sync_id"],
|
||||||
|
project_key=project_key,
|
||||||
|
work_unit_id=None,
|
||||||
|
kind=record["kind"],
|
||||||
|
payload=record,
|
||||||
|
)
|
||||||
|
self.store.mark_summary(project_key, digest)
|
||||||
|
|
||||||
|
def _enqueue_event_lifecycle(self, event: Event) -> None:
|
||||||
|
if event.event_type == "work_unit_started":
|
||||||
|
kind = "TASK_STARTED"
|
||||||
|
elif event.event_type in {"work_unit_completed", "work_unit_failed", "work_unit_blocked"}:
|
||||||
|
kind = "TASK_ENDED"
|
||||||
|
else:
|
||||||
|
kind = "EVENT_RECEIVED"
|
||||||
|
record = lifecycle_record(
|
||||||
|
contract_commit=self.config.contract_commit,
|
||||||
|
project_key=event.project_key,
|
||||||
|
kind=kind,
|
||||||
|
work_unit_id=event.work_unit_id,
|
||||||
|
payload=event.to_dict(),
|
||||||
|
)
|
||||||
|
self.store.enqueue_lifecycle(
|
||||||
|
sync_id=record["sync_id"],
|
||||||
|
project_key=event.project_key,
|
||||||
|
work_unit_id=event.work_unit_id,
|
||||||
|
kind=kind,
|
||||||
|
payload=record,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _enqueue_decision_lifecycle(self, decision: Decision, version: int) -> None:
|
||||||
|
kind = "PROJECT_COMPLETED" if decision.verdict == "COMPLETE" else "DECISION_APPLIED"
|
||||||
|
record = lifecycle_record(
|
||||||
|
contract_commit=self.config.contract_commit,
|
||||||
|
project_key=decision.project_key,
|
||||||
|
kind=kind,
|
||||||
|
work_unit_id=None,
|
||||||
|
payload={"decision": decision.to_dict(), "state_version": version},
|
||||||
|
)
|
||||||
|
self.store.enqueue_lifecycle(
|
||||||
|
sync_id=record["sync_id"],
|
||||||
|
project_key=decision.project_key,
|
||||||
|
work_unit_id=None,
|
||||||
|
kind=kind,
|
||||||
|
payload=record,
|
||||||
|
)
|
||||||
|
summary = self.store.project_summary(decision.project_key)
|
||||||
|
if summary:
|
||||||
|
summary_record = lifecycle_record(
|
||||||
|
contract_commit=self.config.contract_commit,
|
||||||
|
project_key=decision.project_key,
|
||||||
|
kind="PROJECT_SUMMARY",
|
||||||
|
work_unit_id=None,
|
||||||
|
payload=summary,
|
||||||
|
)
|
||||||
|
self.store.enqueue_lifecycle(
|
||||||
|
sync_id=summary_record["sync_id"],
|
||||||
|
project_key=decision.project_key,
|
||||||
|
work_unit_id=None,
|
||||||
|
kind=summary_record["kind"],
|
||||||
|
payload=summary_record,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _project_for_event(self, event_id: str) -> str:
|
||||||
|
rows = self.store._conn().execute(
|
||||||
|
"SELECT project_key FROM events WHERE event_id=?",
|
||||||
|
(event_id,),
|
||||||
|
).fetchone()
|
||||||
|
if rows is None:
|
||||||
|
raise ConflictError("event_not_found")
|
||||||
|
return str(rows["project_key"])
|
||||||
|
|
||||||
|
def _retry_or_dead_letter(self, event: Event, attempts: int, reason: str) -> None:
|
||||||
|
next_attempt = attempts + 1
|
||||||
|
if next_attempt > self.config.max_retries:
|
||||||
|
self.store.update_event_trigger(
|
||||||
|
event.event_id,
|
||||||
|
status="DEAD_LETTER",
|
||||||
|
channel=None,
|
||||||
|
remote_id=None,
|
||||||
|
increment_attempt=True,
|
||||||
|
)
|
||||||
|
record = lifecycle_record(
|
||||||
|
contract_commit=self.config.contract_commit,
|
||||||
|
project_key=event.project_key,
|
||||||
|
kind="TASK_ENDED",
|
||||||
|
work_unit_id=event.work_unit_id,
|
||||||
|
payload={
|
||||||
|
"status": "FAIL",
|
||||||
|
"reason": reason,
|
||||||
|
"event_id": event.event_id,
|
||||||
|
"attempts": next_attempt,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
self.store.enqueue_lifecycle(
|
||||||
|
sync_id=record["sync_id"],
|
||||||
|
project_key=event.project_key,
|
||||||
|
work_unit_id=event.work_unit_id,
|
||||||
|
kind=record["kind"],
|
||||||
|
payload=record,
|
||||||
|
)
|
||||||
|
return
|
||||||
|
self.store.update_event_trigger(
|
||||||
|
event.event_id,
|
||||||
|
status="TRIGGER_FAILED",
|
||||||
|
channel=None,
|
||||||
|
remote_id=None,
|
||||||
|
retry_after=self._backoff(attempts),
|
||||||
|
increment_attempt=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _backoff(self, attempts: int) -> int:
|
||||||
|
return min(3600, self.config.retry_base_seconds * (2 ** min(attempts, 10)))
|
||||||
|
|
||||||
|
|
||||||
|
class Handler(BaseHTTPRequestHandler):
|
||||||
|
runtime: Runtime
|
||||||
|
|
||||||
|
def log_message(self, fmt: str, *args: object) -> None:
|
||||||
|
LOG.info("%s %s", self.client_address[0], fmt % args)
|
||||||
|
|
||||||
|
def _reply(self, status: int, payload: dict[str, Any]) -> None:
|
||||||
|
body = json.dumps(payload, separators=(",", ":"), sort_keys=True).encode()
|
||||||
|
self.send_response(status)
|
||||||
|
self.send_header("Content-Type", "application/json")
|
||||||
|
self.send_header("Content-Length", str(len(body)))
|
||||||
|
self.send_header("Cache-Control", "no-store")
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(body)
|
||||||
|
|
||||||
|
def _body(self) -> dict[str, Any]:
|
||||||
|
try:
|
||||||
|
length = int(self.headers.get("Content-Length", "0"))
|
||||||
|
except ValueError as exc:
|
||||||
|
raise ValidationError("invalid_content_length") from exc
|
||||||
|
if length <= 0 or length > 1_048_576:
|
||||||
|
raise ValidationError("invalid_body_size")
|
||||||
|
try:
|
||||||
|
value = json.loads(self.rfile.read(length))
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise ValidationError("invalid_json") from exc
|
||||||
|
if not isinstance(value, dict):
|
||||||
|
raise ValidationError("json_root_not_object")
|
||||||
|
return value
|
||||||
|
|
||||||
|
def _authenticate(self, expected: str) -> None:
|
||||||
|
header = self.headers.get("Authorization", "")
|
||||||
|
if not header.startswith("Bearer "):
|
||||||
|
raise AuthenticationError("missing_bearer")
|
||||||
|
if not hmac.compare_digest(header[7:], expected):
|
||||||
|
raise AuthenticationError("invalid_bearer")
|
||||||
|
|
||||||
|
def do_GET(self) -> None:
|
||||||
|
path = urlparse(self.path).path.rstrip("/") or "/"
|
||||||
|
try:
|
||||||
|
if path == "/healthz":
|
||||||
|
self._reply(
|
||||||
|
200,
|
||||||
|
{
|
||||||
|
"service": "aios-orchestrator-gateway",
|
||||||
|
"status": "active",
|
||||||
|
"contract_commit": self.runtime.config.contract_commit,
|
||||||
|
"polling": True,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
return
|
||||||
|
if path.startswith("/v1/orchestrator/state/"):
|
||||||
|
self._authenticate(self.runtime.config.inbound_token())
|
||||||
|
key = path.rsplit("/", 1)[-1]
|
||||||
|
state = self.runtime.store.project_summary(key)
|
||||||
|
self._reply(200 if state else 404, state or {"error": "project_not_found"})
|
||||||
|
return
|
||||||
|
self._reply(404, {"error": "not_found"})
|
||||||
|
except AuthenticationError as exc:
|
||||||
|
self._reply(401, {"error": str(exc)})
|
||||||
|
except OrchestratorError as exc:
|
||||||
|
self._reply(400, {"error": str(exc)})
|
||||||
|
|
||||||
|
def do_POST(self) -> None:
|
||||||
|
path = urlparse(self.path).path.rstrip("/")
|
||||||
|
try:
|
||||||
|
if path == "/v1/orchestrator/events":
|
||||||
|
self._authenticate(self.runtime.config.inbound_token())
|
||||||
|
created, event = self.runtime.submit_event(self._body())
|
||||||
|
self._reply(202 if created else 200, {"created": created, "event": event})
|
||||||
|
return
|
||||||
|
if path == "/v1/orchestrator/decisions":
|
||||||
|
self._authenticate(self.runtime.config.callback_token())
|
||||||
|
version = self.runtime.submit_decision(self._body())
|
||||||
|
self._reply(202, {"accepted": True, "state_version": version})
|
||||||
|
return
|
||||||
|
if path == "/v1/orchestrator/poll":
|
||||||
|
self._authenticate(self.runtime.config.inbound_token())
|
||||||
|
self.runtime.poll_once()
|
||||||
|
self._reply(200, {"polled": True})
|
||||||
|
return
|
||||||
|
self._reply(404, {"error": "not_found"})
|
||||||
|
except AuthenticationError as exc:
|
||||||
|
self._reply(401, {"error": str(exc)})
|
||||||
|
except ConflictError as exc:
|
||||||
|
self._reply(409, {"error": str(exc)})
|
||||||
|
except ValidationError as exc:
|
||||||
|
self._reply(422, {"error": str(exc)})
|
||||||
|
except OrchestratorError as exc:
|
||||||
|
self._reply(400, {"error": str(exc)})
|
||||||
|
except Exception:
|
||||||
|
LOG.exception("request_failed path=%s", path)
|
||||||
|
self._reply(500, {"error": "internal_error"})
|
||||||
|
|
||||||
|
|
||||||
|
def serve(config_path: Path) -> None:
|
||||||
|
config = Config.from_file(config_path)
|
||||||
|
runtime = Runtime(config)
|
||||||
|
Handler.runtime = runtime
|
||||||
|
server = ThreadingHTTPServer((config.host, config.port), Handler)
|
||||||
|
runtime.start()
|
||||||
|
LOG.info(
|
||||||
|
"gateway_started host=%s port=%s contract=%s polling=%ss",
|
||||||
|
config.host,
|
||||||
|
config.port,
|
||||||
|
config.contract_commit,
|
||||||
|
config.poll_interval_seconds,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
server.serve_forever(poll_interval=0.5)
|
||||||
|
finally:
|
||||||
|
server.server_close()
|
||||||
|
runtime.stop()
|
||||||
@@ -0,0 +1,826 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
import sqlite3
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .errors import ConflictError
|
||||||
|
from .models import Decision, Event
|
||||||
|
from .securefs import ensure_secure_directory
|
||||||
|
|
||||||
|
|
||||||
|
class Store:
|
||||||
|
"""Durable state machine with idempotency, optimistic versioning, and leases."""
|
||||||
|
|
||||||
|
def __init__(self, path: Path):
|
||||||
|
self.path = Path(path)
|
||||||
|
ensure_secure_directory(self.path.parent)
|
||||||
|
self._local = threading.local()
|
||||||
|
self._initialize()
|
||||||
|
|
||||||
|
def _connect(self) -> sqlite3.Connection:
|
||||||
|
conn = sqlite3.connect(self.path, timeout=30, isolation_level=None)
|
||||||
|
conn.row_factory = sqlite3.Row
|
||||||
|
conn.execute("PRAGMA journal_mode=WAL")
|
||||||
|
conn.execute("PRAGMA synchronous=FULL")
|
||||||
|
conn.execute("PRAGMA foreign_keys=ON")
|
||||||
|
conn.execute("PRAGMA busy_timeout=30000")
|
||||||
|
return conn
|
||||||
|
|
||||||
|
def _conn(self) -> sqlite3.Connection:
|
||||||
|
conn = getattr(self._local, "conn", None)
|
||||||
|
if conn is None:
|
||||||
|
conn = self._connect()
|
||||||
|
self._local.conn = conn
|
||||||
|
return conn
|
||||||
|
|
||||||
|
def _initialize(self) -> None:
|
||||||
|
conn = self._connect()
|
||||||
|
try:
|
||||||
|
conn.executescript(
|
||||||
|
"""
|
||||||
|
CREATE TABLE IF NOT EXISTS project_state (
|
||||||
|
project_key TEXT PRIMARY KEY,
|
||||||
|
contract_commit TEXT NOT NULL,
|
||||||
|
state_version INTEGER NOT NULL,
|
||||||
|
state_json TEXT NOT NULL,
|
||||||
|
hops INTEGER NOT NULL DEFAULT 0,
|
||||||
|
updated_at INTEGER NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS gate_hops (
|
||||||
|
project_key TEXT NOT NULL,
|
||||||
|
gate_id TEXT NOT NULL,
|
||||||
|
hops INTEGER NOT NULL DEFAULT 0,
|
||||||
|
updated_at INTEGER NOT NULL,
|
||||||
|
PRIMARY KEY(project_key, gate_id)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS events (
|
||||||
|
event_id TEXT PRIMARY KEY,
|
||||||
|
project_key TEXT NOT NULL,
|
||||||
|
digest TEXT NOT NULL,
|
||||||
|
event_json TEXT NOT NULL,
|
||||||
|
status TEXT NOT NULL,
|
||||||
|
channel TEXT,
|
||||||
|
remote_id TEXT,
|
||||||
|
attempts INTEGER NOT NULL DEFAULT 0,
|
||||||
|
next_attempt_at INTEGER NOT NULL DEFAULT 0,
|
||||||
|
lease_owner TEXT,
|
||||||
|
lease_until INTEGER,
|
||||||
|
created_at INTEGER NOT NULL,
|
||||||
|
updated_at INTEGER NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_events_poll
|
||||||
|
ON events(status, next_attempt_at, lease_until, created_at);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS decisions (
|
||||||
|
event_id TEXT PRIMARY KEY REFERENCES events(event_id),
|
||||||
|
project_key TEXT NOT NULL,
|
||||||
|
decision_json TEXT NOT NULL,
|
||||||
|
state_version_after INTEGER NOT NULL,
|
||||||
|
created_at INTEGER NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS outbox (
|
||||||
|
work_unit_id TEXT PRIMARY KEY,
|
||||||
|
event_id TEXT NOT NULL REFERENCES events(event_id),
|
||||||
|
assigned_agent TEXT NOT NULL,
|
||||||
|
payload_json TEXT NOT NULL,
|
||||||
|
status TEXT NOT NULL,
|
||||||
|
attempts INTEGER NOT NULL DEFAULT 0,
|
||||||
|
next_attempt_at INTEGER NOT NULL DEFAULT 0,
|
||||||
|
lease_owner TEXT,
|
||||||
|
lease_until INTEGER,
|
||||||
|
created_at INTEGER NOT NULL,
|
||||||
|
updated_at INTEGER NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_outbox_poll
|
||||||
|
ON outbox(status, next_attempt_at, lease_until, created_at);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS lifecycle (
|
||||||
|
sync_id TEXT PRIMARY KEY,
|
||||||
|
project_key TEXT NOT NULL,
|
||||||
|
work_unit_id TEXT,
|
||||||
|
kind TEXT NOT NULL,
|
||||||
|
payload_json TEXT NOT NULL,
|
||||||
|
status TEXT NOT NULL,
|
||||||
|
attempts INTEGER NOT NULL DEFAULT 0,
|
||||||
|
next_attempt_at INTEGER NOT NULL DEFAULT 0,
|
||||||
|
lease_owner TEXT,
|
||||||
|
lease_until INTEGER,
|
||||||
|
created_at INTEGER NOT NULL,
|
||||||
|
updated_at INTEGER NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_lifecycle_poll
|
||||||
|
ON lifecycle(status, next_attempt_at, lease_until, created_at);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS blocker_history (
|
||||||
|
project_key TEXT NOT NULL,
|
||||||
|
gate_id TEXT NOT NULL,
|
||||||
|
blocker TEXT NOT NULL,
|
||||||
|
repetitions INTEGER NOT NULL,
|
||||||
|
updated_at INTEGER NOT NULL,
|
||||||
|
PRIMARY KEY(project_key, gate_id, blocker)
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS summaries (
|
||||||
|
project_key TEXT PRIMARY KEY,
|
||||||
|
last_emitted_at INTEGER NOT NULL DEFAULT 0,
|
||||||
|
last_digest TEXT
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS observed_refs (
|
||||||
|
refname TEXT PRIMARY KEY,
|
||||||
|
object_id TEXT NOT NULL,
|
||||||
|
observed_at INTEGER NOT NULL
|
||||||
|
);
|
||||||
|
|
||||||
|
CREATE TABLE IF NOT EXISTS metadata (
|
||||||
|
key TEXT PRIMARY KEY,
|
||||||
|
value TEXT NOT NULL
|
||||||
|
);
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
def register_event(self, event: Event) -> tuple[bool, dict[str, Any]]:
|
||||||
|
now = int(time.time())
|
||||||
|
conn = self._conn()
|
||||||
|
conn.execute("BEGIN IMMEDIATE")
|
||||||
|
try:
|
||||||
|
existing = conn.execute(
|
||||||
|
"SELECT digest, event_json FROM events WHERE event_id=?",
|
||||||
|
(event.event_id,),
|
||||||
|
).fetchone()
|
||||||
|
if existing:
|
||||||
|
if existing["digest"] != event.digest():
|
||||||
|
raise ConflictError("event_id_payload_conflict")
|
||||||
|
conn.execute("COMMIT")
|
||||||
|
return False, json.loads(existing["event_json"])
|
||||||
|
|
||||||
|
state = conn.execute(
|
||||||
|
"SELECT contract_commit, state_version FROM project_state WHERE project_key=?",
|
||||||
|
(event.project_key,),
|
||||||
|
).fetchone()
|
||||||
|
if state is None:
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO project_state(
|
||||||
|
project_key, contract_commit, state_version, state_json,
|
||||||
|
hops, updated_at
|
||||||
|
) VALUES (?, ?, ?, '{}', 0, ?)
|
||||||
|
""",
|
||||||
|
(event.project_key, event.contract_commit, event.state_version, now),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
if state["contract_commit"] != event.contract_commit:
|
||||||
|
raise ConflictError("contract_version_mismatch")
|
||||||
|
if state["state_version"] != event.state_version:
|
||||||
|
raise ConflictError("stale_state_version")
|
||||||
|
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO events(
|
||||||
|
event_id, project_key, digest, event_json, status,
|
||||||
|
next_attempt_at, created_at, updated_at
|
||||||
|
) VALUES (?, ?, ?, ?, 'RECEIVED', ?, ?, ?)
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
event.event_id,
|
||||||
|
event.project_key,
|
||||||
|
event.digest(),
|
||||||
|
event.canonical_json(),
|
||||||
|
now,
|
||||||
|
now,
|
||||||
|
now,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
conn.execute("COMMIT")
|
||||||
|
return True, event.to_dict()
|
||||||
|
except Exception:
|
||||||
|
conn.execute("ROLLBACK")
|
||||||
|
raise
|
||||||
|
|
||||||
|
def claim_events(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
owner: str,
|
||||||
|
statuses: tuple[str, ...],
|
||||||
|
limit: int,
|
||||||
|
lease_seconds: int,
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
if not statuses:
|
||||||
|
return []
|
||||||
|
now = int(time.time())
|
||||||
|
placeholders = ",".join("?" for _ in statuses)
|
||||||
|
conn = self._conn()
|
||||||
|
conn.execute("BEGIN IMMEDIATE")
|
||||||
|
try:
|
||||||
|
rows = conn.execute(
|
||||||
|
f"""
|
||||||
|
SELECT event_id FROM events
|
||||||
|
WHERE status IN ({placeholders})
|
||||||
|
AND next_attempt_at <= ?
|
||||||
|
AND (lease_until IS NULL OR lease_until < ?)
|
||||||
|
ORDER BY created_at
|
||||||
|
LIMIT ?
|
||||||
|
""",
|
||||||
|
(*statuses, now, now, limit),
|
||||||
|
).fetchall()
|
||||||
|
ids = [row["event_id"] for row in rows]
|
||||||
|
for event_id in ids:
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
UPDATE events SET lease_owner=?, lease_until=?, updated_at=?
|
||||||
|
WHERE event_id=?
|
||||||
|
""",
|
||||||
|
(owner, now + lease_seconds, now, event_id),
|
||||||
|
)
|
||||||
|
conn.execute("COMMIT")
|
||||||
|
except Exception:
|
||||||
|
conn.execute("ROLLBACK")
|
||||||
|
raise
|
||||||
|
if not ids:
|
||||||
|
return []
|
||||||
|
placeholders = ",".join("?" for _ in ids)
|
||||||
|
return [
|
||||||
|
dict(row)
|
||||||
|
for row in conn.execute(
|
||||||
|
f"SELECT * FROM events WHERE event_id IN ({placeholders})",
|
||||||
|
ids,
|
||||||
|
).fetchall()
|
||||||
|
]
|
||||||
|
|
||||||
|
def update_event_trigger(
|
||||||
|
self,
|
||||||
|
event_id: str,
|
||||||
|
*,
|
||||||
|
status: str,
|
||||||
|
channel: str | None,
|
||||||
|
remote_id: str | None,
|
||||||
|
retry_after: int = 0,
|
||||||
|
increment_attempt: bool = False,
|
||||||
|
) -> None:
|
||||||
|
now = int(time.time())
|
||||||
|
self._conn().execute(
|
||||||
|
"""
|
||||||
|
UPDATE events
|
||||||
|
SET status=?, channel=?, remote_id=?,
|
||||||
|
attempts=attempts + ?,
|
||||||
|
next_attempt_at=?,
|
||||||
|
lease_owner=NULL, lease_until=NULL, updated_at=?
|
||||||
|
WHERE event_id=?
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
status,
|
||||||
|
channel,
|
||||||
|
remote_id,
|
||||||
|
1 if increment_attempt else 0,
|
||||||
|
now + max(0, retry_after),
|
||||||
|
now,
|
||||||
|
event_id,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def claim_workspace_timeouts(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
owner: str,
|
||||||
|
older_than: int,
|
||||||
|
limit: int,
|
||||||
|
lease_seconds: int,
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
return self._claim_event_subset(
|
||||||
|
owner=owner,
|
||||||
|
where_sql="status='AWAITING_DECISION' AND channel='workspace_agent' AND updated_at <= ?",
|
||||||
|
where_params=(older_than,),
|
||||||
|
limit=limit,
|
||||||
|
lease_seconds=lease_seconds,
|
||||||
|
)
|
||||||
|
|
||||||
|
def claim_response_polls(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
owner: str,
|
||||||
|
limit: int,
|
||||||
|
lease_seconds: int,
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
now = int(time.time())
|
||||||
|
return self._claim_event_subset(
|
||||||
|
owner=owner,
|
||||||
|
where_sql="status='POLLING_RESPONSE' AND next_attempt_at <= ?",
|
||||||
|
where_params=(now,),
|
||||||
|
limit=limit,
|
||||||
|
lease_seconds=lease_seconds,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _claim_event_subset(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
owner: str,
|
||||||
|
where_sql: str,
|
||||||
|
where_params: tuple[Any, ...],
|
||||||
|
limit: int,
|
||||||
|
lease_seconds: int,
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
allowed = {
|
||||||
|
"status='AWAITING_DECISION' AND channel='workspace_agent' AND updated_at <= ?",
|
||||||
|
"status='POLLING_RESPONSE' AND next_attempt_at <= ?",
|
||||||
|
}
|
||||||
|
if where_sql not in allowed:
|
||||||
|
raise ValueError("invalid_event_claim_query")
|
||||||
|
now = int(time.time())
|
||||||
|
conn = self._conn()
|
||||||
|
conn.execute("BEGIN IMMEDIATE")
|
||||||
|
try:
|
||||||
|
rows = conn.execute(
|
||||||
|
f"""
|
||||||
|
SELECT event_id FROM events
|
||||||
|
WHERE {where_sql}
|
||||||
|
AND (lease_until IS NULL OR lease_until < ?)
|
||||||
|
ORDER BY updated_at
|
||||||
|
LIMIT ?
|
||||||
|
""",
|
||||||
|
(*where_params, now, limit),
|
||||||
|
).fetchall()
|
||||||
|
ids = [str(row["event_id"]) for row in rows]
|
||||||
|
for event_id in ids:
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
UPDATE events SET lease_owner=?, lease_until=?
|
||||||
|
WHERE event_id=?
|
||||||
|
""",
|
||||||
|
(owner, now + lease_seconds, event_id),
|
||||||
|
)
|
||||||
|
conn.execute("COMMIT")
|
||||||
|
except Exception:
|
||||||
|
conn.execute("ROLLBACK")
|
||||||
|
raise
|
||||||
|
if not ids:
|
||||||
|
return []
|
||||||
|
placeholders = ",".join("?" for _ in ids)
|
||||||
|
return [
|
||||||
|
dict(row)
|
||||||
|
for row in conn.execute(
|
||||||
|
f"SELECT * FROM events WHERE event_id IN ({placeholders})",
|
||||||
|
ids,
|
||||||
|
).fetchall()
|
||||||
|
]
|
||||||
|
|
||||||
|
def apply_decision(
|
||||||
|
self,
|
||||||
|
decision: Decision,
|
||||||
|
*,
|
||||||
|
max_hops_per_gate: int,
|
||||||
|
max_same_blocker_retries: int,
|
||||||
|
) -> int:
|
||||||
|
now = int(time.time())
|
||||||
|
conn = self._conn()
|
||||||
|
conn.execute("BEGIN IMMEDIATE")
|
||||||
|
try:
|
||||||
|
event = conn.execute(
|
||||||
|
"SELECT project_key FROM events WHERE event_id=?",
|
||||||
|
(decision.event_id,),
|
||||||
|
).fetchone()
|
||||||
|
if event is None:
|
||||||
|
raise ConflictError("event_not_found")
|
||||||
|
if event["project_key"] != decision.project_key:
|
||||||
|
raise ConflictError("event_project_mismatch")
|
||||||
|
|
||||||
|
existing = conn.execute(
|
||||||
|
"SELECT decision_json, state_version_after FROM decisions WHERE event_id=?",
|
||||||
|
(decision.event_id,),
|
||||||
|
).fetchone()
|
||||||
|
if existing:
|
||||||
|
if json.loads(existing["decision_json"]) != decision.to_dict():
|
||||||
|
raise ConflictError("decision_idempotency_conflict")
|
||||||
|
conn.execute("COMMIT")
|
||||||
|
return int(existing["state_version_after"])
|
||||||
|
|
||||||
|
state = conn.execute(
|
||||||
|
"SELECT * FROM project_state WHERE project_key=?",
|
||||||
|
(decision.project_key,),
|
||||||
|
).fetchone()
|
||||||
|
if state is None:
|
||||||
|
raise ConflictError("project_state_not_found")
|
||||||
|
if state["contract_commit"] != decision.contract_commit:
|
||||||
|
raise ConflictError("contract_version_mismatch")
|
||||||
|
if state["state_version"] != decision.expected_state_version:
|
||||||
|
raise ConflictError("stale_decision_state_version")
|
||||||
|
state_json = json.loads(state["state_json"])
|
||||||
|
gate = str(
|
||||||
|
decision.state_updates.get(
|
||||||
|
"current_gate", state_json.get("current_gate", "unknown")
|
||||||
|
)
|
||||||
|
)
|
||||||
|
gate_hops = conn.execute(
|
||||||
|
"SELECT hops FROM gate_hops WHERE project_key=? AND gate_id=?",
|
||||||
|
(decision.project_key, gate),
|
||||||
|
).fetchone()
|
||||||
|
if gate_hops is not None and int(gate_hops["hops"]) >= max_hops_per_gate:
|
||||||
|
raise ConflictError("max_hops_exceeded")
|
||||||
|
|
||||||
|
state_json.update(decision.state_updates)
|
||||||
|
next_version = int(state["state_version"]) + 1
|
||||||
|
|
||||||
|
if decision.stop_reason:
|
||||||
|
row = conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT repetitions FROM blocker_history
|
||||||
|
WHERE project_key=? AND gate_id=? AND blocker=?
|
||||||
|
""",
|
||||||
|
(decision.project_key, gate, decision.stop_reason),
|
||||||
|
).fetchone()
|
||||||
|
repetitions = (int(row["repetitions"]) if row else 0) + 1
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO blocker_history(
|
||||||
|
project_key, gate_id, blocker, repetitions, updated_at
|
||||||
|
) VALUES (?, ?, ?, ?, ?)
|
||||||
|
ON CONFLICT(project_key, gate_id, blocker)
|
||||||
|
DO UPDATE SET repetitions=excluded.repetitions,
|
||||||
|
updated_at=excluded.updated_at
|
||||||
|
""",
|
||||||
|
(decision.project_key, gate, decision.stop_reason, repetitions, now),
|
||||||
|
)
|
||||||
|
if repetitions > max_same_blocker_retries:
|
||||||
|
raise ConflictError("same_blocker_retry_limit_exceeded")
|
||||||
|
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO gate_hops(project_key, gate_id, hops, updated_at)
|
||||||
|
VALUES (?, ?, 1, ?)
|
||||||
|
ON CONFLICT(project_key, gate_id)
|
||||||
|
DO UPDATE SET hops=gate_hops.hops + 1, updated_at=excluded.updated_at
|
||||||
|
""",
|
||||||
|
(decision.project_key, gate, now),
|
||||||
|
)
|
||||||
|
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
UPDATE project_state
|
||||||
|
SET state_version=?, state_json=?, hops=hops+1, updated_at=?
|
||||||
|
WHERE project_key=?
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
next_version,
|
||||||
|
json.dumps(state_json, separators=(",", ":"), sort_keys=True),
|
||||||
|
now,
|
||||||
|
decision.project_key,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO decisions(
|
||||||
|
event_id, project_key, decision_json,
|
||||||
|
state_version_after, created_at
|
||||||
|
) VALUES (?, ?, ?, ?, ?)
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
decision.event_id,
|
||||||
|
decision.project_key,
|
||||||
|
json.dumps(decision.to_dict(), separators=(",", ":"), sort_keys=True),
|
||||||
|
next_version,
|
||||||
|
now,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for unit in decision.next_work_units:
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO outbox(
|
||||||
|
work_unit_id, event_id, assigned_agent, payload_json,
|
||||||
|
status, next_attempt_at, created_at, updated_at
|
||||||
|
) VALUES (?, ?, ?, ?, 'PENDING', ?, ?, ?)
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
unit.id,
|
||||||
|
decision.event_id,
|
||||||
|
unit.primary_agent,
|
||||||
|
json.dumps(unit.to_dict(), separators=(",", ":"), sort_keys=True),
|
||||||
|
now,
|
||||||
|
now,
|
||||||
|
now,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
UPDATE events SET status='DECIDED', lease_owner=NULL,
|
||||||
|
lease_until=NULL, updated_at=?
|
||||||
|
WHERE event_id=?
|
||||||
|
""",
|
||||||
|
(now, decision.event_id),
|
||||||
|
)
|
||||||
|
conn.execute("COMMIT")
|
||||||
|
return next_version
|
||||||
|
except Exception:
|
||||||
|
conn.execute("ROLLBACK")
|
||||||
|
raise
|
||||||
|
|
||||||
|
def claim_outbox(self, *, owner: str, limit: int, lease_seconds: int) -> list[dict[str, Any]]:
|
||||||
|
return self._claim_table(
|
||||||
|
table="outbox",
|
||||||
|
key="work_unit_id",
|
||||||
|
owner=owner,
|
||||||
|
statuses=("PENDING", "RETRY"),
|
||||||
|
limit=limit,
|
||||||
|
lease_seconds=lease_seconds,
|
||||||
|
)
|
||||||
|
|
||||||
|
def mark_outbox(
|
||||||
|
self,
|
||||||
|
work_unit_id: str,
|
||||||
|
*,
|
||||||
|
status: str,
|
||||||
|
retry_after: int = 0,
|
||||||
|
increment_attempt: bool = False,
|
||||||
|
) -> None:
|
||||||
|
self._mark_table(
|
||||||
|
table="outbox",
|
||||||
|
key="work_unit_id",
|
||||||
|
value=work_unit_id,
|
||||||
|
status=status,
|
||||||
|
retry_after=retry_after,
|
||||||
|
increment_attempt=increment_attempt,
|
||||||
|
)
|
||||||
|
|
||||||
|
def enqueue_lifecycle(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
sync_id: str,
|
||||||
|
project_key: str,
|
||||||
|
work_unit_id: str | None,
|
||||||
|
kind: str,
|
||||||
|
payload: dict[str, Any],
|
||||||
|
) -> None:
|
||||||
|
now = int(time.time())
|
||||||
|
encoded = json.dumps(payload, separators=(",", ":"), sort_keys=True)
|
||||||
|
self._conn().execute(
|
||||||
|
"""
|
||||||
|
INSERT OR IGNORE INTO lifecycle(
|
||||||
|
sync_id, project_key, work_unit_id, kind, payload_json,
|
||||||
|
status, next_attempt_at, created_at, updated_at
|
||||||
|
) VALUES (?, ?, ?, ?, ?, 'PENDING', ?, ?, ?)
|
||||||
|
""",
|
||||||
|
(sync_id, project_key, work_unit_id, kind, encoded, now, now, now),
|
||||||
|
)
|
||||||
|
|
||||||
|
def claim_lifecycle(self, *, owner: str, limit: int, lease_seconds: int) -> list[dict[str, Any]]:
|
||||||
|
return self._claim_table(
|
||||||
|
table="lifecycle",
|
||||||
|
key="sync_id",
|
||||||
|
owner=owner,
|
||||||
|
statuses=("PENDING", "RETRY"),
|
||||||
|
limit=limit,
|
||||||
|
lease_seconds=lease_seconds,
|
||||||
|
)
|
||||||
|
|
||||||
|
def mark_lifecycle(
|
||||||
|
self,
|
||||||
|
sync_id: str,
|
||||||
|
*,
|
||||||
|
status: str,
|
||||||
|
retry_after: int = 0,
|
||||||
|
increment_attempt: bool = False,
|
||||||
|
) -> None:
|
||||||
|
self._mark_table(
|
||||||
|
table="lifecycle",
|
||||||
|
key="sync_id",
|
||||||
|
value=sync_id,
|
||||||
|
status=status,
|
||||||
|
retry_after=retry_after,
|
||||||
|
increment_attempt=increment_attempt,
|
||||||
|
)
|
||||||
|
|
||||||
|
def _claim_table(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
table: str,
|
||||||
|
key: str,
|
||||||
|
owner: str,
|
||||||
|
statuses: tuple[str, ...],
|
||||||
|
limit: int,
|
||||||
|
lease_seconds: int,
|
||||||
|
) -> list[dict[str, Any]]:
|
||||||
|
if table not in {"outbox", "lifecycle"} or key not in {"work_unit_id", "sync_id"}:
|
||||||
|
raise ValueError("invalid_claim_table")
|
||||||
|
now = int(time.time())
|
||||||
|
placeholders = ",".join("?" for _ in statuses)
|
||||||
|
conn = self._conn()
|
||||||
|
conn.execute("BEGIN IMMEDIATE")
|
||||||
|
try:
|
||||||
|
rows = conn.execute(
|
||||||
|
f"""
|
||||||
|
SELECT {key} FROM {table}
|
||||||
|
WHERE status IN ({placeholders})
|
||||||
|
AND next_attempt_at <= ?
|
||||||
|
AND (lease_until IS NULL OR lease_until < ?)
|
||||||
|
ORDER BY created_at
|
||||||
|
LIMIT ?
|
||||||
|
""",
|
||||||
|
(*statuses, now, now, limit),
|
||||||
|
).fetchall()
|
||||||
|
values = [row[key] for row in rows]
|
||||||
|
for item in values:
|
||||||
|
conn.execute(
|
||||||
|
f"""
|
||||||
|
UPDATE {table} SET lease_owner=?, lease_until=?, updated_at=?
|
||||||
|
WHERE {key}=?
|
||||||
|
""",
|
||||||
|
(owner, now + lease_seconds, now, item),
|
||||||
|
)
|
||||||
|
conn.execute("COMMIT")
|
||||||
|
except Exception:
|
||||||
|
conn.execute("ROLLBACK")
|
||||||
|
raise
|
||||||
|
if not values:
|
||||||
|
return []
|
||||||
|
placeholders = ",".join("?" for _ in values)
|
||||||
|
return [
|
||||||
|
dict(row)
|
||||||
|
for row in conn.execute(
|
||||||
|
f"SELECT * FROM {table} WHERE {key} IN ({placeholders})",
|
||||||
|
values,
|
||||||
|
).fetchall()
|
||||||
|
]
|
||||||
|
|
||||||
|
def _mark_table(
|
||||||
|
self,
|
||||||
|
*,
|
||||||
|
table: str,
|
||||||
|
key: str,
|
||||||
|
value: str,
|
||||||
|
status: str,
|
||||||
|
retry_after: int,
|
||||||
|
increment_attempt: bool,
|
||||||
|
) -> None:
|
||||||
|
if table not in {"outbox", "lifecycle"} or key not in {"work_unit_id", "sync_id"}:
|
||||||
|
raise ValueError("invalid_mark_table")
|
||||||
|
now = int(time.time())
|
||||||
|
self._conn().execute(
|
||||||
|
f"""
|
||||||
|
UPDATE {table}
|
||||||
|
SET status=?, attempts=attempts + ?, next_attempt_at=?,
|
||||||
|
lease_owner=NULL, lease_until=NULL, updated_at=?
|
||||||
|
WHERE {key}=?
|
||||||
|
""",
|
||||||
|
(
|
||||||
|
status,
|
||||||
|
1 if increment_attempt else 0,
|
||||||
|
now + max(0, retry_after),
|
||||||
|
now,
|
||||||
|
value,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def project_state(self, project_key: str) -> dict[str, Any] | None:
|
||||||
|
row = self._conn().execute(
|
||||||
|
"SELECT * FROM project_state WHERE project_key=?",
|
||||||
|
(project_key,),
|
||||||
|
).fetchone()
|
||||||
|
if row is None:
|
||||||
|
return None
|
||||||
|
result = dict(row)
|
||||||
|
result["state"] = json.loads(result.pop("state_json"))
|
||||||
|
return result
|
||||||
|
|
||||||
|
def all_project_states(self) -> list[dict[str, Any]]:
|
||||||
|
result = []
|
||||||
|
for row in self._conn().execute("SELECT * FROM project_state ORDER BY project_key").fetchall():
|
||||||
|
item = dict(row)
|
||||||
|
item["state"] = json.loads(item.pop("state_json"))
|
||||||
|
result.append(item)
|
||||||
|
return result
|
||||||
|
|
||||||
|
def summary_due(self, project_key: str, interval: int) -> bool:
|
||||||
|
now = int(time.time())
|
||||||
|
row = self._conn().execute(
|
||||||
|
"SELECT last_emitted_at FROM summaries WHERE project_key=?",
|
||||||
|
(project_key,),
|
||||||
|
).fetchone()
|
||||||
|
return row is None or now - int(row["last_emitted_at"]) >= interval
|
||||||
|
|
||||||
|
def mark_summary(self, project_key: str, digest: str) -> None:
|
||||||
|
now = int(time.time())
|
||||||
|
self._conn().execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO summaries(project_key, last_emitted_at, last_digest)
|
||||||
|
VALUES (?, ?, ?)
|
||||||
|
ON CONFLICT(project_key)
|
||||||
|
DO UPDATE SET last_emitted_at=excluded.last_emitted_at,
|
||||||
|
last_digest=excluded.last_digest
|
||||||
|
""",
|
||||||
|
(project_key, now, digest),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def project_summary(self, project_key: str) -> dict[str, Any] | None:
|
||||||
|
state = self.project_state(project_key)
|
||||||
|
if state is None:
|
||||||
|
return None
|
||||||
|
conn = self._conn()
|
||||||
|
event_counts = {
|
||||||
|
row["status"]: int(row["count"])
|
||||||
|
for row in conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT status, COUNT(*) AS count FROM events
|
||||||
|
WHERE project_key=? GROUP BY status
|
||||||
|
""",
|
||||||
|
(project_key,),
|
||||||
|
).fetchall()
|
||||||
|
}
|
||||||
|
outbox_counts = {
|
||||||
|
row["status"]: int(row["count"])
|
||||||
|
for row in conn.execute(
|
||||||
|
"""
|
||||||
|
SELECT o.status, COUNT(*) AS count
|
||||||
|
FROM outbox o JOIN events e ON e.event_id=o.event_id
|
||||||
|
WHERE e.project_key=? GROUP BY o.status
|
||||||
|
""",
|
||||||
|
(project_key,),
|
||||||
|
).fetchall()
|
||||||
|
}
|
||||||
|
return {
|
||||||
|
"project_key": project_key,
|
||||||
|
"contract_commit": state["contract_commit"],
|
||||||
|
"state_version": state["state_version"],
|
||||||
|
"state": state["state"],
|
||||||
|
"event_counts": event_counts,
|
||||||
|
"work_unit_counts": outbox_counts,
|
||||||
|
"updated_at": state["updated_at"],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def metadata_get(self, key: str) -> str | None:
|
||||||
|
row = self._conn().execute(
|
||||||
|
"SELECT value FROM metadata WHERE key=?",
|
||||||
|
(key,),
|
||||||
|
).fetchone()
|
||||||
|
return None if row is None else str(row["value"])
|
||||||
|
|
||||||
|
def metadata_set(self, key: str, value: str) -> None:
|
||||||
|
self._conn().execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO metadata(key, value) VALUES (?, ?)
|
||||||
|
ON CONFLICT(key) DO UPDATE SET value=excluded.value
|
||||||
|
""",
|
||||||
|
(key, value),
|
||||||
|
)
|
||||||
|
|
||||||
|
def observe_refs(self, refs: dict[str, str]) -> tuple[bool, list[tuple[str, str | None, str]]]:
|
||||||
|
"""Persist a Git snapshot and return changes after the initial baseline."""
|
||||||
|
now = int(time.time())
|
||||||
|
conn = self._conn()
|
||||||
|
conn.execute("BEGIN IMMEDIATE")
|
||||||
|
try:
|
||||||
|
initialized = conn.execute(
|
||||||
|
"SELECT value FROM metadata WHERE key='git_poll_initialized'"
|
||||||
|
).fetchone()
|
||||||
|
previous = {
|
||||||
|
row["refname"]: row["object_id"]
|
||||||
|
for row in conn.execute("SELECT refname, object_id FROM observed_refs").fetchall()
|
||||||
|
}
|
||||||
|
changes: list[tuple[str, str | None, str]] = []
|
||||||
|
if initialized is not None:
|
||||||
|
for refname, oid in refs.items():
|
||||||
|
old = previous.get(refname)
|
||||||
|
if old != oid:
|
||||||
|
changes.append((refname, old, oid))
|
||||||
|
for refname, old in previous.items():
|
||||||
|
if refname not in refs:
|
||||||
|
changes.append((refname, old, "0" * 40))
|
||||||
|
conn.execute("DELETE FROM observed_refs")
|
||||||
|
for refname, oid in refs.items():
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO observed_refs(refname, object_id, observed_at)
|
||||||
|
VALUES (?, ?, ?)
|
||||||
|
""",
|
||||||
|
(refname, oid, now),
|
||||||
|
)
|
||||||
|
conn.execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO metadata(key, value)
|
||||||
|
VALUES ('git_poll_initialized', '1')
|
||||||
|
ON CONFLICT(key) DO UPDATE SET value='1'
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
conn.execute("COMMIT")
|
||||||
|
return initialized is not None, changes
|
||||||
|
except Exception:
|
||||||
|
conn.execute("ROLLBACK")
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
conn = getattr(self._local, "conn", None)
|
||||||
|
if conn is not None:
|
||||||
|
conn.close()
|
||||||
|
self._local.conn = None
|
||||||
@@ -0,0 +1,108 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import stat
|
||||||
|
from pathlib import Path
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .errors import DispatchError
|
||||||
|
from .securefs import atomic_write_bytes_shared, ensure_shared_directory, read_secure_bytes
|
||||||
|
|
||||||
|
|
||||||
|
class OpenClawSynchronizer:
|
||||||
|
"""Synchronize task starts, ends, dispatches, and project summaries.
|
||||||
|
|
||||||
|
A durable file is always emitted. An optional HTTP endpoint is best-effort;
|
||||||
|
failure leaves the lifecycle record pending for retry.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
queue_dir: Path,
|
||||||
|
*,
|
||||||
|
endpoint: str | None = None,
|
||||||
|
bearer_token: str | None = None,
|
||||||
|
timeout_seconds: int = 10,
|
||||||
|
):
|
||||||
|
self.queue_dir = Path(queue_dir)
|
||||||
|
self.endpoint = endpoint
|
||||||
|
self.bearer_token = bearer_token
|
||||||
|
self.timeout_seconds = timeout_seconds
|
||||||
|
ensure_shared_directory(self.queue_dir)
|
||||||
|
|
||||||
|
def emit(self, record: dict[str, Any]) -> Path:
|
||||||
|
sync_id = record.get("sync_id")
|
||||||
|
if not isinstance(sync_id, str) or not sync_id:
|
||||||
|
raise DispatchError("invalid_sync_id")
|
||||||
|
payload = (json.dumps(record, indent=2, sort_keys=True) + "\n").encode()
|
||||||
|
path = self.queue_dir / "pending" / f"{sync_id}.json"
|
||||||
|
ensure_shared_directory(path.parent)
|
||||||
|
try:
|
||||||
|
st = os.lstat(path)
|
||||||
|
except FileNotFoundError:
|
||||||
|
atomic_write_bytes_shared(path, payload, mode=0o640)
|
||||||
|
else:
|
||||||
|
if not stat.S_ISREG(st.st_mode):
|
||||||
|
raise DispatchError("sync_target_not_regular")
|
||||||
|
if read_secure_bytes(path, expected_uid=None, max_bytes=1_048_576) != payload:
|
||||||
|
raise DispatchError("sync_idempotency_conflict")
|
||||||
|
|
||||||
|
if self.endpoint:
|
||||||
|
headers = {
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
"Idempotency-Key": sync_id,
|
||||||
|
"User-Agent": "aios-orchestrator-gateway/0.2",
|
||||||
|
}
|
||||||
|
if self.bearer_token:
|
||||||
|
headers["Authorization"] = f"Bearer {self.bearer_token}"
|
||||||
|
request = urllib.request.Request(
|
||||||
|
self.endpoint,
|
||||||
|
data=payload,
|
||||||
|
method="POST",
|
||||||
|
headers=headers,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(request, timeout=self.timeout_seconds) as response:
|
||||||
|
response.read(65_536)
|
||||||
|
if response.status not in {200, 201, 202, 204}:
|
||||||
|
raise DispatchError(f"openclaw_sync_http_{response.status}")
|
||||||
|
except urllib.error.HTTPError as exc:
|
||||||
|
raise DispatchError(f"openclaw_sync_http_{exc.code}") from exc
|
||||||
|
except (urllib.error.URLError, TimeoutError, OSError) as exc:
|
||||||
|
raise DispatchError("openclaw_sync_failed") from exc
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def lifecycle_record(
|
||||||
|
*,
|
||||||
|
contract_commit: str,
|
||||||
|
project_key: str,
|
||||||
|
kind: str,
|
||||||
|
work_unit_id: str | None,
|
||||||
|
payload: dict[str, Any],
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
canonical = json.dumps(
|
||||||
|
{
|
||||||
|
"contract_commit": contract_commit,
|
||||||
|
"project_key": project_key,
|
||||||
|
"kind": kind,
|
||||||
|
"work_unit_id": work_unit_id,
|
||||||
|
"payload": payload,
|
||||||
|
},
|
||||||
|
separators=(",", ":"),
|
||||||
|
sort_keys=True,
|
||||||
|
)
|
||||||
|
sync_id = hashlib.sha256(canonical.encode()).hexdigest()
|
||||||
|
return {
|
||||||
|
"schema_version": 1,
|
||||||
|
"sync_id": sync_id,
|
||||||
|
"contract_commit": contract_commit,
|
||||||
|
"project_key": project_key,
|
||||||
|
"kind": kind,
|
||||||
|
"work_unit_id": work_unit_id,
|
||||||
|
"payload": payload,
|
||||||
|
}
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=AIOS Codex automatic command watcher
|
||||||
|
After=network-online.target aios-orchestrator-gateway.service
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
User=aios-orchestrator
|
||||||
|
Group=aios-orchestrator
|
||||||
|
UMask=0077
|
||||||
|
Environment=PYTHONPATH=/opt/aios-codex-watcher/current
|
||||||
|
ExecStart=/usr/bin/python3 -m services.aios_codex_watcher --config /etc/aios-codex-watcher/config.json --once
|
||||||
|
NoNewPrivileges=yes
|
||||||
|
PrivateDevices=yes
|
||||||
|
PrivateTmp=yes
|
||||||
|
ProtectClock=yes
|
||||||
|
ProtectControlGroups=yes
|
||||||
|
ProtectHome=yes
|
||||||
|
ProtectHostname=yes
|
||||||
|
ProtectKernelLogs=yes
|
||||||
|
ProtectKernelModules=yes
|
||||||
|
ProtectKernelTunables=yes
|
||||||
|
ProtectSystem=strict
|
||||||
|
ReadOnlyPaths=/etc/aios-orchestrator-gateway/inbound.token
|
||||||
|
ReadWritePaths=/data/aios-gitops/workspace/agents/commands /var/lib/aios-codex-watcher /var/log/aios-codex-watcher
|
||||||
|
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
|
||||||
|
RestrictNamespaces=yes
|
||||||
|
RestrictRealtime=yes
|
||||||
|
RestrictSUIDSGID=yes
|
||||||
|
SystemCallArchitectures=native
|
||||||
|
LockPersonality=yes
|
||||||
|
MemoryDenyWriteExecute=yes
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
u aios-orchestrator - "AIOS Orchestrator" /var/lib/aios-orchestrator-gateway
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Poll AIOS Codex command queue
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnBootSec=30s
|
||||||
|
OnUnitActiveSec=5s
|
||||||
|
AccuracySec=1s
|
||||||
|
Persistent=true
|
||||||
|
Unit=aios-codex-watcher.service
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
d /var/lib/aios-codex-watcher 0700 aios-orchestrator aios-orchestrator -
|
||||||
|
d /var/log/aios-codex-watcher 0700 aios-orchestrator aios-orchestrator -
|
||||||
@@ -0,0 +1,37 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=AIOS Orchestrator Gateway reliability poll
|
||||||
|
After=aios-orchestrator-gateway.service
|
||||||
|
ConditionPathExists=/etc/aios-orchestrator-gateway/config.json
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
User=aios-orchestrator
|
||||||
|
Group=aios-orchestrator
|
||||||
|
SupplementaryGroups=gitops
|
||||||
|
WorkingDirectory=/opt/aios-orchestrator-gateway/current
|
||||||
|
ExecStart=/usr/bin/python3 -m services.aios_orchestrator_gateway --config /etc/aios-orchestrator-gateway/config.json --poll-once
|
||||||
|
UMask=0077
|
||||||
|
NoNewPrivileges=true
|
||||||
|
PrivateTmp=true
|
||||||
|
PrivateDevices=true
|
||||||
|
ProtectSystem=strict
|
||||||
|
ProtectHome=true
|
||||||
|
ProtectKernelTunables=true
|
||||||
|
ProtectKernelModules=true
|
||||||
|
ProtectKernelLogs=true
|
||||||
|
ProtectControlGroups=true
|
||||||
|
ProtectHostname=true
|
||||||
|
ProtectProc=invisible
|
||||||
|
ProcSubset=pid
|
||||||
|
RestrictNamespaces=true
|
||||||
|
MemoryDenyWriteExecute=true
|
||||||
|
RestrictSUIDSGID=true
|
||||||
|
LockPersonality=true
|
||||||
|
CapabilityBoundingSet=
|
||||||
|
AmbientCapabilities=
|
||||||
|
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
|
||||||
|
StateDirectory=aios-orchestrator-gateway
|
||||||
|
ReadOnlyPaths=/etc/aios-orchestrator-gateway /opt/aios-orchestrator-gateway
|
||||||
|
ReadWritePaths=/var/lib/aios-orchestrator-gateway
|
||||||
|
SystemCallFilter=@system-service
|
||||||
|
SystemCallErrorNumber=EPERM
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=Periodically poll AIOS orchestration inboxes and model responses
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnBootSec=45s
|
||||||
|
OnUnitActiveSec=60s
|
||||||
|
RandomizedDelaySec=10s
|
||||||
|
Persistent=true
|
||||||
|
Unit=aios-orchestrator-gateway-poll.service
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=AIOS Orchestrator Gateway
|
||||||
|
After=network-online.target
|
||||||
|
Wants=network-online.target
|
||||||
|
ConditionPathExists=/etc/aios-orchestrator-gateway/config.json
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
User=aios-orchestrator
|
||||||
|
Group=aios-orchestrator
|
||||||
|
SupplementaryGroups=gitops
|
||||||
|
WorkingDirectory=/opt/aios-orchestrator-gateway/current
|
||||||
|
ExecStart=/usr/bin/python3 -m services.aios_orchestrator_gateway --config /etc/aios-orchestrator-gateway/config.json
|
||||||
|
Restart=on-failure
|
||||||
|
RestartSec=5s
|
||||||
|
TimeoutStartSec=30s
|
||||||
|
TimeoutStopSec=30s
|
||||||
|
UMask=0077
|
||||||
|
|
||||||
|
NoNewPrivileges=true
|
||||||
|
PrivateTmp=true
|
||||||
|
PrivateDevices=true
|
||||||
|
ProtectSystem=strict
|
||||||
|
ProtectHome=true
|
||||||
|
ProtectKernelTunables=true
|
||||||
|
ProtectKernelModules=true
|
||||||
|
ProtectKernelLogs=true
|
||||||
|
ProtectControlGroups=true
|
||||||
|
ProtectClock=true
|
||||||
|
ProtectHostname=true
|
||||||
|
ProtectProc=invisible
|
||||||
|
ProcSubset=pid
|
||||||
|
RestrictNamespaces=true
|
||||||
|
MemoryDenyWriteExecute=true
|
||||||
|
RestrictSUIDSGID=true
|
||||||
|
LockPersonality=true
|
||||||
|
RestrictRealtime=true
|
||||||
|
CapabilityBoundingSet=
|
||||||
|
AmbientCapabilities=
|
||||||
|
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
|
||||||
|
SystemCallArchitectures=native
|
||||||
|
StateDirectory=aios-orchestrator-gateway
|
||||||
|
RuntimeDirectory=aios-orchestrator-gateway
|
||||||
|
ReadOnlyPaths=/etc/aios-orchestrator-gateway /opt/aios-orchestrator-gateway
|
||||||
|
ReadWritePaths=/var/lib/aios-orchestrator-gateway
|
||||||
|
SystemCallFilter=@system-service
|
||||||
|
SystemCallErrorNumber=EPERM
|
||||||
|
StandardOutput=journal
|
||||||
|
StandardError=journal
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
g aios-orchestrator -
|
||||||
|
u aios-orchestrator - "AIOS Orchestrator Gateway" /var/lib/aios-orchestrator-gateway /usr/sbin/nologin
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
d /etc/aios-orchestrator-gateway 0700 root root -
|
||||||
|
d /var/lib/aios-orchestrator-gateway 0700 aios-orchestrator aios-orchestrator -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/inbox 0700 aios-orchestrator aios-orchestrator -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/inbox/events 0700 aios-orchestrator aios-orchestrator -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/inbox/events/pending 2770 aios-orchestrator gitops -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/inbox/events/processed 0700 aios-orchestrator aios-orchestrator -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/inbox/events/rejected 0700 aios-orchestrator aios-orchestrator -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/inbox/decisions 0700 aios-orchestrator aios-orchestrator -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/inbox/decisions/pending 2770 aios-orchestrator gitops -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/inbox/decisions/processed 0700 aios-orchestrator aios-orchestrator -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/inbox/decisions/rejected 0700 aios-orchestrator aios-orchestrator -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/outbox 0700 aios-orchestrator aios-orchestrator -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/openclaw-sync 2750 aios-orchestrator gitops -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/openclaw-sync/pending 2750 aios-orchestrator gitops -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/commands 2750 aios-orchestrator gitops -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/commands/openclaw 2750 aios-orchestrator gitops -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/commands/openclaw/pending 2750 aios-orchestrator gitops -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/commands/codex 2750 aios-orchestrator gitops -
|
||||||
|
d /var/lib/aios-orchestrator-gateway/commands/codex/pending 2750 aios-orchestrator gitops -
|
||||||
@@ -1,39 +0,0 @@
|
|||||||
apiVersion: aios.v2/v1
|
|
||||||
kind: AgentTask
|
|
||||||
metadata:
|
|
||||||
name: codex-production-supervisor-task
|
|
||||||
owner: AIOS-Control-Plane
|
|
||||||
spec:
|
|
||||||
executor: codex
|
|
||||||
role: architecture-supervisor
|
|
||||||
objective: >-
|
|
||||||
Supervise AIOS V2R1 production candidate deployment, resolve blocking issues,
|
|
||||||
review patches, and maintain architecture/document consistency.
|
|
||||||
responsibilities:
|
|
||||||
- review OpenClaw deployment evidence
|
|
||||||
- review Hermes runtime issues
|
|
||||||
- analyze failed tests and blockers
|
|
||||||
- approve or reject patch direction
|
|
||||||
- verify docs/code/traceability consistency
|
|
||||||
- require ADR updates for architecture changes
|
|
||||||
- maintain final release readiness assessment
|
|
||||||
prohibited:
|
|
||||||
- bypass GitOps process
|
|
||||||
- directly apply unreviewed production changes
|
|
||||||
- weaken tenant isolation or security gates
|
|
||||||
- approve missing evidence
|
|
||||||
required_checks:
|
|
||||||
- requirement traceability
|
|
||||||
- architecture boundary compliance
|
|
||||||
- test evidence
|
|
||||||
- secret safety
|
|
||||||
- rollback readiness
|
|
||||||
- document synchronization
|
|
||||||
evidence_required:
|
|
||||||
- claim
|
|
||||||
- heartbeat
|
|
||||||
- progress
|
|
||||||
- blocker
|
|
||||||
- review_result
|
|
||||||
- commit
|
|
||||||
- final_recommendation
|
|
||||||
@@ -1,35 +0,0 @@
|
|||||||
apiVersion: aios.v2/v1
|
|
||||||
kind: AgentTask
|
|
||||||
metadata:
|
|
||||||
name: openclaw-production-deploy-task
|
|
||||||
owner: AIOS-Control-Plane
|
|
||||||
spec:
|
|
||||||
executor: openclaw
|
|
||||||
role: deployment-operator
|
|
||||||
objective: >-
|
|
||||||
Deploy AIOS V2R1 candidate runtime, execute health checks, integrate OpenClaw
|
|
||||||
runtime boundary, and fix low-risk runtime/deployment defects.
|
|
||||||
responsibilities:
|
|
||||||
- install systemd or Docker candidate deployment
|
|
||||||
- validate environment configuration
|
|
||||||
- execute healthcheck and smoke tests
|
|
||||||
- verify OpenClaw runtime connectivity
|
|
||||||
- fix small deployment/runtime defects through patch branches
|
|
||||||
- update ExecutionEvidence after every step
|
|
||||||
prohibited:
|
|
||||||
- modify AIOS control-plane architecture
|
|
||||||
- bypass GitOps workflow
|
|
||||||
- commit secrets
|
|
||||||
- directly edit production without a recorded change
|
|
||||||
evidence_required:
|
|
||||||
- claim
|
|
||||||
- heartbeat
|
|
||||||
- progress
|
|
||||||
- blocker
|
|
||||||
- commit
|
|
||||||
- test_status
|
|
||||||
- result
|
|
||||||
escalation:
|
|
||||||
- architecture change -> Codex supervisor
|
|
||||||
- security issue -> AIOS owner
|
|
||||||
- blocked deployment -> Codex supervisor
|
|
||||||
@@ -1,256 +0,0 @@
|
|||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import sys
|
|
||||||
import tempfile
|
|
||||||
import unittest
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
REPO_ROOT = Path(__file__).resolve().parents[2]
|
|
||||||
APP_ROOT = REPO_ROOT / "active-runtime" / "app"
|
|
||||||
if str(APP_ROOT) not in sys.path:
|
|
||||||
sys.path.insert(0, str(APP_ROOT))
|
|
||||||
|
|
||||||
from aios_hermes import HermesAdapter, HermesJobState, HermesWorker
|
|
||||||
from aios_openclaw import (
|
|
||||||
CommandState,
|
|
||||||
InProcessRuntimeTransport,
|
|
||||||
OpenClawAdapter,
|
|
||||||
)
|
|
||||||
from aios_v2 import (
|
|
||||||
ControlPlaneService,
|
|
||||||
ExecutionEvidence,
|
|
||||||
ExecutionOrchestrator,
|
|
||||||
ExecutionTracker,
|
|
||||||
RequestContext,
|
|
||||||
TaskState,
|
|
||||||
TenantIsolationError,
|
|
||||||
)
|
|
||||||
from aios_v2.persistence import DurableControlPlaneService, SQLiteSnapshotRepository
|
|
||||||
|
|
||||||
|
|
||||||
SIGNING_MATERIAL = b"candidate-test-signing-material-0001"
|
|
||||||
|
|
||||||
|
|
||||||
def context(
|
|
||||||
tenant_id: str,
|
|
||||||
principal: str,
|
|
||||||
*roles: str,
|
|
||||||
) -> RequestContext:
|
|
||||||
return RequestContext.create(
|
|
||||||
tenant_id=tenant_id,
|
|
||||||
principal_id=principal,
|
|
||||||
roles=roles,
|
|
||||||
trace_id=f"trace-{tenant_id}-{principal}",
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def bootstrap(service: ControlPlaneService, tenant_id: str = "tenant-a"):
|
|
||||||
platform = context(tenant_id, "platform", "platform_admin")
|
|
||||||
service.register_tenant(
|
|
||||||
platform,
|
|
||||||
tenant_id=tenant_id,
|
|
||||||
display_name=tenant_id,
|
|
||||||
)
|
|
||||||
operator = context(
|
|
||||||
tenant_id,
|
|
||||||
"operator",
|
|
||||||
"tenant_admin",
|
|
||||||
"operator",
|
|
||||||
"approver",
|
|
||||||
"auditor",
|
|
||||||
"runtime",
|
|
||||||
)
|
|
||||||
return platform, operator
|
|
||||||
|
|
||||||
|
|
||||||
class CandidateFlowTests(unittest.TestCase):
|
|
||||||
def test_openclaw_full_flow_reconciles_completed_task(self) -> None:
|
|
||||||
service = ControlPlaneService()
|
|
||||||
_, operator = bootstrap(service)
|
|
||||||
agent = service.register_agent(
|
|
||||||
operator,
|
|
||||||
agent_id="openclaw-agent",
|
|
||||||
display_name="OpenClaw",
|
|
||||||
runtime="openclaw",
|
|
||||||
capabilities=("echo",),
|
|
||||||
)
|
|
||||||
task = service.create_task(
|
|
||||||
operator,
|
|
||||||
agent_id=agent.agent_id,
|
|
||||||
action="echo",
|
|
||||||
payload={"message": "hello"},
|
|
||||||
idempotency_key="openclaw-flow-1",
|
|
||||||
)
|
|
||||||
approved = service.transition_task(
|
|
||||||
operator,
|
|
||||||
task_id=task.task_id,
|
|
||||||
target_state=TaskState.APPROVED,
|
|
||||||
expected_revision=task.revision,
|
|
||||||
)
|
|
||||||
|
|
||||||
transport = InProcessRuntimeTransport(
|
|
||||||
lambda command: {
|
|
||||||
"state": CommandState.COMPLETED.value,
|
|
||||||
"output": {"echo": command.payload["message"]},
|
|
||||||
}
|
|
||||||
)
|
|
||||||
resolver = lambda tenant_id, key_id: SIGNING_MATERIAL
|
|
||||||
adapter = OpenClawAdapter(transport, resolver)
|
|
||||||
orchestrator = ExecutionOrchestrator(
|
|
||||||
service,
|
|
||||||
openclaw=adapter,
|
|
||||||
openclaw_signing_material_resolver=resolver,
|
|
||||||
)
|
|
||||||
|
|
||||||
dispatch = orchestrator.dispatch(operator, task_id=approved.task_id)
|
|
||||||
completed = service.get_task(operator, task.task_id)
|
|
||||||
|
|
||||||
self.assertEqual(dispatch.runtime, "openclaw")
|
|
||||||
self.assertEqual(dispatch.state, CommandState.COMPLETED.value)
|
|
||||||
self.assertEqual(completed.state, TaskState.COMPLETED)
|
|
||||||
self.assertEqual(completed.result, {"echo": "hello"})
|
|
||||||
|
|
||||||
def test_hermes_full_flow_reconciles_worker_result(self) -> None:
|
|
||||||
service = ControlPlaneService()
|
|
||||||
_, operator = bootstrap(service)
|
|
||||||
agent = service.register_agent(
|
|
||||||
operator,
|
|
||||||
agent_id="hermes-agent",
|
|
||||||
display_name="Hermes",
|
|
||||||
runtime="hermes",
|
|
||||||
capabilities=("transform",),
|
|
||||||
metadata={"cell_id": "cell-a", "profile_id": "profile-a"},
|
|
||||||
)
|
|
||||||
task = service.create_task(
|
|
||||||
operator,
|
|
||||||
agent_id=agent.agent_id,
|
|
||||||
action="transform",
|
|
||||||
payload={"value": 7},
|
|
||||||
idempotency_key="hermes-flow-1",
|
|
||||||
)
|
|
||||||
approved = service.transition_task(
|
|
||||||
operator,
|
|
||||||
task_id=task.task_id,
|
|
||||||
target_state=TaskState.APPROVED,
|
|
||||||
expected_revision=task.revision,
|
|
||||||
)
|
|
||||||
|
|
||||||
hermes = HermesAdapter(default_lease_seconds=10)
|
|
||||||
orchestrator = ExecutionOrchestrator(service, hermes=hermes)
|
|
||||||
dispatch = orchestrator.dispatch(operator, task_id=approved.task_id)
|
|
||||||
self.assertEqual(dispatch.runtime, "hermes")
|
|
||||||
self.assertEqual(dispatch.state, HermesJobState.QUEUED.value)
|
|
||||||
|
|
||||||
worker = HermesWorker(
|
|
||||||
hermes,
|
|
||||||
lambda job: {"transformed": int(job.payload["value"]) * 2},
|
|
||||||
worker_id="worker-a",
|
|
||||||
tenant_id="tenant-a",
|
|
||||||
cell_id="cell-a",
|
|
||||||
profile_id="profile-a",
|
|
||||||
heartbeat_seconds=0.05,
|
|
||||||
)
|
|
||||||
run = worker.run_once()
|
|
||||||
self.assertTrue(run.claimed)
|
|
||||||
self.assertEqual(run.state, HermesJobState.COMPLETED.value)
|
|
||||||
|
|
||||||
result = hermes.result(dispatch.runtime_execution_id, tenant_id="tenant-a")
|
|
||||||
completed = orchestrator.reconcile_hermes(operator, result)
|
|
||||||
self.assertEqual(completed.state, TaskState.COMPLETED)
|
|
||||||
self.assertEqual(completed.result, {"transformed": 14})
|
|
||||||
|
|
||||||
def test_cross_tenant_task_read_is_denied(self) -> None:
|
|
||||||
service = ControlPlaneService()
|
|
||||||
_, tenant_a = bootstrap(service, "tenant-a")
|
|
||||||
platform_b = context("tenant-b", "platform", "platform_admin")
|
|
||||||
service.register_tenant(
|
|
||||||
platform_b,
|
|
||||||
tenant_id="tenant-b",
|
|
||||||
display_name="Tenant B",
|
|
||||||
)
|
|
||||||
tenant_b = context("tenant-b", "auditor", "auditor")
|
|
||||||
|
|
||||||
agent = service.register_agent(
|
|
||||||
tenant_a,
|
|
||||||
agent_id="agent-a",
|
|
||||||
display_name="Agent A",
|
|
||||||
runtime="hermes",
|
|
||||||
capabilities=("work",),
|
|
||||||
)
|
|
||||||
task = service.create_task(
|
|
||||||
tenant_a,
|
|
||||||
agent_id=agent.agent_id,
|
|
||||||
action="work",
|
|
||||||
payload={},
|
|
||||||
idempotency_key="tenant-isolation-1",
|
|
||||||
)
|
|
||||||
|
|
||||||
with self.assertRaises(TenantIsolationError):
|
|
||||||
service.get_task(tenant_b, task.task_id)
|
|
||||||
|
|
||||||
def test_durable_service_recovers_state_after_restart(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as temporary:
|
|
||||||
path = Path(temporary) / "state.sqlite3"
|
|
||||||
service = DurableControlPlaneService(SQLiteSnapshotRepository(path))
|
|
||||||
_, operator = bootstrap(service)
|
|
||||||
agent = service.register_agent(
|
|
||||||
operator,
|
|
||||||
agent_id="persistent-agent",
|
|
||||||
display_name="Persistent",
|
|
||||||
runtime="hermes",
|
|
||||||
capabilities=("persist",),
|
|
||||||
)
|
|
||||||
task = service.create_task(
|
|
||||||
operator,
|
|
||||||
agent_id=agent.agent_id,
|
|
||||||
action="persist",
|
|
||||||
payload={"ok": True},
|
|
||||||
idempotency_key="persist-1",
|
|
||||||
)
|
|
||||||
|
|
||||||
recovered = DurableControlPlaneService(SQLiteSnapshotRepository(path))
|
|
||||||
restored_task = recovered.get_task(operator, task.task_id)
|
|
||||||
self.assertEqual(restored_task.payload, {"ok": True})
|
|
||||||
self.assertEqual(restored_task.state, TaskState.CREATED)
|
|
||||||
|
|
||||||
def test_execution_tracking_persists_claim_heartbeat_and_result(self) -> None:
|
|
||||||
with tempfile.TemporaryDirectory() as temporary:
|
|
||||||
tracker = ExecutionTracker(temporary)
|
|
||||||
claimed = tracker.claim(
|
|
||||||
ExecutionEvidence(
|
|
||||||
execution_id="exec-1",
|
|
||||||
tenant_id="tenant-a",
|
|
||||||
task_id="task-1",
|
|
||||||
agent_id="agent-1",
|
|
||||||
executor="codex",
|
|
||||||
branch="task/codex/example",
|
|
||||||
progress_total=3,
|
|
||||||
)
|
|
||||||
)
|
|
||||||
self.assertEqual(claimed.status, "claimed")
|
|
||||||
tracker.heartbeat(
|
|
||||||
tenant_id="tenant-a",
|
|
||||||
execution_id="exec-1",
|
|
||||||
status="running",
|
|
||||||
current_step="tests",
|
|
||||||
progress_completed=2,
|
|
||||||
last_commit="abc123",
|
|
||||||
)
|
|
||||||
tracker.complete(
|
|
||||||
tenant_id="tenant-a",
|
|
||||||
execution_id="exec-1",
|
|
||||||
status="complete",
|
|
||||||
test_status="passed",
|
|
||||||
test_summary={"passed": 5, "failed": 0},
|
|
||||||
)
|
|
||||||
restored = ExecutionTracker(temporary).get(
|
|
||||||
tenant_id="tenant-a",
|
|
||||||
execution_id="exec-1",
|
|
||||||
)
|
|
||||||
self.assertEqual(restored.status, "complete")
|
|
||||||
self.assertEqual(restored.progress_completed, 3)
|
|
||||||
self.assertEqual(restored.test_summary["passed"], 5)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
@@ -0,0 +1,196 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
from pathlib import Path
|
||||||
|
import sqlite3
|
||||||
|
import tempfile
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
from services.aios_codex_watcher.watcher import CommandWatcher, Config, WatcherError, validate_command
|
||||||
|
|
||||||
|
|
||||||
|
CONTRACT = "cbe33ecab8e9794ba5f318bea43aeaf447a56fbc"
|
||||||
|
|
||||||
|
|
||||||
|
class Clock:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.value = 1000.0
|
||||||
|
|
||||||
|
def __call__(self) -> float:
|
||||||
|
return self.value
|
||||||
|
|
||||||
|
|
||||||
|
class WatcherTests(unittest.TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.root = Path(self.temp.name)
|
||||||
|
for name in ("active", "completed", "dead", "state", "log"):
|
||||||
|
(self.root / name).mkdir(mode=0o700)
|
||||||
|
self.token = self.root / "token"
|
||||||
|
self.token.write_text("not-a-real-secret\n", encoding="utf-8")
|
||||||
|
self.token.chmod(0o600)
|
||||||
|
self.clock = Clock()
|
||||||
|
self.events: list[dict] = []
|
||||||
|
self.exec_count = 0
|
||||||
|
self.config = Config(
|
||||||
|
active_dir=self.root / "active", completed_dir=self.root / "completed",
|
||||||
|
dead_letter_dir=self.root / "dead", database_path=self.root / "state/state.sqlite3",
|
||||||
|
audit_log_path=self.root / "log/audit.jsonl",
|
||||||
|
gateway_endpoint="http://127.0.0.1:18920/v1/orchestrator/events",
|
||||||
|
gateway_token_file=self.token, contract_commit=CONTRACT,
|
||||||
|
lease_seconds=10, retry_seconds=2, max_attempts=3, poll_seconds=1,
|
||||||
|
executor_path=None, executor_args=(), executor_timeout_seconds=10,
|
||||||
|
)
|
||||||
|
|
||||||
|
def tearDown(self) -> None:
|
||||||
|
self.temp.cleanup()
|
||||||
|
|
||||||
|
def command(self, command_id: str = "cmd-1", *, contract: str = CONTRACT, kind: str = "acceptance_probe", text: str = "no-op") -> dict:
|
||||||
|
return {
|
||||||
|
"schema_version": 1, "command_id": command_id,
|
||||||
|
"project_key": "a0-codex-watcher-nonproduction",
|
||||||
|
"work_unit_id": "A0-CODEX-WATCHER-ACCEPTANCE",
|
||||||
|
"contract_commit": contract, "state_version": 0,
|
||||||
|
"execution": {"kind": kind, "input": text},
|
||||||
|
}
|
||||||
|
|
||||||
|
def write(self, value: dict, name: str = "command.json") -> Path:
|
||||||
|
path = self.config.active_dir / name
|
||||||
|
path.write_text(json.dumps(value), encoding="utf-8")
|
||||||
|
path.chmod(0o600)
|
||||||
|
return path
|
||||||
|
|
||||||
|
def watcher(self, **kwargs) -> CommandWatcher:
|
||||||
|
def execute(command: dict) -> dict:
|
||||||
|
self.exec_count += 1
|
||||||
|
return {"kind": command["execution"]["kind"], "exit_code": 0}
|
||||||
|
return CommandWatcher(
|
||||||
|
self.config, clock=self.clock,
|
||||||
|
event_poster=kwargs.get("event_poster", self.events.append),
|
||||||
|
executor=kwargs.get("executor", execute), worker_id=kwargs.get("worker_id"),
|
||||||
|
)
|
||||||
|
|
||||||
|
def row(self, command_id: str = "cmd-1") -> sqlite3.Row:
|
||||||
|
db = sqlite3.connect(self.config.database_path)
|
||||||
|
db.row_factory = sqlite3.Row
|
||||||
|
try:
|
||||||
|
return db.execute("SELECT * FROM commands WHERE command_id=?", (command_id,)).fetchone()
|
||||||
|
finally:
|
||||||
|
db.close()
|
||||||
|
|
||||||
|
def test_happy_path_posts_lifecycle_once_and_archives(self) -> None:
|
||||||
|
path = self.write(self.command())
|
||||||
|
self.assertTrue(self.watcher().process_path(path))
|
||||||
|
self.assertEqual([e["event_type"] for e in self.events], ["work_unit_started", "work_unit_ended"])
|
||||||
|
self.assertEqual(self.exec_count, 1)
|
||||||
|
self.assertEqual(self.row()["status"], "COMPLETED")
|
||||||
|
self.assertTrue((self.config.completed_dir / path.name).is_file())
|
||||||
|
markers = [json.loads(line)["marker"] for line in self.config.audit_log_path.read_text().splitlines()]
|
||||||
|
self.assertEqual(markers, ["CODEX_AUTO_CLAIM", "TASK_STARTED", "TASK_EXECUTED", "TASK_ENDED", "EVENT_POSTED"])
|
||||||
|
|
||||||
|
def test_contract_mismatch_is_rejected_without_registration(self) -> None:
|
||||||
|
path = self.write(self.command(contract="0" * 40))
|
||||||
|
self.assertFalse(self.watcher().process_path(path))
|
||||||
|
self.assertTrue((self.config.dead_letter_dir / path.name).is_file())
|
||||||
|
db = sqlite3.connect(self.config.database_path)
|
||||||
|
self.assertEqual(db.execute("SELECT count(*) FROM commands").fetchone()[0], 0)
|
||||||
|
|
||||||
|
def test_duplicate_id_with_different_payload_is_dead_lettered(self) -> None:
|
||||||
|
first = self.write(self.command(), "first.json")
|
||||||
|
self.assertTrue(self.watcher().process_path(first))
|
||||||
|
second = self.write(self.command(text="different"), "second.json")
|
||||||
|
self.assertFalse(self.watcher().process_path(second))
|
||||||
|
self.assertEqual(self.row()["status"], "COMPLETED")
|
||||||
|
self.assertTrue((self.config.dead_letter_dir / second.name).is_file())
|
||||||
|
|
||||||
|
def test_same_work_unit_with_different_command_id_is_not_executed(self) -> None:
|
||||||
|
first = self.write(self.command(), "first.json")
|
||||||
|
self.assertTrue(self.watcher().process_path(first))
|
||||||
|
second_command = self.command(command_id="cmd-2")
|
||||||
|
second = self.write(second_command, "second.json")
|
||||||
|
self.assertFalse(self.watcher().process_path(second))
|
||||||
|
self.assertEqual(self.exec_count, 1)
|
||||||
|
self.assertTrue((self.config.dead_letter_dir / second.name).is_file())
|
||||||
|
|
||||||
|
def test_two_workers_cannot_hold_same_lease(self) -> None:
|
||||||
|
path = self.write(self.command())
|
||||||
|
one = self.watcher(worker_id="worker-one")
|
||||||
|
command, digest = one._read_command(path)
|
||||||
|
one._register(command, digest, path)
|
||||||
|
self.assertIsNotNone(one._claim("cmd-1"))
|
||||||
|
two = self.watcher(worker_id="worker-two")
|
||||||
|
self.assertIsNone(two._claim("cmd-1"))
|
||||||
|
|
||||||
|
def test_stale_lease_is_recovered(self) -> None:
|
||||||
|
path = self.write(self.command())
|
||||||
|
one = self.watcher(worker_id="worker-one")
|
||||||
|
command, digest = one._read_command(path)
|
||||||
|
one._register(command, digest, path)
|
||||||
|
one._claim("cmd-1")
|
||||||
|
self.clock.value += 11
|
||||||
|
self.assertIsNotNone(self.watcher(worker_id="worker-two")._claim("cmd-1"))
|
||||||
|
|
||||||
|
def test_end_post_retry_does_not_execute_twice(self) -> None:
|
||||||
|
calls = 0
|
||||||
|
attempted_end_events: list[dict] = []
|
||||||
|
def poster(event: dict) -> None:
|
||||||
|
nonlocal calls
|
||||||
|
calls += 1
|
||||||
|
if event["event_type"] == "work_unit_ended":
|
||||||
|
attempted_end_events.append(event)
|
||||||
|
if calls == 2:
|
||||||
|
raise WatcherError("temporary")
|
||||||
|
self.events.append(event)
|
||||||
|
path = self.write(self.command())
|
||||||
|
watcher = self.watcher(event_poster=poster)
|
||||||
|
self.assertFalse(watcher.process_path(path))
|
||||||
|
self.assertEqual(self.row()["phase"], "END")
|
||||||
|
self.assertEqual(self.exec_count, 1)
|
||||||
|
self.clock.value += 3
|
||||||
|
self.assertTrue(watcher.process_path(path))
|
||||||
|
self.assertEqual(self.exec_count, 1)
|
||||||
|
self.assertEqual(len(attempted_end_events), 2)
|
||||||
|
self.assertEqual(attempted_end_events[0], attempted_end_events[1])
|
||||||
|
self.assertEqual(self.row()["status"], "COMPLETED")
|
||||||
|
|
||||||
|
def test_failure_exhaustion_moves_to_dead_letter(self) -> None:
|
||||||
|
executions = 0
|
||||||
|
def fail(_command: dict) -> dict:
|
||||||
|
nonlocal executions
|
||||||
|
executions += 1
|
||||||
|
raise WatcherError("executor failure")
|
||||||
|
path = self.write(self.command())
|
||||||
|
watcher = self.watcher(executor=fail)
|
||||||
|
for _ in range(3):
|
||||||
|
self.assertFalse(watcher.process_path(path))
|
||||||
|
self.clock.value += 3
|
||||||
|
self.assertEqual(self.row()["status"], "DEAD_LETTER")
|
||||||
|
self.assertEqual(executions, 1)
|
||||||
|
self.assertTrue((self.config.dead_letter_dir / path.name).is_file())
|
||||||
|
|
||||||
|
def test_symlink_command_is_rejected_without_following(self) -> None:
|
||||||
|
outside = self.root / "outside.json"
|
||||||
|
outside.write_text(json.dumps(self.command()), encoding="utf-8")
|
||||||
|
link = self.config.active_dir / "link.json"
|
||||||
|
os.symlink(outside, link)
|
||||||
|
self.assertFalse(self.watcher().process_path(link))
|
||||||
|
self.assertTrue(outside.is_file())
|
||||||
|
self.assertTrue((self.config.dead_letter_dir / link.name).is_symlink())
|
||||||
|
|
||||||
|
def test_codex_mode_fails_closed_when_executor_absent(self) -> None:
|
||||||
|
path = self.write(self.command(kind="codex"))
|
||||||
|
watcher = CommandWatcher(self.config, clock=self.clock, event_poster=self.events.append)
|
||||||
|
self.assertFalse(watcher.process_path(path))
|
||||||
|
self.assertEqual(self.row()["phase"], "EXECUTE")
|
||||||
|
self.assertIn("codex_executor_unavailable", self.row()["last_error"])
|
||||||
|
|
||||||
|
def test_validation_rejects_unknown_fields(self) -> None:
|
||||||
|
command = self.command()
|
||||||
|
command["secret"] = "must-not-be-accepted"
|
||||||
|
with self.assertRaisesRegex(WatcherError, "invalid_command_fields"):
|
||||||
|
validate_command(command, CONTRACT)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -0,0 +1,774 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import stat
|
||||||
|
from pathlib import Path
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
import unittest
|
||||||
|
import urllib.error
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
from services.aios_orchestrator_gateway.clients import ResponsesClient, WorkspaceAgentClient
|
||||||
|
from services.aios_orchestrator_gateway.config import Config
|
||||||
|
from services.aios_orchestrator_gateway.dispatch import DispatchSink
|
||||||
|
from services.aios_orchestrator_gateway.errors import ConflictError, DispatchError, ValidationError
|
||||||
|
from services.aios_orchestrator_gateway.models import Decision, Event
|
||||||
|
from services.aios_orchestrator_gateway.service import Runtime
|
||||||
|
from services.aios_orchestrator_gateway.store import Store
|
||||||
|
from services.aios_orchestrator_gateway.sync import OpenClawSynchronizer, lifecycle_record
|
||||||
|
from scripts import report_orchestrator_event
|
||||||
|
|
||||||
|
|
||||||
|
CONTRACT = "cbe33ecab8e9794ba5f318bea43aeaf447a56fbc"
|
||||||
|
BASE = "8d9c6da90fc8caa69b691d243c6fefb40b6b0575"
|
||||||
|
|
||||||
|
|
||||||
|
def event_payload(**overrides):
|
||||||
|
value = {
|
||||||
|
"schema_version": 1,
|
||||||
|
"project_key": "v2r1-personal-weixin",
|
||||||
|
"contract_commit": CONTRACT,
|
||||||
|
"event_id": "evt-001",
|
||||||
|
"source_agent": "openclaw",
|
||||||
|
"event_type": "work_unit_completed",
|
||||||
|
"work_unit_id": "P1-REACK",
|
||||||
|
"status": "PASS",
|
||||||
|
"state_version": 0,
|
||||||
|
"implementation_commit": None,
|
||||||
|
"result_commit": None,
|
||||||
|
"blocker": None,
|
||||||
|
"evidence_refs": [],
|
||||||
|
"payload": {},
|
||||||
|
}
|
||||||
|
value.update(overrides)
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def decision_payload(**overrides):
|
||||||
|
value = {
|
||||||
|
"schema_version": 1,
|
||||||
|
"project_key": "v2r1-personal-weixin",
|
||||||
|
"contract_commit": CONTRACT,
|
||||||
|
"event_id": "evt-001",
|
||||||
|
"expected_state_version": 0,
|
||||||
|
"verdict": "DISPATCH",
|
||||||
|
"state_updates": [
|
||||||
|
{"key": "P1_REPAIR3_CODE", "value": "PASS_CANDIDATE_PREPARED"},
|
||||||
|
{"key": "current_gate", "value": "R0_RUNNER_LIVE_RECOVERY"},
|
||||||
|
],
|
||||||
|
"next_work_units": [
|
||||||
|
{
|
||||||
|
"id": "R0-DISCOVER",
|
||||||
|
"track_id": "R_RUNNER",
|
||||||
|
"gate_id": "R0_RUNNER_LIVE_RECOVERY",
|
||||||
|
"primary_agent": "openclaw",
|
||||||
|
"fallback_agent": "codex",
|
||||||
|
"clean_base_commit": BASE,
|
||||||
|
"allowed_paths": [],
|
||||||
|
"required_checks": ["secret_provider_metadata"],
|
||||||
|
"forbidden_operations": ["force_push"],
|
||||||
|
"instruction": "Discover provider metadata without reading secrets.",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"requires_human_input": False,
|
||||||
|
"stop_reason": None,
|
||||||
|
}
|
||||||
|
value.update(overrides)
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
class ModelTests(unittest.TestCase):
|
||||||
|
def test_event_accepts_valid_payload(self):
|
||||||
|
event = Event.from_dict(event_payload())
|
||||||
|
self.assertEqual(event.event_id, "evt-001")
|
||||||
|
self.assertEqual(len(event.digest()), 64)
|
||||||
|
|
||||||
|
def test_event_rejects_extra_fields(self):
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
Event.from_dict(event_payload(extra=True))
|
||||||
|
|
||||||
|
def test_event_rejects_invalid_contract_oid(self):
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
Event.from_dict(event_payload(contract_commit="abc"))
|
||||||
|
|
||||||
|
def test_decision_requires_work_unit_for_dispatch(self):
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
Decision.from_dict(decision_payload(next_work_units=[]))
|
||||||
|
|
||||||
|
def test_decision_rejects_same_primary_fallback(self):
|
||||||
|
value = decision_payload()
|
||||||
|
value["next_work_units"][0]["fallback_agent"] = "openclaw"
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
Decision.from_dict(value)
|
||||||
|
|
||||||
|
def test_non_dispatch_rejects_work_units(self):
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
Decision.from_dict(decision_payload(verdict="BLOCKED"))
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
class ClientTests(unittest.TestCase):
|
||||||
|
def test_workspace_trigger_accepts_202_without_run_id(self):
|
||||||
|
event = Event.from_dict(event_payload())
|
||||||
|
response = mock.MagicMock()
|
||||||
|
response.status = 202
|
||||||
|
response.read.return_value = b""
|
||||||
|
response.__enter__.return_value = response
|
||||||
|
response.__exit__.return_value = False
|
||||||
|
with mock.patch(
|
||||||
|
"services.aios_orchestrator_gateway.clients.urllib.request.urlopen",
|
||||||
|
return_value=response,
|
||||||
|
) as call:
|
||||||
|
WorkspaceAgentClient("agtch_test", "token").trigger(event)
|
||||||
|
request = call.call_args.args[0]
|
||||||
|
self.assertEqual(request.headers["Idempotency-key"], "evt-001")
|
||||||
|
|
||||||
|
def test_responses_background_start_and_poll(self):
|
||||||
|
event = Event.from_dict(event_payload())
|
||||||
|
decision = decision_payload()
|
||||||
|
client = ResponsesClient("key", "gpt-test")
|
||||||
|
with mock.patch.object(
|
||||||
|
client,
|
||||||
|
"_request",
|
||||||
|
side_effect=[
|
||||||
|
{"id": "resp_123", "status": "queued"},
|
||||||
|
{
|
||||||
|
"id": "resp_123",
|
||||||
|
"status": "completed",
|
||||||
|
"output": [
|
||||||
|
{
|
||||||
|
"content": [
|
||||||
|
{
|
||||||
|
"type": "output_text",
|
||||||
|
"text": json.dumps(decision),
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
):
|
||||||
|
response_id = client.start(event)
|
||||||
|
status, parsed = client.poll(response_id)
|
||||||
|
self.assertEqual(response_id, "resp_123")
|
||||||
|
self.assertEqual(status, "completed")
|
||||||
|
self.assertEqual(parsed.event_id, "evt-001")
|
||||||
|
|
||||||
|
def test_responses_poll_keeps_queued_state(self):
|
||||||
|
client = ResponsesClient("key", "gpt-test")
|
||||||
|
with mock.patch.object(
|
||||||
|
client,
|
||||||
|
"_request",
|
||||||
|
return_value={"id": "resp_123", "status": "in_progress"},
|
||||||
|
):
|
||||||
|
status, parsed = client.poll("resp_123")
|
||||||
|
self.assertEqual(status, "in_progress")
|
||||||
|
self.assertIsNone(parsed)
|
||||||
|
|
||||||
|
def test_responses_poll_queued_in_progress_completed_sequence(self):
|
||||||
|
client = ResponsesClient("key", "gpt-test")
|
||||||
|
completed = {
|
||||||
|
"status": "completed",
|
||||||
|
"output": [{"content": [{"type": "output_text", "text": json.dumps(decision_payload())}]}],
|
||||||
|
}
|
||||||
|
with mock.patch.object(
|
||||||
|
client,
|
||||||
|
"_request",
|
||||||
|
side_effect=[{"status": "queued"}, {"status": "in_progress"}, completed],
|
||||||
|
):
|
||||||
|
self.assertEqual(client.poll("resp_123"), ("queued", None))
|
||||||
|
self.assertEqual(client.poll("resp_123"), ("in_progress", None))
|
||||||
|
status, decision = client.poll("resp_123")
|
||||||
|
self.assertEqual(status, "completed")
|
||||||
|
self.assertEqual(decision.event_id, "evt-001")
|
||||||
|
|
||||||
|
def test_responses_invalid_completed_decision_fails_closed(self):
|
||||||
|
client = ResponsesClient("key", "gpt-test")
|
||||||
|
payload = {"status": "completed", "output": [{"content": [{"type": "output_text", "text": "{}"}]}]}
|
||||||
|
with mock.patch.object(client, "_request", return_value=payload):
|
||||||
|
with self.assertRaises(ValidationError):
|
||||||
|
client.poll("resp_123")
|
||||||
|
|
||||||
|
|
||||||
|
class StoreTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.store = Store(Path(self.temp.name) / "state.sqlite3")
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.store.close()
|
||||||
|
self.temp.cleanup()
|
||||||
|
|
||||||
|
def test_event_idempotency(self):
|
||||||
|
event = Event.from_dict(event_payload())
|
||||||
|
created, _ = self.store.register_event(event)
|
||||||
|
duplicate, _ = self.store.register_event(event)
|
||||||
|
self.assertTrue(created)
|
||||||
|
self.assertFalse(duplicate)
|
||||||
|
|
||||||
|
def test_event_payload_conflict_rejected(self):
|
||||||
|
self.store.register_event(Event.from_dict(event_payload()))
|
||||||
|
with self.assertRaises(ConflictError):
|
||||||
|
self.store.register_event(Event.from_dict(event_payload(status="FAIL")))
|
||||||
|
|
||||||
|
def test_stale_state_version_rejected(self):
|
||||||
|
self.store.register_event(Event.from_dict(event_payload()))
|
||||||
|
with self.assertRaises(ConflictError):
|
||||||
|
self.store.register_event(
|
||||||
|
Event.from_dict(event_payload(event_id="evt-002", state_version=1))
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_lease_claim_prevents_double_claim(self):
|
||||||
|
self.store.register_event(Event.from_dict(event_payload()))
|
||||||
|
first = self.store.claim_events(
|
||||||
|
owner="one", statuses=("RECEIVED",), limit=5, lease_seconds=60
|
||||||
|
)
|
||||||
|
second_store = Store(Path(self.temp.name) / "state.sqlite3")
|
||||||
|
try:
|
||||||
|
second = second_store.claim_events(
|
||||||
|
owner="two", statuses=("RECEIVED",), limit=5, lease_seconds=60
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
second_store.close()
|
||||||
|
self.assertEqual(len(first), 1)
|
||||||
|
self.assertEqual(second, [])
|
||||||
|
|
||||||
|
def test_workspace_timeout_claim_prevents_double_worker(self):
|
||||||
|
self.store.register_event(Event.from_dict(event_payload()))
|
||||||
|
self.store.update_event_trigger("evt-001", status="AWAITING_DECISION", channel="workspace_agent", remote_id=None)
|
||||||
|
self.store._conn().execute("UPDATE events SET updated_at=0 WHERE event_id='evt-001'")
|
||||||
|
first = self.store.claim_workspace_timeouts(owner="one", older_than=1, limit=5, lease_seconds=60)
|
||||||
|
second = self.store.claim_workspace_timeouts(owner="two", older_than=1, limit=5, lease_seconds=60)
|
||||||
|
self.assertEqual(len(first), 1)
|
||||||
|
self.assertEqual(second, [])
|
||||||
|
|
||||||
|
def test_response_poll_claim_prevents_double_worker(self):
|
||||||
|
self.store.register_event(Event.from_dict(event_payload()))
|
||||||
|
self.store.update_event_trigger("evt-001", status="POLLING_RESPONSE", channel="responses_api", remote_id="resp_123")
|
||||||
|
first = self.store.claim_response_polls(owner="one", limit=5, lease_seconds=60)
|
||||||
|
second = self.store.claim_response_polls(owner="two", limit=5, lease_seconds=60)
|
||||||
|
self.assertEqual(len(first), 1)
|
||||||
|
self.assertEqual(second, [])
|
||||||
|
|
||||||
|
def test_max_hops_limit(self):
|
||||||
|
self.store.register_event(Event.from_dict(event_payload()))
|
||||||
|
self.store._conn().execute(
|
||||||
|
"""
|
||||||
|
INSERT INTO gate_hops(project_key, gate_id, hops, updated_at)
|
||||||
|
VALUES ('v2r1-personal-weixin', 'R0_RUNNER_LIVE_RECOVERY', 1, 0)
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
with self.assertRaisesRegex(ConflictError, "max_hops_exceeded"):
|
||||||
|
self.store.apply_decision(
|
||||||
|
Decision.from_dict(decision_payload()),
|
||||||
|
max_hops_per_gate=1,
|
||||||
|
max_same_blocker_retries=3,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_hop_limit_is_scoped_per_gate(self):
|
||||||
|
self.store.register_event(Event.from_dict(event_payload()))
|
||||||
|
first = Decision.from_dict(decision_payload())
|
||||||
|
self.assertEqual(
|
||||||
|
self.store.apply_decision(
|
||||||
|
first, max_hops_per_gate=1, max_same_blocker_retries=3
|
||||||
|
),
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.store.register_event(
|
||||||
|
Event.from_dict(event_payload(event_id="evt-002", state_version=1))
|
||||||
|
)
|
||||||
|
second_value = decision_payload(
|
||||||
|
event_id="evt-002",
|
||||||
|
expected_state_version=1,
|
||||||
|
state_updates=[{"key": "current_gate", "value": "P2_REAL_B3_EVIDENCE"}],
|
||||||
|
)
|
||||||
|
second_value["next_work_units"][0]["id"] = "P2-REVIEW"
|
||||||
|
second_value["next_work_units"][0]["gate_id"] = "P2_REAL_B3_EVIDENCE"
|
||||||
|
self.assertEqual(
|
||||||
|
self.store.apply_decision(
|
||||||
|
Decision.from_dict(second_value),
|
||||||
|
max_hops_per_gate=1,
|
||||||
|
max_same_blocker_retries=3,
|
||||||
|
),
|
||||||
|
2,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.store.register_event(
|
||||||
|
Event.from_dict(event_payload(event_id="evt-003", state_version=2))
|
||||||
|
)
|
||||||
|
third_value = decision_payload(
|
||||||
|
event_id="evt-003",
|
||||||
|
expected_state_version=2,
|
||||||
|
state_updates=[{"key": "current_gate", "value": "P2_REAL_B3_EVIDENCE"}],
|
||||||
|
)
|
||||||
|
third_value["next_work_units"][0]["id"] = "P2-REVIEW-RETRY"
|
||||||
|
third_value["next_work_units"][0]["gate_id"] = "P2_REAL_B3_EVIDENCE"
|
||||||
|
with self.assertRaisesRegex(ConflictError, "max_hops_exceeded"):
|
||||||
|
self.store.apply_decision(
|
||||||
|
Decision.from_dict(third_value),
|
||||||
|
max_hops_per_gate=1,
|
||||||
|
max_same_blocker_retries=3,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_decision_advances_state_and_creates_outbox(self):
|
||||||
|
self.store.register_event(Event.from_dict(event_payload()))
|
||||||
|
version = self.store.apply_decision(
|
||||||
|
Decision.from_dict(decision_payload()),
|
||||||
|
max_hops_per_gate=20,
|
||||||
|
max_same_blocker_retries=3,
|
||||||
|
)
|
||||||
|
self.assertEqual(version, 1)
|
||||||
|
state = self.store.project_state("v2r1-personal-weixin")
|
||||||
|
self.assertEqual(state["state_version"], 1)
|
||||||
|
rows = self.store.claim_outbox(owner="x", limit=10, lease_seconds=60)
|
||||||
|
self.assertEqual(len(rows), 1)
|
||||||
|
|
||||||
|
def test_decision_idempotency(self):
|
||||||
|
self.store.register_event(Event.from_dict(event_payload()))
|
||||||
|
decision = Decision.from_dict(decision_payload())
|
||||||
|
first = self.store.apply_decision(
|
||||||
|
decision, max_hops_per_gate=20, max_same_blocker_retries=3
|
||||||
|
)
|
||||||
|
second = self.store.apply_decision(
|
||||||
|
decision, max_hops_per_gate=20, max_same_blocker_retries=3
|
||||||
|
)
|
||||||
|
self.assertEqual(first, second)
|
||||||
|
|
||||||
|
def test_same_blocker_limit(self):
|
||||||
|
self.store.register_event(Event.from_dict(event_payload()))
|
||||||
|
decision = Decision.from_dict(
|
||||||
|
decision_payload(
|
||||||
|
verdict="BLOCKED",
|
||||||
|
next_work_units=[],
|
||||||
|
stop_reason="SAME_BLOCKER",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.store.apply_decision(
|
||||||
|
decision, max_hops_per_gate=20, max_same_blocker_retries=1
|
||||||
|
)
|
||||||
|
# New event at next state with the same blocker exceeds the configured limit.
|
||||||
|
self.store.register_event(
|
||||||
|
Event.from_dict(event_payload(event_id="evt-002", state_version=1))
|
||||||
|
)
|
||||||
|
decision2 = Decision.from_dict(
|
||||||
|
decision_payload(
|
||||||
|
event_id="evt-002",
|
||||||
|
expected_state_version=1,
|
||||||
|
verdict="BLOCKED",
|
||||||
|
next_work_units=[],
|
||||||
|
stop_reason="SAME_BLOCKER",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
with self.assertRaises(ConflictError):
|
||||||
|
self.store.apply_decision(
|
||||||
|
decision2, max_hops_per_gate=20, max_same_blocker_retries=1
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_project_summary_contains_counts(self):
|
||||||
|
self.store.register_event(Event.from_dict(event_payload()))
|
||||||
|
summary = self.store.project_summary("v2r1-personal-weixin")
|
||||||
|
self.assertEqual(summary["event_counts"]["RECEIVED"], 1)
|
||||||
|
|
||||||
|
|
||||||
|
class DispatchTests(unittest.TestCase):
|
||||||
|
def test_dispatch_is_idempotent(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
outbox = Path(temp) / "outbox"
|
||||||
|
queue = Path(temp) / "queue"
|
||||||
|
outbox.mkdir(mode=0o700)
|
||||||
|
queue.mkdir(mode=0o2770)
|
||||||
|
queue.chmod(0o2770)
|
||||||
|
sink = DispatchSink(outbox, queue, CONTRACT)
|
||||||
|
record = {
|
||||||
|
"work_unit_id": "R0-DISCOVER",
|
||||||
|
"event_id": "evt-001",
|
||||||
|
"assigned_agent": "openclaw",
|
||||||
|
"payload_json": json.dumps(decision_payload()["next_work_units"][0]),
|
||||||
|
}
|
||||||
|
first = sink.dispatch(record)
|
||||||
|
second = sink.dispatch(record)
|
||||||
|
self.assertEqual(first, second)
|
||||||
|
self.assertTrue(first[0].exists())
|
||||||
|
self.assertTrue(first[1].exists())
|
||||||
|
self.assertEqual(stat.S_IMODE(first[1].stat().st_mode), 0o640)
|
||||||
|
|
||||||
|
def test_dispatch_rejects_path_escape(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
outbox = Path(temp) / "outbox"
|
||||||
|
queue = Path(temp) / "queue"
|
||||||
|
outbox.mkdir(mode=0o700)
|
||||||
|
queue.mkdir(mode=0o2770)
|
||||||
|
queue.chmod(0o2770)
|
||||||
|
sink = DispatchSink(outbox, queue, CONTRACT)
|
||||||
|
with self.assertRaises(DispatchError):
|
||||||
|
sink.dispatch(
|
||||||
|
{
|
||||||
|
"work_unit_id": "../escape",
|
||||||
|
"event_id": "evt-001",
|
||||||
|
"assigned_agent": "openclaw",
|
||||||
|
"payload_json": "{}",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
def test_dispatch_rejects_existing_symlink_target(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
outbox = Path(temp) / "outbox"
|
||||||
|
queue = Path(temp) / "queue"
|
||||||
|
outbox.mkdir(mode=0o700)
|
||||||
|
queue.mkdir(mode=0o2770)
|
||||||
|
queue.chmod(0o2770)
|
||||||
|
sink = DispatchSink(outbox, queue, CONTRACT)
|
||||||
|
target = Path(temp) / "target"
|
||||||
|
target.write_text("x")
|
||||||
|
pending = outbox / "dispatched"
|
||||||
|
pending.mkdir(mode=0o700)
|
||||||
|
(pending / "R0-DISCOVER.json").symlink_to(target)
|
||||||
|
with self.assertRaises(DispatchError):
|
||||||
|
sink.dispatch(
|
||||||
|
{
|
||||||
|
"work_unit_id": "R0-DISCOVER",
|
||||||
|
"event_id": "evt-001",
|
||||||
|
"assigned_agent": "openclaw",
|
||||||
|
"payload_json": json.dumps(decision_payload()["next_work_units"][0]),
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class SyncTests(unittest.TestCase):
|
||||||
|
def test_lifecycle_record_is_stable(self):
|
||||||
|
one = lifecycle_record(
|
||||||
|
contract_commit=CONTRACT,
|
||||||
|
project_key="v2r1",
|
||||||
|
kind="TASK_STARTED",
|
||||||
|
work_unit_id="R0",
|
||||||
|
payload={"status": "RUNNING"},
|
||||||
|
)
|
||||||
|
two = lifecycle_record(
|
||||||
|
contract_commit=CONTRACT,
|
||||||
|
project_key="v2r1",
|
||||||
|
kind="TASK_STARTED",
|
||||||
|
work_unit_id="R0",
|
||||||
|
payload={"status": "RUNNING"},
|
||||||
|
)
|
||||||
|
self.assertEqual(one["sync_id"], two["sync_id"])
|
||||||
|
|
||||||
|
def test_file_sync_is_idempotent(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
root = Path(temp)
|
||||||
|
root.chmod(0o2750)
|
||||||
|
sync = OpenClawSynchronizer(root)
|
||||||
|
record = lifecycle_record(
|
||||||
|
contract_commit=CONTRACT,
|
||||||
|
project_key="v2r1",
|
||||||
|
kind="PROJECT_SUMMARY",
|
||||||
|
work_unit_id=None,
|
||||||
|
payload={"done": 1},
|
||||||
|
)
|
||||||
|
first = sync.emit(record)
|
||||||
|
second = sync.emit(record)
|
||||||
|
self.assertEqual(first, second)
|
||||||
|
|
||||||
|
def test_http_failure_keeps_file_for_retry(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
root = Path(temp)
|
||||||
|
root.chmod(0o2750)
|
||||||
|
sync = OpenClawSynchronizer(root, endpoint="https://openclaw.invalid/events")
|
||||||
|
record = lifecycle_record(
|
||||||
|
contract_commit=CONTRACT,
|
||||||
|
project_key="v2r1",
|
||||||
|
kind="PROJECT_COMPLETED",
|
||||||
|
work_unit_id=None,
|
||||||
|
payload={"done": 1},
|
||||||
|
)
|
||||||
|
with mock.patch(
|
||||||
|
"services.aios_orchestrator_gateway.sync.urllib.request.urlopen",
|
||||||
|
side_effect=OSError("offline"),
|
||||||
|
):
|
||||||
|
with self.assertRaises(DispatchError):
|
||||||
|
sync.emit(record)
|
||||||
|
pending = root / "pending" / f"{record['sync_id']}.json"
|
||||||
|
self.assertTrue(pending.exists())
|
||||||
|
response = mock.MagicMock(status=202)
|
||||||
|
response.__enter__.return_value = response
|
||||||
|
response.__exit__.return_value = False
|
||||||
|
with mock.patch(
|
||||||
|
"services.aios_orchestrator_gateway.sync.urllib.request.urlopen",
|
||||||
|
return_value=response,
|
||||||
|
):
|
||||||
|
self.assertEqual(sync.emit(record), pending)
|
||||||
|
|
||||||
|
|
||||||
|
class ReporterTests(unittest.TestCase):
|
||||||
|
def test_http_failure_falls_back_to_file_inbox(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
root = Path(temp)
|
||||||
|
inbox = root / "inbox"
|
||||||
|
inbox.mkdir(mode=0o2770)
|
||||||
|
inbox.chmod(0o2770)
|
||||||
|
token = root / "token"
|
||||||
|
token.write_text("not-a-real-secret", encoding="utf-8")
|
||||||
|
token.chmod(0o600)
|
||||||
|
argv = [
|
||||||
|
"report_orchestrator_event.py",
|
||||||
|
"--project-key", "v2r1-personal-weixin",
|
||||||
|
"--contract-commit", CONTRACT,
|
||||||
|
"--event-id", "evt-http-fallback",
|
||||||
|
"--source-agent", "codex",
|
||||||
|
"--event-type", "work_unit_completed",
|
||||||
|
"--work-unit-id", "P1-REACK",
|
||||||
|
"--status", "PASS",
|
||||||
|
"--state-version", "0",
|
||||||
|
"--endpoint", "https://gateway.invalid",
|
||||||
|
"--token-file", str(token),
|
||||||
|
"--inbox-dir", str(inbox),
|
||||||
|
]
|
||||||
|
with mock.patch("sys.argv", argv), mock.patch(
|
||||||
|
"scripts.report_orchestrator_event.urllib.request.urlopen",
|
||||||
|
side_effect=urllib.error.URLError("offline"),
|
||||||
|
):
|
||||||
|
self.assertEqual(report_orchestrator_event.main(), 0)
|
||||||
|
pending = inbox / "pending/evt-http-fallback.json"
|
||||||
|
self.assertTrue(pending.exists())
|
||||||
|
self.assertEqual(stat.S_IMODE(pending.stat().st_mode), 0o640)
|
||||||
|
|
||||||
|
def test_payload_symlink_is_rejected(self):
|
||||||
|
with tempfile.TemporaryDirectory() as temp:
|
||||||
|
root = Path(temp)
|
||||||
|
target = root / "payload.json"
|
||||||
|
target.write_text("{}", encoding="utf-8")
|
||||||
|
target.chmod(0o600)
|
||||||
|
link = root / "payload-link.json"
|
||||||
|
link.symlink_to(target)
|
||||||
|
argv = [
|
||||||
|
"report_orchestrator_event.py",
|
||||||
|
"--project-key", "v2r1-personal-weixin",
|
||||||
|
"--contract-commit", CONTRACT,
|
||||||
|
"--event-id", "evt-symlink",
|
||||||
|
"--source-agent", "codex",
|
||||||
|
"--event-type", "work_unit_completed",
|
||||||
|
"--work-unit-id", "P1-REACK",
|
||||||
|
"--status", "PASS",
|
||||||
|
"--state-version", "0",
|
||||||
|
"--payload-file", str(link),
|
||||||
|
"--inbox-dir", str(root / "inbox"),
|
||||||
|
]
|
||||||
|
with mock.patch("sys.argv", argv), self.assertRaisesRegex(
|
||||||
|
Exception, "symlink_rejected"
|
||||||
|
):
|
||||||
|
report_orchestrator_event.main()
|
||||||
|
|
||||||
|
|
||||||
|
class RuntimePollingTests(unittest.TestCase):
|
||||||
|
def setUp(self):
|
||||||
|
self.temp = tempfile.TemporaryDirectory()
|
||||||
|
self.root = Path(self.temp.name)
|
||||||
|
for name in ("events", "decisions", "outbox", "etc"):
|
||||||
|
(self.root / name).mkdir(mode=0o700)
|
||||||
|
for name in ("sync", "commands"):
|
||||||
|
(self.root / name).mkdir(mode=0o2750)
|
||||||
|
(self.root / name).chmod(0o2750)
|
||||||
|
for name, value in (
|
||||||
|
("inbound.token", "inbound"),
|
||||||
|
("callback.token", "callback"),
|
||||||
|
("workspace.token", "workspace"),
|
||||||
|
):
|
||||||
|
path = self.root / "etc" / name
|
||||||
|
path.write_text(value)
|
||||||
|
path.chmod(0o600)
|
||||||
|
config = {
|
||||||
|
"host": "127.0.0.1",
|
||||||
|
"port": 18920,
|
||||||
|
"project_key": "v2r1-personal-weixin-production",
|
||||||
|
"database_path": str(self.root / "state.sqlite3"),
|
||||||
|
"event_inbox_dir": str(self.root / "events"),
|
||||||
|
"decision_inbox_dir": str(self.root / "decisions"),
|
||||||
|
"dispatch_outbox_dir": str(self.root / "outbox"),
|
||||||
|
"openclaw_sync_dir": str(self.root / "sync"),
|
||||||
|
"command_queue_dir": str(self.root / "commands"),
|
||||||
|
"inbound_token_file": str(self.root / "etc/inbound.token"),
|
||||||
|
"callback_token_file": str(self.root / "etc/callback.token"),
|
||||||
|
"contract_commit": CONTRACT,
|
||||||
|
"workspace_agent": {
|
||||||
|
"trigger_id": "agtch_test",
|
||||||
|
"access_token_file": str(self.root / "etc/workspace.token"),
|
||||||
|
},
|
||||||
|
"responses_fallback": {
|
||||||
|
"api_key_file": None,
|
||||||
|
"model": None,
|
||||||
|
},
|
||||||
|
"polling": {
|
||||||
|
"interval_seconds": 1,
|
||||||
|
"summary_interval_seconds": 10,
|
||||||
|
},
|
||||||
|
"loop_controls": {
|
||||||
|
"callback_timeout_seconds": 30,
|
||||||
|
"lease_seconds": 10,
|
||||||
|
"retry_base_seconds": 1,
|
||||||
|
"max_retries": 2,
|
||||||
|
"max_same_blocker_retries": 2,
|
||||||
|
"max_hops_per_gate": 20,
|
||||||
|
},
|
||||||
|
"openclaw_sync": {"endpoint": None, "token_file": None},
|
||||||
|
"git_poll": {
|
||||||
|
"enabled": False,
|
||||||
|
"bare_repo": None,
|
||||||
|
"ref_prefixes": [],
|
||||||
|
"interval_seconds": 30,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
config_path = self.root / "etc/config.json"
|
||||||
|
config_path.write_text(json.dumps(config))
|
||||||
|
config_path.chmod(0o600)
|
||||||
|
os.environ["AIOS_ORCHESTRATOR_CONFIG_OWNER_UID"] = str(os.getuid())
|
||||||
|
os.environ["AIOS_ORCHESTRATOR_SECRET_OWNER_UID"] = str(os.getuid())
|
||||||
|
self.runtime = Runtime(Config.from_file(config_path))
|
||||||
|
|
||||||
|
def tearDown(self):
|
||||||
|
self.runtime.store.close()
|
||||||
|
self.temp.cleanup()
|
||||||
|
os.environ.pop("AIOS_ORCHESTRATOR_CONFIG_OWNER_UID", None)
|
||||||
|
os.environ.pop("AIOS_ORCHESTRATOR_SECRET_OWNER_UID", None)
|
||||||
|
|
||||||
|
def test_file_event_polling_and_workspace_trigger(self):
|
||||||
|
pending = self.root / "events/pending"
|
||||||
|
pending.mkdir(mode=0o2770)
|
||||||
|
pending.chmod(0o2770)
|
||||||
|
(pending / "evt-001.json").write_text(json.dumps(event_payload()))
|
||||||
|
(pending / "evt-001.json").chmod(0o640)
|
||||||
|
with mock.patch.object(self.runtime.workspace, "trigger") as trigger:
|
||||||
|
self.runtime.poll_once()
|
||||||
|
trigger.assert_called_once()
|
||||||
|
processed = self.root / "events/processed/evt-001.json"
|
||||||
|
self.assertTrue(processed.exists())
|
||||||
|
|
||||||
|
def test_workspace_timeout_uses_responses_fallback(self):
|
||||||
|
self.runtime.submit_event(event_payload())
|
||||||
|
self.runtime.store.update_event_trigger(
|
||||||
|
"evt-001", status="AWAITING_DECISION", channel="workspace_agent", remote_id=None
|
||||||
|
)
|
||||||
|
self.runtime.store._conn().execute("UPDATE events SET updated_at=0 WHERE event_id='evt-001'")
|
||||||
|
fallback = mock.Mock()
|
||||||
|
fallback.start.return_value = "resp_fallback"
|
||||||
|
self.runtime.responses = fallback
|
||||||
|
self.runtime.process_workspace_timeouts()
|
||||||
|
row = self.runtime.store._conn().execute(
|
||||||
|
"SELECT status, channel, remote_id FROM events WHERE event_id='evt-001'"
|
||||||
|
).fetchone()
|
||||||
|
self.assertEqual(tuple(row), ("POLLING_RESPONSE", "responses_api", "resp_fallback"))
|
||||||
|
fallback.start.assert_called_once()
|
||||||
|
|
||||||
|
def test_workspace_callback_timeout_reaches_dead_letter_without_fallback(self):
|
||||||
|
self.runtime.submit_event(event_payload())
|
||||||
|
self.runtime.store.update_event_trigger(
|
||||||
|
"evt-001", status="AWAITING_DECISION", channel="workspace_agent", remote_id=None
|
||||||
|
)
|
||||||
|
self.runtime.store._conn().execute(
|
||||||
|
"UPDATE events SET attempts=2, updated_at=0 WHERE event_id='evt-001'"
|
||||||
|
)
|
||||||
|
self.assertIsNone(self.runtime.responses)
|
||||||
|
self.runtime.process_workspace_timeouts()
|
||||||
|
row = self.runtime.store._conn().execute(
|
||||||
|
"SELECT status, attempts FROM events WHERE event_id='evt-001'"
|
||||||
|
).fetchone()
|
||||||
|
self.assertEqual(tuple(row), ("DEAD_LETTER", 3))
|
||||||
|
|
||||||
|
def test_response_poll_limit_dead_letters_queued_response(self):
|
||||||
|
self.runtime.submit_event(event_payload())
|
||||||
|
self.runtime.store.update_event_trigger(
|
||||||
|
"evt-001",
|
||||||
|
status="POLLING_RESPONSE",
|
||||||
|
channel="responses_api",
|
||||||
|
remote_id="resp-stuck",
|
||||||
|
)
|
||||||
|
self.runtime.store._conn().execute(
|
||||||
|
"UPDATE events SET attempts=2, next_attempt_at=0 WHERE event_id='evt-001'"
|
||||||
|
)
|
||||||
|
fallback = mock.Mock()
|
||||||
|
fallback.poll.return_value = ("queued", None)
|
||||||
|
self.runtime.responses = fallback
|
||||||
|
self.runtime.poll_background_responses()
|
||||||
|
row = self.runtime.store._conn().execute(
|
||||||
|
"SELECT status, attempts FROM events WHERE event_id='evt-001'"
|
||||||
|
).fetchone()
|
||||||
|
self.assertEqual(tuple(row), ("DEAD_LETTER", 3))
|
||||||
|
|
||||||
|
def test_callback_rejects_wrong_contract(self):
|
||||||
|
self.runtime.submit_event(event_payload())
|
||||||
|
with self.assertRaisesRegex(ConflictError, "contract_version_mismatch"):
|
||||||
|
self.runtime.submit_decision(
|
||||||
|
decision_payload(contract_commit="f" * 40)
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_task_start_and_end_are_synced(self):
|
||||||
|
start = Event.from_dict(
|
||||||
|
event_payload(
|
||||||
|
event_id="evt-start",
|
||||||
|
event_type="work_unit_started",
|
||||||
|
status="RUNNING",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.runtime.submit_event(start.to_dict())
|
||||||
|
self.runtime.synchronize_openclaw()
|
||||||
|
pending = list((self.root / "sync/pending").glob("*.json"))
|
||||||
|
self.assertTrue(pending)
|
||||||
|
kinds = {json.loads(p.read_text())["kind"] for p in pending}
|
||||||
|
self.assertIn("TASK_STARTED", kinds)
|
||||||
|
|
||||||
|
def test_decision_dispatch_and_summary_sync(self):
|
||||||
|
self.runtime.submit_event(event_payload())
|
||||||
|
self.runtime.submit_decision(decision_payload())
|
||||||
|
self.runtime.dispatch_work_units()
|
||||||
|
self.runtime.synchronize_openclaw()
|
||||||
|
self.assertTrue(list((self.root / "commands/openclaw/pending").glob("*.json")))
|
||||||
|
kinds = {
|
||||||
|
json.loads(p.read_text())["kind"]
|
||||||
|
for p in (self.root / "sync/pending").glob("*.json")
|
||||||
|
}
|
||||||
|
self.assertIn("TASK_DISPATCHED", kinds)
|
||||||
|
self.assertIn("PROJECT_SUMMARY", kinds)
|
||||||
|
|
||||||
|
def test_poll_once_ingests_decision_file(self):
|
||||||
|
self.runtime.submit_event(event_payload())
|
||||||
|
pending = self.root / "decisions/pending"
|
||||||
|
pending.mkdir(mode=0o2770)
|
||||||
|
pending.chmod(0o2770)
|
||||||
|
(pending / "decision.json").write_text(json.dumps(decision_payload()))
|
||||||
|
(pending / "decision.json").chmod(0o640)
|
||||||
|
self.runtime.poll_once()
|
||||||
|
self.assertTrue((self.root / "decisions/processed/decision.json").exists())
|
||||||
|
|
||||||
|
|
||||||
|
def test_git_ref_polling_baselines_then_emits_change(self):
|
||||||
|
object.__setattr__(self.runtime.config, "git_poll_enabled", True)
|
||||||
|
object.__setattr__(self.runtime.config, "git_poll_bare_repo", self.root / "repo.git")
|
||||||
|
object.__setattr__(self.runtime.config, "git_poll_ref_prefixes", ("refs/heads/main",))
|
||||||
|
oid1 = "1" * 40
|
||||||
|
oid2 = "2" * 40
|
||||||
|
first = mock.Mock(returncode=0, stdout=f"refs/heads/main {oid1}\n", stderr="")
|
||||||
|
second = mock.Mock(returncode=0, stdout=f"refs/heads/main {oid2}\n", stderr="")
|
||||||
|
with mock.patch(
|
||||||
|
"services.aios_orchestrator_gateway.service.subprocess.run",
|
||||||
|
side_effect=[first, second],
|
||||||
|
):
|
||||||
|
self.runtime._next_git_poll = 0
|
||||||
|
self.runtime.poll_git_refs()
|
||||||
|
self.runtime._next_git_poll = 0
|
||||||
|
self.runtime.poll_git_refs()
|
||||||
|
summary = self.runtime.store.project_summary(
|
||||||
|
"v2r1-personal-weixin-production"
|
||||||
|
)
|
||||||
|
self.assertIsNotNone(summary)
|
||||||
|
self.assertEqual(summary["event_counts"]["RECEIVED"], 1)
|
||||||
|
|
||||||
|
def test_retry_backoff_is_bounded(self):
|
||||||
|
self.assertEqual(self.runtime._backoff(0), 1)
|
||||||
|
self.assertLessEqual(self.runtime._backoff(100), 3600)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
+59
@@ -0,0 +1,59 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
set -Eeuo pipefail
|
||||||
|
umask 0077
|
||||||
|
|
||||||
|
SOURCE_ROOT="${1:?source root required}"
|
||||||
|
ROOT="$(mktemp -d /tmp/aios-orchestrator-rollback-test.XXXXXX)"
|
||||||
|
trap 'rm -rf "$ROOT"' EXIT
|
||||||
|
INSTALL_ROOT="$ROOT/opt"
|
||||||
|
STATE_ROOT="$ROOT/state"
|
||||||
|
mkdir -p "$INSTALL_ROOT/releases" "$STATE_ROOT" "$ROOT/bin"
|
||||||
|
chmod 0700 "$STATE_ROOT"
|
||||||
|
|
||||||
|
OLD="$(printf 'a%.0s' {1..64})"
|
||||||
|
NEW="$(printf 'b%.0s' {1..64})"
|
||||||
|
mkdir "$INSTALL_ROOT/releases/$OLD" "$INSTALL_ROOT/releases/$NEW"
|
||||||
|
printf '%s\n' "$OLD" > "$INSTALL_ROOT/releases/$OLD/SOURCE_DIGEST"
|
||||||
|
printf '%s\n' "$NEW" > "$INSTALL_ROOT/releases/$NEW/SOURCE_DIGEST"
|
||||||
|
ln -s "$INSTALL_ROOT/releases/$NEW" "$INSTALL_ROOT/current"
|
||||||
|
python3 - "$STATE_ROOT/install-receipt.json" "$INSTALL_ROOT/releases/$NEW" "$INSTALL_ROOT/releases/$OLD" <<'PY'
|
||||||
|
import json, sys
|
||||||
|
with open(sys.argv[1], "w", encoding="utf-8") as handle:
|
||||||
|
json.dump({
|
||||||
|
"schema_version": 1,
|
||||||
|
"source_digest": "b" * 64,
|
||||||
|
"current_release": sys.argv[2],
|
||||||
|
"previous_release": sys.argv[3],
|
||||||
|
"last_rollback": None,
|
||||||
|
}, handle)
|
||||||
|
PY
|
||||||
|
chmod 0600 "$STATE_ROOT/install-receipt.json"
|
||||||
|
cat > "$ROOT/bin/systemctl" <<'SH'
|
||||||
|
#!/bin/sh
|
||||||
|
printf '%s\n' "$*" >> "${SYSTEMCTL_LOG:?}"
|
||||||
|
exit 0
|
||||||
|
SH
|
||||||
|
chmod 0755 "$ROOT/bin/systemctl"
|
||||||
|
: > "$ROOT/systemctl.log"
|
||||||
|
|
||||||
|
SYSTEMCTL_LOG="$ROOT/systemctl.log" PATH="$ROOT/bin:/usr/bin:/bin" INSTALL_ROOT="$INSTALL_ROOT" STATE_ROOT="$STATE_ROOT" \
|
||||||
|
bash "$SOURCE_ROOT/scripts/rollback_aios_orchestrator_gateway.sh"
|
||||||
|
FIRST="$(readlink -f "$INSTALL_ROOT/current")"
|
||||||
|
SYSTEMCTL_LOG="$ROOT/systemctl.log" PATH="$ROOT/bin:/usr/bin:/bin" INSTALL_ROOT="$INSTALL_ROOT" STATE_ROOT="$STATE_ROOT" \
|
||||||
|
bash "$SOURCE_ROOT/scripts/rollback_aios_orchestrator_gateway.sh"
|
||||||
|
SECOND="$(readlink -f "$INSTALL_ROOT/current")"
|
||||||
|
[[ "$FIRST" == "$INSTALL_ROOT/releases/$OLD" ]]
|
||||||
|
[[ "$SECOND" == "$FIRST" ]]
|
||||||
|
grep -Fxq 'stop aios-orchestrator-gateway-poll.timer' "$ROOT/systemctl.log"
|
||||||
|
grep -Fxq 'stop aios-orchestrator-gateway-poll.service' "$ROOT/systemctl.log"
|
||||||
|
grep -Fxq 'start aios-orchestrator-gateway-poll.timer' "$ROOT/systemctl.log"
|
||||||
|
|
||||||
|
printf 'tampered\n' > "$INSTALL_ROOT/releases/$OLD/SOURCE_DIGEST"
|
||||||
|
if SYSTEMCTL_LOG="$ROOT/systemctl.log" PATH="$ROOT/bin:/usr/bin:/bin" \
|
||||||
|
INSTALL_ROOT="$INSTALL_ROOT" STATE_ROOT="$STATE_ROOT" \
|
||||||
|
bash "$SOURCE_ROOT/scripts/rollback_aios_orchestrator_gateway.sh" >/dev/null 2>&1; then
|
||||||
|
echo 'ROLLBACK_DIGEST_MISMATCH_TEST=FAIL' >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "ROLLBACK_DIGEST_MISMATCH_TEST=PASS"
|
||||||
|
echo "ROLLBACK_REPEAT_TEST=PASS"
|
||||||
@@ -1,11 +1,20 @@
|
|||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import importlib.util
|
import importlib.util
|
||||||
|
import io
|
||||||
|
import os
|
||||||
|
import signal
|
||||||
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
|
import tempfile
|
||||||
|
import time
|
||||||
|
import urllib.error
|
||||||
|
import urllib.parse
|
||||||
|
from contextlib import redirect_stderr, redirect_stdout
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from types import SimpleNamespace
|
from types import SimpleNamespace
|
||||||
from unittest import TestCase
|
from unittest import TestCase
|
||||||
from unittest.mock import patch
|
from unittest.mock import Mock, call, patch
|
||||||
|
|
||||||
|
|
||||||
BACKEND_PATH = Path(__file__).resolve().parents[1] / "scripts" / "wechat_send_backend.py"
|
BACKEND_PATH = Path(__file__).resolve().parents[1] / "scripts" / "wechat_send_backend.py"
|
||||||
@@ -80,6 +89,450 @@ class WechatSendBackendTests(TestCase):
|
|||||||
self.assertEqual(args[1]["content"], "hello")
|
self.assertEqual(args[1]["content"], "hello")
|
||||||
self.assertFalse(args[1]["dry_run"])
|
self.assertFalse(args[1]["dry_run"])
|
||||||
|
|
||||||
|
def test_ucc_auth_is_authorization_header_not_argv_or_output(self):
|
||||||
|
secret = "ucc-super-secret-token"
|
||||||
|
|
||||||
|
class Response:
|
||||||
|
status = 200
|
||||||
|
|
||||||
|
def __enter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *_args):
|
||||||
|
return False
|
||||||
|
|
||||||
|
def read(self, _limit=-1):
|
||||||
|
return b'{"ok":true,"status":"sent"}'
|
||||||
|
|
||||||
|
urlopen = Mock(return_value=Response())
|
||||||
|
output = io.StringIO()
|
||||||
|
with (
|
||||||
|
patch.object(self.backend, "UCC_ADAPTER_TOKEN", secret),
|
||||||
|
patch.object(self.backend, "_open_no_redirect", urlopen),
|
||||||
|
redirect_stdout(output),
|
||||||
|
redirect_stderr(output),
|
||||||
|
):
|
||||||
|
status, payload = self.backend._post_ucc(
|
||||||
|
"/api/internal/wechat/send", {"target": "chat-001", "content": "hello"}
|
||||||
|
)
|
||||||
|
|
||||||
|
request = urlopen.call_args.args[0]
|
||||||
|
self.assertEqual(request.get_header("Authorization"), f"Bearer {secret}")
|
||||||
|
self.assertEqual(status, 200)
|
||||||
|
self.assertEqual(payload, {"ok": True, "status": "sent"})
|
||||||
|
self.assertNotIn(secret, " ".join(self.backend._openclaw_base_args()))
|
||||||
|
self.assertNotIn(secret, output.getvalue())
|
||||||
|
self.assertFalse({"--gateway", "--token", "--password"} & set(self.backend._openclaw_base_args()))
|
||||||
|
|
||||||
|
def test_ucc_paths_are_canonical_first_unique_and_bounded(self):
|
||||||
|
with patch.object(self.backend, "UCC_SEND_PATH", "/api/internal/wechat/send"):
|
||||||
|
paths = self.backend._ucc_send_paths()
|
||||||
|
|
||||||
|
self.assertEqual(paths[0], "/api/internal/wechat/send")
|
||||||
|
self.assertLessEqual(len(paths), 2)
|
||||||
|
self.assertEqual(len(paths), len(set(paths)))
|
||||||
|
self.assertTrue(all(path.startswith("/") for path in paths))
|
||||||
|
|
||||||
|
def test_ucc_accepts_only_explicit_true_and_sent_when_not_dry_run(self):
|
||||||
|
self.assertTrue(self.backend._ucc_response_sent({"ok": True, "status": "sent"}))
|
||||||
|
for response in (
|
||||||
|
{"ok": 1, "status": "sent"},
|
||||||
|
{"ok": True, "status": "accepted"},
|
||||||
|
{"ok": False, "status": "sent"},
|
||||||
|
{"status": "sent"},
|
||||||
|
):
|
||||||
|
with self.subTest(response=response):
|
||||||
|
self.assertFalse(self.backend._ucc_response_sent(response))
|
||||||
|
|
||||||
|
def test_ambiguous_canonical_failures_never_fallback(self):
|
||||||
|
secret = "raw-upstream-secret"
|
||||||
|
failures = (
|
||||||
|
TimeoutError(secret),
|
||||||
|
ConnectionError(secret),
|
||||||
|
self.backend._BackendProtocolError(secret),
|
||||||
|
RuntimeError(secret),
|
||||||
|
)
|
||||||
|
for failure in failures:
|
||||||
|
with self.subTest(failure=type(failure).__name__):
|
||||||
|
with (
|
||||||
|
patch.object(self.backend, "UCC_SEND_PATH", "/api/internal/wechat/send"),
|
||||||
|
patch.object(self.backend, "_post_ucc", side_effect=failure) as post,
|
||||||
|
):
|
||||||
|
result = self.backend.send_ucc_adapter(
|
||||||
|
target="chat-001", content="hello", message_id="msg-001"
|
||||||
|
).to_dict()
|
||||||
|
|
||||||
|
self.assertFalse(result["ok"])
|
||||||
|
self.assertEqual(result["code"], "ucc_adapter_unavailable")
|
||||||
|
self.assertEqual(post.call_count, 1)
|
||||||
|
self.assertNotIn(secret, repr(result))
|
||||||
|
|
||||||
|
def test_only_explicit_http_404_or_405_uses_legacy_fallback(self):
|
||||||
|
for http_status in (404, 405):
|
||||||
|
with self.subTest(http_status=http_status):
|
||||||
|
missing = urllib.error.HTTPError(
|
||||||
|
"http://127.0.0.1/api/internal/wechat/send",
|
||||||
|
http_status,
|
||||||
|
"not found",
|
||||||
|
{},
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
with (
|
||||||
|
patch.object(self.backend, "UCC_SEND_PATH", "/api/internal/wechat/send"),
|
||||||
|
patch.object(
|
||||||
|
self.backend,
|
||||||
|
"_post_ucc",
|
||||||
|
side_effect=[missing, (200, {"ok": True, "status": "sent"})],
|
||||||
|
) as post,
|
||||||
|
):
|
||||||
|
result = self.backend.send_ucc_adapter(
|
||||||
|
target="chat-001", content="hello", message_id="msg-001"
|
||||||
|
).to_dict()
|
||||||
|
|
||||||
|
self.assertTrue(result["ok"])
|
||||||
|
self.assertEqual(result["endpoint"], "/internal/channels/wechat/send")
|
||||||
|
self.assertEqual(post.call_count, 2)
|
||||||
|
|
||||||
|
def test_non_not_found_http_error_never_falls_back(self):
|
||||||
|
denied = urllib.error.HTTPError(
|
||||||
|
"http://127.0.0.1/api/internal/wechat/send", 401, "denied", {}, None
|
||||||
|
)
|
||||||
|
with patch.object(self.backend, "_post_ucc", side_effect=denied) as post:
|
||||||
|
result = self.backend.send_ucc_adapter(target="chat-001", content="hello").to_dict()
|
||||||
|
|
||||||
|
self.assertFalse(result["ok"])
|
||||||
|
self.assertEqual(result["code"], "ucc_adapter_http_error")
|
||||||
|
post.assert_called_once()
|
||||||
|
|
||||||
|
def test_ucc_rejection_does_not_expose_raw_error_or_secret(self):
|
||||||
|
secret = "provider-password=secret-value"
|
||||||
|
with patch.object(
|
||||||
|
self.backend,
|
||||||
|
"_post_ucc",
|
||||||
|
return_value=(200, {"ok": False, "status": "failed", "error": secret}),
|
||||||
|
):
|
||||||
|
result = self.backend.send_ucc_adapter(
|
||||||
|
target="chat-001", content="hello", message_id="msg-001"
|
||||||
|
).to_dict()
|
||||||
|
|
||||||
|
self.assertFalse(result["ok"])
|
||||||
|
self.assertEqual(result["code"], "ucc_adapter_rejected")
|
||||||
|
self.assertNotIn(secret, repr(result))
|
||||||
|
|
||||||
|
def test_dry_run_requires_dry_run_status_and_never_reports_sent(self):
|
||||||
|
with patch.object(
|
||||||
|
self.backend,
|
||||||
|
"_post_ucc",
|
||||||
|
return_value=(200, {"ok": True, "status": "dry_run"}),
|
||||||
|
):
|
||||||
|
result = self.backend.send_ucc_adapter(
|
||||||
|
target="chat-001", content="hello", dry_run=True
|
||||||
|
).to_dict()
|
||||||
|
|
||||||
|
self.assertTrue(result["ok"])
|
||||||
|
self.assertEqual(result["status"], "dry_run")
|
||||||
|
self.assertNotEqual(result["status"], "sent")
|
||||||
|
|
||||||
|
for dry_run, upstream_status in ((True, "sent"), (False, "dry_run")):
|
||||||
|
with self.subTest(dry_run=dry_run, upstream_status=upstream_status):
|
||||||
|
with patch.object(
|
||||||
|
self.backend,
|
||||||
|
"_post_ucc",
|
||||||
|
return_value=(200, {"ok": True, "status": upstream_status}),
|
||||||
|
):
|
||||||
|
rejected = self.backend.send_ucc_adapter(
|
||||||
|
target="chat-001", content="hello", dry_run=dry_run
|
||||||
|
).to_dict()
|
||||||
|
self.assertFalse(rejected["ok"])
|
||||||
|
self.assertEqual(rejected["status"], "failed")
|
||||||
|
|
||||||
|
def test_invalid_provider_message_id_never_reverses_confirmed_send(self):
|
||||||
|
invalid_values = (
|
||||||
|
"\ud800",
|
||||||
|
"contains\x00control",
|
||||||
|
"x" * (self.backend.MAX_IDENTIFIER_BYTES + 1),
|
||||||
|
{"not": "a string"},
|
||||||
|
)
|
||||||
|
for provider_message_id in invalid_values:
|
||||||
|
with self.subTest(value=repr(provider_message_id)[:40]):
|
||||||
|
with patch.object(
|
||||||
|
self.backend,
|
||||||
|
"_post_ucc",
|
||||||
|
return_value=(
|
||||||
|
200,
|
||||||
|
{
|
||||||
|
"ok": True,
|
||||||
|
"status": "sent",
|
||||||
|
"provider_message_id": provider_message_id,
|
||||||
|
},
|
||||||
|
),
|
||||||
|
):
|
||||||
|
result = self.backend.send_ucc_adapter(
|
||||||
|
target="chat-001", content="hello"
|
||||||
|
).to_dict()
|
||||||
|
|
||||||
|
self.assertTrue(result["ok"])
|
||||||
|
self.assertEqual(result["status"], "sent")
|
||||||
|
self.assertNotIn("provider_message_id", result)
|
||||||
|
|
||||||
|
def test_openclaw_2026_6_11_commands_have_no_unsupported_global_gateway(self):
|
||||||
|
completed = SimpleNamespace(returncode=0, stdout='{"channels": []}')
|
||||||
|
with (
|
||||||
|
patch.object(self.backend, "OPENCLAW_BIN", "/usr/local/bin/openclaw"),
|
||||||
|
patch.object(self.backend, "_run_openclaw", return_value=completed) as run,
|
||||||
|
):
|
||||||
|
self.assertIsNone(self.backend.discover_openclaw_wechat_channel())
|
||||||
|
|
||||||
|
command = run.call_args.args[0]
|
||||||
|
self.assertEqual(command, ["/usr/local/bin/openclaw", "channels", "list", "--json"])
|
||||||
|
self.assertFalse({"--gateway", "--token", "--password"} & set(command))
|
||||||
|
|
||||||
|
def test_openclaw_output_overflow_terminates_while_process_is_running(self):
|
||||||
|
writer = (
|
||||||
|
"import os,time;"
|
||||||
|
f"os.write(1,b'x'*{self.backend.MAX_RESPONSE_BYTES + 4096});"
|
||||||
|
"time.sleep(10)"
|
||||||
|
)
|
||||||
|
started = time.monotonic()
|
||||||
|
completed = self.backend._run_openclaw([sys.executable, "-c", writer], timeout=4)
|
||||||
|
elapsed = time.monotonic() - started
|
||||||
|
|
||||||
|
self.assertLess(elapsed, 2.0)
|
||||||
|
self.assertNotEqual(completed.returncode, 0)
|
||||||
|
self.assertEqual(completed.stdout, "")
|
||||||
|
self.assertEqual(completed.stderr, "")
|
||||||
|
|
||||||
|
def test_openclaw_timeout_returns_failure_and_terminates_promptly(self):
|
||||||
|
started = time.monotonic()
|
||||||
|
completed = self.backend._run_openclaw(
|
||||||
|
[sys.executable, "-c", "import time;time.sleep(10)"], timeout=1
|
||||||
|
)
|
||||||
|
elapsed = time.monotonic() - started
|
||||||
|
|
||||||
|
self.assertLess(elapsed, 3.0)
|
||||||
|
self.assertNotEqual(completed.returncode, 0)
|
||||||
|
self.assertEqual(completed.stdout, "")
|
||||||
|
self.assertEqual(completed.stderr, "")
|
||||||
|
|
||||||
|
def test_timeout_kills_inherited_pipe_descendant_after_parent_exits(self):
|
||||||
|
if os.name != "posix" or not Path("/proc").is_dir():
|
||||||
|
self.skipTest("POSIX process-group and procfs contract")
|
||||||
|
|
||||||
|
def process_is_running(pid):
|
||||||
|
try:
|
||||||
|
state = Path(f"/proc/{pid}/stat").read_text(encoding="utf-8").split()[2]
|
||||||
|
except (FileNotFoundError, IndexError, OSError):
|
||||||
|
return False
|
||||||
|
return state not in {"X", "Z"}
|
||||||
|
|
||||||
|
with tempfile.TemporaryDirectory() as directory:
|
||||||
|
pid_path = Path(directory) / "descendant.pid"
|
||||||
|
descendant = "import time;time.sleep(30)"
|
||||||
|
launcher = (
|
||||||
|
"import pathlib,subprocess,sys;"
|
||||||
|
f"child=subprocess.Popen([sys.executable,'-c',{descendant!r}]);"
|
||||||
|
f"pathlib.Path({str(pid_path)!r}).write_text(str(child.pid),encoding='utf-8')"
|
||||||
|
)
|
||||||
|
started = time.monotonic()
|
||||||
|
completed = self.backend._run_openclaw(
|
||||||
|
[sys.executable, "-c", launcher], timeout=1
|
||||||
|
)
|
||||||
|
elapsed = time.monotonic() - started
|
||||||
|
self.assertTrue(pid_path.is_file())
|
||||||
|
descendant_pid = int(pid_path.read_text(encoding="utf-8"))
|
||||||
|
try:
|
||||||
|
deadline = time.monotonic() + 2
|
||||||
|
while process_is_running(descendant_pid) and time.monotonic() < deadline:
|
||||||
|
time.sleep(0.02)
|
||||||
|
self.assertFalse(process_is_running(descendant_pid))
|
||||||
|
finally:
|
||||||
|
if process_is_running(descendant_pid):
|
||||||
|
os.kill(descendant_pid, signal.SIGKILL)
|
||||||
|
|
||||||
|
self.assertLess(elapsed, 3.0)
|
||||||
|
self.assertNotEqual(completed.returncode, 0)
|
||||||
|
|
||||||
|
def test_process_tree_termination_uses_posix_process_group(self):
|
||||||
|
if os.name != "posix":
|
||||||
|
self.skipTest("POSIX process-group contract")
|
||||||
|
|
||||||
|
class Process:
|
||||||
|
pid = 43210
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.wait_count = 0
|
||||||
|
|
||||||
|
def poll(self):
|
||||||
|
return None
|
||||||
|
|
||||||
|
def wait(self, timeout):
|
||||||
|
self.wait_count += 1
|
||||||
|
if self.wait_count == 1:
|
||||||
|
raise subprocess.TimeoutExpired(["openclaw"], timeout)
|
||||||
|
return -signal.SIGKILL
|
||||||
|
|
||||||
|
process = Process()
|
||||||
|
with patch.object(self.backend.os, "killpg") as killpg:
|
||||||
|
self.backend._terminate_process_tree(process)
|
||||||
|
|
||||||
|
self.assertEqual(
|
||||||
|
killpg.call_args_list,
|
||||||
|
[
|
||||||
|
call(process.pid, signal.SIGTERM),
|
||||||
|
call(process.pid, signal.SIGKILL),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_unsupported_and_generic_chat_backends_fail_closed(self):
|
||||||
|
with (
|
||||||
|
patch.object(self.backend, "SEND_BACKEND", "openclaw-chat"),
|
||||||
|
patch.object(self.backend, "ucc_adapter_alive", return_value=False),
|
||||||
|
):
|
||||||
|
selected = self.backend.select_backend()
|
||||||
|
result = self.backend.send_message(target="chat-001", content="hello")
|
||||||
|
|
||||||
|
self.assertEqual(selected["backend"], "unavailable")
|
||||||
|
self.assertEqual(selected["reason"], "unsupported_backend")
|
||||||
|
self.assertFalse(result["ok"])
|
||||||
|
self.assertEqual(result["code"], "unsupported_backend")
|
||||||
|
|
||||||
|
def test_invalid_or_oversized_input_is_rejected_before_network(self):
|
||||||
|
with patch.object(self.backend, "_post_ucc") as post:
|
||||||
|
empty = self.backend.send_ucc_adapter(target="", content="hello").to_dict()
|
||||||
|
oversized = self.backend.send_ucc_adapter(
|
||||||
|
target="chat-001", content="x" * (self.backend.MAX_CONTENT_BYTES + 1)
|
||||||
|
).to_dict()
|
||||||
|
|
||||||
|
self.assertEqual(empty["code"], "invalid_input")
|
||||||
|
self.assertEqual(oversized["code"], "input_too_large")
|
||||||
|
post.assert_not_called()
|
||||||
|
|
||||||
|
def test_unpaired_surrogate_inputs_fail_closed_before_network(self):
|
||||||
|
cases = (
|
||||||
|
{"target": "\ud800", "content": "hello"},
|
||||||
|
{"target": "chat-001", "content": "\ud800"},
|
||||||
|
{"target": "chat-001", "content": "hello", "account_id": "\ud800"},
|
||||||
|
{"target": "chat-001", "content": "hello", "message_id": "\ud800"},
|
||||||
|
{"target": "chat-001", "content": "hello", "metadata": {"bad": "\ud800"}},
|
||||||
|
)
|
||||||
|
with patch.object(self.backend, "_post_ucc") as post:
|
||||||
|
for kwargs in cases:
|
||||||
|
with self.subTest(field=next(reversed(kwargs))):
|
||||||
|
result = self.backend.send_ucc_adapter(**kwargs).to_dict()
|
||||||
|
self.assertEqual(result["code"], "invalid_input")
|
||||||
|
post.assert_not_called()
|
||||||
|
|
||||||
|
def test_deeply_nested_metadata_is_bounded_before_network(self):
|
||||||
|
metadata = {}
|
||||||
|
for _ in range(1200):
|
||||||
|
metadata = {"nested": metadata}
|
||||||
|
|
||||||
|
with patch.object(self.backend, "_post_ucc") as post:
|
||||||
|
result = self.backend.send_ucc_adapter(
|
||||||
|
target="chat-001", content="hello", metadata=metadata
|
||||||
|
).to_dict()
|
||||||
|
|
||||||
|
self.assertEqual(result["code"], "input_too_large")
|
||||||
|
post.assert_not_called()
|
||||||
|
|
||||||
|
def test_adapter_origin_must_be_allowlisted(self):
|
||||||
|
with (
|
||||||
|
patch.object(self.backend, "UCC_ADAPTER_URL", "https://evil.example:8443"),
|
||||||
|
patch.object(
|
||||||
|
self.backend,
|
||||||
|
"UCC_ADAPTER_ALLOWED_ORIGINS",
|
||||||
|
("http://127.0.0.1:18800",),
|
||||||
|
),
|
||||||
|
):
|
||||||
|
self.assertIsNone(self.backend._adapter_base_url())
|
||||||
|
|
||||||
|
with (
|
||||||
|
patch.object(self.backend, "UCC_ADAPTER_URL", "https://adapter.example:8443"),
|
||||||
|
patch.object(
|
||||||
|
self.backend,
|
||||||
|
"UCC_ADAPTER_ALLOWED_ORIGINS",
|
||||||
|
("https://adapter.example:8443",),
|
||||||
|
),
|
||||||
|
):
|
||||||
|
self.assertEqual(
|
||||||
|
self.backend._adapter_base_url(), "https://adapter.example:8443"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_adapter_origin_rejects_scheme_userinfo_fragment_and_path(self):
|
||||||
|
invalid_origins = (
|
||||||
|
"ftp://127.0.0.1:18800",
|
||||||
|
"http://user:password@127.0.0.1:18800",
|
||||||
|
"http://127.0.0.1:18800#fragment",
|
||||||
|
"http://127.0.0.1:18800/base-path",
|
||||||
|
)
|
||||||
|
for origin in invalid_origins:
|
||||||
|
with self.subTest(origin=origin):
|
||||||
|
with (
|
||||||
|
patch.object(self.backend, "UCC_ADAPTER_URL", origin),
|
||||||
|
patch.object(
|
||||||
|
self.backend,
|
||||||
|
"UCC_ADAPTER_ALLOWED_ORIGINS",
|
||||||
|
("http://127.0.0.1:18800",),
|
||||||
|
),
|
||||||
|
):
|
||||||
|
self.assertIsNone(self.backend._adapter_base_url())
|
||||||
|
|
||||||
|
def test_adapter_path_rejects_decoded_traversal(self):
|
||||||
|
for path in (
|
||||||
|
"/api/%2e%2e/send",
|
||||||
|
"/api/%2E%2E/send",
|
||||||
|
"/api/%2525252e%2525252e/send",
|
||||||
|
"/api/%252525252e%252525252e/send",
|
||||||
|
"/api/../send",
|
||||||
|
"/api/%zz/send",
|
||||||
|
):
|
||||||
|
with self.subTest(path=path):
|
||||||
|
self.assertIsNone(self.backend._safe_path(path))
|
||||||
|
|
||||||
|
def test_adapter_path_keeps_intended_safe_canonical_paths(self):
|
||||||
|
for path in (
|
||||||
|
"/api/internal/wechat/send",
|
||||||
|
"/internal/channels/wechat/send",
|
||||||
|
"/api/%7Ehealth/status",
|
||||||
|
):
|
||||||
|
with self.subTest(path=path):
|
||||||
|
self.assertEqual(self.backend._safe_path(path), path)
|
||||||
|
|
||||||
|
def test_redirects_are_refused_and_authorization_cannot_cross_origin(self):
|
||||||
|
handler = self.backend._NoRedirectHandler()
|
||||||
|
self.assertIsNone(
|
||||||
|
handler.redirect_request(
|
||||||
|
None,
|
||||||
|
None,
|
||||||
|
302,
|
||||||
|
"redirect",
|
||||||
|
{"Location": "https://evil.example/steal"},
|
||||||
|
"https://evil.example/steal",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
redirected = urllib.error.HTTPError(
|
||||||
|
"http://127.0.0.1:18800/api/internal/wechat/send",
|
||||||
|
302,
|
||||||
|
"redirect",
|
||||||
|
{"Location": "https://evil.example/steal"},
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
with (
|
||||||
|
patch.object(self.backend, "UCC_ADAPTER_TOKEN", "header-secret"),
|
||||||
|
patch.object(self.backend, "_open_no_redirect", side_effect=redirected) as opened,
|
||||||
|
):
|
||||||
|
result = self.backend.send_ucc_adapter(
|
||||||
|
target="chat-001", content="hello"
|
||||||
|
).to_dict()
|
||||||
|
|
||||||
|
request = opened.call_args.args[0]
|
||||||
|
self.assertEqual(urllib.parse.urlsplit(request.full_url).netloc, "127.0.0.1:18800")
|
||||||
|
self.assertEqual(request.get_header("Authorization"), "Bearer header-secret")
|
||||||
|
opened.assert_called_once()
|
||||||
|
self.assertFalse(result["ok"])
|
||||||
|
self.assertEqual(result["code"], "ucc_adapter_http_error")
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
import unittest
|
import unittest
|
||||||
|
|||||||
@@ -0,0 +1,31 @@
|
|||||||
|
# AIOS Central Orchestrator Workspace Agent
|
||||||
|
|
||||||
|
Contract commit: `cbe33ecab8e9794ba5f318bea43aeaf447a56fbc`
|
||||||
|
|
||||||
|
When an API-triggered input arrives:
|
||||||
|
|
||||||
|
1. Parse the AIOS event JSON from the trigger message.
|
||||||
|
2. Verify `contract_commit` exactly matches the configured contract commit.
|
||||||
|
3. Use connected GitOps tools to read the authoritative Git tree, runtime status,
|
||||||
|
gate state, test evidence and full OIDs.
|
||||||
|
4. Never invent evidence, credentials, secrets, tests, approvals, Git results,
|
||||||
|
model availability or deployment status.
|
||||||
|
5. Apply the adaptive dispatch contract:
|
||||||
|
- Prefer OpenClaw when quality is comparable, work is large/repetitive,
|
||||||
|
monitoring-heavy, infrastructure-heavy or cost-sensitive.
|
||||||
|
- Prefer the strongest actually available Codex model for complex code,
|
||||||
|
security boundaries, architecture and final high-risk review.
|
||||||
|
- On timeout/unavailability, mark the attempt `TIMEOUT` or `NOT_EXECUTED`
|
||||||
|
and assign the same contract to the fallback agent.
|
||||||
|
- Never infer PASS from timeout or missing output.
|
||||||
|
6. Every next work unit must include a clean base commit, allowed paths, required
|
||||||
|
checks, forbidden operations, primary agent and fallback agent.
|
||||||
|
7. Keep real credential, raw-wire, reviewer, secret-provider and owner-only
|
||||||
|
release gates fail-closed.
|
||||||
|
8. Submit exactly one strict decision through the
|
||||||
|
`submit_orchestration_decision` action.
|
||||||
|
9. Do not write a second decision for the same event ID.
|
||||||
|
10. Stop after the action succeeds.
|
||||||
|
|
||||||
|
The decision must satisfy the JSON schema in the connected action. Use the
|
||||||
|
incoming `event_id`, `project_key`, `contract_commit`, and `state_version`.
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
openapi: 3.1.0
|
||||||
|
info:
|
||||||
|
title: AIOS Orchestrator Decision Callback
|
||||||
|
version: 1.0.0
|
||||||
|
servers:
|
||||||
|
- url: https://REPLACE_WITH_GATEWAY_HOST
|
||||||
|
paths:
|
||||||
|
/v1/orchestrator/decisions:
|
||||||
|
post:
|
||||||
|
operationId: submit_orchestration_decision
|
||||||
|
summary: Submit one strict orchestration decision for a triggered event
|
||||||
|
security:
|
||||||
|
- bearerAuth: []
|
||||||
|
requestBody:
|
||||||
|
required: true
|
||||||
|
content:
|
||||||
|
application/json:
|
||||||
|
schema:
|
||||||
|
$ref: '#/components/schemas/Decision'
|
||||||
|
responses:
|
||||||
|
'202':
|
||||||
|
description: Accepted
|
||||||
|
'409':
|
||||||
|
description: Contract, idempotency, or state version conflict
|
||||||
|
'422':
|
||||||
|
description: Schema validation failed
|
||||||
|
components:
|
||||||
|
securitySchemes:
|
||||||
|
bearerAuth:
|
||||||
|
type: http
|
||||||
|
scheme: bearer
|
||||||
|
schemas:
|
||||||
|
WorkUnit:
|
||||||
|
type: object
|
||||||
|
additionalProperties: false
|
||||||
|
required:
|
||||||
|
- id
|
||||||
|
- track_id
|
||||||
|
- gate_id
|
||||||
|
- primary_agent
|
||||||
|
- fallback_agent
|
||||||
|
- clean_base_commit
|
||||||
|
- allowed_paths
|
||||||
|
- required_checks
|
||||||
|
- forbidden_operations
|
||||||
|
- instruction
|
||||||
|
properties:
|
||||||
|
id: {type: string}
|
||||||
|
track_id: {type: string}
|
||||||
|
gate_id: {type: string}
|
||||||
|
primary_agent:
|
||||||
|
type: string
|
||||||
|
enum: [openclaw, codex]
|
||||||
|
fallback_agent:
|
||||||
|
type: string
|
||||||
|
enum: [openclaw, codex]
|
||||||
|
clean_base_commit:
|
||||||
|
type: string
|
||||||
|
pattern: '^[0-9a-f]{40}$'
|
||||||
|
allowed_paths:
|
||||||
|
type: array
|
||||||
|
items: {type: string}
|
||||||
|
required_checks:
|
||||||
|
type: array
|
||||||
|
items: {type: string}
|
||||||
|
forbidden_operations:
|
||||||
|
type: array
|
||||||
|
items: {type: string}
|
||||||
|
instruction: {type: string}
|
||||||
|
Decision:
|
||||||
|
type: object
|
||||||
|
additionalProperties: false
|
||||||
|
required:
|
||||||
|
- schema_version
|
||||||
|
- project_key
|
||||||
|
- contract_commit
|
||||||
|
- event_id
|
||||||
|
- expected_state_version
|
||||||
|
- verdict
|
||||||
|
- state_updates
|
||||||
|
- next_work_units
|
||||||
|
- requires_human_input
|
||||||
|
- stop_reason
|
||||||
|
properties:
|
||||||
|
schema_version:
|
||||||
|
type: integer
|
||||||
|
const: 1
|
||||||
|
project_key: {type: string}
|
||||||
|
contract_commit:
|
||||||
|
type: string
|
||||||
|
pattern: '^[0-9a-f]{40}$'
|
||||||
|
event_id: {type: string}
|
||||||
|
expected_state_version:
|
||||||
|
type: integer
|
||||||
|
minimum: 0
|
||||||
|
verdict:
|
||||||
|
type: string
|
||||||
|
enum: [DISPATCH, BLOCKED, COMPLETE, NOOP]
|
||||||
|
state_updates:
|
||||||
|
type: array
|
||||||
|
items:
|
||||||
|
type: object
|
||||||
|
additionalProperties: false
|
||||||
|
required: [key, value]
|
||||||
|
properties:
|
||||||
|
key: {type: string}
|
||||||
|
value: {type: string}
|
||||||
|
next_work_units:
|
||||||
|
type: array
|
||||||
|
items:
|
||||||
|
$ref: '#/components/schemas/WorkUnit'
|
||||||
|
requires_human_input: {type: boolean}
|
||||||
|
stop_reason:
|
||||||
|
type: [string, 'null']
|
||||||
Reference in New Issue
Block a user