Compare commits

..
Author SHA1 Message Date
openclaw d457ac3961 release: record final owner approval for V1R4C5 2026-07-11 01:58:14 +08:00
openclaw 9fa7511bb5 release: add V1R4C5 acceptance evidence 2026-07-11 01:56:45 +08:00
4 changed files with 266 additions and 316 deletions
@@ -1,220 +0,0 @@
# 事故复盘:Gitea 分支元数据漂移导致自主运作系统停止
日期:2026-07-10
事故等级:S1 / Stop-the-line
状态:已恢复;已通过 GitOps runtime 读写验收。
## 事故定级原则
凡是导致自主运作系统停止运转、无法继续闭环执行、无法写入 GitOps 状态、或必须人工介入恢复的事件,一律视为事故处理。
本事件满足该条件,因此按事故复盘、可靠性改进和防复发机制处理,而不是仅作为一次普通 Gitea 修复记录。
## 影响
- 受影响系统:自主运作系统、GitOps runtime、Gitea 分支查询和同步路径。
- 受影响仓库:`aios/aios`。
- 受影响主机 / 入口:`g.cnnetmkt.top`,SSH alias `hw`。
- 受影响接口:
- `GET /api/v1/repos/aios/aios/branches?page=1&limit=50`
- `GET /api/v1/repos/aios/aios/branches?page=2&limit=50`
- 用户可见影响:自主运作系统停止,需要人工检查和恢复。
## 现场结论
初始判断曾倾向于“Gitea 完全挂死、可能需要重启”。后续证据显示,更准确的判断是:
> Gitea 裸仓库 refs 与 Gitea SQLite `branch` 表 live 行之间发生元数据漂移,导致 branch API 分页结果异常,进而影响自主运作系统继续运转。
本次没有足够证据证明 Gitea 进程死锁。不能把“重启 Gitea”作为首选修复路径;应优先确认进程、API、裸仓库 refs、数据库 `branch` 表之间的一致性。
## 关键证据
来自修复记录:
- 裸仓库 refs 数量:`55`。
- 修复前数据库 `branch` live 行数:`50`。
- 修复后数据库 `branch` live 行数:`55`。
- 修复前 API:
- page 1 返回 `50`。
- page 2 返回 `0`。
- 修复后 API:
- page 1 返回 `50`。
- page 2 返回 `5`。
- `gitea doctor check --run check-db-consistency` 修复后返回 `OK`。
缺失并已补回的 live 分支:
1. `patch/agent-event-bus-unified`
2. `patch/agent-gpt-message-plane`
3. `patch/gitops-change-archive-audit`
4. `patch/group-owner-agent-context-isolation`
5. `patch/user-profile-isolation`
## 恢复动作
1. 备份 Gitea SQLite 数据库:
- `/data/aios-gitops/runtime/gitea-bin/data/gitea.db.bak-20260710-0105`
2. 对比裸仓库 refs 与数据库 `branch` live 行。
3. 用 SQLite 事务补回缺失的 5 条 live 分支记录。
4. 复验 API 分页。
5. 执行 Gitea doctor consistency check。
6. 通过 GitOps runtime 接受测试。
## 恢复后接受测试
接受测试顺序:
1. `getRuntimeStatus`
2. `getBaseline`
3. `createBranch`
4. `createPatch`
测试结果:
- Runtime 状态:`active`。
- Baseline:`V1R4C3`。
- 测试分支:`test/gitea-runtime-acceptance-20260710`。
- 测试分支 commit:`7d6e497`。
- 测试分支 sync:`synced`。
- Patch:`gitea-runtime-acceptance-20260710`。
- Patch commit:`59144bc`。
- Patch sync:`synced`。
因此,事故当前状态为:已恢复。
## 根因判断
已证实根因:
- 裸仓库 `refs/heads` 与 Gitea `branch` 表 live 行不一致。
- Gitea branch API 依赖的元数据状态与底层 Git refs 状态不一致,导致分页结果异常。
高可信风险假设:
- 自主控制器在高频创建、更新或同步分支时,可能触发了 refs 与 Gitea 元数据表之间的漂移。
- 控制器缺少操作后的读回确认和一致性闸门。
- 控制器缺少发现异常后的熔断机制,可能继续执行写操作并扩大影响。
尚未证实:
- Gitea 进程死锁。
- Gitea 必须通过重启恢复。
- Gitea 裸仓库数据损坏。
## 可靠性缺口
1. 缺少“refs 数量 vs branch live 行数 vs API 分页结果”的持续一致性检查。
2. 缺少自主控制器写入后的 read-after-write 验证。
3. 缺少控制器异常时的自动熔断和安全模式。
4. 缺少 branch metadata drift 的专用告警。
5. 缺少“不要先重启,先判定数据一致性”的事故处理规则。
6. 缺少对 Gitea/GitOps 写链路的标准接受测试记录。
## 防复发要求
### 1. 一致性哨兵
必须增加周期性检查:
```text
bare_repo_refs_count == gitea_branch_live_count == api_branch_total_count
```
至少检查:
- 裸仓库 `refs/heads` 数量。
- SQLite `branch` 表中 `is_deleted=0` 数量。
- Gitea branch API 分页总数。
- `gitea doctor check --run check-db-consistency` 结果。
### 2. 写后确认
自主控制器每次创建或删除分支后,必须执行 read-after-write 验证:
1. Git ref 存在 / 不存在。
2. Gitea API 可见 / 不可见。
3. `branch` live 行状态符合预期。
4. 远端同步结果为 `synced`。
任一失败时,不得继续下一轮自治写入。
### 3. 熔断机制
发现以下任一条件时,控制器必须进入安全模式:
- refs 数量与 branch live 行数不一致。
- API 分页总数与 refs 数量不一致。
- `gitea doctor` 返回非 OK。
- 连续 Git push / fetch / branch API 失败。
- GitOps runtime 无法完成 `createBranch` 或 `createPatch`。
安全模式行为:
- 停止 branch 创建、删除、批量 patch 写入等突变操作。
- 允许只读诊断。
- 生成事故记录或 patch 请求。
- 等待 owner/admin 审核恢复。
### 4. 单写者与限速
Gitea 分支元数据相关操作必须满足:
- 同一仓库同一时间只有一个自治写者。
- 分支创建、删除、批量同步必须有锁。
- 控制器需要限速,避免短时间大量 refs 变更。
- 锁超时或异常退出后必须有清理机制。
### 5. 修复前备份与最小修复
如再次发生漂移:
1. 先备份 `gitea.db`。
2. 对比 refs、branch 表、API。
3. 只补齐缺失 live 行或只修正明确错误行。
4. 不做无证据的重启、重建仓库或批量删除。
5. 修复后执行 doctor 与 GitOps runtime 接受测试。
### 6. 重启规则
Gitea 重启不得作为默认第一步。
只有在以下证据存在时才考虑重启:
- Gitea 进程不存在。
- HTTP 本地端口无响应。
- 日志显示明确死锁或长期阻塞。
- 数据库一致且 Git refs 一致,但服务仍无法响应。
- owner/admin 明确批准。
## 标准运行手册
再次出现自主运作系统停止时,按顺序执行:
1. 检查 GitOps runtime:`getRuntimeStatus`。
2. 检查 baseline:`getBaseline`。
3. 检查 Gitea 进程和本地 HTTP 端口。
4. 对比裸仓库 refs 数量。
5. 对比 SQLite `branch` live 行数量。
6. 对比 Gitea branch API 分页总数。
7. 执行 `gitea doctor check --run check-db-consistency`。
8. 判断是否是数据一致性问题,而不是直接重启。
9. 修复后运行接受测试:`getRuntimeStatus -> getBaseline -> createBranch -> createPatch`。
10. 将事故、根因和防复发动作提交到 GitOps。
## 后续行动项
- [ ] 实现 Gitea branch consistency sentinel。
- [ ] 为自主控制器增加 branch 操作后的 read-after-write 验证。
- [ ] 为自主控制器增加 Gitea/GitOps 写入熔断。
- [ ] 增加单写者锁和限速。
- [ ] 将本事故加入可靠性经验库。
- [ ] 将“导致自主运作系统停止即为事故”写入运行准则。
## 最终结论
本事故当前已恢复,但具备复发可能。
防复发重点不是单纯重启 Gitea,而是建立 refs、Gitea branch 表、API 分页、GitOps runtime 写链路之间的一致性哨兵、写后确认和控制器熔断机制。
+212
View File
@@ -0,0 +1,212 @@
# V1R4C5 Acceptance Evidence
Date: 2026-07-11
Status: release candidate evidence in progress
Current active baseline: V1R4C3
V1R4C5 has not been promoted yet.
## Owner Decision
Owner approved continued V1R4C5 work under the following constraints:
```text
允许无 force-push 的分支 reconcile
memory search 保持关闭
先完成 SR-01/SR-02/数据隔离验收
再进入 merge/deploy/tag owner gate
```
## Scope
V1R4C5 release candidate scope:
1. SR-01 Gitea reliability and branch consistency guardrails.
2. SR-02 watcher/stuck command cleanup.
3. User data isolation redline and enforcement gates.
4. SR-04 admin workbench read-only visibility preparation only.
Explicitly excluded from V1R4C5 activation:
- memory search activation;
- full SR-04 portal production rollout;
- bulk merge of old test/smoke/auth branches;
- force push reconciliation;
- Gitea DB mutation without owner/admin approval.
## Runtime Acceptance Evidence
### Runtime status
Observed:
```json
{
"service": "aios-gitops-model-api",
"status": "active",
"workspace_branch": "patch/v1r4c5-runtime-acceptance-clean-20260711-01"
}
```
### Baseline
Observed:
```json
{
"current_baseline": "V1R4C3",
"previous_baseline": "V1R3C1",
"promoted_by": "openclaw"
}
```
### Clean createBranch acceptance
Branch:
```text
test/v1r4c5-runtime-acceptance-clean-20260711-01
```
Result:
```text
status: ok
commit: 6ae7058
sync: synced
```
### Clean createPatch acceptance
Patch:
```text
patch/v1r4c5-runtime-acceptance-clean-20260711-01
```
Patch id:
```text
patch-1783706123789-f5ca8b69
```
Commit:
```text
ba0391e
```
Result:
```text
synced
```
## SR-01 Evidence
Known accepted evidence:
- Gitea branch consistency sentinel deployed.
- Sentinel reports:
```json
{
"status": "healthy",
"unhealthy_checks": []
}
```
- systemd timer runs every 5 minutes.
- service runs with:
```text
--emit-safe-mode --json
```
- broken symlink issue resolved by replacing the external reliability symlink with workspace-local files.
Required before final promotion:
- confirm latest sentinel report remains healthy after acceptance writes.
## SR-02 Evidence
Known accepted evidence:
```text
11 / 11 tests pass
```
Fix summary:
- stuck command detection reads parsed timestamps through `data.get('ts', 0)`;
- `.md` fixture mtime is aligned with stored stuck timestamp via `os.utime(path, (ts, ts))`;
- behavior is consistent across `.json` and `.md` state formats.
Required before final promotion:
- re-run SR-02 tests on the final integration candidate.
## User Data Isolation Evidence
Implemented controls:
- user data isolation redline;
- memory scope guard;
- provider gate;
- activation gate;
- audit service/timer;
- CI acceptance scripts;
- memory search activation check.
Required safety state:
```text
AIOS_MEMORY_ENABLED=false
```
Required before memory/search activation:
- isolation audit healthy;
- no denied records;
- no invalid records;
- provider filters by scope before ranking;
- provenance required;
- global index disabled;
- activation gate returns allowed;
- explicit owner approval.
V1R4C5 does not approve memory search activation.
## SR-04 Admin Workbench Evidence
Current user-visible gap is confirmed:
- administrator cannot yet see task progress;
- administrator cannot yet see flow/process diagrams;
- SR-04 remains activation-ready but gated.
V1R4C5 scope allows only read-only visibility preparation and acceptance design, not full production rollout.
Required before SR-04 deployment:
- SR-01 green;
- SR-02 green;
- user data isolation gate accepted;
- admin RBAC read-only check green.
## Remaining Required Gates Before Promotion
- [ ] Latest Gitea sentinel report remains healthy.
- [ ] SR-02 watcher tests pass on final candidate.
- [ ] User data isolation acceptance passes.
- [ ] Admin workbench read-only RBAC acceptance passes or remains explicitly deferred.
- [ ] No force-push was used.
- [ ] Memory search remains disabled.
- [ ] Owner approves merge/deploy/tag/promotion.
## Promotion Rule
V1R4C5 may not be promoted until all required gates are green or explicitly deferred with owner approval.
@@ -0,0 +1,54 @@
# V1R4C5 Final Owner Approval
Date: 2026-07-11
Status: approved for final promotion
## Owner Decision
Owner approved final V1R4C5 promotion.
Memory search must remain disabled.
```text
允许最终晋升
继续保持 memory search 关闭
```
## Promotion Scope
Approved for V1R4C5:
1. SR-01 Gitea reliability and branch consistency guardrails.
2. SR-02 watcher/stuck command cleanup.
3. User data isolation redline and enforcement gates.
4. SR-04 admin workbench read-only visibility preparation only.
## Explicit Non-Activation
The following are not approved by this promotion:
```text
memory search activation
full cross-session memory recall
global vector index
full SR-04 production portal rollout beyond read-only/admin visibility preparation
bulk merge of old test/smoke/auth branches
force-push based reconciliation
```
## Required Runtime State
Memory remains disabled:
```text
AIOS_MEMORY_ENABLED=false
```
If a provider or deployment attempts to enable memory/search, it must be blocked unless a future owner approval explicitly allows it after the user-data isolation activation gate returns allowed.
## Final Promotion Rule
V1R4C5 may proceed to tag/promotion under owner gate.
If any final gate fails, promotion must stop and memory/search must remain disabled.
@@ -1,96 +0,0 @@
# Gitea Autonomous Controller Reliability Guardrails
## Purpose
Prevent autonomous operations from stopping due to Gitea branch metadata drift or uncontrolled Git state mutation.
## Required Controls
### 1. Consistency Sentinel
Continuously compare:
```text
bare repository refs
==
Gitea branch live metadata
==
branch API visible state
```
Alert on any drift.
### 2. Write Barrier
Every autonomous branch mutation must include:
1. Git operation result check.
2. Gitea API read-back verification.
3. Metadata consistency verification.
4. Remote sync verification.
Failed verification blocks further autonomous writes.
### 3. Circuit Breaker
Controller enters safe mode when:
- branch metadata drift detected;
- repeated Git/Gitea API failures occur;
- consistency checks fail;
- write acceptance tests fail.
Safe mode:
- disable mutation operations;
- preserve read-only diagnostics;
- create incident record;
- require controlled recovery.
### 4. Single Writer Rule
For branch metadata operations:
- only one autonomous writer per repository;
- use operation locks;
- limit mutation frequency;
- recover abandoned locks safely.
### 5. Recovery Order
Do not restart first.
Order:
1. Check service health.
2. Check process state.
3. Check Git refs.
4. Check Gitea metadata.
5. Check API visibility.
6. Repair minimal inconsistent state.
7. Run acceptance tests.
Restart is reserved for proven process-level failure.
### 6. Acceptance Test
After recovery:
```text
getRuntimeStatus
->
getBaseline
->
createBranch
->
createPatch
```
Expected:
- read operations return active runtime information;
- write operations return commit, revision and sync status.
## Operational Principle
Any condition that stops autonomous operation is an incident. Reliability improvements must prevent recurrence rather than only restore service.