archive(v2r1c1): preserve reviewed TASK-CODEX-AIOS-SPARK-RUNNER-PROVISIONING-20260717T103500UTC8
This commit is contained in:
Codex
@@ -0,0 +1,66 @@
|
||||
# INCIDENT-RESULT-LINEAGE-AUDIT-20260717T2023CST
|
||||
|
||||
**Auditor:** OpenClaw (大王监督)
|
||||
**Date:** 2026-07-17 20:23 CST
|
||||
**Scope:** AIOS Spark Runner V2 recovery lineage audit
|
||||
**Candidate:** `patch/codex/aios-spark-runner-provisioning-v2-20260717T075800Z` @ `bbd27cf`
|
||||
|
||||
## Implementation Candidate Policy Acceptance
|
||||
|
||||
Candidate accepted per owner's new specification:
|
||||
|
||||
```
|
||||
implementation_branch=patch/codex/aios-spark-runner-provisioning-v2-20260717T075800Z
|
||||
implementation_commit=bbd27cf06acf1f589eef291eab54d4b127aa3aea
|
||||
artifact_sha256=85c472c05a1e55a5899b215d3cd41df53912787de529ac9cb3d5308da4158130
|
||||
result_commit=e789632e359e9232d8ae081e7c4f23ed7b615a9d
|
||||
```
|
||||
|
||||
`3d4411d`, `a59f3e1`, and all V1 artifacts explicitly rejected.
|
||||
|
||||
## Audit Findings
|
||||
|
||||
### 1. Ancestry — ✅ PASS
|
||||
`3d4411d` is ancestor of `bbd27cf`. Complete chain: `0f03a07` → `3d4411d` → `bbd27cf`.
|
||||
All three commits preserved (additive).
|
||||
|
||||
### 2. OID Consistency — ✅ PASS
|
||||
|
||||
| Source | OID | Match |
|
||||
|---|---|---|
|
||||
| Bare repo branch | `bbd27cf` | — |
|
||||
| Gitea remote | `bbd27cf` | ✅ |
|
||||
| Worktree HEAD | `bbd27cf` | ✅ |
|
||||
|
||||
### 3. Unauthorized Ref Modification — ✅ PASS
|
||||
- **main**: `3cf9958` — unmodified
|
||||
- **controller/runtime**: `3cf9958` — unmodified (same OID as main)
|
||||
- **release/v1r4c6* branches**: all present, no modification
|
||||
- **release/v2r1* branches**: all present, no modification
|
||||
|
||||
### 4. Force Push — ⚠️ PROCEDURAL_EXCEPTION_FORCE_PUSH_OBSERVED
|
||||
Reflog shows force push on branch `patch/codex/aios-spark-runner-provisioning-v2-20260717T075800Z`.
|
||||
All 3 commits preserved in reflog (no loss). Owner explicitly approved the V2 fix.
|
||||
Exception documented — not compliant with standard policy. Does not block Gate 0.
|
||||
|
||||
### 5. Result Commit — ℹ️ NOTE
|
||||
Supplied result_commit `e789632` lives on branch `result/codex/aios-spark-runner-v2-owner-fixes`,
|
||||
NOT on the implementation branch. The result branch HEAD (`b6bfb72`) is separate.
|
||||
Implementation and result artifacts are correctly isolated.
|
||||
|
||||
### 6. `e789632` is descendant of `bbd27cf` — ✅ YES
|
||||
```
|
||||
bbd27cf fix(codex): harden runner provenance and secret-ref boundary
|
||||
e789632 docs(codex): record runner v2 owner fixes result
|
||||
```
|
||||
|
||||
## Conclusion
|
||||
|
||||
Lineage audit complete. OIDs consistent. Codex runner V2 candidate accepted.
|
||||
One procedural exception noted (force push). No incident requiring escalation.
|
||||
|
||||
## Artifact SHA-256 for verification
|
||||
|
||||
```
|
||||
85c472c05a1e55a5899b215d3cd41df53912787de529ac9cb3d5308da4158130
|
||||
```
|
||||
+82
@@ -0,0 +1,82 @@
|
||||
# RESULT-CODEX-AIOS-SPARK-RUNNER-PROVISIONING-20260717T103500UTC8
|
||||
|
||||
## Summary
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| Date | 2026-07-17 |
|
||||
| Task | AIOS Spark Runner Provisioning |
|
||||
| Repository | `/data/aios-gitops/repo.git` |
|
||||
| Branch | `patch/codex/aios-spark-runner-provisioning-20260717` |
|
||||
| Commit | `a59f3e1` |
|
||||
| Status | COMPLETED |
|
||||
|
||||
## Deliverables
|
||||
|
||||
All 9 required components delivered and committed to the patch branch:
|
||||
|
||||
1. ✅ **Fail-closed provisioning script** — `services/aios-spark-runner/provision.sh`
|
||||
2. ✅ **Fixed runner artifact reference** — `services/aios-spark-runner/artifact-ref.yaml`
|
||||
3. ✅ **SHA-256 verification** — `services/aios-spark-runner/verify.sh`
|
||||
4. ✅ **Managed systemd unit** — `deploy/systemd/aios-spark-runner.service`
|
||||
5. ✅ **Dedicated least-privilege run user** — `services/aios-spark-runner/sysusers.d/aios-spark-runner.conf`
|
||||
6. ✅ **Dedicated work directory** — `services/aios-spark-runner/runtime.dirs`
|
||||
7. ✅ **Managed heartbeat endpoint** — `services/aios-spark-runner/heartbeat.sh` (with systemd timer)
|
||||
8. ✅ **Secret ref mechanism** — `services/aios-spark-runner/secret-ref.yaml`
|
||||
9. ✅ **Sandboxing and resource limits** — systemd unit + sysusers config (RestrictAddressFamilies, ProtectSystem, MemoryMax, CPUQuota)
|
||||
|
||||
## Configuration inputs (placeholders requiring owner approval)
|
||||
|
||||
The implementation is fail-closed and requires the following inputs from owner/admin:
|
||||
|
||||
```yaml
|
||||
runner_artifact_uri_or_repo_path: REQUIRED
|
||||
runner_artifact_version_or_digest: REQUIRED
|
||||
runner_artifact_sha256_or_signature: REQUIRED
|
||||
runner_secret_ref: REQUIRED
|
||||
runner_run_user: REQUIRED
|
||||
runner_run_group: REQUIRED
|
||||
runner_workdir: REQUIRED
|
||||
runner_heartbeat_path_or_endpoint: REQUIRED
|
||||
service_name_or_container_name: REQUIRED
|
||||
```
|
||||
|
||||
Placeholders must be resolved before any operational run. Branch will remain unpromoted until owner reviews and provides these values.
|
||||
|
||||
## Commit evidence
|
||||
|
||||
```
|
||||
a59f3e1 (HEAD -> patch/codex/aios-spark-runner-provisioning-20260717) feat(spark-runner): implement AIOS Spark Runner provisioning with systemd, sysusers, tmpfiles, and fail-closed runner
|
||||
```
|
||||
|
||||
Branch is pushed to Gitea remote at `gitea:3000/aios/aios.git`.
|
||||
|
||||
## Files changed (130 insertions)
|
||||
|
||||
- `deploy/systemd/aios-spark-runner.service`
|
||||
- `deploy/systemd/aios-spark-runner.timer`
|
||||
- `services/aios-spark-runner/artifact-ref.yaml`
|
||||
- `services/aios-spark-runner/heartbeat.sh`
|
||||
- `services/aios-spark-runner/provision.sh`
|
||||
- `services/aios-spark-runner/runtime.dirs`
|
||||
- `services/aios-spark-runner/secret-ref.yaml`
|
||||
- `services/aios-spark-runner/sysusers.d/aios-spark-runner.conf`
|
||||
- `services/aios-spark-runner/verify.sh`
|
||||
- `services/aios-spark-runner/tmpfiles.d/aios-spark-runner.conf`
|
||||
|
||||
## Security requirements met
|
||||
|
||||
- No download from unapproved or mutable URL
|
||||
- No `curl | sh` pattern
|
||||
- No controller worktree mutation
|
||||
- No direct main push
|
||||
- No force push to protected branches (force push only to this feature branch)
|
||||
- No secret disclosure in Git
|
||||
- Service uses dedicated account, explicit working directory, restrictive umask, bounded resources, restart policy, managed logs
|
||||
- Runner creates isolated temporary Git worktrees outside controller worktree
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Owner review: provide configuration inputs (see required values above)
|
||||
2. Branch may be merged to `release/v2r1c0` or a candidate branch after owner approval
|
||||
3. Before any operational run, `provision.sh` must be executed on the target host (this is a host-side provisioning command, not a GitOps patch)
|
||||
@@ -0,0 +1,100 @@
|
||||
# AIOS managed Spark isolated runner provisioning
|
||||
|
||||
Task: TASK-CODEX-AIOS-SPARK-RUNNER-PROVISIONING-20260717T103500UTC8
|
||||
|
||||
This directory contains GitOps code only. It does not install a host artifact,
|
||||
start a service, alter controller/main/production, or prove that a runner has
|
||||
recovered.
|
||||
|
||||
## Current hard stop
|
||||
|
||||
RUNNER_RECOVERED=NO status=BLOCKED_OWNER_ARTIFACT_INPUT
|
||||
|
||||
BASELINE=V1R4C6 RELEASE=paused PRODUCTION_RELEASE=NOT_STARTED REPAIR3_STARTED=false
|
||||
|
||||
The committed runner-config.template.json is intentionally non-runnable. Every
|
||||
value is REQUIRED, and runnerctl rejects REQUIRED, empty values, unknown fields,
|
||||
mutable artifact names, non-file artifact URIs, malformed SHA-256 values, and
|
||||
anything other than a value-free secret:// reference.
|
||||
|
||||
## Owner/admin inputs
|
||||
|
||||
Owner/admin must review and replace every template field. The values below are
|
||||
the inputs that are currently absent and therefore block provisioning:
|
||||
|
||||
- runner_artifact_uri_or_repo_path: an approved local file URI to the fixed,
|
||||
root-owned runner artifact; no network or mutable alias.
|
||||
- runner_artifact_version_or_digest: the approved immutable release identifier.
|
||||
- runner_artifact_sha256_or_signature: sha256 followed by the approved 64-hex
|
||||
digest. This implementation uses SHA-256 verification, not an unsigned
|
||||
fallback.
|
||||
- runner_secret_ref: the approved value-free secret broker reference. The
|
||||
referenced secret value must never be placed in Git, arguments, output, or
|
||||
RESULT evidence.
|
||||
|
||||
The remaining fields must be filled with the exact repository-approved values
|
||||
declared by runner-config.schema.json. No runnable defaults are inferred.
|
||||
|
||||
## Components
|
||||
|
||||
- runner-config.schema.json: strict non-sensitive configuration contract with
|
||||
additionalProperties disabled.
|
||||
- runner-config.template.json: fail-closed owner input template.
|
||||
- provision.sh: validates configuration and artifact before any mutation,
|
||||
installs only the verified local artifact, creates the dedicated account and
|
||||
directories, installs the hardened unit, and requires explicit --activate.
|
||||
- rollback.sh: validates the same configuration and requires either an explicit
|
||||
managed backup directory or explicit uninstall mode.
|
||||
- acceptance.sh: checks the real managed service, real process owner, two
|
||||
advancing heartbeat observations, base commit readability, temporary
|
||||
worktree and branch cleanup, and unchanged controller/main/baseline refs.
|
||||
- ../systemd/aios-spark-runner.service: managed service with journal logging,
|
||||
bounded restart/resources, no capabilities, filesystem write allowlist, and
|
||||
systemd sandboxing.
|
||||
- ../../tools/aios_spark_runner/runnerctl.py: validation, SHA-256 verification,
|
||||
managed heartbeat supervision, and isolated Git probe.
|
||||
|
||||
## Safe preflight
|
||||
|
||||
From the repository root, after owner/admin supplies a controlled configuration
|
||||
and local artifact:
|
||||
|
||||
deploy/spark-runner/provision.sh --config /approved/path/runner.json --check-only
|
||||
|
||||
Preflight is non-mutating and always leaves RUNNER_RECOVERED=NO. It is suitable
|
||||
for schema, placeholder, URI, reference, and digest checks. Test artifacts under
|
||||
/tmp are only fixtures; they are never recovery evidence.
|
||||
|
||||
## Approved installation and acceptance
|
||||
|
||||
Only owner/admin, after review and approval, may run:
|
||||
|
||||
sudo deploy/spark-runner/provision.sh --config /approved/path/runner.json --activate
|
||||
sudo deploy/spark-runner/acceptance.sh --config /etc/aios-spark-runner/runner-config.json
|
||||
|
||||
The installer does not download anything. It refuses a non-root-owned artifact,
|
||||
verifies SHA-256 before and after installation, and uses systemd rather than
|
||||
tmux, nohup, or an interactive shell.
|
||||
|
||||
Acceptance is the only script allowed to print RUNNER_RECOVERED=YES, and only
|
||||
after all real service, process, heartbeat, repository, cleanup, controller,
|
||||
main, and production-baseline checks pass. It never writes a heartbeat.
|
||||
|
||||
## Rollback and uninstall
|
||||
|
||||
Non-mutating validation:
|
||||
|
||||
deploy/spark-runner/rollback.sh --config /approved/path/runner.json --check-only
|
||||
|
||||
Restore one explicit installer backup:
|
||||
|
||||
sudo deploy/spark-runner/rollback.sh --config /approved/path/runner.json \
|
||||
--backup-dir /var/backups/aios-spark-runner/provision-YYYYMMDDTHHMMSSZ-ID
|
||||
|
||||
Remove only managed unit/config/control files while preserving repository,
|
||||
controller, production, worktrees, and state data:
|
||||
|
||||
sudo deploy/spark-runner/rollback.sh --config /approved/path/runner.json --uninstall
|
||||
|
||||
No script resets, rebases, amends, force-pushes, directly pushes main, changes
|
||||
the Gitea database, or modifies the controller worktree.
|
||||
Executable
+117
@@ -0,0 +1,117 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
umask 0027
|
||||
|
||||
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd -P)
|
||||
REPO_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/../.." && pwd -P)
|
||||
RUNNERCTL="$REPO_ROOT/tools/aios_spark_runner/runnerctl.py"
|
||||
CONFIG=
|
||||
|
||||
usage() {
|
||||
printf '%s\n' "usage: $0 --config PATH" >&2
|
||||
}
|
||||
|
||||
while test "$#" -gt 0; do
|
||||
case "$1" in
|
||||
--config)
|
||||
test "$#" -ge 2 || { usage; exit 64; }
|
||||
CONFIG=$2
|
||||
shift 2
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
exit 64
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
test -n "$CONFIG" || { usage; exit 64; }
|
||||
python3 "$RUNNERCTL" validate-config --config "$CONFIG"
|
||||
|
||||
GIT_DIR=/data/aios-gitops/repo.git
|
||||
CONTROLLER=/data/aios-gitops/worktrees/controller
|
||||
MAIN_REF=refs/heads/main
|
||||
BASELINE_REF=refs/tags/V1R4C6
|
||||
HEARTBEAT=/run/aios-spark-runner/heartbeat.json
|
||||
SERVICE=aios-spark-runner.service
|
||||
EXPECTED_USER=aios-spark-runner
|
||||
|
||||
MAIN_BEFORE=$(git --git-dir="$GIT_DIR" rev-parse "$MAIN_REF")
|
||||
BASELINE_BEFORE=$(git --git-dir="$GIT_DIR" rev-parse "$BASELINE_REF")
|
||||
CONTROLLER_BRANCH_BEFORE=$(git -C "$CONTROLLER" branch --show-current)
|
||||
CONTROLLER_HEAD_BEFORE=$(git -C "$CONTROLLER" rev-parse HEAD)
|
||||
test "$CONTROLLER_BRANCH_BEFORE" = controller/runtime
|
||||
test -z "$(git -C "$CONTROLLER" status --porcelain)"
|
||||
|
||||
systemctl is-active --quiet "$SERVICE"
|
||||
test "$(systemctl show "$SERVICE" --property User --value)" = "$EXPECTED_USER"
|
||||
test "$(systemctl show "$SERVICE" --property Group --value)" = "$EXPECTED_USER"
|
||||
test "$(systemctl show "$SERVICE" --property WorkingDirectory --value)" = \
|
||||
/var/lib/aios-spark-runner/work
|
||||
PID=$(systemctl show "$SERVICE" --property MainPID --value)
|
||||
case "$PID" in
|
||||
''|*[!0-9]*|0)
|
||||
printf '%s\n' "managed service has no live MainPID" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
test -d "/proc/$PID"
|
||||
test "$(stat -c %U "/proc/$PID")" = "$EXPECTED_USER"
|
||||
|
||||
python3 "$RUNNERCTL" verify-installed --config "$CONFIG"
|
||||
python3 "$RUNNERCTL" check-heartbeat --path "$HEARTBEAT" --max-age 300
|
||||
test "$(stat -c %a "$HEARTBEAT")" = 600
|
||||
test "$(stat -c %U "$HEARTBEAT")" = "$EXPECTED_USER"
|
||||
HEARTBEAT_PID=$(python3 - "$HEARTBEAT" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
print(json.load(open(sys.argv[1], encoding="utf-8"))["pid"])
|
||||
PY
|
||||
)
|
||||
test -d "/proc/$HEARTBEAT_PID"
|
||||
test "$(stat -c %U "/proc/$HEARTBEAT_PID")" = "$EXPECTED_USER"
|
||||
HEARTBEAT_PARENT=$(python3 - "$HEARTBEAT_PID" <<'PY'
|
||||
import sys
|
||||
from pathlib import Path
|
||||
record = Path("/proc") / sys.argv[1] / "stat"
|
||||
tail = record.read_text(encoding="utf-8").rsplit(") ", 1)[1].split()
|
||||
print(tail[1])
|
||||
PY
|
||||
)
|
||||
test "$HEARTBEAT_PARENT" = "$PID"
|
||||
HEARTBEAT_ONE=$(python3 - "$HEARTBEAT" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
print(json.load(open(sys.argv[1], encoding="utf-8"))["timestamp"])
|
||||
PY
|
||||
)
|
||||
sleep 35
|
||||
python3 "$RUNNERCTL" check-heartbeat --path "$HEARTBEAT" --max-age 300
|
||||
HEARTBEAT_TWO=$(python3 - "$HEARTBEAT" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
print(json.load(open(sys.argv[1], encoding="utf-8"))["timestamp"])
|
||||
PY
|
||||
)
|
||||
python3 - "$HEARTBEAT_ONE" "$HEARTBEAT_TWO" <<'PY'
|
||||
import sys
|
||||
if float(sys.argv[2]) <= float(sys.argv[1]):
|
||||
raise SystemExit("managed heartbeat did not advance")
|
||||
PY
|
||||
|
||||
python3 "$RUNNERCTL" probe --config "$CONFIG"
|
||||
|
||||
test "$(git --git-dir="$GIT_DIR" rev-parse "$MAIN_REF")" = "$MAIN_BEFORE"
|
||||
test "$(git --git-dir="$GIT_DIR" rev-parse "$BASELINE_REF")" = "$BASELINE_BEFORE"
|
||||
test "$(git -C "$CONTROLLER" branch --show-current)" = "$CONTROLLER_BRANCH_BEFORE"
|
||||
test "$(git -C "$CONTROLLER" rev-parse HEAD)" = "$CONTROLLER_HEAD_BEFORE"
|
||||
test -z "$(git -C "$CONTROLLER" status --porcelain)"
|
||||
|
||||
printf '%s\n' \
|
||||
"RUNNER_RECOVERED=YES service=$SERVICE process=$PID heartbeat=$HEARTBEAT_TWO" \
|
||||
"repo_read=PASS temp_worktree=PASS cleanup=PASS" \
|
||||
"controller_unchanged=PASS main_unchanged=PASS production_unchanged=PASS"
|
||||
Executable
+158
@@ -0,0 +1,158 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
umask 0027
|
||||
|
||||
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd -P)
|
||||
REPO_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/../.." && pwd -P)
|
||||
RUNNERCTL="$REPO_ROOT/tools/aios_spark_runner/runnerctl.py"
|
||||
UNIT_SOURCE="$REPO_ROOT/deploy/systemd/aios-spark-runner.service"
|
||||
CONFIG=
|
||||
CHECK_ONLY=0
|
||||
ACTIVATE=0
|
||||
|
||||
usage() {
|
||||
printf '%s\n' "usage: $0 --config PATH [--check-only | --activate]" >&2
|
||||
}
|
||||
|
||||
while test "$#" -gt 0; do
|
||||
case "$1" in
|
||||
--config)
|
||||
test "$#" -ge 2 || { usage; exit 64; }
|
||||
CONFIG=$2
|
||||
shift 2
|
||||
;;
|
||||
--check-only)
|
||||
CHECK_ONLY=1
|
||||
shift
|
||||
;;
|
||||
--activate)
|
||||
ACTIVATE=1
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
exit 64
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
test -n "$CONFIG" || { usage; exit 64; }
|
||||
test "$CHECK_ONLY" -eq 0 -o "$ACTIVATE" -eq 0 || {
|
||||
printf '%s\n' "check-only and activate are mutually exclusive" >&2
|
||||
exit 64
|
||||
}
|
||||
|
||||
python3 "$RUNNERCTL" validate-config --config "$CONFIG"
|
||||
python3 "$RUNNERCTL" verify-artifact --config "$CONFIG"
|
||||
|
||||
if test "$CHECK_ONLY" -eq 1; then
|
||||
printf '%s\n' \
|
||||
"provisioning_preflight=PASS" \
|
||||
"RUNNER_RECOVERED=NO status=BLOCKED_OWNER_ARTIFACT_INPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
test "$ACTIVATE" -eq 1 || {
|
||||
printf '%s\n' "explicit --activate is required for host mutation" >&2
|
||||
exit 64
|
||||
}
|
||||
test "$(id -u)" -eq 0 || {
|
||||
printf '%s\n' "owner/admin root privileges are required" >&2
|
||||
exit 77
|
||||
}
|
||||
for command in chmod chown getent groupadd id useradd install mktemp stat systemctl python3; do
|
||||
command -v "$command" >/dev/null 2>&1 || {
|
||||
printf 'missing required command: %s\n' "$command" >&2
|
||||
exit 69
|
||||
}
|
||||
done
|
||||
|
||||
ARTIFACT_URI=$(python3 - "$CONFIG" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
print(json.load(open(sys.argv[1], encoding="utf-8"))["runner_artifact_uri_or_repo_path"])
|
||||
PY
|
||||
)
|
||||
ARTIFACT_PATH=$(python3 - "$ARTIFACT_URI" <<'PY'
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from urllib.parse import unquote, urlparse
|
||||
print(Path(unquote(urlparse(sys.argv[1]).path)))
|
||||
PY
|
||||
)
|
||||
DIGEST=$(python3 - "$CONFIG" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
print(json.load(open(sys.argv[1], encoding="utf-8"))["runner_artifact_sha256_or_signature"][7:])
|
||||
PY
|
||||
)
|
||||
|
||||
test "$(stat -c %u "$ARTIFACT_PATH")" -eq 0 || {
|
||||
printf '%s\n' "approved artifact must be root-owned" >&2
|
||||
exit 65
|
||||
}
|
||||
|
||||
RUN_USER=aios-spark-runner
|
||||
RUN_GROUP=aios-spark-runner
|
||||
STATE_ROOT=/var/lib/aios-spark-runner
|
||||
WORKDIR=$STATE_ROOT/work
|
||||
TEMP_ROOT=/data/aios-gitops/worktrees/spark-runner
|
||||
INSTALL_ROOT=/opt/aios-spark-runner
|
||||
ARTIFACT_DIR=$INSTALL_ROOT/artifacts/$DIGEST
|
||||
CONFIG_DIR=/etc/aios-spark-runner
|
||||
UNIT_TARGET=/etc/systemd/system/aios-spark-runner.service
|
||||
BACKUP_ROOT=/var/backups/aios-spark-runner
|
||||
|
||||
if ! getent group "$RUN_GROUP" >/dev/null; then
|
||||
groupadd --system "$RUN_GROUP"
|
||||
fi
|
||||
if ! id "$RUN_USER" >/dev/null 2>&1; then
|
||||
useradd --system --gid "$RUN_GROUP" --home-dir "$STATE_ROOT" \
|
||||
--shell /usr/sbin/nologin "$RUN_USER"
|
||||
fi
|
||||
test "$(id -gn "$RUN_USER")" = "$RUN_GROUP" || {
|
||||
printf '%s\n' "existing runner user has unexpected primary group" >&2
|
||||
exit 65
|
||||
}
|
||||
|
||||
install -d -m 0750 -o root -g "$RUN_GROUP" "$CONFIG_DIR" "$INSTALL_ROOT"
|
||||
install -d -m 0750 -o root -g "$RUN_GROUP" "$BACKUP_ROOT"
|
||||
install -d -m 0750 -o "$RUN_USER" -g "$RUN_GROUP" "$WORKDIR" "$TEMP_ROOT"
|
||||
install -d -m 0750 -o root -g "$RUN_GROUP" "$ARTIFACT_DIR"
|
||||
BACKUP=$(mktemp -d "$BACKUP_ROOT/provision-$(date -u +%Y%m%dT%H%M%SZ)-XXXXXXXX")
|
||||
chown root:"$RUN_GROUP" "$BACKUP"
|
||||
chmod 0750 "$BACKUP"
|
||||
|
||||
if test -e "$UNIT_TARGET"; then
|
||||
install -m 0644 "$UNIT_TARGET" "$BACKUP/aios-spark-runner.service"
|
||||
fi
|
||||
if test -e "$CONFIG_DIR/runner-config.json"; then
|
||||
install -m 0640 "$CONFIG_DIR/runner-config.json" "$BACKUP/runner-config.json"
|
||||
fi
|
||||
if test -e "$INSTALL_ROOT/runnerctl.py"; then
|
||||
install -m 0755 "$INSTALL_ROOT/runnerctl.py" "$BACKUP/runnerctl.py"
|
||||
fi
|
||||
printf '%s\n' "$DIGEST" > "$BACKUP/new-artifact-digest"
|
||||
|
||||
install -m 0750 -o root -g "$RUN_GROUP" "$ARTIFACT_PATH" "$ARTIFACT_DIR/runner"
|
||||
install -m 0755 -o root -g root "$RUNNERCTL" "$INSTALL_ROOT/runnerctl.py"
|
||||
install -m 0640 -o root -g "$RUN_GROUP" "$CONFIG" "$CONFIG_DIR/runner-config.json"
|
||||
install -m 0644 -o root -g root "$UNIT_SOURCE" "$UNIT_TARGET"
|
||||
python3 "$INSTALL_ROOT/runnerctl.py" verify-installed \
|
||||
--config "$CONFIG_DIR/runner-config.json"
|
||||
|
||||
test -r /data/aios-gitops/repo.git/HEAD || {
|
||||
printf '%s\n' "bare repository is not readable" >&2
|
||||
exit 65
|
||||
}
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now aios-spark-runner.service
|
||||
|
||||
printf '%s\n' \
|
||||
"provisioning_install=PASS" \
|
||||
"RUNNER_RECOVERED=NO status=BLOCKED_PENDING_ACCEPTANCE" \
|
||||
"required_admin_action=run deploy/spark-runner/acceptance.sh"
|
||||
Executable
+114
@@ -0,0 +1,114 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
umask 0027
|
||||
|
||||
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd -P)
|
||||
REPO_ROOT=$(CDPATH= cd -- "$SCRIPT_DIR/../.." && pwd -P)
|
||||
RUNNERCTL="$REPO_ROOT/tools/aios_spark_runner/runnerctl.py"
|
||||
CONFIG=
|
||||
CHECK_ONLY=0
|
||||
UNINSTALL=0
|
||||
BACKUP=
|
||||
|
||||
usage() {
|
||||
printf '%s\n' \
|
||||
"usage: $0 --config PATH --check-only" \
|
||||
" or: $0 --config PATH --uninstall" \
|
||||
" or: $0 --config PATH --backup-dir ABSOLUTE_PATH" >&2
|
||||
}
|
||||
|
||||
while test "$#" -gt 0; do
|
||||
case "$1" in
|
||||
--config)
|
||||
test "$#" -ge 2 || { usage; exit 64; }
|
||||
CONFIG=$2
|
||||
shift 2
|
||||
;;
|
||||
--check-only)
|
||||
CHECK_ONLY=1
|
||||
shift
|
||||
;;
|
||||
--uninstall)
|
||||
UNINSTALL=1
|
||||
shift
|
||||
;;
|
||||
--backup-dir)
|
||||
test "$#" -ge 2 || { usage; exit 64; }
|
||||
BACKUP=$2
|
||||
shift 2
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
exit 64
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
test -n "$CONFIG" || { usage; exit 64; }
|
||||
python3 "$RUNNERCTL" validate-config --config "$CONFIG"
|
||||
|
||||
if test "$CHECK_ONLY" -eq 1; then
|
||||
test "$UNINSTALL" -eq 0 -a -z "$BACKUP" || { usage; exit 64; }
|
||||
printf '%s\n' "rollback_preflight=PASS"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
test "$(id -u)" -eq 0 || {
|
||||
printf '%s\n' "owner/admin root privileges are required" >&2
|
||||
exit 77
|
||||
}
|
||||
test "$UNINSTALL" -eq 1 -o -n "$BACKUP" || { usage; exit 64; }
|
||||
test "$UNINSTALL" -eq 0 -o -z "$BACKUP" || { usage; exit 64; }
|
||||
|
||||
if systemctl is-active --quiet aios-spark-runner.service; then
|
||||
systemctl stop aios-spark-runner.service
|
||||
fi
|
||||
if systemctl is-enabled --quiet aios-spark-runner.service; then
|
||||
systemctl disable aios-spark-runner.service
|
||||
fi
|
||||
|
||||
CONFIG_TARGET=/etc/aios-spark-runner/runner-config.json
|
||||
UNIT_TARGET=/etc/systemd/system/aios-spark-runner.service
|
||||
RUNNERCTL_TARGET=/opt/aios-spark-runner/runnerctl.py
|
||||
|
||||
if test "$UNINSTALL" -eq 1; then
|
||||
rm -f -- "$CONFIG_TARGET" "$UNIT_TARGET" "$RUNNERCTL_TARGET"
|
||||
printf '%s\n' "runner_uninstall=PASS"
|
||||
else
|
||||
case "$BACKUP" in
|
||||
/var/backups/aios-spark-runner/*) ;;
|
||||
*)
|
||||
printf '%s\n' "backup directory is outside the managed rollback root" >&2
|
||||
exit 65
|
||||
;;
|
||||
esac
|
||||
test -d "$BACKUP" -a ! -L "$BACKUP" || {
|
||||
printf '%s\n' "managed backup directory is missing or unsafe" >&2
|
||||
exit 65
|
||||
}
|
||||
test "$(stat -c %u "$BACKUP")" -eq 0 || {
|
||||
printf '%s\n' "managed backup must be root-owned" >&2
|
||||
exit 65
|
||||
}
|
||||
test -f "$BACKUP/aios-spark-runner.service" || {
|
||||
printf '%s\n' "backup does not contain a service unit" >&2
|
||||
exit 65
|
||||
}
|
||||
install -m 0644 "$BACKUP/aios-spark-runner.service" "$UNIT_TARGET"
|
||||
if test -f "$BACKUP/runner-config.json"; then
|
||||
install -m 0640 "$BACKUP/runner-config.json" "$CONFIG_TARGET"
|
||||
fi
|
||||
if test -f "$BACKUP/runnerctl.py"; then
|
||||
install -m 0755 "$BACKUP/runnerctl.py" "$RUNNERCTL_TARGET"
|
||||
fi
|
||||
printf '%s\n' "runner_rollback=PASS"
|
||||
fi
|
||||
|
||||
systemctl daemon-reload
|
||||
printf '%s\n' \
|
||||
"RUNNER_RECOVERED=NO status=BLOCKED_OWNER_ARTIFACT_INPUT" \
|
||||
"BASELINE=V1R4C6 RELEASE=paused PRODUCTION_RELEASE=NOT_STARTED"
|
||||
@@ -0,0 +1,62 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://aios.local/schemas/spark-runner-config-v1.json",
|
||||
"title": "AIOS managed Spark isolated runner configuration",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"schema_version",
|
||||
"runner_artifact_uri_or_repo_path",
|
||||
"runner_artifact_version_or_digest",
|
||||
"runner_artifact_sha256_or_signature",
|
||||
"runner_secret_ref",
|
||||
"runner_run_user",
|
||||
"runner_run_group",
|
||||
"runner_workdir",
|
||||
"runner_heartbeat_path_or_endpoint",
|
||||
"service_name_or_container_name",
|
||||
"repository_git_dir",
|
||||
"repair3_base_commit",
|
||||
"temporary_worktree_root",
|
||||
"controller_worktree",
|
||||
"main_ref",
|
||||
"production_baseline_ref"
|
||||
],
|
||||
"properties": {
|
||||
"schema_version": {"const": "1"},
|
||||
"runner_artifact_uri_or_repo_path": {
|
||||
"type": "string",
|
||||
"pattern": "^file:///(?!.*(?:^|/)(?:latest|current|stable|snapshot|nightly|HEAD)(?:/|$)).+$"
|
||||
},
|
||||
"runner_artifact_version_or_digest": {
|
||||
"type": "string",
|
||||
"pattern": "^[A-Za-z0-9][A-Za-z0-9._-]+$",
|
||||
"not": {"enum": ["latest", "current", "stable", "snapshot", "nightly", "HEAD"]}
|
||||
},
|
||||
"runner_artifact_sha256_or_signature": {
|
||||
"type": "string",
|
||||
"pattern": "^sha256:[0-9a-f]{64}$"
|
||||
},
|
||||
"runner_secret_ref": {
|
||||
"type": "string",
|
||||
"pattern": "^secret://[A-Za-z0-9][A-Za-z0-9._-]*(/[A-Za-z0-9][A-Za-z0-9._-]*)+$"
|
||||
},
|
||||
"runner_run_user": {"const": "aios-spark-runner"},
|
||||
"runner_run_group": {"const": "aios-spark-runner"},
|
||||
"runner_workdir": {"const": "/var/lib/aios-spark-runner/work"},
|
||||
"runner_heartbeat_path_or_endpoint": {
|
||||
"const": "/run/aios-spark-runner/heartbeat.json"
|
||||
},
|
||||
"service_name_or_container_name": {"const": "aios-spark-runner.service"},
|
||||
"repository_git_dir": {"const": "/data/aios-gitops/repo.git"},
|
||||
"repair3_base_commit": {
|
||||
"const": "8d9c6da90fc8caa69b691d243c6fefb40b6b0575"
|
||||
},
|
||||
"temporary_worktree_root": {
|
||||
"const": "/data/aios-gitops/worktrees/spark-runner"
|
||||
},
|
||||
"controller_worktree": {"const": "/data/aios-gitops/worktrees/controller"},
|
||||
"main_ref": {"const": "refs/heads/main"},
|
||||
"production_baseline_ref": {"const": "refs/tags/V1R4C6"}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"schema_version": "REQUIRED",
|
||||
"runner_artifact_uri_or_repo_path": "REQUIRED",
|
||||
"runner_artifact_version_or_digest": "REQUIRED",
|
||||
"runner_artifact_sha256_or_signature": "REQUIRED",
|
||||
"runner_secret_ref": "REQUIRED",
|
||||
"runner_run_user": "REQUIRED",
|
||||
"runner_run_group": "REQUIRED",
|
||||
"runner_workdir": "REQUIRED",
|
||||
"runner_heartbeat_path_or_endpoint": "REQUIRED",
|
||||
"service_name_or_container_name": "REQUIRED",
|
||||
"repository_git_dir": "REQUIRED",
|
||||
"repair3_base_commit": "REQUIRED",
|
||||
"temporary_worktree_root": "REQUIRED",
|
||||
"controller_worktree": "REQUIRED",
|
||||
"main_ref": "REQUIRED",
|
||||
"production_baseline_ref": "REQUIRED"
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
[Unit]
|
||||
Description=AIOS managed Spark isolated runner
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
StartLimitIntervalSec=300
|
||||
StartLimitBurst=5
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=aios-spark-runner
|
||||
Group=aios-spark-runner
|
||||
WorkingDirectory=/var/lib/aios-spark-runner/work
|
||||
UMask=0007
|
||||
ExecStartPre=/usr/bin/python3 /opt/aios-spark-runner/runnerctl.py validate-config --config /etc/aios-spark-runner/runner-config.json
|
||||
ExecStartPre=/usr/bin/python3 /opt/aios-spark-runner/runnerctl.py verify-installed --config /etc/aios-spark-runner/runner-config.json
|
||||
ExecStart=/usr/bin/python3 /opt/aios-spark-runner/runnerctl.py supervise --config /etc/aios-spark-runner/runner-config.json
|
||||
Restart=on-failure
|
||||
RestartSec=10s
|
||||
TimeoutStartSec=30s
|
||||
TimeoutStopSec=35s
|
||||
KillMode=mixed
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=aios-spark-runner
|
||||
|
||||
NoNewPrivileges=yes
|
||||
PrivateTmp=yes
|
||||
PrivateDevices=yes
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
ProtectHostname=yes
|
||||
ProtectClock=yes
|
||||
ProtectKernelTunables=yes
|
||||
ProtectKernelModules=yes
|
||||
ProtectKernelLogs=yes
|
||||
ProtectControlGroups=yes
|
||||
ProtectProc=invisible
|
||||
ProcSubset=pid
|
||||
RestrictSUIDSGID=yes
|
||||
RestrictRealtime=yes
|
||||
RestrictNamespaces=yes
|
||||
LockPersonality=yes
|
||||
MemoryDenyWriteExecute=yes
|
||||
RemoveIPC=yes
|
||||
CapabilityBoundingSet=
|
||||
AmbientCapabilities=
|
||||
DevicePolicy=closed
|
||||
SystemCallArchitectures=native
|
||||
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
|
||||
|
||||
RuntimeDirectory=aios-spark-runner
|
||||
RuntimeDirectoryMode=0700
|
||||
StateDirectory=aios-spark-runner
|
||||
StateDirectoryMode=0750
|
||||
ReadOnlyPaths=/etc/aios-spark-runner /opt/aios-spark-runner
|
||||
ReadWritePaths=/data/aios-gitops/repo.git /data/aios-gitops/worktrees/spark-runner /var/lib/aios-spark-runner /run/aios-spark-runner
|
||||
|
||||
CPUAccounting=yes
|
||||
MemoryAccounting=yes
|
||||
TasksAccounting=yes
|
||||
CPUQuota=200%
|
||||
MemoryMax=2G
|
||||
TasksMax=256
|
||||
LimitNOFILE=4096
|
||||
LimitCORE=0
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,267 @@
|
||||
"""Contract tests for managed Spark runner provisioning.
|
||||
|
||||
Fixtures use pytest's /tmp-backed tmp_path only. Passing does not claim that a
|
||||
real artifact, secret reference, service, or heartbeat has been accepted.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
from types import ModuleType
|
||||
import pytest
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
RUNNERCTL = ROOT / "tools/aios_spark_runner/runnerctl.py"
|
||||
SCHEMA = ROOT / "deploy/spark-runner/runner-config.schema.json"
|
||||
TEMPLATE = ROOT / "deploy/spark-runner/runner-config.template.json"
|
||||
UNIT = ROOT / "deploy/systemd/aios-spark-runner.service"
|
||||
PROVISION = ROOT / "deploy/spark-runner/provision.sh"
|
||||
ROLLBACK = ROOT / "deploy/spark-runner/rollback.sh"
|
||||
ACCEPTANCE = ROOT / "deploy/spark-runner/acceptance.sh"
|
||||
README = ROOT / "deploy/spark-runner/README.md"
|
||||
FILES = (RUNNERCTL, SCHEMA, TEMPLATE, UNIT, PROVISION, ROLLBACK, ACCEPTANCE, README)
|
||||
REQUIRED = {
|
||||
"schema_version", "runner_artifact_uri_or_repo_path",
|
||||
"runner_artifact_version_or_digest", "runner_artifact_sha256_or_signature",
|
||||
"runner_secret_ref", "runner_run_user", "runner_run_group", "runner_workdir",
|
||||
"runner_heartbeat_path_or_endpoint", "service_name_or_container_name",
|
||||
"repository_git_dir", "repair3_base_commit", "temporary_worktree_root",
|
||||
"controller_worktree", "main_ref", "production_baseline_ref",
|
||||
}
|
||||
|
||||
def module() -> ModuleType:
|
||||
assert RUNNERCTL.is_file(), f"missing module: {RUNNERCTL}"
|
||||
spec = importlib.util.spec_from_file_location("spark_runnerctl", RUNNERCTL)
|
||||
assert spec and spec.loader
|
||||
loaded = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(loaded)
|
||||
return loaded
|
||||
|
||||
def config_file(tmp_path: Path, **changes: object) -> Path:
|
||||
artifact = tmp_path / "spark-runner-1.0.0"
|
||||
artifact.write_bytes(b"approved-fixture\n")
|
||||
artifact.chmod(0o644)
|
||||
digest = hashlib.sha256(artifact.read_bytes()).hexdigest()
|
||||
values: dict[str, object] = {
|
||||
"schema_version": "1",
|
||||
"runner_artifact_uri_or_repo_path": artifact.as_uri(),
|
||||
"runner_artifact_version_or_digest": "spark-runner-1.0.0",
|
||||
"runner_artifact_sha256_or_signature": f"sha256:{digest}",
|
||||
"runner_secret_ref": "secret://runtime/spark-runner/credential-v1",
|
||||
"runner_run_user": "aios-spark-runner",
|
||||
"runner_run_group": "aios-spark-runner",
|
||||
"runner_workdir": "/var/lib/aios-spark-runner/work",
|
||||
"runner_heartbeat_path_or_endpoint": "/run/aios-spark-runner/heartbeat.json",
|
||||
"service_name_or_container_name": "aios-spark-runner.service",
|
||||
"repository_git_dir": "/data/aios-gitops/repo.git",
|
||||
"repair3_base_commit": "8d9c6da90fc8caa69b691d243c6fefb40b6b0575",
|
||||
"temporary_worktree_root": "/data/aios-gitops/worktrees/spark-runner",
|
||||
"controller_worktree": "/data/aios-gitops/worktrees/controller",
|
||||
"main_ref": "refs/heads/main",
|
||||
"production_baseline_ref": "refs/tags/V1R4C6",
|
||||
}
|
||||
values.update(changes)
|
||||
path = tmp_path / "config.json"
|
||||
path.write_text(json.dumps(values), encoding="utf-8")
|
||||
return path
|
||||
|
||||
def git(*args: str, cwd: Path | None = None) -> str:
|
||||
return subprocess.run(
|
||||
["git", *args], cwd=cwd, check=True, capture_output=True, text=True
|
||||
).stdout.strip()
|
||||
|
||||
def test_deliverables_schema_and_non_runnable_template() -> None:
|
||||
for path in FILES:
|
||||
assert path.is_file(), f"missing deliverable: {path}"
|
||||
schema = json.loads(SCHEMA.read_text(encoding="utf-8"))
|
||||
template = json.loads(TEMPLATE.read_text(encoding="utf-8"))
|
||||
assert schema["type"] == "object" and schema["additionalProperties"] is False
|
||||
assert set(schema["required"]) == REQUIRED
|
||||
assert set(template) == REQUIRED
|
||||
assert all(value == "REQUIRED" for value in template.values())
|
||||
|
||||
def test_valid_fixture_and_digest(tmp_path: Path) -> None:
|
||||
runnerctl = module()
|
||||
config = runnerctl.load_config(config_file(tmp_path))
|
||||
result = runnerctl.verify_artifact(config)
|
||||
assert result["artifact_integrity"] == "PASS"
|
||||
|
||||
def test_provision_check_only_is_cwd_independent_and_template_fails_closed(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
config = config_file(tmp_path)
|
||||
result = subprocess.run(
|
||||
[str(PROVISION), "--config", str(config), "--check-only"],
|
||||
cwd=tmp_path,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
assert "provisioning_preflight=PASS" in result.stdout
|
||||
assert "RUNNER_RECOVERED=NO" in result.stdout
|
||||
rejected = subprocess.run(
|
||||
[str(PROVISION), "--config", str(TEMPLATE), "--check-only"],
|
||||
cwd=tmp_path,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
assert rejected.returncode != 0
|
||||
|
||||
@pytest.mark.parametrize("bad", ["", " ", "REQUIRED", "required", None])
|
||||
def test_required_inputs_reject_placeholders(tmp_path: Path, bad: object) -> None:
|
||||
runnerctl = module()
|
||||
for key in REQUIRED:
|
||||
with pytest.raises(runnerctl.ConfigError):
|
||||
runnerctl.load_config(config_file(tmp_path, **{key: bad}))
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("key", "bad"),
|
||||
[
|
||||
("runner_artifact_uri_or_repo_path", "https://example.invalid/runner"),
|
||||
("runner_artifact_uri_or_repo_path", "file:///tmp/latest/runner"),
|
||||
("runner_artifact_version_or_digest", "latest"),
|
||||
("runner_artifact_version_or_digest", "snapshot"),
|
||||
("runner_artifact_sha256_or_signature", "sha256:1234"),
|
||||
("runner_artifact_sha256_or_signature", "md5:" + "a" * 32),
|
||||
("runner_secret_ref", "plain-value"),
|
||||
("runner_secret_ref", "secret://runtime/../key"),
|
||||
("runner_secret_ref", "secret://runtime/key?version=1"),
|
||||
],
|
||||
)
|
||||
def test_mutable_artifact_digest_and_secret_ref_rejected(
|
||||
tmp_path: Path, key: str, bad: str
|
||||
) -> None:
|
||||
runnerctl = module()
|
||||
with pytest.raises(runnerctl.ConfigError):
|
||||
runnerctl.load_config(config_file(tmp_path, **{key: bad}))
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("key", "bad"),
|
||||
[
|
||||
("runner_run_user", "root"),
|
||||
("runner_run_group", "root"),
|
||||
("runner_workdir", "/tmp/runner"),
|
||||
("repository_git_dir", "/tmp/repo.git"),
|
||||
("service_name_or_container_name", "runner.service"),
|
||||
],
|
||||
)
|
||||
def test_fixed_policy_values_cannot_be_overridden(
|
||||
tmp_path: Path, key: str, bad: str
|
||||
) -> None:
|
||||
runnerctl = module()
|
||||
with pytest.raises(runnerctl.ConfigError):
|
||||
runnerctl.load_config(config_file(tmp_path, **{key: bad}))
|
||||
|
||||
def test_inline_secret_digest_mismatch_and_symlink_rejected(tmp_path: Path) -> None:
|
||||
runnerctl = module()
|
||||
path = config_file(tmp_path)
|
||||
values = json.loads(path.read_text(encoding="utf-8"))
|
||||
values["runner_secret"] = "not-accepted"
|
||||
path.write_text(json.dumps(values), encoding="utf-8")
|
||||
with pytest.raises(runnerctl.ConfigError):
|
||||
runnerctl.load_config(path)
|
||||
config = runnerctl.load_config(
|
||||
config_file(tmp_path, runner_artifact_sha256_or_signature="sha256:" + "0" * 64)
|
||||
)
|
||||
with pytest.raises(runnerctl.IntegrityError):
|
||||
runnerctl.verify_artifact(config)
|
||||
target = tmp_path / "target-1.0.0"
|
||||
target.write_bytes(b"fixture\n")
|
||||
link = tmp_path / "linked-1.0.0"
|
||||
link.symlink_to(target)
|
||||
digest = hashlib.sha256(target.read_bytes()).hexdigest()
|
||||
config = runnerctl.load_config(config_file(
|
||||
tmp_path, runner_artifact_uri_or_repo_path=link.as_uri(),
|
||||
runner_artifact_sha256_or_signature=f"sha256:{digest}",
|
||||
))
|
||||
with pytest.raises(runnerctl.IntegrityError):
|
||||
runnerctl.verify_artifact(config)
|
||||
writable = tmp_path / "group-writable-1.0.0"
|
||||
writable.write_bytes(b"fixture\n")
|
||||
writable.chmod(0o664)
|
||||
digest = hashlib.sha256(writable.read_bytes()).hexdigest()
|
||||
config = runnerctl.load_config(config_file(
|
||||
tmp_path, runner_artifact_uri_or_repo_path=writable.as_uri(),
|
||||
runner_artifact_sha256_or_signature=f"sha256:{digest}",
|
||||
))
|
||||
with pytest.raises(runnerctl.IntegrityError):
|
||||
runnerctl.verify_artifact(config)
|
||||
|
||||
def test_heartbeat_freshness_and_atomic_cleanup(tmp_path: Path) -> None:
|
||||
runnerctl = module()
|
||||
path = tmp_path / "run" / "heartbeat.json"
|
||||
now = time.time()
|
||||
runnerctl.write_heartbeat(path, state="active", pid=1234, now=now)
|
||||
assert runnerctl.check_heartbeat(
|
||||
path, max_age_seconds=300, now=now + 2
|
||||
)["heartbeat_age_under_5_minutes"] == "PASS"
|
||||
assert not list(path.parent.glob(".heartbeat.json.*"))
|
||||
with pytest.raises(runnerctl.HeartbeatError):
|
||||
runnerctl.check_heartbeat(path, max_age_seconds=300, now=now + 301)
|
||||
|
||||
def test_temporary_worktree_probe_and_cleanup(tmp_path: Path) -> None:
|
||||
runnerctl = module()
|
||||
source = tmp_path / "source"
|
||||
source.mkdir()
|
||||
git("init", cwd=source)
|
||||
git("config", "user.name", "Fixture", cwd=source)
|
||||
git("config", "user.email", "fixture@example.invalid", cwd=source)
|
||||
(source / "README").write_text("probe\n", encoding="utf-8")
|
||||
git("add", "README", cwd=source)
|
||||
git("commit", "-m", "fixture", cwd=source)
|
||||
commit = git("rev-parse", "HEAD", cwd=source)
|
||||
bare = tmp_path / "repo.git"
|
||||
git("clone", "--bare", str(source), str(bare))
|
||||
root = tmp_path / "isolated-worktrees"
|
||||
before = git(f"--git-dir={bare}", "worktree", "list", "--porcelain")
|
||||
result = runnerctl.temporary_worktree_probe(bare, commit, root)
|
||||
assert set(result.values()) == {"PASS"}
|
||||
assert git(f"--git-dir={bare}", "worktree", "list", "--porcelain") == before
|
||||
assert not root.exists() or not any(root.iterdir())
|
||||
assert git(f"--git-dir={bare}", "for-each-ref", "refs/heads/aios-spark-runner-probe/") == ""
|
||||
with pytest.raises(runnerctl.ProbeError):
|
||||
runnerctl.temporary_worktree_probe(bare, "0" * 40, root)
|
||||
assert git(f"--git-dir={bare}", "worktree", "list", "--porcelain") == before
|
||||
assert not root.exists() or not any(root.iterdir())
|
||||
|
||||
def test_systemd_and_script_static_security() -> None:
|
||||
unit = UNIT.read_text(encoding="utf-8")
|
||||
for item in (
|
||||
"User=aios-spark-runner", "Group=aios-spark-runner",
|
||||
"WorkingDirectory=/var/lib/aios-spark-runner/work", "UMask=0007",
|
||||
"Restart=on-failure", "StandardOutput=journal", "StandardError=journal",
|
||||
"NoNewPrivileges=yes", "PrivateTmp=yes", "PrivateDevices=yes",
|
||||
"ProtectSystem=strict", "ProtectHome=yes", "ProtectKernelTunables=yes",
|
||||
"ProtectKernelModules=yes", "ProtectControlGroups=yes",
|
||||
"CapabilityBoundingSet=", "AmbientCapabilities=", "RestrictSUIDSGID=yes",
|
||||
"LockPersonality=yes", "MemoryDenyWriteExecute=yes", "ReadWritePaths=",
|
||||
"RuntimeDirectory=aios-spark-runner", "StateDirectory=aios-spark-runner",
|
||||
"MemoryMax=", "CPUQuota=", "TasksMax=",
|
||||
):
|
||||
assert item in unit
|
||||
assert "EnvironmentFile=" not in unit
|
||||
texts = [path.read_text(encoding="utf-8") for path in (PROVISION, ROLLBACK, ACCEPTANCE)]
|
||||
assert sum(text.count("set -euo pipefail") for text in texts) == 3
|
||||
combined = "\n".join(texts)
|
||||
for bad in (
|
||||
"curl | sh", "wget | sh", "nohup", "tmux", "git reset", "git rebase",
|
||||
"--force", "push origin main", "push gitea main",
|
||||
):
|
||||
assert bad not in combined
|
||||
assert "--check-only" in texts[0] and "systemctl" in texts[0]
|
||||
assert "worktree" in texts[2]
|
||||
assert "write_heartbeat" not in texts[2]
|
||||
runner_source = RUNNERCTL.read_text(encoding="utf-8")
|
||||
assert "shell=True" not in runner_source
|
||||
assert "subprocess.Popen([str(artifact)]" in runner_source
|
||||
|
||||
def test_readme_preserves_blocked_state() -> None:
|
||||
text = README.read_text(encoding="utf-8")
|
||||
for marker in (
|
||||
"RUNNER_RECOVERED=NO", "BLOCKED_OWNER_ARTIFACT_INPUT", "BASELINE=V1R4C6",
|
||||
"RELEASE=paused", "PRODUCTION_RELEASE=NOT_STARTED", "REPAIR3_STARTED=false",
|
||||
):
|
||||
assert marker in text
|
||||
@@ -0,0 +1,25 @@
|
||||
"""Managed AIOS Spark runner provisioning helpers."""
|
||||
|
||||
from .runnerctl import (
|
||||
ConfigError,
|
||||
HeartbeatError,
|
||||
IntegrityError,
|
||||
ProbeError,
|
||||
check_heartbeat,
|
||||
load_config,
|
||||
temporary_worktree_probe,
|
||||
verify_artifact,
|
||||
write_heartbeat,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"ConfigError",
|
||||
"HeartbeatError",
|
||||
"IntegrityError",
|
||||
"ProbeError",
|
||||
"check_heartbeat",
|
||||
"load_config",
|
||||
"temporary_worktree_probe",
|
||||
"verify_artifact",
|
||||
"write_heartbeat",
|
||||
]
|
||||
Executable
+469
@@ -0,0 +1,469 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail-closed control plane for the managed AIOS Spark isolated runner."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import signal
|
||||
import stat
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import NoReturn
|
||||
from urllib.parse import unquote, urlparse
|
||||
|
||||
EXPECTED = {
|
||||
"schema_version": "1",
|
||||
"runner_run_user": "aios-spark-runner",
|
||||
"runner_run_group": "aios-spark-runner",
|
||||
"runner_workdir": "/var/lib/aios-spark-runner/work",
|
||||
"runner_heartbeat_path_or_endpoint": "/run/aios-spark-runner/heartbeat.json",
|
||||
"service_name_or_container_name": "aios-spark-runner.service",
|
||||
"repository_git_dir": "/data/aios-gitops/repo.git",
|
||||
"repair3_base_commit": "8d9c6da90fc8caa69b691d243c6fefb40b6b0575",
|
||||
"temporary_worktree_root": "/data/aios-gitops/worktrees/spark-runner",
|
||||
"controller_worktree": "/data/aios-gitops/worktrees/controller",
|
||||
"main_ref": "refs/heads/main",
|
||||
"production_baseline_ref": "refs/tags/V1R4C6",
|
||||
}
|
||||
REQUIRED_CONFIG_KEYS = {
|
||||
"schema_version",
|
||||
"runner_artifact_uri_or_repo_path",
|
||||
"runner_artifact_version_or_digest",
|
||||
"runner_artifact_sha256_or_signature",
|
||||
"runner_secret_ref",
|
||||
"runner_run_user",
|
||||
"runner_run_group",
|
||||
"runner_workdir",
|
||||
"runner_heartbeat_path_or_endpoint",
|
||||
"service_name_or_container_name",
|
||||
"repository_git_dir",
|
||||
"repair3_base_commit",
|
||||
"temporary_worktree_root",
|
||||
"controller_worktree",
|
||||
"main_ref",
|
||||
"production_baseline_ref",
|
||||
}
|
||||
MUTABLE_TOKENS = {
|
||||
"current", "head", "latest", "main", "master", "nightly", "snapshot", "stable"
|
||||
}
|
||||
IDENTIFIER = re.compile(r"^[A-Za-z0-9][A-Za-z0-9._-]*$")
|
||||
FULL_OID = re.compile(r"^[0-9a-f]{40}$")
|
||||
SHA256_REF = re.compile(r"^sha256:([0-9a-f]{64})$")
|
||||
|
||||
|
||||
class ConfigError(ValueError):
|
||||
"""Configuration is absent, ambiguous, or outside the approved policy."""
|
||||
|
||||
|
||||
class IntegrityError(RuntimeError):
|
||||
"""An artifact failed local-file or SHA-256 verification."""
|
||||
|
||||
|
||||
class HeartbeatError(RuntimeError):
|
||||
"""The managed heartbeat is missing, malformed, future-dated, or stale."""
|
||||
|
||||
|
||||
class ProbeError(RuntimeError):
|
||||
"""The isolated Git probe or its cleanup failed."""
|
||||
|
||||
|
||||
def _run(*argv: str, cwd: Path | None = None) -> subprocess.CompletedProcess[str]:
|
||||
env = os.environ.copy()
|
||||
env["LC_ALL"] = "C"
|
||||
return subprocess.run(
|
||||
argv,
|
||||
cwd=str(cwd) if cwd else None,
|
||||
env=env,
|
||||
check=True,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
)
|
||||
|
||||
|
||||
def _string(payload: dict[str, object], key: str) -> str:
|
||||
value = payload.get(key)
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise ConfigError(f"{key} must be an explicit non-empty string")
|
||||
if value != value.strip() or value.casefold() == "required":
|
||||
raise ConfigError(f"{key} contains an empty or placeholder value")
|
||||
return value
|
||||
|
||||
|
||||
def _absolute(value: str, key: str) -> Path:
|
||||
path = Path(value)
|
||||
if not path.is_absolute() or ".." in path.parts:
|
||||
raise ConfigError(f"{key} must be a normalized absolute path")
|
||||
return path
|
||||
|
||||
|
||||
def _artifact_path(uri: str) -> Path:
|
||||
parsed = urlparse(uri)
|
||||
if parsed.scheme != "file" or parsed.netloc:
|
||||
raise ConfigError("runner_artifact_uri_or_repo_path must be a local file URI")
|
||||
if parsed.query or parsed.fragment:
|
||||
raise ConfigError("artifact URI must not contain query or fragment")
|
||||
path = _absolute(unquote(parsed.path), "runner_artifact_uri_or_repo_path")
|
||||
if any(part.casefold() in MUTABLE_TOKENS for part in path.parts):
|
||||
raise ConfigError("artifact URI contains a mutable path token")
|
||||
return path
|
||||
|
||||
|
||||
def _secret_ref(value: str) -> None:
|
||||
parsed = urlparse(value)
|
||||
if parsed.scheme != "secret" or parsed.query or parsed.fragment:
|
||||
raise ConfigError("runner_secret_ref must be a value-free secret:// reference")
|
||||
segments = []
|
||||
if parsed.netloc:
|
||||
segments.append(parsed.netloc)
|
||||
segments.extend(part for part in parsed.path.split("/") if part)
|
||||
if len(segments) < 2 or any(
|
||||
part in {".", ".."} or not IDENTIFIER.fullmatch(part) for part in segments
|
||||
):
|
||||
raise ConfigError("runner_secret_ref has an invalid namespace or path")
|
||||
|
||||
|
||||
def load_config(path: str | Path) -> dict[str, str]:
|
||||
config_path = Path(path)
|
||||
if config_path.is_symlink() or not config_path.is_file():
|
||||
raise ConfigError("config must be a regular non-symlink file")
|
||||
try:
|
||||
payload = json.loads(config_path.read_text(encoding="utf-8"))
|
||||
except (OSError, UnicodeError, json.JSONDecodeError) as exc:
|
||||
raise ConfigError("config is not valid UTF-8 JSON") from exc
|
||||
if not isinstance(payload, dict):
|
||||
raise ConfigError("config root must be an object")
|
||||
keys = set(payload)
|
||||
if keys != REQUIRED_CONFIG_KEYS:
|
||||
missing = sorted(REQUIRED_CONFIG_KEYS - keys)
|
||||
extra = sorted(keys - REQUIRED_CONFIG_KEYS)
|
||||
raise ConfigError(f"config keys differ: missing={missing} extra={extra}")
|
||||
config = {key: _string(payload, key) for key in REQUIRED_CONFIG_KEYS}
|
||||
for key, expected in EXPECTED.items():
|
||||
if config[key] != expected:
|
||||
raise ConfigError(f"{key} must equal the repository-approved value")
|
||||
_artifact_path(config["runner_artifact_uri_or_repo_path"])
|
||||
version = config["runner_artifact_version_or_digest"]
|
||||
if (
|
||||
not IDENTIFIER.fullmatch(version)
|
||||
or version.casefold() in MUTABLE_TOKENS
|
||||
or any(token in MUTABLE_TOKENS for token in re.split(r"[._-]+", version.casefold()))
|
||||
):
|
||||
raise ConfigError("runner_artifact_version_or_digest is mutable or malformed")
|
||||
if not SHA256_REF.fullmatch(config["runner_artifact_sha256_or_signature"]):
|
||||
raise ConfigError("runner_artifact_sha256_or_signature must be sha256 plus 64 hex")
|
||||
_secret_ref(config["runner_secret_ref"])
|
||||
return config
|
||||
|
||||
|
||||
def _reject_symlink_chain(path: Path) -> None:
|
||||
current = Path(path.anchor)
|
||||
for part in path.parts[1:]:
|
||||
current = current / part
|
||||
if current.exists() or current.is_symlink():
|
||||
if current.is_symlink():
|
||||
raise IntegrityError(f"symlink path component rejected: {current}")
|
||||
|
||||
|
||||
def _sha256_regular_file(path: Path) -> str:
|
||||
_reject_symlink_chain(path)
|
||||
flags = os.O_RDONLY
|
||||
if hasattr(os, "O_NOFOLLOW"):
|
||||
flags |= os.O_NOFOLLOW
|
||||
try:
|
||||
descriptor = os.open(path, flags)
|
||||
except OSError as exc:
|
||||
raise IntegrityError("artifact cannot be opened safely") from exc
|
||||
digest = hashlib.sha256()
|
||||
try:
|
||||
info = os.fstat(descriptor)
|
||||
if not stat.S_ISREG(info.st_mode):
|
||||
raise IntegrityError("artifact is not a regular file")
|
||||
if info.st_mode & 0o022:
|
||||
raise IntegrityError("artifact must not be group/world writable")
|
||||
with os.fdopen(descriptor, "rb", closefd=False) as stream:
|
||||
for chunk in iter(lambda: stream.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def verify_artifact(config: dict[str, str]) -> dict[str, str]:
|
||||
path = _artifact_path(config["runner_artifact_uri_or_repo_path"])
|
||||
actual = _sha256_regular_file(path)
|
||||
match = SHA256_REF.fullmatch(config["runner_artifact_sha256_or_signature"])
|
||||
if match is None or not hmac.compare_digest(actual, match.group(1)):
|
||||
raise IntegrityError("artifact SHA-256 mismatch")
|
||||
return {"artifact_integrity": "PASS", "sha256": actual}
|
||||
|
||||
|
||||
def installed_artifact_path(config: dict[str, str]) -> Path:
|
||||
digest = config["runner_artifact_sha256_or_signature"][7:]
|
||||
return Path("/opt/aios-spark-runner/artifacts") / digest / "runner"
|
||||
|
||||
|
||||
def verify_installed_artifact(config: dict[str, str]) -> dict[str, str]:
|
||||
actual = _sha256_regular_file(installed_artifact_path(config))
|
||||
expected = config["runner_artifact_sha256_or_signature"][7:]
|
||||
if not hmac.compare_digest(actual, expected):
|
||||
raise IntegrityError("installed artifact SHA-256 mismatch")
|
||||
return {"artifact_integrity": "PASS", "sha256": actual}
|
||||
|
||||
|
||||
def write_heartbeat(
|
||||
path: str | Path,
|
||||
*,
|
||||
state: str,
|
||||
pid: int,
|
||||
now: float | None = None,
|
||||
) -> None:
|
||||
if state not in {"active", "stopping"} or not isinstance(pid, int) or pid <= 0:
|
||||
raise HeartbeatError("invalid heartbeat state or pid")
|
||||
heartbeat = Path(path)
|
||||
heartbeat.parent.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
if heartbeat.parent.is_symlink() or heartbeat.is_symlink():
|
||||
raise HeartbeatError("heartbeat path must not be a symlink")
|
||||
timestamp = time.time() if now is None else now
|
||||
payload = json.dumps(
|
||||
{"pid": pid, "state": state, "timestamp": timestamp},
|
||||
sort_keys=True,
|
||||
separators=(",", ":"),
|
||||
).encode("utf-8")
|
||||
temporary = heartbeat.parent / f".{heartbeat.name}.{uuid.uuid4().hex}"
|
||||
descriptor = os.open(temporary, os.O_CREAT | os.O_EXCL | os.O_WRONLY, 0o600)
|
||||
try:
|
||||
with os.fdopen(descriptor, "wb", closefd=False) as stream:
|
||||
stream.write(payload)
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
finally:
|
||||
os.close(descriptor)
|
||||
try:
|
||||
os.replace(temporary, heartbeat)
|
||||
directory = os.open(heartbeat.parent, os.O_RDONLY)
|
||||
try:
|
||||
os.fsync(directory)
|
||||
finally:
|
||||
os.close(directory)
|
||||
finally:
|
||||
if temporary.exists():
|
||||
temporary.unlink()
|
||||
|
||||
|
||||
def check_heartbeat(
|
||||
path: str | Path,
|
||||
*,
|
||||
max_age_seconds: int,
|
||||
now: float | None = None,
|
||||
) -> dict[str, str]:
|
||||
heartbeat = Path(path)
|
||||
if heartbeat.is_symlink() or not heartbeat.is_file():
|
||||
raise HeartbeatError("heartbeat is missing or unsafe")
|
||||
try:
|
||||
payload = json.loads(heartbeat.read_text(encoding="utf-8"))
|
||||
timestamp = payload["timestamp"]
|
||||
pid = payload["pid"]
|
||||
state = payload["state"]
|
||||
except (OSError, UnicodeError, json.JSONDecodeError, KeyError, TypeError) as exc:
|
||||
raise HeartbeatError("heartbeat is malformed") from exc
|
||||
if (
|
||||
not isinstance(timestamp, (int, float))
|
||||
or isinstance(timestamp, bool)
|
||||
or not isinstance(pid, int)
|
||||
or pid <= 0
|
||||
or state != "active"
|
||||
):
|
||||
raise HeartbeatError("heartbeat fields are invalid")
|
||||
age = (time.time() if now is None else now) - float(timestamp)
|
||||
if max_age_seconds <= 0 or age < 0 or age > max_age_seconds:
|
||||
raise HeartbeatError("heartbeat is not fresh")
|
||||
return {"heartbeat_age_under_5_minutes": "PASS"}
|
||||
|
||||
|
||||
def temporary_worktree_probe(
|
||||
git_dir: str | Path,
|
||||
commit: str,
|
||||
temporary_worktree_root: str | Path,
|
||||
) -> dict[str, str]:
|
||||
repository = Path(git_dir)
|
||||
root = Path(temporary_worktree_root)
|
||||
if not FULL_OID.fullmatch(commit):
|
||||
raise ProbeError("probe commit must be a full object ID")
|
||||
if repository.is_symlink() or not repository.is_dir():
|
||||
raise ProbeError("bare repository is unavailable or unsafe")
|
||||
if root.is_symlink():
|
||||
raise ProbeError("temporary worktree root is a symlink")
|
||||
root.mkdir(parents=True, exist_ok=True, mode=0o700)
|
||||
token = uuid.uuid4().hex
|
||||
branch = f"aios-spark-runner-probe/{token}"
|
||||
branch_ref = f"refs/heads/{branch}"
|
||||
worktree = root / token
|
||||
options = (f"--git-dir={repository}",)
|
||||
created = False
|
||||
primary: BaseException | None = None
|
||||
cleanup_errors: list[str] = []
|
||||
result = {
|
||||
"repo_read": "PASS",
|
||||
"temporary_worktree_create": "FAIL",
|
||||
"temporary_worktree_read_only_probe": "FAIL",
|
||||
"temporary_worktree_remove": "FAIL",
|
||||
"temporary_branch_remove": "FAIL",
|
||||
}
|
||||
try:
|
||||
_run("git", *options, "cat-file", "-e", f"{commit}^{{commit}}")
|
||||
_run("git", *options, "worktree", "add", "-b", branch, str(worktree), commit)
|
||||
created = True
|
||||
result["temporary_worktree_create"] = "PASS"
|
||||
if _run("git", "-C", str(worktree), "rev-parse", "HEAD").stdout.strip() != commit:
|
||||
raise ProbeError("temporary worktree OID mismatch")
|
||||
if _run("git", "-C", str(worktree), "status", "--porcelain").stdout:
|
||||
raise ProbeError("temporary worktree is not clean")
|
||||
_run("git", "-C", str(worktree), "ls-tree", "-r", "--name-only", "HEAD")
|
||||
result["temporary_worktree_read_only_probe"] = "PASS"
|
||||
except BaseException as exc:
|
||||
primary = exc
|
||||
finally:
|
||||
if created:
|
||||
try:
|
||||
_run("git", *options, "worktree", "remove", "--force", str(worktree))
|
||||
result["temporary_worktree_remove"] = "PASS"
|
||||
except BaseException as exc:
|
||||
cleanup_errors.append(f"worktree removal failed: {type(exc).__name__}")
|
||||
elif worktree.exists():
|
||||
try:
|
||||
shutil.rmtree(worktree)
|
||||
except OSError as exc:
|
||||
cleanup_errors.append(f"partial worktree cleanup failed: {type(exc).__name__}")
|
||||
try:
|
||||
_run("git", *options, "update-ref", "-d", branch_ref, commit)
|
||||
if _run("git", *options, "show-ref", "--verify", "--quiet", branch_ref).returncode == 0:
|
||||
cleanup_errors.append("temporary branch remains")
|
||||
else:
|
||||
result["temporary_branch_remove"] = "PASS"
|
||||
except subprocess.CalledProcessError as exc:
|
||||
if exc.returncode == 1:
|
||||
result["temporary_branch_remove"] = "PASS"
|
||||
else:
|
||||
cleanup_errors.append("temporary branch cleanup failed")
|
||||
try:
|
||||
root.rmdir()
|
||||
except OSError:
|
||||
pass
|
||||
if primary is not None or cleanup_errors:
|
||||
detail = cleanup_errors[0] if cleanup_errors else type(primary).__name__
|
||||
raise ProbeError(f"isolated probe failed: {detail}") from primary
|
||||
return result
|
||||
|
||||
|
||||
def _stop_child(child: subprocess.Popen[bytes]) -> None:
|
||||
if child.poll() is not None:
|
||||
return
|
||||
child.terminate()
|
||||
try:
|
||||
child.wait(timeout=20)
|
||||
except subprocess.TimeoutExpired:
|
||||
child.kill()
|
||||
child.wait(timeout=10)
|
||||
|
||||
|
||||
def supervise(config_path: Path) -> int:
|
||||
config = load_config(config_path)
|
||||
verify_installed_artifact(config)
|
||||
workdir = Path(config["runner_workdir"])
|
||||
artifact = installed_artifact_path(config)
|
||||
environment = os.environ.copy()
|
||||
environment.update(
|
||||
{
|
||||
"AIOS_RUNNER_SECRET_REF": config["runner_secret_ref"],
|
||||
"AIOS_RUNNER_GIT_DIR": config["repository_git_dir"],
|
||||
"AIOS_RUNNER_WORKTREE_ROOT": config["temporary_worktree_root"],
|
||||
"AIOS_RUNNER_BASE_COMMIT": config["repair3_base_commit"],
|
||||
}
|
||||
)
|
||||
child = subprocess.Popen([str(artifact)], cwd=workdir, env=environment, shell=False)
|
||||
stopping = False
|
||||
|
||||
def request_stop(_signum: int, _frame: object) -> None:
|
||||
nonlocal stopping
|
||||
stopping = True
|
||||
|
||||
signal.signal(signal.SIGTERM, request_stop)
|
||||
signal.signal(signal.SIGINT, request_stop)
|
||||
heartbeat = Path(config["runner_heartbeat_path_or_endpoint"])
|
||||
last_write = 0.0
|
||||
try:
|
||||
while child.poll() is None and not stopping:
|
||||
current = time.time()
|
||||
if current - last_write >= 30:
|
||||
write_heartbeat(heartbeat, state="active", pid=child.pid, now=current)
|
||||
last_write = current
|
||||
time.sleep(1)
|
||||
if stopping:
|
||||
write_heartbeat(heartbeat, state="stopping", pid=child.pid)
|
||||
_stop_child(child)
|
||||
return int(child.returncode or 0)
|
||||
finally:
|
||||
_stop_child(child)
|
||||
|
||||
|
||||
def _parser() -> argparse.ArgumentParser:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
commands = parser.add_subparsers(dest="command", required=True)
|
||||
for name in ("validate-config", "verify-artifact", "verify-installed", "supervise"):
|
||||
command = commands.add_parser(name)
|
||||
command.add_argument("--config", type=Path, required=True)
|
||||
heartbeat = commands.add_parser("check-heartbeat")
|
||||
heartbeat.add_argument("--path", type=Path, required=True)
|
||||
heartbeat.add_argument("--max-age", type=int, default=300)
|
||||
probe = commands.add_parser("probe")
|
||||
probe.add_argument("--config", type=Path, required=True)
|
||||
return parser
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = _parser().parse_args(argv)
|
||||
config = None
|
||||
if hasattr(args, "config"):
|
||||
config = load_config(args.config)
|
||||
if args.command == "validate-config":
|
||||
print("config_validation=PASS")
|
||||
elif args.command == "verify-artifact":
|
||||
print(json.dumps(verify_artifact(config), sort_keys=True))
|
||||
elif args.command == "verify-installed":
|
||||
print(json.dumps(verify_installed_artifact(config), sort_keys=True))
|
||||
elif args.command == "check-heartbeat":
|
||||
print(json.dumps(check_heartbeat(args.path, max_age_seconds=args.max_age), sort_keys=True))
|
||||
elif args.command == "probe":
|
||||
print(
|
||||
json.dumps(
|
||||
temporary_worktree_probe(
|
||||
config["repository_git_dir"],
|
||||
config["repair3_base_commit"],
|
||||
config["temporary_worktree_root"],
|
||||
),
|
||||
sort_keys=True,
|
||||
)
|
||||
)
|
||||
elif args.command == "supervise":
|
||||
return supervise(args.config)
|
||||
return 0
|
||||
|
||||
|
||||
def _fatal(exc: BaseException) -> NoReturn:
|
||||
print(f"RUNNER_RECOVERED=NO error={type(exc).__name__}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
raise SystemExit(main())
|
||||
except (ConfigError, HeartbeatError, IntegrityError, ProbeError, OSError, subprocess.SubprocessError) as error:
|
||||
_fatal(error)
|
||||
Reference in New Issue
Block a user