From f2ec9ec4343f2d79fef7d44b55dbf25548097903 Mon Sep 17 00:00:00 2001 From: Your Name Date: Thu, 18 Jun 2026 16:15:41 +0800 Subject: [PATCH] docs(aiops): record host runaway production readback [skip ci] --- docs/LOGBOOK.md | 27 +++++++++++++++++++ docs/runbooks/FULL-STACK-COLD-START-SOP.md | 13 +++++++++ ...oot-cold-start-backup-recovery-workplan.md | 9 +++++++ 3 files changed, 49 insertions(+) diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index 4ff55e3c2..462a4fbc5 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -1,3 +1,30 @@ +## 2026-06-18｜P3-009 Host Runaway AIOps 閉環正式部署驗證 + +**背景**：110 CPU 滿載事件已從人工 `top/ps` 處理收斂成 read-only exporter、Prometheus alert lane、Telegram / AI event packet、PlayBook / KM contract 與 gated remediation helper；最後缺口是讓 governance automation inventory 能用單一讀模型回看「監控 -> 告警 -> 事件卡 -> PlayBook -> KM / Verifier -> gated remediation」是否真的串起來，而不是散在文件和 log。 + +**完成內容**： +- 新增並部署 `host_runaway_aiops_loop_readiness_v1` production read model：`GET /api/v1/agents/agent-host-runaway-aiops-loop-readiness`。 +- Governance automation inventory 已顯示 `P3-009 Host Runaway AIOps 閉環`，可讀到 `6` 個 loop stage、`2` 條 alert lane、`5` 份 asset writeback contract、host `110` live readback、orphan browser groups `0`、active CI containers `2`、runtime writes `0`。 +- `P2-407 AI 報表 no-write 分析` 與 P3-009 在同一正式 deploy marker 上共存；最新主線包含 `0e72a6f4 feat(aiops): expose host runaway loop readiness`。 +- CD runner lock 修補由平行工作流收斂後，最終有效 deploy marker 為 `42c08ece chore(cd): deploy 27143fb [skip ci]`；舊的 `3168 / 3171 / 3174` 不作成功依據。 + +**Production readback / smoke**： +- Gitea `code-review.yaml #3178`：成功。 +- Gitea `cd.yaml #3177`：成功；有效部署 commit `27143fb0 fix(cd): 補齊 runner lock 解析工具`，其 ancestry 包含 Host Runaway AIOps commit `0e72a6f4`。 +- API health：`https://awoooi.wooo.work/api/v1/health` 回 `healthy / prod / mock_mode=false`；PostgreSQL、Redis、OpenClaw、SignOz、Ollama GCP-A / GCP-B / local route 均為 `up`。 +- P3-009 API：`schema_version=host_runaway_aiops_loop_readiness_v1`、`current_task_id=P3-009`、`next_task_id=P3-010`、`overall_completion_percent=100`、`loop_stage_count=6`、`alert_lane_count=2`、`asset_writeback_contract_count=5`、`runtime_remediation_authorized_count=0`、`host_write_count=0`、`process_termination_count=0`、`docker_restart_count=0`、`production_write_count=0`。 +- Desktop `1440x1100`：`/zh-TW/governance?tab=automation-inventory&_v=42c08ece-host-runaway-aiops-desktop-1440x1100` 必要文案缺漏 `0`、console error `0`、page error `0`、horizontalOverflow `false`、overflowing elements `0`；截圖 `/tmp/awoooi-host-runaway-aiops-desktop-1440x1100-42c08ece.png`。 +- Mobile `390x844`：`/zh-TW/governance?tab=automation-inventory&_v=42c08ece-host-runaway-aiops-mobile-390x844` 必要文案缺漏 `0`、console error `0`、page error `0`、horizontalOverflow `false`、overflowing elements `0`；截圖 `/tmp/awoooi-host-runaway-aiops-mobile-390x844-42c08ece.png`。 +- Smoke JSON：`/tmp/awoooi-host-runaway-aiops-smoke-42c08ece.json`。可見文字未出現原始 repo 名稱；HTML / client payload 仍有既有 governance metadata term 與舊 blocker key，另列 public bundle hygiene，不作本輪 Host Runaway card 可見外洩或 runtime gate。 + +**完成度同步**： +- Host Runaway AIOps product-visible loop readback：`100%`。 +- 110 CPU orphan Chrome recurrence guard：`READY`，正式站可回讀。 +- CD runner lock 修補到可部署本輪變更：`100%` for this deploy evidence。 +- Runtime auto-remediation：仍 `0 / false`。`SIGTERM` 仍必須有 owner approval、maintenance window、evidence ref、dry-run、post-check、KM / PlayBook / Verifier writeback；不得把 API 200、UI 可見、CD success 或 route smoke 當作主機寫入授權。 + +**邊界**：本段沒有 SSH、沒有 kill process、沒有 Docker / systemd / Nginx / firewall / K8s runtime write、沒有 Telegram live send、沒有 Gateway queue write、沒有 Bot API call、沒有 secret 讀取、沒有 force push。此完成代表監控 / 告警 / PlayBook / KM / Verifier / gated remediation 的產品可視讀模型已正式部署，不代表自動修復可直接執行。 + ## 2026-06-18｜監控告警訊息專業化：主機 / runner raw dump 必須轉 AI 自動化事件卡 **背景**：統帥指出 Telegram / SRE 戰情室收到的主機告警仍把 CPU、容器 root Node.js 行程、完整工具路徑、套件 JSON、外部檢查 endpoint 與 raw process dump 直接貼出。這種訊息不可讀、不可處置，也會擴大資訊外洩面；AWOOOI / IwoooS 作為 AI 自動化產品，監控告警必須先轉成可判讀、可分流、可驗證、可回寫的事件卡，而不是把底層 log 丟給值班者。 diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index 47e24cf1f..36fa42089 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -56,6 +56,19 @@ Allowed declaration: monitoring, alert rules, live scrape, AI event packet routi Forbidden declaration: Telegram send, Bot API call, Gateway queue write, process kill, Docker/systemd restart, Nginx reload, firewall/K8s action, or runtime remediation is authorized. ``` +2026-06-18 16:08 P3-009 Host Runaway AIOps product readback: + +```text +Host runaway AIOps closed-loop read model is deployed in production. +Deploy marker: 42c08ece chore(cd): deploy 27143fb [skip ci]. +API endpoint: /api/v1/agents/agent-host-runaway-aiops-loop-readiness. +Production readback: schema_version=host_runaway_aiops_loop_readiness_v1, current_task_id=P3-009, next_task_id=P3-010, completion=100, loop_stage_count=6, alert_lane_count=2, asset_writeback_contract_count=5. +Host 110 live readback in the model: orphan browser groups=0, active CI containers=2, remediation_authorized=0, runtime/write counters=0. +Governance route: /zh-TW/governance?tab=automation-inventory shows P3-009 on desktop 1440x1100 and mobile 390x844 with missing text=0, console/page errors=0, horizontal overflow=false. +Allowed declaration: monitoring, alert rules, AI event packet, PlayBook / KM contract, Verifier/writeback contract, gated remediation dry-run boundary, and product-visible readback are complete for this evidence set. +Forbidden declaration: AI runtime remediation is enabled. Process termination, Docker/systemd restart, Nginx reload, firewall/K8s action, Telegram live send, Gateway queue write, Bot API call, production write, and secret read remain forbidden without owner approval, maintenance window, evidence ref, dry-run, and post-check. +``` + | 項目 | 2026-06-18 13:43 Asia/Taipei live result | 判定 | |------|-------------------------------------------|------| | Overall recovery readiness | `99%` | `SERVICE_GREEN_DR_ESCROW_BLOCKED` | diff --git a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md index e69eebaa0..c1a35fab6 100644 --- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md +++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md @@ -254,6 +254,15 @@ Blocked: No for product readback; yes for runtime remediation by design. Next: If a real or fixture alert fires, verify Telegram card, AwoooP Work Item, KM / PlayBook / Verifier fields agree before considering any owner-approved non-production gated SIGTERM drill. Completion: host runaway AIOps product-visible loop readback 100%; runtime auto-remediation remains safely gated at 0. +2026-06-18 16:08 Asia/Taipei +Phase: P3-009 Host runaway AIOps loop production verification +Before: P3-009 source, API, UI and tests were pushed, but production still needed deploy marker, API readback, desktop/mobile browser smoke, and CD runner lock recovery evidence. +After: Final deploy marker `42c08ece chore(cd): deploy 27143fb [skip ci]` is live after CD runner lock fixes `fc6c01ee` / `84ca8423` / `27143fb0`; `cd.yaml #3177` and `code-review.yaml #3178` are successful. Production endpoint `/api/v1/agents/agent-host-runaway-aiops-loop-readiness` returns `schema_version=host_runaway_aiops_loop_readiness_v1`, `current_task_id=P3-009`, `next_task_id=P3-010`, completion `100`, loop stages `6`, alert lanes `2`, writeback contracts `5`, host `110`, orphan browser groups `0`, active CI containers `2`, and every runtime/write/remediation counter `0`. +Evidence: API health `healthy / prod / mock_mode=false`; desktop `1440x1100` and mobile `390x844` governance smoke with deploy marker `42c08ece` have required text missing `0`, console/page errors `0`, horizontal overflow `false`, overflowing elements `0`; screenshots are `/tmp/awoooi-host-runaway-aiops-desktop-1440x1100-42c08ece.png` and `/tmp/awoooi-host-runaway-aiops-mobile-390x844-42c08ece.png`. +Blocked: No for production product readback. Yes for runtime remediation by design: process termination, Docker/systemd restart, Nginx reload, firewall/K8s action, Telegram live send, Gateway queue write, Bot API call, production write, and secret read remain `0 / false`. +Next: Treat the next real or fixture `HostOrphanBrowserSmokeHighCpu` as the acceptance drill for end-to-end Telegram card / AwoooP work item / KM / PlayBook / Verifier field agreement. Any actual SIGTERM remains owner-approved, maintenance-windowed, dry-run-first, and post-check-gated. +Completion: host runaway AIOps product-visible loop readback and production verification 100%; runtime auto-remediation remains safely gated at 0. + 2026-06-18 13:43 Asia/Taipei Phase: P1/P2/P3 live readback Before: live cold-start was `PASS=83 WARN=1 BLOCKED=0`, result `DEGRADED`, because retained stale `km-vectorize-29689620` failed Job evidence was still counted as a service warning.