wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 3 Packages Projects Releases Wiki Activity

docs(aiops): record host runaway production readback [skip ci]

Browse Source
This commit is contained in:
Your Name
2026-06-18 16:15:41 +08:00
parent 513dafab7c
commit f2ec9ec434
3 changed files with 49 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

27
docs/LOGBOOK.md
View File

@@ -1,3 +1,30 @@
## 2026-06-18｜P3-009 Host Runaway AIOps 閉環正式部署驗證
**背景**：110 CPU 滿載事件已從人工 `top/ps` 處理收斂成 read-only exporter、Prometheus alert lane、Telegram / AI event packet、PlayBook / KM contract 與 gated remediation helper；最後缺口是讓 governance automation inventory 能用單一讀模型回看「監控 -> 告警 -> 事件卡 -> PlayBook -> KM / Verifier -> gated remediation」是否真的串起來，而不是散在文件和 log。
**完成內容**：
- 新增並部署 `host_runaway_aiops_loop_readiness_v1` production read model：`GET /api/v1/agents/agent-host-runaway-aiops-loop-readiness`。
- Governance automation inventory 已顯示 `P3-009 Host Runaway AIOps 閉環`，可讀到 `6` 個 loop stage、`2` 條 alert lane、`5` 份 asset writeback contract、host `110` live readback、orphan browser groups `0`、active CI containers `2`、runtime writes `0`。
- `P2-407 AI 報表 no-write 分析` 與 P3-009 在同一正式 deploy marker 上共存；最新主線包含 `0e72a6f4 feat(aiops): expose host runaway loop readiness`。
- CD runner lock 修補由平行工作流收斂後，最終有效 deploy marker 為 `42c08ece chore(cd): deploy 27143fb [skip ci]`；舊的 `3168 / 3171 / 3174` 不作成功依據。
**Production readback / smoke**：
- Gitea `code-review.yaml #3178`：成功。
- Gitea `cd.yaml #3177`：成功；有效部署 commit `27143fb0 fix(cd): 補齊 runner lock 解析工具`，其 ancestry 包含 Host Runaway AIOps commit `0e72a6f4`。
- API health：`https://awoooi.wooo.work/api/v1/health` 回 `healthy / prod / mock_mode=false`；PostgreSQL、Redis、OpenClaw、SignOz、Ollama GCP-A / GCP-B / local route 均為 `up`。
- P3-009 API：`schema_version=host_runaway_aiops_loop_readiness_v1`、`current_task_id=P3-009`、`next_task_id=P3-010`、`overall_completion_percent=100`、`loop_stage_count=6`、`alert_lane_count=2`、`asset_writeback_contract_count=5`、`runtime_remediation_authorized_count=0`、`host_write_count=0`、`process_termination_count=0`、`docker_restart_count=0`、`production_write_count=0`。
- Desktop `1440x1100`：`/zh-TW/governance?tab=automation-inventory&_v=42c08ece-host-runaway-aiops-desktop-1440x1100` 必要文案缺漏 `0`、console error `0`、page error `0`、horizontalOverflow `false`、overflowing elements `0`；截圖 `/tmp/awoooi-host-runaway-aiops-desktop-1440x1100-42c08ece.png`。
- Mobile `390x844`：`/zh-TW/governance?tab=automation-inventory&_v=42c08ece-host-runaway-aiops-mobile-390x844` 必要文案缺漏 `0`、console error `0`、page error `0`、horizontalOverflow `false`、overflowing elements `0`；截圖 `/tmp/awoooi-host-runaway-aiops-mobile-390x844-42c08ece.png`。
- Smoke JSON：`/tmp/awoooi-host-runaway-aiops-smoke-42c08ece.json`。可見文字未出現原始 repo 名稱；HTML / client payload 仍有既有 governance metadata term 與舊 blocker key，另列 public bundle hygiene，不作本輪 Host Runaway card 可見外洩或 runtime gate。
**完成度同步**：
- Host Runaway AIOps product-visible loop readback：`100%`。
- 110 CPU orphan Chrome recurrence guard：`READY`，正式站可回讀。
- CD runner lock 修補到可部署本輪變更：`100%` for this deploy evidence。
- Runtime auto-remediation：仍 `0 / false`。`SIGTERM` 仍必須有 owner approval、maintenance window、evidence ref、dry-run、post-check、KM / PlayBook / Verifier writeback；不得把 API 200、UI 可見、CD success 或 route smoke 當作主機寫入授權。
**邊界**：本段沒有 SSH、沒有 kill process、沒有 Docker / systemd / Nginx / firewall / K8s runtime write、沒有 Telegram live send、沒有 Gateway queue write、沒有 Bot API call、沒有 secret 讀取、沒有 force push。此完成代表監控 / 告警 / PlayBook / KM / Verifier / gated remediation 的產品可視讀模型已正式部署，不代表自動修復可直接執行。
## 2026-06-18｜監控告警訊息專業化：主機 / runner raw dump 必須轉 AI 自動化事件卡
**背景**：統帥指出 Telegram / SRE 戰情室收到的主機告警仍把 CPU、容器 root Node.js 行程、完整工具路徑、套件 JSON、外部檢查 endpoint 與 raw process dump 直接貼出。這種訊息不可讀、不可處置，也會擴大資訊外洩面；AWOOOI / IwoooS 作為 AI 自動化產品，監控告警必須先轉成可判讀、可分流、可驗證、可回寫的事件卡，而不是把底層 log 丟給值班者。

13
docs/runbooks/FULL-STACK-COLD-START-SOP.md
View File

@@ -56,6 +56,19 @@ Allowed declaration: monitoring, alert rules, live scrape, AI event packet routi
Forbidden declaration: Telegram send, Bot API call, Gateway queue write, process kill, Docker/systemd restart, Nginx reload, firewall/K8s action, or runtime remediation is authorized.
```
2026-06-18 16:08 P3-009 Host Runaway AIOps product readback:
```text
Host runaway AIOps closed-loop read model is deployed in production.
Deploy marker: 42c08ece chore(cd): deploy 27143fb [skip ci].
API endpoint: /api/v1/agents/agent-host-runaway-aiops-loop-readiness.
Production readback: schema_version=host_runaway_aiops_loop_readiness_v1, current_task_id=P3-009, next_task_id=P3-010, completion=100, loop_stage_count=6, alert_lane_count=2, asset_writeback_contract_count=5.
Host 110 live readback in the model: orphan browser groups=0, active CI containers=2, remediation_authorized=0, runtime/write counters=0.
Governance route: /zh-TW/governance?tab=automation-inventory shows P3-009 on desktop 1440x1100 and mobile 390x844 with missing text=0, console/page errors=0, horizontal overflow=false.
Allowed declaration: monitoring, alert rules, AI event packet, PlayBook / KM contract, Verifier/writeback contract, gated remediation dry-run boundary, and product-visible readback are complete for this evidence set.
Forbidden declaration: AI runtime remediation is enabled. Process termination, Docker/systemd restart, Nginx reload, firewall/K8s action, Telegram live send, Gateway queue write, Bot API call, production write, and secret read remain forbidden without owner approval, maintenance window, evidence ref, dry-run, and post-check.
```
| 項目 | 2026-06-18 13:43 Asia/Taipei live result | 判定 |
|------|-------------------------------------------|------|
| Overall recovery readiness | `99%` | `SERVICE_GREEN_DR_ESCROW_BLOCKED` |

9
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
View File

@@ -254,6 +254,15 @@ Blocked: No for product readback; yes for runtime remediation by design.
Next: If a real or fixture alert fires, verify Telegram card, AwoooP Work Item, KM / PlayBook / Verifier fields agree before considering any owner-approved non-production gated SIGTERM drill.
Completion: host runaway AIOps product-visible loop readback 100%; runtime auto-remediation remains safely gated at 0.
2026-06-18 16:08 Asia/Taipei
Phase: P3-009 Host runaway AIOps loop production verification
Before: P3-009 source, API, UI and tests were pushed, but production still needed deploy marker, API readback, desktop/mobile browser smoke, and CD runner lock recovery evidence.
After: Final deploy marker `42c08ece chore(cd): deploy 27143fb [skip ci]` is live after CD runner lock fixes `fc6c01ee` / `84ca8423` / `27143fb0`; `cd.yaml #3177` and `code-review.yaml #3178` are successful. Production endpoint `/api/v1/agents/agent-host-runaway-aiops-loop-readiness` returns `schema_version=host_runaway_aiops_loop_readiness_v1`, `current_task_id=P3-009`, `next_task_id=P3-010`, completion `100`, loop stages `6`, alert lanes `2`, writeback contracts `5`, host `110`, orphan browser groups `0`, active CI containers `2`, and every runtime/write/remediation counter `0`.
Evidence: API health `healthy / prod / mock_mode=false`; desktop `1440x1100` and mobile `390x844` governance smoke with deploy marker `42c08ece` have required text missing `0`, console/page errors `0`, horizontal overflow `false`, overflowing elements `0`; screenshots are `/tmp/awoooi-host-runaway-aiops-desktop-1440x1100-42c08ece.png` and `/tmp/awoooi-host-runaway-aiops-mobile-390x844-42c08ece.png`.
Blocked: No for production product readback. Yes for runtime remediation by design: process termination, Docker/systemd restart, Nginx reload, firewall/K8s action, Telegram live send, Gateway queue write, Bot API call, production write, and secret read remain `0 / false`.
Next: Treat the next real or fixture `HostOrphanBrowserSmokeHighCpu` as the acceptance drill for end-to-end Telegram card / AwoooP work item / KM / PlayBook / Verifier field agreement. Any actual SIGTERM remains owner-approved, maintenance-windowed, dry-run-first, and post-check-gated.
Completion: host runaway AIOps product-visible loop readback and production verification 100%; runtime auto-remediation remains safely gated at 0.
2026-06-18 13:43 Asia/Taipei
Phase: P1/P2/P3 live readback
Before: live cold-start was `PASS=83 WARN=1 BLOCKED=0`, result `DEGRADED`, because retained stale `km-vectorize-29689620` failed Job evidence was still counted as a service warning.