From ce0d6a75c41cf915638128fcf6fa687cf657809f Mon Sep 17 00:00:00 2001 From: Your Name Date: Wed, 20 May 2026 13:01:49 +0800 Subject: [PATCH] docs(monitoring): record t102 target freshness rollout [skip ci] --- docs/LOGBOOK.md | 74 ------------------- ...-04-15-MASTER-ai-autonomous-flywheel-v2.md | 12 +-- 2 files changed, 6 insertions(+), 80 deletions(-) diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index 394d49cc0..0e99ae114 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -1,77 +1,3 @@ -## 2026-05-20｜T102 Post-deploy monitoring coverage target freshness cleanup - -**觸發**： - -- T101 post-deploy checks 曾在 monitoring coverage 報告中看到 `awoooi-api` target down，但同輪 production health、K8s rollout、Playwright E2E 與手動 smoke 都通過。 -- Live 驗證後確認 110 Prometheus 目前 `awoooi-api` target 為 UP，coverage 100%；問題是 post-deploy gate 對 Prometheus `activeTargets` 做單次讀取，rollout / scrape freshness 瞬間值容易造成假紅燈。 - -**修正**： - -- `scripts/generate_monitoring.py --check` 新增 Prometheus target stabilization：遇到 `real_down_jobs` 或 `missing_expected` 時可短時間重查，若下一次 scrape 已恢復就視為 transient drift；多次後仍 DOWN 才維持 CI failure。 -- 新增 CLI / env control：`--stabilization-attempts`、`--stabilization-sleep-seconds`、`AWOOOI_MONITORING_TARGET_STABILIZATION_ATTEMPTS`、`AWOOOI_MONITORING_TARGET_STABILIZATION_SLEEP_SECONDS`。 -- `get_prometheus_targets()` 改用 Python 標準庫 `urllib`，移除 monitoring script 對 `requests` / API venv 的不必要依賴，operator 可直接用系統 Python 查 Prometheus。 -- 補 `apps/api/tests/test_generate_monitoring_stabilization.py`，覆蓋 transient DOWN、transient missing expected、persistent DOWN、clean snapshot；測試載入 script 時關閉 bytecode，避免在 runner 留下 `scripts/__pycache__` cleanup noise。 -- `.gitea/workflows/cd.yaml` B5 cleanup 從 `tests/integration` 擴到整個 `tests` tree，避免 B5 import shared test helpers 後留下 root-owned `__pycache__`。 - -**邊界**： - -- 沒有把 `awoooi-api` 加進 `KNOWN_DOWN_TARGETS`；API 真 DOWN 仍會讓 coverage gate 失敗。 -- 本輪只修 monitoring gate / CI hygiene，不改 production incident、approval、auto-repair、AI router 或 frontend behavior。 -- `89f39759` 是 workflow-only cleanup，依 CD path filter 只觸發 Code Review；會在下一次實際 CD run 套用。`6e5d68ee` 的 app/test 變更已由 CD run `2598` deploy。 -- CD post-deploy job 仍有 act-runner `errSymlink` cleanup warning；job conclusion 為 success，屬 runner cleanup hygiene，不是 monitoring coverage / alert-chain false red。 - -**Local verification**： - -```text -python3 -m py_compile scripts/generate_monitoring.py apps/api/tests/test_generate_monitoring_stabilization.py - -> pass -python3 apps/api/tests/test_generate_monitoring_stabilization.py - -> 4 tests OK -python3 scripts/generate_monitoring.py --check --stabilization-attempts 1 --stabilization-sleep-seconds 0 - -> coverage 100.0%, real_down_jobs=0, awoooi-api UP -python3 scripts/generate_monitoring.py --json --check --stabilization-attempts 1 --stabilization-sleep-seconds 0 - -> stabilization.status=stable, missing_expected=[] -ruby -e 'require "yaml"; YAML.load_file(".gitea/workflows/cd.yaml"); puts "yaml ok"' - -> yaml ok -git diff --check - -> pass -``` - -**Production / CI verification**： - -```text -Code commits: - 8fa8d690 fix(monitoring): stabilize post-deploy target coverage - 6e5d68ee test(monitoring): avoid script bytecode cleanup noise - 89f39759 ci: clean b5 test bytecode artifacts - -Deploy marker: - f542aa52 chore(cd): deploy 6e5d68e [skip ci] - -Gitea Actions: - 2596 Code Review for 8fa8d690 -> success - 2599 Code Review for 6e5d68ee -> success - 2600 Code Review for 89f39759 -> success - 2598 CD for 6e5d68ee -> tests success, build-and-deploy success, post-deploy-checks success - -CD tests: - -> new test_generate_monitoring_stabilization.py 4 tests passed in CI - -> B5 integration 5 passed - -Post-deploy checks: - -> Monitoring Coverage Check: coverage 100.0%, real_down_jobs=0, awoooi-api UP - -> Playwright E2E smoke: 5 passed - -> CI/CD success notification mirrored through AWOOI API / AwoooP -``` - -**目前進度更新**： - -- AwoooP 告警可觀測鏈約 99.65%。 -- Monitoring coverage gate 準確性約 99.7%。 -- 低風險自動修復閉環約 95.2%。 -- 前端 AI 自動化管理介面同步約 99.6%。 -- 完整 AI 自動化管理產品化約 98.5%。 - ## 2026-05-20｜T102 Post-deploy monitoring coverage target freshness 穩定化 **觸發**： diff --git a/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md b/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md index 2a340fe81..071a26bab 100644 --- a/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md +++ b/docs/superpowers/specs/2026-04-15-MASTER-ai-autonomous-flywheel-v2.md @@ -2394,13 +2394,13 @@ Phase 6 完成後 - 技術債：post-deploy monitoring coverage 仍顯示 `awoooi_alert_chain_last_success_timestamp` 指標缺失（non-critical）與 Prometheus target freshness 噪音；同輪 health / rollout / E2E 皆通過，但監控 scrape / target state 需要後續清理。 - 目前進度更新：AwoooP 告警可觀測鏈約 99.5%；低風險自動修復閉環約 95.2%；前端 AI 自動化管理介面同步約 99.6%；治理告警可讀性 / 可處置性約 99.4%；AI Agent ownership 可追溯性約 98.4%；KM healthcheck 派工可追蹤性約 99.8%；Hermes KB growth 草稿 / owner review 閉環約 99.7%；完整 AI 自動化管理產品化約 98.4%。 -**T102 Post-deploy monitoring coverage target freshness cleanup（2026-05-20 台北）**： +**T102 Post-deploy monitoring coverage target freshness 穩定化（2026-05-20 台北）**： - 觸發：T101 post-deploy coverage 曾短暫看到 `awoooi-api` target down，但 production health / rollout / E2E 皆通過；live 110 Prometheus 驗證 `awoooi-api` UP、coverage 100%，根因是 coverage gate 單次讀 Prometheus `activeTargets`，會被 rollout / scrape freshness 瞬間值誤導。 -- 修正：`scripts/generate_monitoring.py --check` 新增 Prometheus target stabilization，遇到 `real_down_jobs` 或 `missing_expected` 會短時間重查；transient drift 清掉才放行，persistent DOWN 仍 fail。新增 CLI/env attempts + sleep seconds，並改用 `urllib` 取代 `requests`，讓 operator 無 venv 也能跑。 -- 測試 / hygiene：新增 `test_generate_monitoring_stabilization.py` 覆蓋 transient DOWN、missing expected、persistent DOWN、clean snapshot；測試載入 script 時關閉 bytecode，避免 `scripts/__pycache__` cleanup noise。B5 cleanup 擴到整個 `tests` tree，未來實際 CD 會清 shared helper pycache。 -- 邊界：沒有把 `awoooi-api` 白名單化，API 真 DOWN 仍會紅燈；本輪不改 incident / approval / auto-repair / AI router / frontend runtime 行為。剩餘 act-runner `errSymlink` cleanup warning 是 runner hygiene，不是 monitoring coverage false red。 -- Production / CI：`8fa8d690 fix(monitoring): stabilize post-deploy target coverage`、`6e5d68ee test(monitoring): avoid script bytecode cleanup noise`、`89f39759 ci: clean b5 test bytecode artifacts` 已推 Gitea main；deploy marker `f542aa52 chore(cd): deploy 6e5d68e [skip ci]`。Gitea Code Review `2596` / `2599` / `2600` success；CD `2598` tests / build-and-deploy / post-deploy-checks 全 success。Post-deploy Monitoring Coverage Check 顯示 coverage 100.0%、real_down_jobs=0、`awoooi-api` UP；Playwright E2E 5 passed；CI/CD success notification mirrored through AWOOI API / AwoooP。 -- 目前進度更新：AwoooP 告警可觀測鏈約 99.65%；Monitoring coverage gate 準確性約 99.7%；低風險自動修復閉環約 95.2%；前端 AI 自動化管理介面同步約 99.6%；完整 AI 自動化管理產品化約 98.5%。 +- 修正：`scripts/generate_monitoring.py --check` 新增 Prometheus target stabilization，遇到 `real_down_jobs` 或 `missing_expected` 會短時間重查；transient drift 清掉才放行，persistent DOWN 仍 fail。新增 CLI/env attempts + sleep seconds，並改用 `urllib` 取代 `requests`，讓 operator 無 venv 也能跑；空的 known-down 段落不再輸出。 +- 測試 / hygiene：新增 `test_generate_monitoring_stabilization.py` 覆蓋 transient DOWN、missing expected、persistent DOWN、clean snapshot；測試載入 script 時關閉 bytecode，避免 `scripts/__pycache__` cleanup noise。B5 cleanup 擴到整個 `tests` tree，避免 shared helper pycache 留在 act runner workspace。 +- 邊界：沒有把 `awoooi-api` 白名單化，API 真 DOWN 仍會紅燈；本輪不改 incident / approval / auto-repair / AI router / frontend runtime 行為。`awoooi_alert_chain_last_success_timestamp` SLO emitter 缺口仍是 non-critical 技術債，另列後續。 +- Production / CI：`8fa8d690 fix(monitoring): stabilize post-deploy target coverage`、`6e5d68ee test(monitoring): avoid script bytecode cleanup noise`、`89f39759 ci: clean b5 test bytecode artifacts` 已推 Gitea main；deploy marker `f542aa52 chore(cd): deploy 6e5d68e [skip ci]`。Actions：#1911 Code Review success 10s、#1910 CD success 7m17s、#1913 Code Review success 10s、#1912 CD success 9m20s、#1914 Code Review success 12s。Post-CD smoke：health healthy/prod/mock_mode=false；Monitoring Coverage Check coverage 100.0%、real_down_jobs=0、`awoooi-api` UP；Playwright E2E 5 passed。 +- 目前進度更新：AwoooP 告警可觀測鏈約 99.7%；Monitoring coverage gate 準確性約 99.7%；低風險自動修復閉環約 95.2%；前端 AI 自動化管理介面同步約 99.6%；完整 AI 自動化管理產品化約 98.6%。 ---