fix(ops): close Gitea backup restore verification
This commit is contained in:
ogt
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"schema_version": "agent99_enterprise_ai_automation_work_items_v1",
|
||||
"generated_at": "2026-07-10T16:22:00+08:00",
|
||||
"generated_at": "2026-07-10T19:40:00+08:00",
|
||||
"status": "in_progress_p0_alert_routing_grouping_dedupe",
|
||||
"scope_complete": false,
|
||||
"north_star": "Make Windows host 192.168.0.99 the policy-controlled on-prem AI execution node for all hosts, VMs, services, products, sites, tools, packages, alerts and recovery workflows, with external monitoring when host 99 is unavailable.",
|
||||
@@ -20,8 +20,8 @@
|
||||
"id": "AG99-P0-002",
|
||||
"title": "Alert taxonomy, routing, grouping and dedupe",
|
||||
"reason": "Explicit event kind and suggested mode are not consistently authoritative, and forwarding before grouping can create wrong-domain or duplicate Agent99 work.",
|
||||
"implementation_status": "in_progress_replay_corpus_and_pre_bridge_grouping",
|
||||
"next_action": "Add routing precedence and a replay corpus for backup, provider, reboot, PostgreSQL, host and K3s events; move grouping and single-flight ahead of Agent99 dispatch."
|
||||
"implementation_status": "local_package_and_contract_green_runtime_deploy_pending",
|
||||
"next_action": "Finish the current Gitea source receipt, atomically deploy the tested Agent99 package to Windows 99 when an authenticated no-secret management channel is available, then replay backup, provider, reboot, PostgreSQL, host and K3s events against the runtime manifest."
|
||||
},
|
||||
"current_runtime_truth": {
|
||||
"observed_at": "2026-07-10T15:04:27+08:00",
|
||||
@@ -316,7 +316,10 @@
|
||||
"verifier": "Replay routing corpus and assert one active incident and expected mode per fingerprint.",
|
||||
"callback": "Publish normalized route decision and dedupe reason to AWOOOI and AwoooP.",
|
||||
"acceptance": "Known backup, provider, reboot, PostgreSQL, host and K3s samples route to the correct lane with no duplicate same-state dispatch.",
|
||||
"next_action": "Add routing precedence and pre-bridge grouping regression corpus."
|
||||
"completion_percent": 80,
|
||||
"implementation_progress": "Local Agent99 contract and synthetic replay are green; Windows 99 runtime deployment and runtime replay remain pending.",
|
||||
"evidence_refs": ["local Agent99 PowerShell contract 19/19", "local Agent99 synthetic replay 8/8", "isolated atomic deploy test passed"],
|
||||
"next_action": "Deploy the tested package atomically to Windows 99, verify the runtime manifest, and rerun the routing/dedupe corpus through the live ingress."
|
||||
},
|
||||
{
|
||||
"id": "AG99-P0-003",
|
||||
@@ -536,7 +539,10 @@
|
||||
"verifier": "Each baseline domain has schedule, freshness, integrity and isolated restore evidence.",
|
||||
"callback": "Report one lifecycle incident per failed backup domain and update DR scorecard.",
|
||||
"acceptance": "No unknown backup target; failures name the domain, age, SLA, owner, evidence and safe next action.",
|
||||
"next_action": "Reconcile failed readbacks and add missing full Gitea, DB, packages, LFS and tool/log restore receipts."
|
||||
"completion_percent": 78,
|
||||
"implementation_progress": "The 110/188 backup heartbeat, Gitea full-dump isolated restore, private bundle off-host copy, sample restore, Velero missed-run backfill and Prometheus alert slice are live green.",
|
||||
"evidence_refs": ["110 backup 13/13", "188 backup 4/4", "Gitea dump CRC and inventory 15/15", "Gitea product bundles and remote SHA-256 12/12 on 188", "Velero backfill job succeeded", "Prometheus source/canonical/active/runtime SHA matched", "Backup/Gitea non-inactive alerts 0"],
|
||||
"next_action": "Merge and deploy the source branch through Gitea, then complete the remaining DB, registry, package, tool, log and AI-artifact restore-receipt matrix."
|
||||
},
|
||||
{
|
||||
"id": "AG99-P0-013",
|
||||
@@ -580,7 +586,10 @@
|
||||
"verifier": "All expected repos and refs, full dump components, registry artifacts, CD run and runtime markers read back successfully.",
|
||||
"callback": "Update source-control, release and DR scorecards with one trace.",
|
||||
"acceptance": "All 12 products have source and release proof and the platform can be reconstructed in isolation from verified backups.",
|
||||
"next_action": "Add full Gitea state and registry restore receipts to the existing repo-bundle proof."
|
||||
"completion_percent": 72,
|
||||
"implementation_progress": "Gitea full dump and off-host private bundle restore-confidence are live verified; Harbor/Registry artifacts, runner/CD and complete platform reconstruction remain open.",
|
||||
"evidence_refs": ["Gitea full dump 6928031514 bytes", "regular repository inventory 15/15", "product bundle manifest 12/12", "188 sample clone and fsck passed"],
|
||||
"next_action": "Merge/deploy the Gitea DR source, then add Harbor/Registry artifact restore receipts and runner/CD/runtime-marker reconstruction proof."
|
||||
},
|
||||
{
|
||||
"id": "AG99-P1-001",
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# AWOOOI 全棧冷啟動與主機重啟 SOP
|
||||
|
||||
> Version: v1.131
|
||||
> Last updated: 2026-07-09 Asia/Taipei
|
||||
> Version: v1.132
|
||||
> Last updated: 2026-07-10 Asia/Taipei
|
||||
> Scope: 99 / 110 / 111 / 112 / 120 / 121 / 188 全棧重啟恢復。112 仍是 Kali / VM guest 訊號，但 2026-06-30 全主機重啟後已納入 10 分鐘 SLO 的必要 boot / power signal；此納入不代表授權任何破壞性 runtime apply。
|
||||
|
||||
---
|
||||
@@ -10,6 +10,14 @@
|
||||
|
||||
本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。
|
||||
|
||||
### v1.132 2026-07-10 備份與 Gitea 可還原性閉環
|
||||
|
||||
重啟恢復不得再以「備份檔存在」或「Gitea HTTP 200」宣告備份正常。固定證據鏈為：backup exporter/status freshness -> Gitea full dump isolated restore -> private bundle complete-set sync -> sample clone/fsck -> Prometheus rule/alert readback。任一層缺失都只能標示 degraded，不能標示 restore-ready。
|
||||
|
||||
2026-07-10 19:40 live receipt：110 backup heartbeat 為 `13/13`、188 為 `4/4`、`core_blockers=0`、`dr_warnings=0`；分母由 live `awoooi_backup_job_fresh` series 動態計算，不再使用硬編碼 `13/2`，缺 metric family 會 fail closed。Gitea isolated full-dump drill 驗證 snapshot `29c8b7a8`、dump `6928031514` bytes、ZIP CRC、DB/config required artifacts、backup/live regular repositories `15/15`、wiki `0/0` 與 privacy-preserving inventory digest，全程未覆寫 production。12 個產品 private bundles 已由 110 local Gitea container 產生完整集合、no-delete 同步至 188，remote manifest `12/12` 並完成 sample clone + `git fsck`。Velero weekly restore dry-run 的 `startingDeadlineSeconds` 已由 `600` 修正為 `604800`，停機期間漏掉的排程由 controller 自動補跑且成功。
|
||||
|
||||
Prometheus active/canonical rules 在 19:40 先通過排程週期；20:17 進一步發現 host active path 與 Docker read-only bind mount 曾因 inode replacement 分叉。drift guard 現在同時比較 canonical、host active 與 container runtime SHA，canonical 先經 promtool，只有 runtime mount 分叉才受控 recreate Prometheus container，並在 rules/health 未收斂時回滾 active file。live 自癒驗收後 source/canonical/active/runtime 四方 SHA 均為 `34c0359e01d608d6300b92ca878aed414180ec725aace3ade0a19d39cc362697`，guard `status=ok`、`repaired=1`、`missing_required_count=0`、`current_matches_canonical=1`、`runtime_matches_current=1`，active alerts file 載入 `145` rules，Backup/Gitea firing 或 pending alerts 為 `0`。20:20 下一個正式 cron 週期使用相同 container ID/StartedAt，回 `repaired=0`、三方 match `1`，證明穩態不會每五分鐘 recreate。這只完成本輪 Gitea/backup isolated verification；不等於執行 production disaster restore，也不等於所有產品版本與資料 freshness 已完成。
|
||||
|
||||
v1.131 AI log triage controlled auto-recovery rule：每次全主機重啟後，10 分鐘 SLO verifier 不得只跑 read-only summary。`awoooi-reboot-auto-recovery-slo.service` 必須以 `AI_LOG_TRIAGE_AUTO_RECOVERY_MODE=apply` 先執行 `scripts/reboot-recovery/ai-log-triage-auto-recovery.sh`，bounded 收集 99 / 110 / 120 / 121 / 188 logs、K3s events、VMware no-secret readback 與 public route status，自動輸出 findings / actions / summary，再接 host probe、post-reboot summary、scorecard 與 textfile metrics。2026-07-09 16:23 orchestrator check artifact `/tmp/awoooi-full-host-reboot-orchestrator-codex/20260709T082305Z` 的目前 truth 是 `RECOVERY_BLOCKED_P0`、`FINDING_COUNT=3`、`FINDING_P0_COUNT=2`：`STOCK_UPSTREAM_NOT_READY host=110_188`、`EDGE_502_UPSTREAM_REFUSED host=188_edge`，另有 `HOST188_POSTGRES_SYSTEMD_FAILED` P2 hygiene；public route readback 為 AWOOOI health `200`、AWOOOI web `307`、Stock freshness `200`、Gitea `200`、registry `/v2/` `401`、Harbor `200`。這些不得再被當成噪音或人工 SOP；必須進 AI findings、Prometheus/Telegram blocker 與下一步 action matrix。允許的 `--apply` 只限低風險 allowlist：start inactive docker/nginx、Harbor watchdog log-first repair、registry ready 後刪 failed-pull pods 讓 K3s 重建。仍禁止 reboot、node drain、firewall cutover、DB destructive action、Docker prune、secret/session/.env/raw-volume reads。
|
||||
|
||||
v1.130 Windows99 VMX backup/source search package rule：2026-07-03 18:19 最新 via-110 no-secret locator 已把目前 truth 從早先 `candidate_source_count=6` 修正為 `candidate_source_count=0`、`source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found`，且 `raw_path_output=false`、`path_fingerprint_only=true`、`secret_value_read=false`、`remote_write_performed=false`、`vm_power_change_performed=false`。因此 P0-4 下一步不得再引用 generic Ubuntu candidate；必須讀 `/api/v1/agents/windows99-vmx-source-backup-search-package` 與 priority readback 的 `windows99_vmx_source_backup_search_*` 欄位。當前 backup/source search receipt 顯示 120 / 121 / local 均未找到 verified 111 source，99 alias 無 Linux backup host mapping，110 / 188 / 112 SSH BatchMode 不可用，`candidate_source_count=0`、`blocked_host_count=4`、`apply_allowed_by_this_package=false`。固定下一步是離線備份 inventory 或授權 console source path，再取得 verified source path 後建立 scoped relink dry-run；仍不得 power on VM、restore、讀 VMX 內容、讀 secret、重啟 Windows / VMware service / host / Docker / Nginx / K3s / DB / firewall。
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# AWOOOI 重開機恢復 SOP
|
||||
|
||||
> **版本**: v5.22
|
||||
> **最後更新**: 2026-07-09 (台北時間)
|
||||
> **版本**: v5.23
|
||||
> **最後更新**: 2026-07-10 (台北時間)
|
||||
> **更新者**: Codex
|
||||
> **觸發事件**: 2026-07-03 18:19 Windows99 VMX backup/source blocked-host truth 上卷
|
||||
> **觸發事件**: 2026-07-10 全主機重啟後備份/Gitea 可還原性與排程補跑收斂
|
||||
|
||||
---
|
||||
|
||||
@@ -21,6 +21,16 @@
|
||||
|
||||
## 架構概覽與依賴圖
|
||||
|
||||
### 2026-07-10 P0 備份/Gitea 自動驗證覆蓋
|
||||
|
||||
每次 host reboot 後，備份 lane 必須依序執行 `/backup/scripts/backup-status.sh --no-notify`、`/backup/scripts/verify-gitea-backup.sh --no-notify`，再由 110 的每日排程執行 `/backup/scripts/gitea-bundle-backup-sync-188.sh`。最後讀回 110/188 `backup_health.prom`、Gitea restore status、bundle sync status、188 sample restore metric 與 Prometheus Backup/Gitea alerts；不得只看檔案 mtime、HTTP 200 或命令 exit code。
|
||||
|
||||
本輪 live 驗證已證明：110 `13/13`、188 `4/4`、core blocker `0`；Gitea full dump 為 `6928031514` bytes，CRC、DB/config、15/15 regular repo inventory digest 一致；12/12 product private bundles 已從 110 local container 產生並完整同步 188，remote SHA-256 `12/12`、sample clone/fsck 成功；Velero missed weekly dry-run 已由 `startingDeadlineSeconds=604800` 自動 backfill 成功。20:17 drift guard 自動修復 read-only bind-mount inode drift 後，source/canonical/active/runtime rule SHA 一致，145 rules 已載入，Backup/Gitea 非 inactive alerts 為空；20:20 正式 cron 回 `repaired=0` 且 container ID/StartedAt 未變，穩態無重複 recreate。
|
||||
|
||||
Prometheus rules 部署不得用會置換 bind source inode 的 `install` / atomic rename 直接覆蓋 `/home/wooo/monitoring/alerts.yml`。正式 guard 會先 promtool canonical，再以保留 host path 的方式更新 active；若 container runtime SHA 仍不同，才用 `/home/wooo/monitoring/docker-compose.yml` 單獨 `--no-deps --force-recreate prometheus` 重新掛載，TSDB volume 保留。post-verifier 必須同時通過 `145` rules health、runtime SHA、required rules 與 active alert query；失敗則還原舊 active file 並重建 Prometheus，不得只看 reload HTTP 200。
|
||||
|
||||
失敗處置固定為 fail closed：full dump 或 bundle incomplete 不得更新 complete symlink；remote sync 不使用 delete；restore drill 只能在隔離暫存目錄；不得讀 raw backup contents、secret 或 private repo 名稱，不得 restore over production、prune 或刪遠端備份。Telegram 只在狀態改變或失敗時送一張繁中摘要卡，成功心跳維持每日一次。
|
||||
|
||||
### 2026-07-09 P0 AI Log Triage / Controlled Auto-Recovery 覆蓋
|
||||
|
||||
主機重啟後不得再靠人工逐台看 log 才判斷異常。固定第一步改為執行 `scripts/reboot-recovery/ai-log-triage-auto-recovery.sh --apply`；若只允許讀取，則用 `--check`，但 scorecard / 告警仍必須吃同一份 artifact。此 runner 會 bounded 收集 110 Docker / containerd / Harbor / Nginx、120 / 121 K3s events、188 edge Nginx / Docker / systemd、99 VMware no-secret readback 與 public route HTTP status，然後輸出 `findings.tsv`、`actions.tsv`、`summary.txt`。
|
||||
|
||||
@@ -25,7 +25,7 @@
|
||||
| P0-2 | DEPLOYED_RUNTIME_EDGE_DRIFT_BLOCKED | Deploy / reboot 期間 public 502 維護頁與外部 fallback | 2026-07-03 15:35 source fix 已部署讀回：production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 保留 `edge_public_maintenance_live_config_drift`，`runtime_metric_active_blocker_count=8`，且 route fallback ready 不再清掉 edge Nginx drift。110 SLO artifact `/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260703-151219` 與 textfile 證明 edge 未 ready：`awoooi_public_maintenance_edge_fallback_ready{scope="public_routes"} 0`、active blocker `edge_public_maintenance_live_config_drift`。 | 後續才進 privileged edge apply 或 L1 cloud/CDN fallback decision；完成條件是 edge fallback live config / maintenance file / `nginx -t` / reload / public header verifier 或外部 L1 fallback 全綠。不得讀密碼、不得用 app / Docker / host restart 掩蓋 edge fallback drift。 |
|
||||
| P0-3 | PARTIAL_GREEN_DEPLOYED_SOURCE | 所有產品 / 網站版本與資料最新性 | Gitea `main=bc8d3af3f chore(cd): deploy dacdd90 [skip ci]`；CD `#4558` log 顯示 API/Web build/push wrapper 全部 `rc=0`、deploy marker `dacdd90`、production deploy readback matched。Production SLO route 已讀回最新 API 行為：`runtime_metric_runtime_readback_added_blockers=["windows99_vmware_autostart_config_not_ready","windows99_update_no_auto_reboot_policy_not_ready"]`、`windows99_update_no_auto_reboot_ready=false`。Stock freshness 仍為 `status=ok`、`latest_trading_date=2026-07-02`、blockers `[]`。 | 版本層已補到 AWOOOI production；完成整體 P0 仍需收斂 Windows99 VMX / service / policy / power 與 host reboot-event blockers。後續每個 public product 仍要維持 source SHA、deploy marker、runtime endpoint、public route watch、freshness 四層一致。 |
|
||||
| P0-4 | BLOCKED_WINDOWS99_VMX_SOURCE_AND_TASK_NOT_TRIGGERED | 192.168.0.99 VMware 自動啟動與 VM guest 111 / 188 / 120 / 121 / 112 | 2026-07-03 17:34 已確認 110→99 SSH channel 可用，via-110 locator `rc=0`，`expected_vmx_path_present_count=0`、`candidate_source_count=0`、`source_locator_status=blocked_missing_expected_vmx_source_no_candidate_found`。full verify 在 90 秒曾 timeout，但 180 秒成功讀回 `VMX alias=111 present=0`、`MISSING_VMX_ALIASES=111`、`VMWARE_AUTOSTART_CONFIG_READY=0`；因此 source 已調整 Windows99 remote verify 預設 timeout 為 `180` 秒。前一輪 CD `#4618` 已將 `36536adcf` 投產，deploy marker `acbf96c`，production package endpoint live。 | 下一步固定為取得 111 既有 VMX source / verified backup / 授權 console candidate identity，產生 scoped relink dry-run + rollback + post-verifier，再 rerun no-secret collector + SLO scorecard。仍不得從 scorecard 直接 power on VM、重啟 Windows host / VMware service、改 scheduled task / registry、猜 generic Ubuntu candidate、restore 到 production 或讀 Windows 密碼；也不得把 90 秒 timeout 當成 SSH channel 故障。 |
|
||||
| P0-5 | RUNTIME_READY_BACKUP_RECEIPT_GAP | Gitea / 主機 / DB / 網站 / 服務 / 套件 / 工具 / log 備份監控告警 | Gitea repo bundle readback ready：expected `12`、rows `12`、missing `0`、failed `0`、sample restore dry-run ok；backup core green。2026-07-03 source / runtime 已部署 `awoooi_backup_alert_receipt_*` 指標與 Prometheus rules；110 exporter 讀回 88 個 stage requirement、188 讀回 12 個 stage requirement，`BackupAlertReceiptMetricMissing*` inactive，`BackupAlertReceiptStageMissing` 已修成每 `host / receipt_channel` 聚合 pending：110 一條、188 一條。 | 補 `/backup/alert-receipts/*.last_success` 脫敏 marker；下一層仍要補 Gitea full dump、DB/settings/issues/packages/LFS、所有工具與 log 全量備份監控。 |
|
||||
| P0-5 | LIVE_GREEN_SOURCE_BRANCH_PENDING_MAIN | Gitea / 主機 / DB / 網站 / 服務 / 套件 / 工具 / log 備份監控告警 | 2026-07-10 live：backup status 動態分母讀回 110 `13/13`、188 `4/4`、`core_blockers=0`、`dr_warnings=0`；Gitea full dump isolated drill 驗證 `6928031514` bytes、CRC、DB/config 與 regular repo inventory `15/15`；12/12 product private bundles 由 110 local container 產生、no-delete 同步 188、remote SHA-256 `12/12` 並 sample clone/fsck；Velero missed weekly dry-run 自動 backfill 成功。20:17 guard 修復 Prometheus bind-mount inode drift 後，source/canonical/active/runtime SHA 一致、145 rules、Backup/Gitea non-inactive alerts `0`。 | 本輪 Gitea/backup slice 已 live verified；把 branch source 推進 Gitea main/CD 後，依總帳繼續補其餘 DB、registry、package、tool、log 與 AI artifact restore matrix，不得把本列誤報成全域 DR complete。 |
|
||||
| P0-6 | RUNTIME_READY_ALERT_RECEIPT_GAP | 主機關機 / 重啟 / SLO miss / backup failure Telegram 告警 | Reboot per-blocker alert 與 backup receipt alert rules 已 deploy/readback；backup receipt 缺段不再產生 100 條 stage 噪音，現在聚合成 110 / 188 兩條 host-level pending。scorecard 目前有 10 個 reboot active blockers，其中 Windows99 config / policy blocker 已可供 per-blocker alert routing 使用；尚未完成 shutdown / reboot / backup alert 的 production 脫敏 delivery receipt 全矩陣。 | 補 alert receipt readback：host down、host up、SLO miss、Windows99 blocker、backup stale/failed、deploy 502、freshness stale；完成條件是每類告警都有 sent / received / dedup / escalation evidence。 |
|
||||
| P0-7 | SOURCE_READY_SLA_AUTOMATION | 固定排查順序、ETA / wait reason、自動化判斷與修復 | Scorecard 已固定 `current_phase`、`eta_or_wait_reason`、`primary_blocker`、fixed triage order 與 action matrix；2026-07-03 新增 `windows99_vmware_autostart_task_never_ran`，其 next action 固定為 `run_windows99_vmware_autostart_check_mode_package_for_powered_off_aliases_then_rerun_no_secret_collector_no_vm_power_change`，不再退回泛用「再收一次 verify」。 | 把每個 blocker 的 next_safe_action、post_verifier、forbidden_actions 接到自動 work item / Telegram / scorecard；完成條件是重啟後自動判斷、主動告警、主動 rerun verifier，不再人工臨場猜流程。 |
|
||||
| P0-8 | BLOCKED_POLICY_READBACK | Windows99 禁止 Windows Update 無預警重啟 | 10:44 production readback 已 fail-closed：`windows99_update_no_auto_reboot_ready=false`，active blockers 含 `windows99_update_no_auto_reboot_policy_not_ready`，來源是 nested verifier 的 `policy_blockers=["windows_update_policy_readback_missing"]`。 | 保留 no-secret verifier，補週期性 policy readback 與 Telegram drift alert；完成條件是 Windows Update policy drift / missing readback 會自動告警且不需讀 secret，不得從 scorecard 直接 apply registry。 |
|
||||
@@ -225,6 +225,8 @@ Next: <single next action>
|
||||
| P1-015 | DONE | 100 | Restore 188 MinIO / Velero backup freshness and DB exporters | 2026-06-24 06:35 resolved real backup / exporter red lights: 188 PostgreSQL exporter and Redis exporter now expose `pg_up=1` / `redis_up=1`; 188 MinIO health is live; 120 Velero BSL is `Available`; one-off backup `reboot-recovery-202606240456` completed; 110 backup-health textfile reports latest Velero backup fresh. 110 disk pressure was reduced from 92% to 73% by Docker image/build-cache cleanup only. | Reconcile MinIO `userns_mode: host` override into formal source-of-truth or data ownership fix; keep Docker volume prune forbidden without explicit owner approval. | `VeleroBackupNotRun`、`PostgreSQLDown`、`RedisDown`、110 disk-pressure alerts are resolved, and SOP includes restore helpers. |
|
||||
| P1-016 | DONE | 100 | Control repeated Telegram notification noise without hiding real alerts | 2026-06-24 confirmed MOMO Pro 5-minute spam came from a legacy 110 script checking `http://192.168.0.188/health`; live script now uses `https://mo.wooo.work/health` as primary truth. Heartbeat warning dedupe now hashes stable actionable fingerprints so HTTP status / timeout / latency drift does not create a new Telegram event every 30 minutes. `MoWoooWorkDown` now labels `component=momo-pro-system`, disables blind auto-repair, and requires public/local/container/data-freshness triage. Generic docker-health monitor keeps 5-minute repair checks but adds a separate 30-minute direct Telegram fallback cooldown. Bitan public-content cleanliness keeps failure notification with same-fingerprint cooldown and one recovery notice. | Fold remaining cross-product direct Telegram egress into the unified notification gateway over time; do not disable real warning/failure/recovery signals. Production deployment/readback must confirm the code and Prometheus rules are live before declaring runtime closure. | Healthy heartbeat is quiet, same actionable heartbeat warning is deduped, MOMO public health success produces no alert, repeated same-failure direct fallback paths are cooled, and real failure/recovery/new-warning notifications remain enabled. |
|
||||
| P1-017 | DONE | 100 | Restore 188 nginx-exporter and post-CD monitoring coverage | CD `#3294` deployed marker `622bc372` but failed post-deploy checks because `scripts/generate_monitoring.py --check` saw Prometheus job `nginx-exporter` down at `192.168.0.188:9113`. 188 `stub_status` and compose config were healthy, so the correct fix was restoring the stateless exporter from `/home/ollama/nginx-exporter.yml`, not reloading Nginx or restarting products. New helper `scripts/ops/188-nginx-exporter-restore.sh` defaults to read-only `--check` and exposes explicit `--apply` for maintenance-window restore. `high-value-config-change-gate.py` now classifies `scripts/ops/**/*exporter*` as `monitoring_alerting_observability` P1 / C1. | Keep this check in post-reboot and post-CD recovery. Do not mark historical CD `#3294` as success; use the next CD run plus monitoring coverage as future proof. | `bash scripts/ops/188-nginx-exporter-restore.sh --check` reports `nginx_up 1`; `python3 scripts/generate_monitoring.py --check --stabilization-sleep-seconds 0` reports `Jobs=14`, `全部 UP=14`, `真實問題=0`, coverage `100.0%`; high-value gate matches the helper as P1 / C1, not unmanaged. |
|
||||
| P1-018 | DONE_LIVE_SOURCE_BRANCH_PENDING_MAIN | 95 | Gitea full dump isolated restore and metrics | The previous generic drill could report fallback success without proving the 6.9 GB Gitea dump, database/config artifacts or repository inventory. A dedicated verifier now restores to an isolated temporary directory, capacity-gates, checks ZIP CRC, required DB/config files and a privacy-preserving live/backup inventory digest, then deletes temporary data. | Merge and deploy the source branch through Gitea; retain isolated-only execution and never restore over production. | Live status is `success=1`, `reason=verified`, snapshot `29c8b7a8`, dump `6928031514`, backup/live regular repos `15/15`, inventory match `1`, CRC `1`. |
|
||||
| P1-019 | DONE_LIVE_SOURCE_BRANCH_PENDING_MAIN | 95 | Gitea private bundle generation and 188 restore confidence | 188 did not have Gitea SSH key access, so generation moved to the 110 local Gitea container. The controller creates a complete 12-repo set, syncs to 188 without deletion, validates the remote manifest before symlink promotion, then runs sample clone/fsck and refreshes metrics. | Merge/deploy via Gitea and keep the 00:15 110 cron as the only generator; 188 remains verifier/exporter, not a key-bearing source client. | Live local and remote manifests are `12/12`, sync status `success=1`, sample restore metric `1`, and 188 bundle/mirror freshness metrics are green. |
|
||||
|
||||
---
|
||||
|
||||
@@ -292,6 +294,17 @@ Do not run `truncate`, whole DB restore, force-push, DROP, or online root filesy
|
||||
|
||||
## 9. Progress Updates
|
||||
|
||||
```text
|
||||
2026-07-10 19:40 Asia/Taipei
|
||||
Phase: P0 backup/Gitea recovery truth-chain
|
||||
Before: backup status used a stale hard-coded 188 denominator, generic Gitea restore evidence did not prove the full dump, the weekly Velero dry-run could be skipped permanently after downtime, and a five-minute drift guard could overwrite manually loaded Prometheus rules.
|
||||
After: dynamic healthy/actual backup ratios are live; Gitea full-dump isolated restore and 110-to-188 complete private bundle flow are verified; Velero automatically backfilled the missed schedule; canonical/current Prometheus rules stay identical across the guard cycle.
|
||||
Evidence: 110 `13/13`, 188 `4/4`, core blockers `0`; Gitea dump `6928031514` bytes, CRC/database/config/inventory match `1`, regular repositories `15/15`; bundles `12/12` and sample clone/fsck `1`; active rules `145`; Backup/Gitea non-inactive alerts `0`.
|
||||
Additional verifier: remote bundle SHA-256 `12/12`; Prometheus source/canonical/active/runtime SHA `34c0359e01d608d6300b92ca878aed414180ec725aace3ade0a19d39cc362697`; guard runtime match `1` after controlled Prometheus-only recreate; next 20:20 cron kept the same container and returned `repaired=0`.
|
||||
Blocked: no for this live Gitea/backup slice. Source still needs Gitea branch commit/main/CD convergence; full platform DR and credential escrow remain separate incomplete work items.
|
||||
Next: finish source tests, commit/push the Gitea branch, then return to the ordered current Agent99 P0 without reopening this slice unless the verifier regresses.
|
||||
```
|
||||
|
||||
```text
|
||||
2026-06-18 14:20 Asia/Taipei
|
||||
Phase: P3 AI Ops runaway process automation
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Agent99 全域 AI 自動化主計畫
|
||||
|
||||
> 建立時間：2026-07-10 15:05 CST
|
||||
> 建立時間：2026-07-10 15:05 CST；最近更新：2026-07-10 19:40 CST
|
||||
> 狀態：P0 執行中，尚未達成全域 AI 自動化
|
||||
> 機器可讀總帳：`docs/operations/agent99-enterprise-ai-automation-work-items.snapshot.json`
|
||||
> 歷史需求來源：`docs/workplans/2026-07-02-commander-inserted-requirements-priority-ledger.md`
|
||||
@@ -92,7 +92,7 @@
|
||||
| 順序 | ID | 工作 | 狀態 | 驗收 |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| 1 | AG99-P0-001 | 修正 `exitCode=0 => resolved` 假綠，對齊 99 runtime script 與 Gitea source SHA | **完成** | Production 99 outcome self-test 通過；runtime manifest `7/7` matched、drift `0`；operator replay 為 verified resolved，alert replay 缺 source-event resolved 時維持 verifying |
|
||||
| 2 | AG99-P0-002 | 修正 alert taxonomy、wrong-domain routing、bridge-before-grouping 與重複事件 | **進行中，current P0** | `suggestedMode`/kind 優先於 source 字串；5 分鐘先 group；同 fingerprint 只建立一個 active incident |
|
||||
| 2 | AG99-P0-002 | 修正 alert taxonomy、wrong-domain routing、bridge-before-grouping 與重複事件 | **進行中，current P0；本機 package/contract 綠，99 runtime 待部署** | `suggestedMode`/kind 優先於 source 字串；5 分鐘先 group；同 fingerprint 只建立一個 active incident |
|
||||
| 3 | AG99-P0-003 | 強化 ingress：relay token 必填、Telegram chat+sender allowlist、idempotency/single consumer | 待辦 | 未認證 LAN request 拒絕；producer `auto_repair`/policy 被保留；所有 request 可追 trace |
|
||||
| 4 | AG99-P0-004 | 新增 EvidenceCollect / RepairCandidate / MCP lane | 待辦 | 缺 evidence 的事件會建立可查 candidate，不再錯送 Perf/AwoooRepair；`INC-*` 可從收件追到 candidate |
|
||||
|
||||
@@ -112,9 +112,9 @@
|
||||
| 9 | AG99-P0-009 | 外部 down/up watcher、L0/L1 維護頁與 502 failover | 進行中 | 99/內網全斷仍能告警；edge 失聯自動顯示維護頁；恢復後撤回並驗證，不改正式 DNS 作為日常修復 |
|
||||
| 10 | AG99-P0-010 | 12 產品、全部網站/API 的版本與資料 freshness 矩陣 | 進行中 | 每列具 source SHA、CD、deploy marker、runtime、HTTP、功能 smoke、data timestamp；任何缺欄不宣稱最新 |
|
||||
| 11 | AG99-P0-011 | 全域 log source registry、AI 異常判讀與 correlation | 待辦 | host/product/site/service/package/tool logs 有 freshness、redaction、tag、cluster、primary/secondary/noise 判定 |
|
||||
| 12 | AG99-P0-012 | 備份 observability、integrity、offsite 與 restore drill | 進行中 | 所有 backup domain 有 schedule/last success/freshness/restore receipt；失敗可定位且 TG 只報一次狀態變化 |
|
||||
| 12 | AG99-P0-012 | 備份 observability、integrity、offsite 與 restore drill | 進行中；Gitea/Velero/110-188 slice live green | 所有 backup domain 有 schedule/last success/freshness/restore receipt；失敗可定位且 TG 只報一次狀態變化 |
|
||||
| 13 | AG99-P0-013 | 110/188 及全主機效能 AIOps 與安全降載 | 進行中 | CPU/memory/disk/IO/queue/DB attribution；allowlisted action 後 verifier，無空值 perf alert、無單次下降假結案 |
|
||||
| 14 | AG99-P0-014 | Gitea/Harbor/Registry/Gitea-only CD 與完整 DR | 進行中 | repo/main/dev、full dump、DB/settings/issues/packages/LFS、registry、sample restore、runner/CD/runtime marker 全部可回讀 |
|
||||
| 14 | AG99-P0-014 | Gitea/Harbor/Registry/Gitea-only CD 與完整 DR | 進行中；Gitea full dump/bundle restore slice live green | repo/main/dev、full dump、DB/settings/issues/packages/LFS、registry、sample restore、runner/CD/runtime marker 全部可回讀 |
|
||||
|
||||
### Wave 3：真正 AI Agent 與產品化
|
||||
|
||||
@@ -142,6 +142,14 @@
|
||||
- Gitea inventory 有 12 個產品的 main/dev SSH refs；這只證明 repo/branch 存在，不證明每個產品 production 都是最新版本。
|
||||
- Agent99 現況仍是規則式 SRE automation；在 planner、tool loop、mode verifier、callback、secure ingress 完成前，不得宣稱完整 AI Agent。
|
||||
|
||||
### 2026-07-10 19:40 本輪已驗證增量
|
||||
|
||||
- Backup heartbeat 已以 live metric 動態分母讀回 110 `13/13`、188 `4/4`，`core_blockers=0`、`dr_warnings=0`。
|
||||
- Gitea full dump isolated restore 已驗證 6.9 GB archive、CRC、DB/config 與 15/15 regular repository inventory digest；production 未被 restore 或改寫。
|
||||
- 12/12 product private bundles 已由 110 local container 產生、no-delete 同步 188，remote SHA-256 `12/12` 與 sample clone/fsck 通過。
|
||||
- Velero missed weekly restore dry-run 已由 controller 自動 backfill 成功；20:17 Prometheus guard 修復 bind-mount inode drift 後，source/canonical/active/runtime 四方 SHA 一致、145 rules healthy，Backup/Gitea active alerts 為 0。
|
||||
- 以上只關閉 `AG99-P0-012/014` 的 Gitea/backup 子切片；全域 backup domain、Harbor/Registry/CD DR、99 Agent runtime 與 10 分鐘 reboot SLO 仍未完成。
|
||||
|
||||
## 7. 工作項強制欄位與治理
|
||||
|
||||
每個工作項必須先有：`id`、priority/order/status、scope、owner lane、dependencies、acceptance 與 next action。任何工作項升到 `candidate_ready`、`in_progress` 或 `executing` 前，還必須補齊 problem、trigger、evidence sources、AI role、action mode、controlled actions、prohibited actions、rollback、mode-specific verifier、callback、evidence refs 與 last update；缺一不可執行。
|
||||
@@ -157,4 +165,4 @@
|
||||
|
||||
## 8. 立即下一步
|
||||
|
||||
目前唯一 P0 是 `AG99-P0-002`：以 explicit event kind / suggested mode 建立 routing precedence，將 5 分鐘 grouping、fingerprint dedupe 與 single-flight 移到 Agent99 dispatch 之前，並用 backup、provider、reboot、PostgreSQL、host、K3s replay corpus 證明不再 wrong-domain 或重複派工。完成後依序做 secure ingress、RepairCandidate，再回到 reboot state machine live rerun。
|
||||
目前唯一 P0 仍是 `AG99-P0-002`。本機 Agent99 package 已完成 routing/dedupe/card/reboot identity 契約與 synthetic replay；下一步是先完成本分支 source/test/Gitea receipt，再將測試過的 Agent99 package 原子部署到 99，重跑 backup、provider、reboot、PostgreSQL、host、K3s replay corpus 並讀回 runtime manifest。99 尚未取得可用的 no-secret authenticated management channel前，狀態只能是 local green/runtime pending；不能把 `exitCode=0` 或本機測試當 production resolved。其後依序做 secure ingress、RepairCandidate、mode verifier/callback，再回到 reboot state machine live rerun。
|
||||
|
||||
Reference in New Issue
Block a user