fix(recovery): ignore stale docker cpu attribution
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Failing after 33s
AWOOOI Harbor 110 Local Repair / workflow-shape (push) Successful in 0s
CD Pipeline / build-and-deploy (push) Has been skipped
AWOOOI Harbor 110 Local Repair / harbor-110-local-repair (push) Failing after 1m41s
CD Pipeline / post-deploy-checks (push) Has been skipped

This commit is contained in:
Your Name
2026-07-01 14:06:51 +08:00
parent e580954e82
commit 1ac8808607
7 changed files with 318 additions and 4 deletions

View File

@@ -1,3 +1,23 @@
## 2026-07-01 — 14:05 110 stale Docker stats attribution hardening
**照主線修正的問題**
- live 110 diagnosis 又回到高壓:`NODE_LOAD1=29.9``NODE_LOAD5=22.66``NODE_LOAD1_PER_CPU=2.49``NODE_LOAD_CLASSIFIER=high_load`,但 `docker_stats.prom``node_textfile_mtime_seconds` 已 stale 約 `107565s`;因此舊值 `docker_container_cpu_cores{container_name="gitea"}=3.4019` 不能再作為當下 CPU 元兇。
- `scripts/ops/host-sustained-load-controller.py` 已新增 Docker stats freshness gate。當 textfile 超過 `300s` 時,`top_container_cpu` 會變 `null`,舊 container 樣本只保留在 `top_container_cpu_untrusted`;同一份 live metrics 現在分類為 `blocked_unknown_sustained_load_requires_source_specific_playbook`,不再誤報 `blocked_gitea_queue_or_hook_backlog_requires_playbook`
- `scripts/ops/host-sustained-load-evidence.py` 同步把 stale container 樣本放到 `top_containers_untrusted``top_containers=[]`recommendation 轉成 `source_specific_playbook_required`
- `scripts/reboot-recovery/reboot-auto-recovery-slo-scorecard.py` 已支援 `docker_stats.fresh=false` / `top_containers_fresh=false`SLO 仍保留 `host_pressure_high_load`,但不再產生 `host_110_gitea_cpu_pressure`;下一步改為 `restore_docker_stats_textfile_exporter_then_collect_sanitized_host_pressure_no_restart_no_secret_read`
- live route / cold-start 仍 blocked`https://registry.wooo.work/v2/` 502、`https://harbor.wooo.work/api/v2.0/health` 502`/tmp/awoooi-full-stack-cold-start-after-e580954e8-20260701-135922.log``PASS=61 WARN=9 BLOCKED=6`。核心 blocker 仍是 110 control path / local recovery package / Harbor `/v2`,不是已證實的 Gitea CPU backlog。
**驗證**
- `python3.11 -m pytest scripts/ops/tests/test_host_runaway_process_exporter.py -q``19 passed`
- `python3.11 -m pytest scripts/reboot-recovery/tests/test_reboot_auto_recovery_slo_scorecard.py -q``9 passed`
- `python3.11 -m py_compile scripts/ops/host-sustained-load-controller.py scripts/ops/host-sustained-load-evidence.py scripts/reboot-recovery/reboot-auto-recovery-slo-scorecard.py`:通過。
- `python3.11 ops/runner/guard-gitea-runner-pressure.py --root .``GITEA_RUNNER_PRESSURE_GUARD_OK workflow_files=12 scheduled_workflows=4 auto_branch_events_on_110=0 generic_runner_labels=0`
**邊界**:只讀 node exporter / public routes / public Gitea queue未讀 secret / token / `.env` / raw sessions / SQLite / auth未使用 GitHub / `gh` / GitHub API未 workflow_dispatch未重啟主機、未 restart Docker / Nginx / K3s / DB / firewall未發 process signal。
**下一步**
- P0 不切支線:先讓 110 local recovery package 可在 110 console 或恢復後 SSH control path 執行,再跑 `recover-110-control-path-and-harbor-local.sh --check`;若只剩 exporter freshness 缺口,先恢復 Docker stats textfile exporter再收集 sanitized host pressure。
## 2026-07-01 — 14:55 Work Items 顯示 AI Loop LOG source tags
**照主線修正的問題**

View File

@@ -72,6 +72,8 @@ v1.82 bounded summary rule`post-start-quick-check.sh` 與 `188-host-hygiene-m
2026-07-01 13:35 110 CPU/load 判讀規則更新:`docker_stats.prom` 必須先看 mtime / freshness超過 300 秒不得作為當下 container CPU 歸因或 blocker若 110 `node_load5` 高但 Prometheus CPU mode 仍有 idle、`awoooi_host_gitea_actions_active_process_count=0`、orphan browser count=0主 blocker 不得誤寫成 Gitea / Playwright / Stock smoke CPU。此時優先看 `diagnose-110-ssh-publickey-auth.sh``NODE_LOAD_CLASSIFIER``DOCKER_STATS_TEXTFILE_FRESHNESS``SYSTEMD_UNIT ... classifier=systemctl_show_timeout|systemctl_timeout_budget_exhausted`。外部 SSH userauth timeout 時cold-start 必須輸出 `SSH_110_BLOCKER remote_control_channel_unavailable``SSH_110_NEXT_ACTION local_console_run_recover_110_control_path_and_harbor_local_check`;下一步是 110 本機 console / 已恢復 control path 執行 `recover-110-control-path-and-harbor-local.sh --check`,不是重跑 Harbor workflow 或用舊 docker stats 指認 Gitea。
2026-07-01 14:05 追加 controller / SLO stale-attribution guard`host-sustained-load-controller.py``host-sustained-load-evidence.py` 必須把超過 `300s` 的 Docker stats 樣本標成 untrusted`top_container_cpu` / `top_containers` 不得使用 stale `docker_container_cpu_cores`,舊值只能留在 `top_container_cpu_untrusted` / `top_containers_untrusted` 當證據。`reboot-auto-recovery-slo-scorecard.py` 若收到 `docker_stats.fresh=false``top_containers_fresh=false`,只能保留 `host_pressure_high_load``host_container_cpu_attribution_stale`,不得產生 `host_110_gitea_cpu_pressure`。此時下一步固定為恢復 Docker stats textfile exporter 或收集 sanitized host pressure且仍不得重啟 Docker / Nginx / K3s / DB / firewall、不得恢復 generic runner、不得用 stale Gitea CPU 樣本取消或 drain 任何工作。
2026-06-25 20:25 orphan Chrome cleanup / scorecard refresh supersedes the 20:11 CPU wording. 110 high CPU was traced to two `stockplatform-review-bulk-ux` Chrome process groups `2756503` and `2829627` with root Chrome process `PPID=1`, elapsed about 5h, no active parent smoke, and sustained GPU/renderer CPU. With user approval, only those two process groups received targeted `SIGTERM` at 20:24. Post-check showed no remaining PGID entries; `vmstat` showed CPU idle around `85-90%`, `si/so=0`, and no immediate swap thrash. No Docker/systemd/Nginx/firewall/K8s action, CI cancellation, manual data ingestion, manual DB write, Wazuh/SOC runtime change, or secret read was performed. The 20:25 full post-start wrapper then returned cold-start `PASS=89 WARN=0 BLOCKED=0`, but overall `POST_START_QUICK_CHECK PASS=37 WARN=2 BLOCKED=1`, `RESULT=BLOCKED`, because StockPlatform data freshness was still blocked at that time and DR remained incomplete.
2026-06-25 20:11 StockPlatform cron-source recovery supersedes the 19:35 source-version wording. StockPlatform Gitea `main` and live `/home/wooo/stockplatform-v2` are now at `fb91aa4c6272469d1d26e0820169629eac17d28a fix(ops): restore production cron recovery entrypoints`; six missing production cron entrypoint scripts are restored, `run-intelligence-sync.sh` contains the Docker-backed `psql` shim, and live contract check confirms every `scripts/ops/*.sh` referenced by `install-production-cron.sh` exists. The only live write performed for StockPlatform recovery was a fast-forward `git pull --ff-only origin main` on 110; no Docker/systemd/Nginx/firewall/K8s restart, manual ingestion run, manual DB write, or secret read was performed. Natural cron evidence after the pull is now green for the repaired entrypoints: `source-remediation-queue` 19:56 and 20:00 succeeded, `market-index-ingestion` 20:00 succeeded, `price-ingestion` 20:02 succeeded, `margin-short-ingestion` 20:05 succeeded, `chips-ingestion` 20:06 succeeded, and `ai-recommendation-pipeline` 20:10 ran but correctly produced the internal blocker `core_margin_short_daily_incomplete,official_margin_short_daily_official_pending`. StockPlatform `/api/v1/system/freshness` therefore still returns `status=blocked` because the 2026-06-25 official margin-short source is pending and `ai.recommendations` must stay on 2026-06-24 until that gate clears. This is no longer a route, source-version, or missing-cron-script blocker; it is a product-data freshness blocker waiting on official source availability and the next valid AI pipeline run.