fix(recovery): expose reboot slo machine readback
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 33s
CD Pipeline / post-deploy-checks (push) Has been cancelled
CD Pipeline / build-and-deploy (push) Has been cancelled

This commit is contained in:
ogt
2026-07-01 20:35:22 +08:00
parent 68c982fe6a
commit 17b6bf36ce
6 changed files with 358 additions and 3 deletions

View File

@@ -1,3 +1,26 @@
## 2026-07-01 — 20:34 P0-006 reboot SLO machine-readback source closure
**照主線修正的問題**
- Gitea / deploy closure 已恢復後production `/api/v1/agents/delivery-closure-workbench` 仍讀回 P0-006 `blocked_reboot_auto_recovery_slo_not_ready`、readiness `18%`、active blockers `12`;這是目前主線,不是 GitHub / Gitea repo 消失。
- `docs/operations/awoooi-reboot-auto-recovery-slo-scorecard.snapshot.json` 只有 raw blocker/source fields缺少 machine-readable `readback` / `rollups` / `summary`,容易讓 MCP / RAG / KM / PlayBook /泛用 readback 讀成空值或誤判。
- `scripts/reboot-recovery/reboot-auto-recovery-slo-scorecard.py` 現在會直接輸出 `required_checks``readback``rollups``summary`,保留 fail-closed truth`active_blocker_count=12``readiness_percent=18``runtime_write_authorized=false``host_reboot_authorized=false``workflow_trigger_authorized=false``secret_value_collection_allowed=false`
- API loader `reboot_auto_recovery_slo_scorecard.py` 額外回傳 `source_controls`,並修正「無 blocker 且可 claim SLO 時 readiness 必須為 100」的一致性規則。
- committed P0-006 snapshot 已回填 derived readback 欄位,但沒有把任何 runtime blocker 改成 green。
**驗證**
- `python3 -m py_compile scripts/reboot-recovery/reboot-auto-recovery-slo-scorecard.py apps/api/src/services/reboot_auto_recovery_slo_scorecard.py`:通過。
- `DATABASE_URL=sqlite+aiosqlite:////tmp/awoooi-codex-api-test.db PYTHONPATH=apps/api python3.11 -m pytest scripts/reboot-recovery/tests/test_reboot_auto_recovery_slo_scorecard.py apps/api/tests/test_reboot_auto_recovery_slo_scorecard_api.py apps/api/tests/test_delivery_closure_workbench_api.py -q``18 passed`
- `DATABASE_URL=sqlite+aiosqlite:////tmp/awoooi-codex-api-test.db PYTHONPATH=apps/api python3.11 -m pytest ops/runner/test_cd_controlled_runtime_profile.py -q``39 passed`
- `git diff --check`:通過。
**仍維持**
- 沒有使用 GitHub / `gh` / GitHub API / GitHub Actions。
- 沒有讀 secret / token / `.env` / raw sessions / SQLite / auth。
- 沒有重啟主機,沒有 Docker / Nginx / K3s / DB / firewall restart沒有 workflow_dispatch沒有 runtime write。
**下一步**
- commit / push 後讀回 Gitea CD部署後 production 應直接讀到 P0-006 `readback` / `rollups` / `summary``source_controls`,再依 safe_next_step 繼續處理 boot-triggered SLO timer / host probe evidence仍不得把一般「繼續」解讀為 reboot 授權。
## 2026-07-01 — 19:32 110 startup fail-closed stub 修復 / systemd degraded 清零
**照主線修正的問題**
@@ -48,7 +71,6 @@
**邊界**:未讀 secret / token / `.env` / raw sessions / SQLite / auth未使用 GitHub / `gh` / GitHub API未重啟主機未 restart Docker / Nginx / K3s / DB / firewall未 workflow_dispatch未恢復 generic runner。
**下一步**P0 轉入 backup / DR escrow / alert warning 收斂;先處理 `ESCROW_MISSING_COUNT=5` 與 backup stale / failed-component warnings再回來做 188 Nginx privileged source apply 取代 SignOz temporary bridge。
## 2026-07-01 — 18:47 SignOz public route source drift 修正
**照主線修正的問題**

View File

@@ -146,6 +146,49 @@
"summary_present": true,
"wazuh_dashboard_degraded": true
},
"readback": {
"active_blocker_count": 12,
"active_blockers": [
"all_required_hosts_not_in_10_minute_reboot_window",
"backup_core_green_not_1",
"host_188_service_green_not_1",
"host_boot_observation_older_than_target_window",
"host_unreachable_after_reboot",
"host_uptime_unknown",
"local_disk_free_below_minimum",
"post_start_blocked_not_zero",
"product_data_green_not_1",
"reboot_event_required_host_unreachable",
"service_green_not_1",
"wazuh_dashboard_degraded"
],
"blocked_by_fresh_reboot_window_only": false,
"host_reboot_authorized_by_this_scorecard": false,
"readiness_percent": 18,
"required_checks": {
"backup_core_green": false,
"can_claim_slo": false,
"fresh_reboot_window_observed": false,
"host_188_service_green": false,
"product_data_green": false,
"required_hosts_observed": true,
"required_hosts_reachable": false,
"service_green": false,
"source_controls_present": true,
"stockplatform_freshness_ok": false,
"stockplatform_ingestion_ok": false
},
"runtime_write_authorized_by_this_scorecard": false,
"safe_next_step": "deploy_boot_triggered_slo_timer_and_collect_all_host_boot_probe_then_rerun_scorecard_until_status_slo_ready",
"secret_value_collection_allowed": false,
"source_controls_present": true,
"source_id": "reboot_auto_recovery_slo_scorecard",
"status": "blocked_reboot_auto_recovery_slo_not_ready",
"target_minutes": 10,
"workflow_trigger_authorized_by_this_scorecard": false,
"workplan_id": "P0-006",
"workplan_title": "主機重啟自動偵測、自動觸發與 10 分鐘恢復 SLO"
},
"reboot_event_detection": {
"all_required_hosts_in_reboot_window": false,
"all_required_hosts_observed": false,
@@ -165,6 +208,53 @@
"111"
]
},
"required_checks": {
"backup_core_green": false,
"can_claim_slo": false,
"fresh_reboot_window_observed": false,
"host_188_service_green": false,
"product_data_green": false,
"required_hosts_observed": true,
"required_hosts_reachable": false,
"service_green": false,
"source_controls_present": true,
"stockplatform_freshness_ok": false,
"stockplatform_ingestion_ok": false
},
"rollups": {
"active_blocker_count": 12,
"backup_core_green": false,
"blocked_by_fresh_reboot_window_only": false,
"can_claim_all_services_recovered_within_target": false,
"capacity_below_minimum": true,
"capacity_checked": true,
"capacity_free_gib": 0.751,
"capacity_min_free_gib": 2.0,
"completed_check_count": 2,
"host_188_service_green": false,
"latest_verify_only_metric_blocker_count": 0,
"latest_verify_only_metric_last_run_timestamp": 0,
"latest_verify_only_metric_max_host_uptime_seconds": 0,
"latest_verify_only_metric_present": false,
"latest_verify_only_metric_ready": 0,
"missing_host_count": 0,
"observed_host_count": 7,
"post_start_blocked": 8,
"product_data_green": false,
"readiness_percent": 18,
"required_check_count": 11,
"service_green": false,
"source_control_count": 14,
"source_control_ready_count": 14,
"source_controls_present": true,
"stale_host_count": 5,
"stockplatform_freshness_blocker_count": 1,
"stockplatform_freshness_status": "not_configured",
"stockplatform_ingestion_blocker_count": 1,
"stockplatform_ingestion_status": "not_configured",
"unknown_uptime_host_count": 1,
"unreachable_host_count": 1
},
"safe_next_step": "deploy_boot_triggered_slo_timer_and_collect_all_host_boot_probe_then_rerun_scorecard_until_status_slo_ready",
"schema_version": "awoooi_reboot_auto_recovery_slo_scorecard_v1",
"sla_recovery_eta": {
@@ -259,6 +349,27 @@
"successful_source_run_ids": []
}
},
"summary": {
"github_api_used": false,
"reboot_auto_recovery_active_blocker_count": 12,
"reboot_auto_recovery_backup_core_green": false,
"reboot_auto_recovery_can_claim_slo": false,
"reboot_auto_recovery_host_188_service_green": false,
"reboot_auto_recovery_observed_host_count": 7,
"reboot_auto_recovery_product_data_green": false,
"reboot_auto_recovery_readiness_percent": 18,
"reboot_auto_recovery_safe_next_step": "deploy_boot_triggered_slo_timer_and_collect_all_host_boot_probe_then_rerun_scorecard_until_status_slo_ready",
"reboot_auto_recovery_service_green": false,
"reboot_auto_recovery_source_controls_present": true,
"reboot_auto_recovery_stale_host_count": 5,
"reboot_auto_recovery_status": "blocked_reboot_auto_recovery_slo_not_ready",
"reboot_auto_recovery_stockplatform_freshness_status": "not_configured",
"reboot_auto_recovery_stockplatform_ingestion_status": "not_configured",
"reboot_auto_recovery_workplan_id": "P0-006",
"runtime_write_authorized": false,
"secret_values_collected": false,
"workflow_trigger_performed": false
},
"target_minutes": 10,
"target_seconds": 600
}