From 5963687e6293c7d5f1ef4c727f9a4f90fe0410b8 Mon Sep 17 00:00:00 2001 From: Your Name Date: Fri, 3 Jul 2026 15:14:41 +0800 Subject: [PATCH] fix(reboot): include edge nginx source in slo package --- docs/LOGBOOK.md | 22 +++++++++++++++++++ docs/runbooks/FULL-STACK-COLD-START-SOP.md | 4 +++- docs/runbooks/REBOOT-RECOVERY-SOP.md | 14 ++++++++++-- ...oot-cold-start-backup-recovery-workplan.md | 2 +- .../install-reboot-auto-recovery-slo-110.sh | 3 ++- ...test_reboot_auto_recovery_slo_installer.py | 2 ++ 6 files changed, 42 insertions(+), 5 deletions(-) diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index 91d3cb6a8..c6c733e93 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -1,3 +1,25 @@ +## 2026-07-03 — 15:13 Public maintenance edge SLO runtime readback 收斂 + +**完成內容**： +- 110 `awoooi-reboot-auto-recovery-slo.service` controlled install 已重新套用新版 exporter，timer `enabled/active`，service `Result=success`、`ExecMainStatus=0`；未重啟 Docker / Nginx / K3s / DB / firewall，未讀 secret / token / `.env` / raw sessions / SQLite / auth。 +- 修正 `install-reboot-auto-recovery-slo-110.sh` payload：把 `ops/nginx` 納入 110 SLO package，避免 `public-maintenance-edge-fallback-apply.sh --check` 在 110 runtime 因缺 source Nginx config 被誤判成 `local_nginx_conf_missing`。 +- 110 最新 artifact `/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260703-151219` 已正確讀到 188 live edge：`REMOTE_CONF_READABLE=1`、`LIVE_CONF_HAS_INTERCEPT=0`、`LIVE_CONF_HAS_ERROR_PAGE_503=0`、`LIVE_CONF_HAS_FALLBACK_HEADER=0`、`LIVE_CONF_HAS_INTERNAL_LOCATION=0`、`LIVE_CONF_HAS_API_VIP=1`、`LIVE_CONF_HAS_WEB_VIP=1`、`MAINTENANCE_FILE_PRESENT=0`、`SUDO_N_READY=0`、`EDGE_FALLBACK_READY=0`。 +- 110 textfile 現在輸出 `awoooi_public_maintenance_edge_fallback_ready{scope="public_routes"} 0`，`awoooi_reboot_auto_recovery_slo_blocker_count=8`，active blocker 已包含 `edge_public_maintenance_live_config_drift`，category=`public_maintenance_fallback`、owner_lane=`public_route_maintenance_fallback`。 +- Public Gitea CD 最新可見 run `#4599` / source `803a90f` 已推 deploy marker `20db53a`，但 public jobs API 仍 stale/mismatched；production closeout 仍需 marker readback matched + job succeeded + API scorecard readback。 + +**已跑驗證**： +- `PYTHONPATH=apps/api python3.11 -m pytest scripts/reboot-recovery/tests/test_reboot_auto_recovery_slo_installer.py -q -p no:cacheprovider`：`6 passed`。 +- `bash -n scripts/reboot-recovery/install-reboot-auto-recovery-slo-110.sh scripts/reboot-recovery/reboot-auto-recovery-slo-exporter.sh scripts/reboot-recovery/public-maintenance-edge-fallback-apply.sh`：通過。 +- `python3.11 ops/runner/guard-gitea-runner-pressure.py --root .`：`GITEA_RUNNER_PRESSURE_GUARD_OK workflow_files=12 scheduled_workflows=4 auto_branch_events_on_110=0 generic_runner_labels=0`。 +- `git diff --check`：通過。 + +**仍維持**： +- P0-006 尚未完成：不能宣稱全主機重啟後 10 分鐘內全部恢復。剩餘 active blockers 是 fresh all-host reboot event / 10-minute window / host reachability / Windows99 guest power readback，加上 public maintenance edge live config drift。 +- P0-2 runtime 尚未完成：188 edge 缺 maintenance fallback live config 與 maintenance file，且 `SUDO_N_READY=0`；不得讀密碼、不得直接 reload Nginx、不得用 app / Docker / host restart 掩蓋 raw 502 UX blocker。 + +**下一步**： +- 等最新 CD marker readback matched 後，讀 production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 是否上卷 `edge_public_maintenance_live_config_drift`；若 production API 尚未上卷，先處理 deploy/readback，而不是重跑新的支線。 + ## 2026-07-03 — P0-006 Windows99 VMware autostart check-mode package **完成內容**： diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index 2acac3342..b7656eda6 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -1,6 +1,6 @@ # AWOOOI 全棧冷啟動與主機重啟 SOP -> Version: v1.122 +> Version: v1.123 > Last updated: 2026-07-03 Asia/Taipei > Scope: 99 / 110 / 111 / 112 / 120 / 121 / 188 全棧重啟恢復。112 仍是 Kali / VM guest 訊號，但 2026-06-30 全主機重啟後已納入 10 分鐘 SLO 的必要 boot / power signal；此納入不代表授權任何破壞性 runtime apply。 @@ -76,6 +76,8 @@ v1.121 deployed Windows99 runtime metric overlay readback rule：2026-07-03 14:2 v1.122 public maintenance edge fallback drift metric rule：deploy / reboot window 期間 AWOOOI API / SLO route 只要曾回 raw `502` 且缺 `X-AWOOOI-Fallback` / `Retry-After` / maintenance body，即使稍後自癒為 `200`，仍要保留 P0 blocker。`reboot-auto-recovery-slo-exporter.sh` 必須執行 `public-maintenance-edge-fallback-apply.sh --check`，把 live 188 edge Nginx drift merge 進 `public-maintenance-fallback.json`，並輸出 `awoooi_public_maintenance_edge_fallback_ready{scope="public_routes"}`。當 edge check 顯示 `EDGE_FALLBACK_READY=0` 時，scorecard / API 必須新增 `edge_public_maintenance_live_config_drift`，讓 `public_maintenance_fallback.ready=false`；Alertmanager 必須透過 `PublicMaintenanceEdgeFallbackConfigDrift` 告警。此規則只授權 no-write check / metric / alert；實際 `--apply` 必須有 passwordless sudo、backup、`nginx -t`、reload 與 public header verifier，不得讀密碼、不得重啟 app / Docker / host。 +v1.123 SLO installer edge-source packaging rule：110 SLO installer payload 必須包含 `ops/nginx`、`ops/maintenance` 與 `ops/monitoring`。若 110 runtime `public-maintenance-edge-fallback-apply.sh --check` 先回 `local_nginx_conf_missing`，那是 verifier source package 不完整，必須修 installer payload 並重新套用 verifier；不得把它當 live edge 已驗證或直接跳到 Nginx apply。2026-07-03 15:13 controlled install 已證明修正後 artifact `/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260703-151219` 正確讀到 188 live edge：`REMOTE_CONF_READABLE=1`、VIP `192.168.0.125` API/Web upstream 存在，但 `proxy_intercept_errors` / `error_page` / fallback header / internal maintenance location / maintenance file 均缺，`SUDO_N_READY=0`，textfile 輸出 `awoooi_public_maintenance_edge_fallback_ready{scope="public_routes"} 0` 與 `edge_public_maintenance_live_config_drift`，blocker count `8`。這仍只授權 no-write verifier 與告警，不授權讀密碼、reload Nginx、重啟服務或 host。 + 2026-07-02 110 control-path / Harbor recovery receipt rule：若 Gitea Harbor repair queue 仍保留 `harbor_110_remote_ssh_publickey_auth_stalled`、remote-control unavailable、jobs stale 或 historical failure，但同一輪本地證據同時證明 `wooo` command path ready、110 local Harbor `/v2/` ready、public/internal registry `/v2/` 回 `401`，則該 Gitea Harbor repair 失敗只能列為 historical queue metadata，不得再當成 current SSH blocker。必須用 `/api/v1/agents/harbor-registry-controlled-recovery-receipt` 或同等 validator 合併 `diagnose-110-ssh-publickey-auth.sh`、`recover-110-control-path-and-harbor-local.sh --check`、public Gitea queue readback 與 registry `/v2/` verifier，並把機器可讀結果寫入 `docs/operations/harbor-110-control-path-recovery-readback-2026-07-02.snapshot.json` 類型的 snapshot。2026-07-02 live receipt 顯示：public/internal registry `/v2/` 均為 `401`、latest visible CD `#4335` 為 `Success`、Gitea Harbor repair failure 已是 `historical_after_latest_cd_success=true`；active blockers 收斂為 110 controlled CD lane config / binary / registration / service guardrail、active action container pressure，以及 Gitea CD jobs head-SHA / stale readback mismatch。若 local-console output 只有 `AWOOOI_110_CONTROLLED_CD_LANE_READY` marker，non110 runner parser 不得從 110 `BLOCKER` 行推導 non110 blocker；non110 只有看到 `AWOOOI_NON110_RUNNER_READY` marker 才能列入 active blocker。 2026-07-02 110 controlled CD lane fail-closed enforcer staging rule：110 runner 壓力事故後，legacy / generic runner 仍必須 fail-closed；但 `awoooi-cd-lane-drain.service` 的非 secret staging artifact 不得再被 enforcer 無差別封回 stub。`scripts/reboot-recovery/enforce-110-runner-failclosed.sh` 只有在 `config.yaml` 符合 `capacity <= 1`、只含 `awoooi-host:host` 與 `awoooi-ubuntu:docker://192.168.0.110:5000/awoooi/ci-runner:act-22.04`、binary 是 executable ELF、systemd unit 具備 `ConditionPathExists=/home/wooo/awoooi-cd-lane-drain/data/.runner`、`CPUAccounting` / `MemoryAccounting` / `TasksAccounting` / `NoNewPrivileges` 等 guardrail，且 service `inactive`、`MainPID=0`、未 enabled / 未 masked 時，才可保留 drain config / binary / unit，並輸出 `CONTROLLED_DRAIN_STAGING_ALLOWED=1` 與 textfile metric。此 staging 規則不得讀 token、不得讀 `.runner` 內容、不得註冊 runner、不得啟動 service；若 registration 缺失，readiness verifier 仍必須只留下 `controlled_cd_lane_registration_missing` / `controlled_cd_lane_service_not_active` 類 blocker。若 `CONTROLLED_DRAIN_STAGING_ALLOWED=0` 且 config / binary 又被搬走，優先修 source enforcer / unit guardrail，不要手工反覆補同一組 artifact。 diff --git a/docs/runbooks/REBOOT-RECOVERY-SOP.md b/docs/runbooks/REBOOT-RECOVERY-SOP.md index 801d0ccf6..33deebec8 100644 --- a/docs/runbooks/REBOOT-RECOVERY-SOP.md +++ b/docs/runbooks/REBOOT-RECOVERY-SOP.md @@ -1,9 +1,9 @@ # AWOOOI 重開機恢復 SOP -> **版本**: v5.14 +> **版本**: v5.15 > **最後更新**: 2026-07-03 (台北時間) > **更新者**: Codex -> **觸發事件**: 2026-07-03 14:40 public maintenance edge fallback drift 已接入 SLO exporter / scorecard / alert；raw 502 不再等人工發現 +> **觸發事件**: 2026-07-03 15:13 110 SLO runtime 已正確上卷 public maintenance edge live drift，installer payload 補入 `ops/nginx` --- @@ -21,6 +21,16 @@ ## 架構概覽與依賴圖 +### 2026-07-03 15:13 P0-2 Runtime 覆蓋 + +110 SLO controlled install 已完成，`awoooi-reboot-auto-recovery-slo.timer` 為 `enabled/active`，service `Result=success`、`ExecMainStatus=0`。本輪只 stage SLO verifier source、啟動 verifier oneshot 與寫 textfile metric；未重啟 Docker / Nginx / K3s / DB / firewall，未讀 secret / token / `.env` / raw sessions / SQLite / auth。 + +installer v5.15 規定 payload 必須包含 `ops/nginx`。若缺少 `ops/nginx/awoooi.wooo.work.conf`，110 runtime edge check 會在本地 source gate 先失敗成 `local_nginx_conf_missing`，導致無法讀到 188 live edge truth；這是 verifier packaging bug，不是 Nginx runtime 狀態。完成條件是 110 artifact 內 `public-maintenance-edge-fallback.txt` 進入遠端 readback，且 blocker 必須是 live edge drift。 + +2026-07-03 15:13 live artifact `/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260703-151219` 已讀到 188 edge：`REMOTE_CONF_READABLE=1`、`LIVE_CONF_HAS_INTERCEPT=0`、`LIVE_CONF_HAS_ERROR_PAGE_503=0`、`LIVE_CONF_HAS_FALLBACK_HEADER=0`、`LIVE_CONF_HAS_INTERNAL_LOCATION=0`、`LIVE_CONF_HAS_API_VIP=1`、`LIVE_CONF_HAS_WEB_VIP=1`、`MAINTENANCE_FILE_PRESENT=0`、`SUDO_N_READY=0`、`EDGE_FALLBACK_READY=0`、`BLOCKER edge_public_maintenance_live_config_drift`。110 textfile 必須輸出 `awoooi_public_maintenance_edge_fallback_ready{scope="public_routes"} 0` 與 active blocker `edge_public_maintenance_live_config_drift`；目前 blocker count 為 `8`。 + +此狀態仍不授權 `--apply`。P0-2 完成需要 passwordless sudo / privileged edge apply channel、live conf backup、maintenance file install、`nginx -t`、reload 與 public route header verifier 全綠，或完成外部 L1 cloud/CDN fallback decision record。`SUDO_N_READY=0` 時不得讀密碼、不得 reload Nginx、不得重啟 app / Docker / host。 + ### 2026-07-03 14:40 P0-2 Public Maintenance 現況覆蓋 最新 main `3ebeeeb4b` 的 Gitea CD `#4592` 已推 deploy marker `ee0da93`，production deploy readback matched、job succeeded；deploy window 期間 AWOOOI API health 與 SLO route 曾回 raw `502 Bad Gateway`，沒有 `X-AWOOOI-Fallback`、`Retry-After` 或維護頁 body。後續 public route probe 自癒回 `200`，但 raw 502 exposure 仍必須列為 P0 public UX blocker。 diff --git a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md index 52eeb3155..7366efc36 100644 --- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md +++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md @@ -20,7 +20,7 @@ | 優先 | 狀態 | 工作項 | 最新證據 | 下一步 / 完成條件 | |------|------|--------|----------|-------------------| | P0-1 | BLOCKED_HOST_WINDOWS | 全主機 reboot auto-detection / auto-trigger / 10 分鐘恢復 SLO | 2026-07-03 10:44 production scorecard：`status=blocked_reboot_auto_recovery_slo_not_ready`、`active_blocker_count=10`、`readiness_percent=60`、active blockers 仍含 reboot event / host unreachable / 99+111+Windows99 VMX / guest power，且新增 `windows99_vmware_autostart_config_not_ready`、`windows99_update_no_auto_reboot_policy_not_ready`。no-write artifact `/tmp/awoooi-reboot-continue-20260703-085205`：111 socket probe 仍 timeout / unreachable；99 RDP / VMConnect reachable 但 uptime unknown；188 socket probe reachable，前次 host probe 仍為 `systemd_state=degraded` / `startup_active=failed`。 | 先收斂 99 / Windows99 / VMware 與 111：恢復 no-secret management channel 或取得 validator 可接受的完整 console Verify stdout，讀回 VMX / VM power / host uptime，再 rerun host probe + reboot-event detector；不得 reboot、不得 VM power change、不得讀 Windows 密碼，不得用 RDP clipboard 片段當完成證據。 | -| P0-2 | SOURCE_READY_RUNTIME_EDGE_DRIFT_BLOCKED | Deploy / reboot 期間 public 502 維護頁與外部 fallback | 2026-07-03 最新 main `3ebeeeb4b` 的 Gitea CD `#4592` 已推 deploy marker `ee0da93` 且 deploy readback matched，但 deploy window 期間 AWOOOI API health / SLO route 仍出現 raw `502`、無 fallback header/body。live 188 edge check 固定 drift：`LIVE_CONF_HAS_INTERCEPT=0`、`LIVE_CONF_HAS_ERROR_PAGE_503=0`、`LIVE_CONF_HAS_FALLBACK_HEADER=0`、`LIVE_CONF_HAS_INTERNAL_LOCATION=0`、`MAINTENANCE_FILE_PRESENT=0`、`SUDO_N_READY=0`、`EDGE_FALLBACK_READY=0`。source 已新增 `edge_public_maintenance_live_config_drift` scorecard blocker、`awoooi_public_maintenance_edge_fallback_ready` metric 與 `PublicMaintenanceEdgeFallbackConfigDrift` alert rule。 | 部署新版 exporter / API 後，110 textfile 與 production scorecard 必須主動顯示 edge fallback drift blocker；具備 privileged channel 後才可執行受控 `public-maintenance-edge-fallback-apply.sh --apply`，要求 backup、`nginx -t`、reload、public header verifier 全綠；或完成 L1 external cloud/CDN fallback。不得讀密碼、不得用 app / Docker / host restart 掩蓋 edge fallback drift。 | +| P0-2 | RUNTIME_EDGE_DRIFT_ACTIVE_PROD_API_READBACK_PENDING | Deploy / reboot 期間 public 502 維護頁與外部 fallback | 2026-07-03 15:13 110 SLO controlled install 已套用新版 exporter，並修正 installer payload 缺 `ops/nginx` 導致的 `local_nginx_conf_missing` 誤判。最新 artifact `/home/wooo/reboot-recovery/reboot-auto-recovery-slo-20260703-151219` 正確讀到 188 live edge：`REMOTE_CONF_READABLE=1`、`LIVE_CONF_HAS_INTERCEPT=0`、`LIVE_CONF_HAS_ERROR_PAGE_503=0`、`LIVE_CONF_HAS_FALLBACK_HEADER=0`、`LIVE_CONF_HAS_INTERNAL_LOCATION=0`、`LIVE_CONF_HAS_API_VIP=1`、`LIVE_CONF_HAS_WEB_VIP=1`、`MAINTENANCE_FILE_PRESENT=0`、`SUDO_N_READY=0`、`EDGE_FALLBACK_READY=0`。110 textfile 已輸出 `awoooi_public_maintenance_edge_fallback_ready{scope="public_routes"} 0`、`awoooi_reboot_auto_recovery_slo_blocker_count=8` 與 active blocker `edge_public_maintenance_live_config_drift`。 | 等最新 CD deploy marker readback matched 後，production `/api/v1/agents/reboot-auto-recovery-slo-scorecard` 必須上卷 edge drift；具備 privileged channel 後才可受控 `public-maintenance-edge-fallback-apply.sh --apply`，要求 backup、`nginx -t`、reload、public header verifier 全綠；或完成 L1 external cloud/CDN fallback。不得讀密碼、不得用 app / Docker / host restart 掩蓋 edge fallback drift。 | | P0-3 | PARTIAL_GREEN_DEPLOYED_SOURCE | 所有產品 / 網站版本與資料最新性 | Gitea `main=bc8d3af3f chore(cd): deploy dacdd90 [skip ci]`；CD `#4558` log 顯示 API/Web build/push wrapper 全部 `rc=0`、deploy marker `dacdd90`、production deploy readback matched。Production SLO route 已讀回最新 API 行為：`runtime_metric_runtime_readback_added_blockers=["windows99_vmware_autostart_config_not_ready","windows99_update_no_auto_reboot_policy_not_ready"]`、`windows99_update_no_auto_reboot_ready=false`。Stock freshness 仍為 `status=ok`、`latest_trading_date=2026-07-02`、blockers `[]`。 | 版本層已補到 AWOOOI production；完成整體 P0 仍需收斂 Windows99 VMX / service / policy / power 與 host reboot-event blockers。後續每個 public product 仍要維持 source SHA、deploy marker、runtime endpoint、public route watch、freshness 四層一致。 | | P0-4 | BLOCKED_WINDOWS99_AUTOSTART | 192.168.0.99 VMware 自動啟動與 VM guest 111 / 188 / 120 / 121 / 112 | 10:44 production readback 已把 nested Windows99 evidence 上卷到 active blockers：`windows99_vmware_vmx_missing`、`windows99_vmware_guest_power_not_ready`、`windows99_vmware_autostart_config_not_ready`、`windows99_update_no_auto_reboot_policy_not_ready`；`runtime_metric_runtime_readback_added_blockers` 含 config / policy 兩項，`windows99_update_no_auto_reboot_ready=false`。09:53 no-secret management probe：99 reachable、RDP / Hyper-V VMConnect reachable，WinRM timeout，Mac vantage collector 仍為 `ssh_batchmode_auth_ready=0` / `blocked_ssh_publickey_auth_missing`；completion 仍取決於 VMX / service / policy / power 全綠。 | 先修 111 VMX source 與 VMware service / policy readback，使 check-mode package 能解釋 powered-off aliases；不得從 scorecard 直接重啟 Windows service、改 registry、啟動 / 關閉 VM 或 host reboot。完成條件是 VMX config ready、guest power ready、99 uptime known、all required host reachable，且 active blocker matrix 顯示 service / policy / VMX / power 分開收斂。 | | P0-5 | RUNTIME_READY_BACKUP_RECEIPT_GAP | Gitea / 主機 / DB / 網站 / 服務 / 套件 / 工具 / log 備份監控告警 | Gitea repo bundle readback ready：expected `12`、rows `12`、missing `0`、failed `0`、sample restore dry-run ok；backup core green。2026-07-03 source / runtime 已部署 `awoooi_backup_alert_receipt_*` 指標與 Prometheus rules；110 exporter 讀回 88 個 stage requirement、188 讀回 12 個 stage requirement，`BackupAlertReceiptMetricMissing*` inactive，`BackupAlertReceiptStageMissing` 已修成每 `host / receipt_channel` 聚合 pending：110 一條、188 一條。 | 補 `/backup/alert-receipts/*.last_success` 脫敏 marker；下一層仍要補 Gitea full dump、DB/settings/issues/packages/LFS、所有工具與 log 全量備份監控。 | diff --git a/scripts/reboot-recovery/install-reboot-auto-recovery-slo-110.sh b/scripts/reboot-recovery/install-reboot-auto-recovery-slo-110.sh index 6e9ed9340..88130148f 100644 --- a/scripts/reboot-recovery/install-reboot-auto-recovery-slo-110.sh +++ b/scripts/reboot-recovery/install-reboot-auto-recovery-slo-110.sh @@ -218,6 +218,7 @@ create_payload() { scripts/reboot-recovery \ scripts/security \ apps/api/migrations/awooop_conversation_event_hot_path_indexes_2026-07-01.sql \ + ops/nginx \ ops/maintenance \ ops/monitoring \ docs/runbooks/PUBLIC-MAINTENANCE-FALLBACK-RUNBOOK.md @@ -230,7 +231,7 @@ case "$MODE" in cat < None: assert "verify_command=bash scripts/reboot-recovery/install-reboot-auto-recovery-slo-110.sh --verify-only" in result.stdout assert "would_include_source_gates=" in result.stdout assert "awooop_conversation_event_hot_path_indexes_2026-07-01.sql" in result.stdout + assert "ops/nginx" in result.stdout assert "would_enable_timer=awoooi-reboot-auto-recovery-slo.timer" in result.stdout @@ -58,6 +59,7 @@ def test_installer_is_limited_to_verifier_timer_not_product_restarts() -> None: assert 'payload_dir="$(mktemp -d "${TMPDIR:-/tmp}/${UNIT_NAME}.XXXXXX")"' in text assert 'payload="${payload_dir}/payload.tar.gz"' in text assert "apps/api/migrations/awooop_conversation_event_hot_path_indexes_2026-07-01.sql" in text + assert "ops/nginx" in text assert "ops/maintenance" in text assert "ops/monitoring" in text assert "PUBLIC-MAINTENANCE-FALLBACK-RUNBOOK.md" in text