1415 lines
81 KiB
Markdown
1415 lines
81 KiB
Markdown
# AWOOOI 全棧冷啟動與主機重啟 SOP
|
||
|
||
> Version: v1.20
|
||
> Last updated: 2026-06-14 Asia/Taipei
|
||
> Scope: 110 / 120 / 121 / 188 full-stack reboot recovery. 112 Kali is recorded as P3 optional and is not part of this recovery path.
|
||
|
||
---
|
||
|
||
## 0. 最新 Live Baseline 與釋出判定
|
||
|
||
本節是每次接手、開機、關機、重啟後的第一個判定錨點。若日期不是今天，必須先重跑 live check，再更新本節與 `docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md`。
|
||
|
||
| 項目 | 2026-06-14 18:15 Asia/Taipei live result | 判定 |
|
||
|------|-------------------------------------------|------|
|
||
| Overall recovery readiness | `97%` | `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` |
|
||
| P0 host / K3s recovery | `100%` | `DONE` |
|
||
| P1 backup / alert / escrow | `92%` | `BLOCKED_DR_ESCROW` |
|
||
| P2 service / data truth | `99%` | `VERIFIED_WORKLOAD_BALANCED_WITH_KM_WARN` |
|
||
| P3 docs / automation contracts | `100%` | `DONE_WITH_VALIDATION_GAP` |
|
||
| 110 host runtime | `fwupd-refresh.timer` intentionally disabled/inactive after non-runtime firmware metadata refresh failed units were classified; `systemctl --failed` returns `0 loaded units listed`; rollback is `sudo systemctl enable --now fwupd-refresh.timer` | `GREEN_WITH_FWUPD_TIMER_DISABLED` |
|
||
| 120 reachability | ping OK, SSH OK, boot around `2026-06-14 02:23`, K3s active, node `mon Ready` | `GREEN` |
|
||
| 121 reachability | ping OK, SSH OK, failed units `0` | `GREEN` |
|
||
| 188 host runtime | production services green; host degraded only by `certbot.service` and `snap.certbot.renew.service` | `GREEN_WITH_CERTBOT_DEBT` |
|
||
| K3s node state | `mon Ready control-plane`, `mon1 Ready control-plane`; latest readback keeps API/Web split across 120 / 121; bad pods `0`; retained failed Job `km-vectorize-29689620` is the only WARN | `GREEN_WORKLOAD_BALANCED_WITH_KM_WARN` |
|
||
| 110 -> 120 / 188 SSH trust | 00:33 cold-start exposed stale `known_hosts`; backup `/home/wooo/.ssh/known_hosts.before-120-188-refresh.20260613-003416`; final repair backup `/home/wooo/.ssh/known_hosts.before-120-188-final-refresh.20260613-011949`; CD fix `80e6ec1a` moves deploy trust to `/home/wooo/.ssh/deploy_known_hosts`; 01:28 global `known_hosts` still contains 120 / 188 and was not clobbered by deploy marker `e4a349bc` | `GREEN_WITH_GUARDRAIL` |
|
||
| Backup status | 18:14 status: 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-14 02:40:22` | `GREEN_WITH_DR_ESCROW_WARNING` |
|
||
| Offsite sync / verify | 01:28 textfile: `awoooi_backup_offsite_remote_verify_ok=1`, `full_verify_fresh=1`, all 13 repos have `snapshot_count=1` and `snapshot_latest_only=1`; latest scheduled verifier log is 2026-06-12 07:20 | `GREEN` |
|
||
| Backup / cold-start alerts | 01:27 live visibility check confirms Prometheus and Alertmanager expose the 5 required credential escrow gap alerts; Prometheus rules API has all five required alert names healthy; label contract check loads 24 baseline backup alert rules | `GREEN_WITH_EXPECTED_REDLIGHTS` |
|
||
| Cold-start scorecard | 18:15 高價值配置 Owner Packet 前台同步後只讀 scorecard：`PASS=82 WARN=1 BLOCKED=0`；remaining WARN 仍是 K8s failed Job `km-vectorize-29689620`。Public routes / TLS、momo DB parity、backup exporters、120/121 K3s 與 AWOOOI API/Web 皆通過。 | `DEGRADED_NO_BLOCKERS` |
|
||
| momo DB parity | `4571|4571|2026-06-01|2026-06-07|2026-06-01|2026-06-07` | `GREEN` |
|
||
| momo scheduler | container healthy; scorecard reads `SCHEDULER_RECENT_ACTIVITY 1009`; detector widened and deployed to 110 | `GREEN` |
|
||
| ArgoCD app health | Latest live app source revision `0a4766ddc94b0690824ce3deba5c6b9a69764f94` remains `Synced / Degraded` because official Job `km-vectorize-29689620` failed and CronJob health reports `CronJob has not completed its last execution successfully`. Latest repo docs head is `0a4766dd`, but runtime image readback is from deploy marker `16c6b983` / feature `e999c16b`. CronJob `lastScheduleTime=2026-06-13T19:00:00Z` and `lastSuccessfulTime=2026-06-04T11:00:37Z`; failed Job is retained, but its failed Pod/log are not currently available, so the next official 03:00 run remains the required read-only evidence gate. | `WAITING_OFFICIAL_KM_VECTORIZE_RUN` |
|
||
| Workload balancing | Live API/Web/Worker/CronJob image is `e999c16b3435f197b78fe2adfeec1c4faa6c4675`; API/Web pods remain split across `mon` / `mon1`, Worker single replica remains healthy on `mon` | `GREEN` |
|
||
| Credential escrow | 5 non-secret evidence markers missing | `BLOCKED` |
|
||
|
||
Release rule:
|
||
|
||
```text
|
||
Do not declare full cold-start green unless the latest scorecard has `WARN=0` and `BLOCKED=0`.
|
||
Do not declare aggregate backup green unless latest `backup-status` has `core_blockers=0`.
|
||
Do not declare DR scorecard complete while credential escrow markers are missing.
|
||
```
|
||
|
||
2026-06-14 18:15 live rule:
|
||
|
||
```text
|
||
110 / 120 / 121 / 188 core service recovery remains available, but the latest 18:15 scorecard is DEGRADED because `WARN=1`.
|
||
GO for controlled runner/CD release; keep AI auto-remediation governed by normal gates.
|
||
NO-GO for "DR complete" while credential escrow evidence markers are missing.
|
||
Do not fake or silence credential escrow alerts; they are the remaining correct DR red light.
|
||
GO for "AWOOOI core workload balanced"; topology spread is in Gitea main / ArgoCD live and API/Web placement proves max skew <= 1.
|
||
NO-GO for "full cold-start green" until `km-vectorize` failed Job is cleared by an official successful run.
|
||
NO-GO for "ArgoCD fully healthy" until `km-vectorize` updates `lastSuccessfulTime` after an official scheduled Job, not a manual `UnexpectedJob`.
|
||
NO-GO for any CD workflow that writes deploy host keys into `/home/wooo/.ssh/known_hosts`; deploy jobs must use an isolated `UserKnownHostsFile=/home/wooo/.ssh/deploy_known_hosts`.
|
||
Current allowed wording: "core service and backup are available; 110 failed units are cleared after intentionally disabling `fwupd-refresh.timer`; high-value config Owner Packet 前台同步後 recovery readback shows no service regression; cold-start is degraded only by the `km-vectorize` official Job failure; DR complete still blocked by credential escrow; `km-vectorize` failed Job is retained but failed Pod/log are currently absent, so the next official 03:00 run remains the evidence gate."
|
||
```
|
||
|
||
After any future 120 recovery, rerun this exact chain from 110:
|
||
|
||
```bash
|
||
/backup/scripts/backup-configs.sh
|
||
/backup/scripts/backup-all.sh
|
||
/backup/scripts/sync-offsite-backups.sh --mode sync
|
||
/backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color
|
||
/home/wooo/scripts/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1
|
||
```
|
||
|
||
## 0.1 When To Use This
|
||
|
||
Use this SOP when any of these happen:
|
||
|
||
- 110/120/121/188 reboot unexpectedly.
|
||
- All services are abnormal after a power/network event.
|
||
- K3s is stuck `activating`.
|
||
- Host load remains high during startup and service health is mixed.
|
||
- Monitoring, alerting, CD, AI auto-repair, and Docker Compose services disagree about the real state.
|
||
|
||
The rule is simple: **recover the dependency chain, not the loudest symptom.**
|
||
|
||
## 0.2 啟動判定分層
|
||
|
||
重啟後不能只用一個訊號宣稱完成。每台主機與整個平台都必須分四層判定：
|
||
|
||
| 層級 | 代表意義 | 最低證據 | 不代表 |
|
||
|------|----------|----------|--------|
|
||
| `HOST_POWERED` | 主機或 VM 看起來已通電 | console / hypervisor 顯示 running，或 LAN ARP 開始出現 | OS 已完成開機 |
|
||
| `HOST_BOOTED` | OS 已進入可互動狀態 | ping OK、SSH port open、`who -b` 有本次 boot time | systemd / Docker / K3s 已健康 |
|
||
| `HOST_READY` | 主機基礎服務可承接下一層 | `systemctl is-system-running` 非 degraded；failed units 可解釋；cron / docker / DB / K3s 依角色正常 | public route 或業務資料已正常 |
|
||
| `SERVICE_READY` | 主機承載服務可用 | 服務 health、port、container health、DB / Redis / K3s / Harbor / Alertmanager checks 通過 | 備份、排程、告警、資料一致性已驗證 |
|
||
| `FULL_STACK_GREEN` | 可以宣稱重啟恢復完成 | cold-start scorecard `WARN=0`、`BLOCKED=0`，備份/offsite/DB/告警/排程都綠 | 120 不可達時永遠不能宣稱 |
|
||
|
||
2026-06-12 的 110/120 事故收斂判定是：
|
||
|
||
```text
|
||
110 HOST_READY = yes
|
||
120 HOST_READY = yes
|
||
Core public services SERVICE_READY = yes
|
||
FULL_STACK_GREEN = yes, because cold-start scorecard is PASS=83 WARN=0 BLOCKED=0
|
||
DR_COMPLETE = no, because credential escrow evidence is incomplete
|
||
```
|
||
|
||
所有回報必須使用這組詞，避免把「服務面可用」誤報成「整體 DR 完成」。
|
||
|
||
---
|
||
|
||
## 1. Golden Startup Order
|
||
|
||
```text
|
||
0. Freeze automation and preserve evidence
|
||
1. Physical/network layer
|
||
2. 188 data layer
|
||
3. 110 registry/observability layer
|
||
4. 120/121 K3s layer
|
||
5. AWOOOI workload layer
|
||
6. Public routes and alert chain
|
||
7. High-load batch/consumer/crawler services
|
||
8. Runner/CD
|
||
9. AI auto-remediation
|
||
10. 112 Kali scanner, if needed
|
||
```
|
||
|
||
Never start runner/CD before 188 PostgreSQL, 110 Harbor, K3s nodes, and AWOOOI API are healthy.
|
||
|
||
### 1.1 Dependency Graph
|
||
|
||
```mermaid
|
||
flowchart TD
|
||
network["P0 network: LAN, ARP, SSH"] --> data188["188 data: PostgreSQL, Redis, momo DB, SignOz"]
|
||
network --> obs110["110 registry/observability: Harbor, Gitea, Prometheus, Alertmanager, Sentry"]
|
||
data188 --> k3s["120/121 K3s: server, agent, VIP, NodePorts"]
|
||
obs110 --> k3s
|
||
k3s --> workload["AWOOOI workload: API, Web, K8s Secrets"]
|
||
workload --> alertchain["Alert chain: Alertmanager webhook, Telegram"]
|
||
workload --> public["Public routes: awoooi.wooo.work, mo.wooo.work"]
|
||
public --> schedules["Schedules: cron, CronJobs, backups, exporters"]
|
||
schedules --> highload["High-load release: crawlers, Snuba, ClickHouse merges, runners/CD"]
|
||
highload --> ai["AI auto-remediation: limited execution"]
|
||
```
|
||
|
||
This is also captured in the machine-readable baseline:
|
||
|
||
```text
|
||
ops/reboot-recovery/full-stack-cold-start-baseline.yml
|
||
```
|
||
|
||
The YAML baseline is the source of truth for:
|
||
|
||
- hosts, roles, and SSH users
|
||
- phase ordering
|
||
- service startup dependencies
|
||
- endpoint success codes
|
||
- schedule freshness thresholds
|
||
- stateful-service protection boundaries
|
||
- AI automation release gates
|
||
|
||
### 1.2 Phase Gate Logic
|
||
|
||
Each phase has the same decision rule:
|
||
|
||
| Result | Meaning | Action |
|
||
|--------|---------|--------|
|
||
| `BLOCKED` | A dependency required by later phases is down. | Stop phase release and fix the first blocked gate. |
|
||
| `WARN` | Core dependency passed, but confidence is incomplete. | Continue diagnosis, but do not release runner/CD/AI full execution. |
|
||
| `GREEN` | All checks in scope passed. | Release the next phase only. |
|
||
|
||
The cold-start flow is intentionally conservative:
|
||
|
||
```text
|
||
P0 network green
|
||
-> P0 188 data green
|
||
-> P0 110 registry/observability green
|
||
-> P1 K3s green
|
||
-> P2 workload + alert chain green
|
||
-> P2 public routes green
|
||
-> P2 schedules green
|
||
-> P3 high-load services and runners/CD
|
||
-> AI auto-remediation limited execution
|
||
```
|
||
|
||
The final release condition is not "containers are running". It is:
|
||
|
||
```text
|
||
PASS > 0
|
||
WARN = 0
|
||
BLOCKED = 0
|
||
Result: GREEN
|
||
```
|
||
|
||
### 1.3 重啟 GO / NO-GO 決策樹
|
||
|
||
每次維護前先用這張表決定是否可以重啟，以及重啟後可以宣稱到哪個層級。
|
||
|
||
| 情境 | GO / NO-GO | 可做範圍 | 完成宣告上限 |
|
||
|------|------------|----------|--------------|
|
||
| 03:00 offsite sync 正在跑 | `NO-GO` | 只讀觀察，等待 sync 結束後 verifier | 不宣告維護完成 |
|
||
| 120 不可達，但只重啟 110 | `CONDITIONAL GO` | 只可宣稱 110 / public service recovery；不可跑 120 backup fix | `SERVICE_READY`，不可 `FULL_STACK_GREEN` |
|
||
| 188 data layer 不健康 | `NO-GO` | 先修 PostgreSQL / Redis / Docker / SignOz / momo DB | 不釋出 K3s / runner / AI |
|
||
| 110 Harbor / Registry 不健康 | `NO-GO for K3s deploy` | 先修 registry；K3s 可能 image pull 失敗 | 不釋出 CD / deploy |
|
||
| 120 / 121 都 Ready，offsite verifier 綠 | `GO` | 可做完整 cold-start release chain | 需 scorecard `WARN=0 BLOCKED=0` |
|
||
| credential escrow marker 缺失 | `GO for service reboot`，`NO-GO for DR complete` | 可恢復服務；不可宣稱 DR scorecard complete | `SERVICE_READY` 或 `BLOCKED_DR_ESCROW` |
|
||
| Alertmanager required rules 不可見 | `NO-GO for unattended window` | 先修 alert rules / drift guard | 不釋出 AI auto-remediation |
|
||
|
||
GO 只代表允許執行指定範圍，不代表完成。完成一定要回到 §15 Done Criteria。
|
||
|
||
---
|
||
|
||
## 2. Automation Freeze
|
||
|
||
Cold start creates noisy metrics and partial failures. During P0/P1, keep automation in observe-only mode.
|
||
|
||
| Item | Cold-start policy | Reason |
|
||
|------|-------------------|--------|
|
||
| Gitea/GitHub runners | Last | Build jobs can saturate 110 CPU/RAM. |
|
||
| momo-scheduler / crawlers | Last | Chrome and batch work can saturate 188. |
|
||
| Sentry/Snuba consumers | Controlled | Kafka backlog and ClickHouse merge can create temporary high load. |
|
||
| Alertmanager outbound notification | Gate | Avoid alert storms before API webhook and Telegram are verified. |
|
||
| AI auto-repair | Observe-only | Metrics, Redis, KM, and playbooks may be incomplete. |
|
||
| Stateful DB restart | Human approval | PostgreSQL, Redis, ClickHouse, Harbor DB, Sentry DB are not generic restart targets. |
|
||
|
||
### 2.1 Freeze 執行清單
|
||
|
||
進入維護窗口後，先把「會放大事故」的來源降到 observe-only 或延後釋出。若沒有做到這一步，後續負載和告警會很難判讀。
|
||
|
||
| 順序 | 對象 | 只讀確認 | 允許動作 | 禁止動作 |
|
||
|------|------|----------|----------|----------|
|
||
| 1 | runner / CD | `systemctl list-units "actions.runner.*"`、Gitea Actions running jobs | 暫停新 job、等待可完成 job 結束 | 重啟 Docker daemon 來中斷 job |
|
||
| 2 | AI auto-remediation | Prometheus / Alertmanager / cold-start monitor 狀態 | 切 observe-only、保留告警 | 自動 restart stateful service |
|
||
| 3 | momo scheduler / crawler | container health、recent logs、DB parity | 延後 heavy import、保留現有資料 | 在 DB 未綠時強行匯入 |
|
||
| 4 | Sentry / Snuba | ClickHouse / Kafka health、consumer restart loop | 控制 consumer 釋出順序 | generic compose down/up 全套重啟 |
|
||
| 5 | K3s workload | node readiness、pods、events | 依 node 狀態 cordon/drain | 120 不可達時宣稱 drain 成功 |
|
||
|
||
多個工作視窗同時處理事故時，第一優先是避免互相打斷：只要有人在收斂 Docker / Nginx / firewall / K3s 寫操作，其他視窗先只讀觀察，直到明確交接。
|
||
|
||
### 2.2 CD / SSH Trust Guardrail
|
||
|
||
2026-06-13 的冷啟動假紅燈顯示：CD workflow 若用 `ssh-keyscan ... > /home/wooo/.ssh/known_hosts`，會覆蓋 110 使用者層的全域 SSH trust，導致 110 到 120 / 188 的 strict SSH 檢查失敗。這會把實際已恢復的主機誤判成 blocked。
|
||
|
||
固定規則：
|
||
|
||
| 項目 | 正確做法 | 禁止 |
|
||
|------|----------|------|
|
||
| Deploy 專用 host key | 寫入 `/home/wooo/.ssh/deploy_known_hosts` | 寫入或覆蓋 `/home/wooo/.ssh/known_hosts` |
|
||
| Deploy SSH options | `-o UserKnownHostsFile=/home/wooo/.ssh/deploy_known_hosts` | 共用 operator / cold-start 的 `known_hosts` |
|
||
| 冷啟動 SSH trust | 保留 120 / 188 的已驗證 fingerprint；修復前先備份 | 無 fingerprint 交叉驗證就 `ssh-keygen -R` 或重建全檔 |
|
||
| 驗證 | CD 後檢查 `known_hosts` mtime、120 / 188 entries、strict SSH | 只看 CD success badge |
|
||
|
||
2026-06-13 修復錨點：
|
||
|
||
- Source fix：Gitea main 包含 `80e6ec1a fix(ci): avoid clobbering runner known hosts`。
|
||
- Deploy marker：`e4a349bc chore(cd): deploy 414413a [skip ci]` 後，`/home/wooo/.ssh/known_hosts` mtime 仍停在 `2026-06-13 01:20:02 +0800`，未被 CD 覆蓋。
|
||
- Deploy isolated file：`/home/wooo/.ssh/deploy_known_hosts` mtime `2026-06-13 01:24:05 +0800`。
|
||
- Global strict entries：120 ED25519 line 4、188 ED25519 line 5 仍存在；strict SSH 到 `wooo@192.168.0.120` 與 `ollama@192.168.0.188` 必須通過。
|
||
|
||
---
|
||
|
||
## 3. P0 Evidence And Network
|
||
|
||
Run from any machine on the same LAN:
|
||
|
||
```bash
|
||
for h in 110 120 121 188; do
|
||
ping -c 2 -W 2 192.168.0.$h >/dev/null && echo "PING_OK 192.168.0.$h" || echo "PING_FAIL 192.168.0.$h"
|
||
done
|
||
|
||
arp -an | grep -E '192\.168\.0\.(110|120|121|188)'
|
||
for h in 110 120 121 188; do
|
||
nc -G 3 -z 192.168.0.$h 22 && echo "SSH_OK 192.168.0.$h" || echo "SSH_FAIL 192.168.0.$h"
|
||
done
|
||
```
|
||
|
||
Then capture reboot evidence:
|
||
|
||
```bash
|
||
ssh ollama@192.168.0.188 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20'
|
||
ssh wooo@192.168.0.110 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20'
|
||
ssh wooo@192.168.0.120 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20'
|
||
ssh wooo@192.168.0.121 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20'
|
||
```
|
||
|
||
If any host has ARP `incomplete` or SSH port down, stop here and fix physical/network first.
|
||
|
||
### 3.1 主機已啟動判定標準
|
||
|
||
每台主機重啟後先跑「四段式啟動判定」。只有全部符合角色期望，才進入服務恢復。
|
||
|
||
```bash
|
||
for h in 110 120 121 188; do
|
||
ip="192.168.0.$h"
|
||
echo "=== $ip ==="
|
||
ping -c 2 -W 2 "$ip" >/dev/null && echo "HOST_POWERED_OR_LAN_OK=1" || echo "HOST_POWERED_OR_LAN_OK=0"
|
||
arp -an | grep "$ip" || true
|
||
nc -G 3 -z "$ip" 22 && echo "SSH_PORT_OPEN=1" || echo "SSH_PORT_OPEN=0"
|
||
done
|
||
```
|
||
|
||
可 SSH 後：
|
||
|
||
```bash
|
||
ssh wooo@192.168.0.110 'hostname; date; who -b; uptime; systemctl is-system-running || true; systemctl --failed --no-pager --plain || true; free -h; swapon --show'
|
||
ssh wooo@192.168.0.121 'hostname; date; who -b; uptime; systemctl is-system-running || true; systemctl --failed --no-pager --plain || true'
|
||
ssh ollama@192.168.0.188 'hostname; date; who -b; uptime; systemctl is-system-running || true; systemctl --failed --no-pager --plain || true; free -h'
|
||
```
|
||
|
||
120 若不可 SSH，狀態只能是 `HOST_POWERED_UNKNOWN` 或 `HOST_BOOTED_UNKNOWN`，必須走 console / VM / network 檢查，不可用 Kubernetes stale node object 代替主機現況。
|
||
|
||
| 判定 | 必要條件 | 下一步 |
|
||
|------|----------|--------|
|
||
| `HOST_BOOTED` | ping 或 ARP 有回應、SSH port open、`who -b` 顯示本次 boot time | 檢查角色服務 |
|
||
| `HOST_READY` | `systemctl is-system-running` 為 `running`，或 degraded units 已逐一解釋且不影響本 host 角色 | 進入服務層驗證 |
|
||
| `HOST_DEGRADED` | failed units 存在且影響本 host 角色，或 swap 滿、root readonly、boot storage error | 先修 host，不釋出下一層 |
|
||
| `HOST_UNREACHABLE` | ping/SSH/ARP 失敗 | 停止遠端修復假設，改 console/VM/network |
|
||
|
||
2026-06-12 110 事故教訓：failed unit 若指向不存在的 legacy 路徑，要先確認是否仍屬現役服務。停用 stale timer 可以解除 host degraded，但必須同步 source-of-truth 後續清理，不能靠反覆 `reset-failed` 掩蓋。
|
||
|
||
---
|
||
|
||
## 4. P0 188 Data Layer
|
||
|
||
188 is the first real service dependency because K3s datastore and AWOOOI DB depend on PostgreSQL.
|
||
|
||
### 4.1 Startup order
|
||
|
||
1. `containerd`
|
||
2. `docker`
|
||
3. `postgresql@14-main`
|
||
4. `k3s_datastore.kine` maintenance
|
||
5. `redis-server` on `6380`
|
||
6. `ollama` or current AI proxy dependencies
|
||
7. `nginx`
|
||
8. Docker networks
|
||
9. MinIO / OpenClaw / SignOz
|
||
10. momo / litellm / batch services after load is stable
|
||
|
||
### 4.2 Read-only check
|
||
|
||
```bash
|
||
ssh ollama@192.168.0.188 '
|
||
hostname; date; uptime; free -h
|
||
systemctl is-active containerd docker postgresql@14-main redis-server ollama nginx || true
|
||
pg_isready -h localhost -p 5432 || true
|
||
redis-cli -p 6380 ping 2>/dev/null || redis-cli ping 2>/dev/null || true
|
||
docker ps --format "{{.Names}}\t{{.Status}}\t{{.Ports}}" | head -120
|
||
'
|
||
```
|
||
|
||
### 4.3 PostgreSQL WAL checkpoint damage
|
||
|
||
Signature:
|
||
|
||
```text
|
||
PANIC: could not locate a valid checkpoint record
|
||
invalid primary checkpoint record
|
||
unexpected pageaddr ... in log segment ...
|
||
```
|
||
|
||
This blocks:
|
||
|
||
- `188:5432`
|
||
- K3s startup on 120/121
|
||
- AWOOOI API DB access
|
||
- Alertmanager webhook if API cannot start
|
||
|
||
Human-approved recovery command on 188:
|
||
|
||
```bash
|
||
sudo systemctl stop postgresql@14-main
|
||
sudo install -d -m 700 -o postgres -g postgres /var/backups/postgresql
|
||
sudo tar -C /var/lib/postgresql/14 -czf /var/backups/postgresql/14-main-before-pg-resetwal-$(date +%Y%m%d-%H%M%S).tgz main
|
||
sudo -u postgres /usr/lib/postgresql/14/bin/pg_resetwal -f /var/lib/postgresql/14/main
|
||
sudo systemctl start postgresql@14-main
|
||
pg_isready -h localhost -p 5432
|
||
sudo -u postgres psql -d k3s_datastore -c "VACUUM ANALYZE kine;"
|
||
```
|
||
|
||
Do not run `DROP`, reinitialize the cluster, delete `/var/lib/postgresql`, or restore an old backup unless the commander explicitly approves it.
|
||
|
||
---
|
||
|
||
## 5. P0/P1 110 Registry And Observability
|
||
|
||
110 must recover Harbor/Gitea/Monitoring early, but runners last.
|
||
|
||
### 5.1 Startup order
|
||
|
||
1. `docker`
|
||
2. Remove `Exited (128)` / `Exited (137)` orphan containers
|
||
3. Harbor `harbor-log`
|
||
4. Harbor full stack
|
||
5. Gitea
|
||
6. Prometheus / Alertmanager / Grafana / exporters
|
||
7. Langfuse
|
||
8. SignOz
|
||
9. Sentry DB layer
|
||
10. Sentry web/worker/consumer layer
|
||
11. Gitea host runner and actions runners
|
||
|
||
### 5.2 Checks
|
||
|
||
```bash
|
||
ssh wooo@192.168.0.110 '
|
||
hostname; date; uptime; free -h
|
||
systemctl is-active docker || true
|
||
curl -s -o /dev/null -w "harbor=%{http_code}\n" --max-time 5 http://127.0.0.1:5000/v2/ || true
|
||
curl -s -o /dev/null -w "gitea=%{http_code}\n" --max-time 5 http://127.0.0.1:3001/ || true
|
||
curl -s --max-time 5 http://127.0.0.1:9090/-/ready || true
|
||
curl -s --max-time 5 http://127.0.0.1:9093/-/healthy || true
|
||
curl -s -o /dev/null -w "sentry=%{http_code}\n" --max-time 10 http://127.0.0.1:9000/ || true
|
||
docker ps --format "{{.Names}}\t{{.Status}}" | head -120
|
||
'
|
||
```
|
||
|
||
Harbor healthy means `/v2/` returns `200` or `401`. Do not treat `401` as failure.
|
||
|
||
### 5.3 Runner gate
|
||
|
||
Runner may start only after all are true:
|
||
|
||
- `188 PostgreSQL` ready
|
||
- `110 Harbor` ready
|
||
- `110 Gitea` ready
|
||
- `120/121 K3s` nodes ready
|
||
- AWOOOI API health passes
|
||
- 110 load/core is below `1.0` for at least 15 minutes
|
||
- runner systemd guardrails are active: `CPUQuota=200%`, `MemoryMax=2G`, `WatchdogUSec=0`
|
||
|
||
Check:
|
||
|
||
```bash
|
||
ssh wooo@192.168.0.110 '
|
||
for u in $(systemctl list-units "actions.runner.*" --all --no-legend --plain | awk "{print \$1}"); do
|
||
echo "=== $u ==="
|
||
systemctl show "$u" -p ActiveState -p SubState -p CPUQuotaPerSecUSec -p MemoryMax -p WatchdogUSec -p NRestarts
|
||
done
|
||
'
|
||
```
|
||
|
||
If `WatchdogUSec` is not `0`, apply the guardrail script manually with sudo:
|
||
|
||
```bash
|
||
sudo /home/wooo/scripts/apply-runner-systemd-guardrails.sh --apply
|
||
```
|
||
|
||
---
|
||
|
||
## 6. P1 120/121 K3s
|
||
|
||
K3s must wait for 188 PostgreSQL and 110 Harbor.
|
||
|
||
### 6.1 Startup order
|
||
|
||
1. 120 `k3s.service`
|
||
2. 121 `k3s.service`, `k3s-agent.service`, or its live role
|
||
3. CNI / kube-proxy
|
||
4. Nodes Ready
|
||
5. Core pods
|
||
6. `awoooi-prod` pods
|
||
7. keepalived VIP `192.168.0.125`
|
||
8. NodePorts `32334` and `32335`
|
||
|
||
### 6.2 Checks
|
||
|
||
```bash
|
||
ssh wooo@192.168.0.120 '
|
||
hostname; uptime
|
||
pg_isready -h 192.168.0.188 -p 5432 || true
|
||
systemctl is-active k3s k3s-agent keepalived 2>/dev/null || true
|
||
kubectl get nodes -o wide 2>/dev/null || true
|
||
kubectl get pods -A 2>/dev/null | grep -v -E "Running|Completed" || true
|
||
kubectl get pods -n awoooi-prod -o wide 2>/dev/null || true
|
||
ip addr show | grep 192.168.0.125 || true
|
||
'
|
||
|
||
ssh wooo@192.168.0.121 '
|
||
hostname; uptime
|
||
systemctl is-active k3s k3s-agent keepalived 2>/dev/null || true
|
||
ip addr show | grep 192.168.0.125 || true
|
||
'
|
||
```
|
||
|
||
If K3s is `activating` while 188 PostgreSQL is down, fix PostgreSQL first. Restarting K3s repeatedly will not solve it.
|
||
|
||
### 6.3 120 / 121 AA / AS 與負載平衡判定
|
||
|
||
2026-06-12 15:19 live check 確認 120 / 121 都是 K3s `control-plane`，且兩台都是 `k3s active`、`k3s-agent inactive`。因此它們是 K3s 控制面 AA，不是傳統一主一從 AS。
|
||
|
||
但控制面 AA 不等於業務 workload AA。120 剛從 root filesystem fsck 恢復後，大多數 ArgoCD / AWOOOI / Velero / kube-system workload 仍集中在 121；120 主要只有 DaemonSet 類 Pod。每次 120 / 121 重啟或恢復後，都要額外跑 Pod 落點檢查：
|
||
|
||
```bash
|
||
ssh wooo@192.168.0.120 '
|
||
sudo kubectl get nodes -o wide
|
||
sudo kubectl get pods -A -o wide
|
||
sudo kubectl top nodes 2>/dev/null || true
|
||
sudo kubectl top pods -A --sort-by=cpu 2>/dev/null | head -30 || true
|
||
'
|
||
```
|
||
|
||
判定規則：
|
||
|
||
| 判定 | 條件 | 可宣稱 |
|
||
|------|------|--------|
|
||
| `K3S_CONTROL_PLANE_AA` | 120 / 121 都是 `Ready control-plane` | 控制面雙節點可用 |
|
||
| `WORKLOAD_IMBALANCED` | 主要 deployment / pod 都落在單一節點 | 不可宣稱服務 AA；需排程治理 |
|
||
| `WORKLOAD_BALANCED` | replicas >= 2 的核心 API / Web 跨 120 / 121 分散 | 可宣稱承載層分散 |
|
||
| `STATEFUL_AA` | storage replication、backup / restore drill、failover drill 都通過 | 才可宣稱資料層 AA |
|
||
|
||
負載平衡與遷移評估的正式基準文件是 `docs/runbooks/HOST-ROLE-LOAD-BALANCING-ASSESSMENT.md`。恢復期先完成 P0 備份鏈與 cold-start scorecard，再做 topology spread 或服務搬遷。
|
||
|
||
---
|
||
|
||
## 7. P2 AWOOOI Workloads
|
||
|
||
Run after K3s nodes are Ready:
|
||
|
||
```bash
|
||
ssh wooo@192.168.0.120 '
|
||
kubectl get deploy -n awoooi-prod
|
||
kubectl get pods -n awoooi-prod -o wide
|
||
kubectl get svc -n awoooi-prod
|
||
kubectl get events -n awoooi-prod --sort-by=.lastTimestamp | tail -40
|
||
'
|
||
|
||
curl -s --max-time 8 http://192.168.0.125:32334/api/v1/health
|
||
curl -s -o /dev/null -w "web=%{http_code}\n" --max-time 8 http://192.168.0.125:32335/
|
||
```
|
||
|
||
If pods are `ImagePullBackOff`, go back to 110 Harbor.
|
||
|
||
If API health fails because DB/Redis is down, go back to 188.
|
||
|
||
---
|
||
|
||
## 8. P2 Alert Chain
|
||
|
||
Current main path:
|
||
|
||
```text
|
||
Prometheus/Alertmanager on 110
|
||
-> http://192.168.0.125:32334/api/v1/webhooks/alertmanager
|
||
-> AWOOOI API
|
||
-> TelegramGateway
|
||
-> Telegram
|
||
```
|
||
|
||
Alertmanager health alone is not enough. Run E2E:
|
||
|
||
```bash
|
||
curl -s -X POST http://192.168.0.125:32334/api/v1/webhooks/alertmanager \
|
||
-H 'Content-Type: application/json' \
|
||
-d '{"receiver":"cold-start-test","status":"firing","alerts":[{"status":"firing","labels":{"alertname":"ColdStartE2ETest","severity":"info"},"annotations":{"summary":"Cold start E2E test, ignore"},"startsAt":"2026-05-05T11:00:00Z","endsAt":"0001-01-01T00:00:00Z","generatorURL":""}],"groupLabels":{},"commonLabels":{},"commonAnnotations":{},"externalURL":"","version":"4","groupKey":"cold-start-test"}'
|
||
```
|
||
|
||
Expected: API returns success and Telegram receives the test alert.
|
||
|
||
---
|
||
|
||
## 9. P2 Schedules And Delayed Work
|
||
|
||
Do not mark the reboot complete until scheduled work is proven runnable. A container can be healthy while its cron path is broken.
|
||
|
||
| Host / Layer | Required check | Success baseline |
|
||
|--------------|----------------|------------------|
|
||
| 188 cron | `systemctl is-active cron` and `crontab -l` | cron active; backup, restart exporter, stats exporter entries present |
|
||
| 188 backup-from-110 | `backup_110_last_success_timestamp` in textfile/Prometheus | last success age `< 25h` |
|
||
| 188 momo-scheduler | `docker inspect momo-scheduler` and `docker logs --since 6h momo-scheduler` | container `running healthy`; `全部排程任務已註冊`; Google Drive auth works; dashboard URLs use container-reachable hostnames |
|
||
| 188 momo import | manual `run_auto_import_task()` after parser changes | selected sheet is `即時業績明細`; imported date range has matching rows in `daily_sales_snapshot` and `realtime_sales_monthly` |
|
||
| 110 cron | `systemctl is-active cron` | cron active; Docker/systemd textfile exporters fresh |
|
||
| 110 startup units | `systemctl --failed` | zero failed units; stale `momo-startup-complete` and `wooo-staggered-startup` disabled |
|
||
| 120 K8s CronJobs | `kubectl get cronjobs -n awoooi-prod` | unsuspended; no failed Jobs remain after current validation |
|
||
| 121 DR drill | `crontab -l` | DR drill cron present unless explicitly paused |
|
||
|
||
Useful checks:
|
||
|
||
```bash
|
||
ssh ollama@192.168.0.188 'systemctl is-active cron; crontab -l; ls -l /home/ollama/node_exporter_textfiles/*.prom'
|
||
ssh wooo@192.168.0.110 'systemctl --failed --no-pager; systemctl is-active cron; crontab -l'
|
||
ssh wooo@192.168.0.120 'sudo kubectl get cronjobs,jobs -n awoooi-prod'
|
||
ssh wooo@192.168.0.121 'systemctl is-active cron; crontab -l'
|
||
```
|
||
|
||
If a schedule succeeds but emits a false verification alert, fix the verification rule before releasing AI auto-remediation. False positives train operators to ignore real alarms.
|
||
|
||
---
|
||
|
||
## 10. P2/P3 Stateful Service Guardrails
|
||
|
||
| Tier | Examples | Automation |
|
||
|------|----------|------------|
|
||
| BLOCK | PostgreSQL data dir, ClickHouse data dir, Harbor DB, Sentry DB | No automatic destructive action. Human approval only. |
|
||
| CRITICAL_HITL | Redis, Kafka, MinIO, SignOz ClickHouse, Sentry ClickHouse | Human-in-the-loop restart/repair. |
|
||
| STANDARD_HITL | API/Web/worker, OpenClaw, litellm | Restart only with evidence and blast-radius check. |
|
||
| AUTO | Stateless exporters, blackbox, nginx exporter | Auto restart allowed after verification. |
|
||
|
||
Never use generic `docker restart $(docker ps -q)` during cold start.
|
||
|
||
### 10.1 Dirty-Reboot Storage Corruption
|
||
|
||
Treat these log signatures as storage corruption, not ordinary service flakiness:
|
||
|
||
- `Bad message`
|
||
- `Structure needs cleaning`
|
||
- `Unknown codec`
|
||
- `PANIC: could not locate a valid checkpoint record`
|
||
- Kafka `Malformed line` in checkpoint files
|
||
- ClickHouse `broken and needs manual correction`
|
||
|
||
Cold-start automation may stop a restart storm and collect evidence, but it must not delete the original data directory. If a filesystem returns `Bad message` or `Structure needs cleaning`, the real root cause is below the container layer. Online recovery can restore service from readable data, but complete historical recovery requires an offline filesystem check or backup restore.
|
||
|
||
### 10.2 ClickHouse Clean-Clone Recovery Pattern
|
||
|
||
Use this pattern for Sentry ClickHouse or SignOz ClickHouse when individual corrupted parts cannot be moved because the host filesystem rejects reads.
|
||
|
||
```text
|
||
1. Stop the compose stack or at least stop dependent consumers.
|
||
2. Disable restart loops for the failing container.
|
||
3. Save logs and build an exclude list from unreadable store paths.
|
||
4. Preserve the original volume as _data.corrupt-YYYYMMDD-HHMMSS.
|
||
5. Create a clean _data clone with readable files only.
|
||
6. Add flags/force_restore_data.
|
||
7. Start ClickHouse first, then web/API, then consumers.
|
||
8. Verify HTTP, merge backlog, and restart count before releasing high-load services.
|
||
```
|
||
|
||
Do not replace this with `rm -rf store/...` unless the unreadable path is already backed up or the commander explicitly accepts data loss. The preferred incident artifact is:
|
||
|
||
```text
|
||
/var/lib/docker/volumes/<volume>/_data.corrupt-YYYYMMDD-HHMMSS
|
||
/var/backups/<service>-<component>-YYYYMMDD-HHMMSS
|
||
```
|
||
|
||
### 10.3 Kafka Checkpoint Recovery Pattern
|
||
|
||
If Kafka refuses to start with malformed checkpoint files after a dirty reboot, preserve and move only checkpoint files:
|
||
|
||
```text
|
||
log-start-offset-checkpoint
|
||
recovery-point-offset-checkpoint
|
||
replication-offset-checkpoint
|
||
```
|
||
|
||
Then start Kafka and confirm health before starting Snuba/Sentry consumers. Do not delete topic directories or Kafka logs during cold-start recovery.
|
||
|
||
---
|
||
|
||
## 11. P3 High-Load Services
|
||
|
||
Only release these after P0/P1/P2 gates are green:
|
||
|
||
| Host | Service | Release condition |
|
||
|------|---------|-------------------|
|
||
| 188 | momo-scheduler / crawler | load/core < 1.0 for 15 minutes and DB healthy |
|
||
| 188 | SignOz ClickHouse | healthy and merge backlog trending down |
|
||
| 188 | litellm | `/health/liveliness` good and provider route verified |
|
||
| 110 | Sentry Snuba consumers | ClickHouse healthy and Kafka backlog decreasing |
|
||
| 110 | Sentry uptime-checker | Sentry web/DB healthy |
|
||
| 110 | runners | all previous gates green and load/core < 1.0 for 15 minutes |
|
||
|
||
---
|
||
|
||
## 12. Baseline And AI Auto-Remediation Gate
|
||
|
||
### 12.1 Stable Runtime Baseline
|
||
|
||
These are release gates after the first cold-start recovery pass:
|
||
|
||
| Area | Baseline |
|
||
|------|----------|
|
||
| 188 host | PostgreSQL accepting, Redis PONG, momo `/health` 200, SignOz HTTP reachable, load/core < 1.0 sustained before crawlers |
|
||
| 110 host | Harbor `/v2/` 200/401, Gitea 200/302, Prometheus ready, Alertmanager healthy, Sentry HTTP 200/302/400, no ClickHouse/Kafka restart loop |
|
||
| K3s | 120/121 nodes Ready, VIP `192.168.0.125` present, AWOOOI API 2xx/3xx, Web 2xx/3xx |
|
||
| Public routes | `https://awoooi.wooo.work/api/v1/health` 2xx/3xx, `https://mo.wooo.work/health` 2xx/3xx |
|
||
| Guardrails | Docker/systemd textfile exporters fresh, runner `CPUQuota=200%`, `MemoryMax=2G`, `WatchdogUSec=0` |
|
||
| Schedules | cron active on 110/188/120/121; K8s CronJobs unsuspended; no current failed Jobs; 188 backup success `< 25h` |
|
||
| Backlog | ClickHouse merges and Kafka/Snuba lag trending down, not increasing for two consecutive checks |
|
||
|
||
If service health is green but load average remains high, check live CPU and IO before changing memory limits. High load after Sentry/Snuba or ClickHouse startup can be backlog drain; high CPU from runners/builds/crawlers is a release-order problem.
|
||
|
||
### 12.2 AI Auto-Remediation Gate
|
||
|
||
AI auto-repair can move from observe-only to limited execution only after:
|
||
|
||
- Prometheus rules are loaded.
|
||
- docker/systemd textfile exporter files are fresh.
|
||
- blackbox probes have stable results.
|
||
- cron/CronJob schedule checks are green.
|
||
- AWOOOI API `/api/v1/health` passes.
|
||
- Alertmanager E2E webhook passes.
|
||
- Redis/KM/playbook health is available.
|
||
- No active restart storm.
|
||
- Host load/core remains below `1.0` for 15 minutes.
|
||
|
||
Until then:
|
||
|
||
- diagnose only
|
||
- notify only
|
||
- require human approval for remediation
|
||
- no DB/ClickHouse/Harbor/Sentry destructive action
|
||
- no generic restart action against stateful services
|
||
|
||
---
|
||
|
||
## 13. One-Command Readiness Script
|
||
|
||
### 13.1 Single Pass
|
||
|
||
Run this when you want one read-only snapshot:
|
||
|
||
```bash
|
||
bash scripts/reboot-recovery/full-stack-cold-start-check.sh
|
||
```
|
||
|
||
The script is read-only. It does not restart services, delete data, change memory/CPU limits, or patch Kubernetes. It reports gates:
|
||
|
||
- `P0-NETWORK`
|
||
- `P0-188-DATA`
|
||
- `P0-110-REGISTRY-OBSERVABILITY`
|
||
- `P1-K3S`
|
||
- `P2-WORKLOAD-ALERTCHAIN`
|
||
- `P2-PUBLIC-ROUTES`
|
||
- `P2-SCHEDULES`
|
||
- runner guardrail state inside `P0-110-REGISTRY-OBSERVABILITY`
|
||
|
||
If it prints `BLOCKED`, fix the first blocked gate before moving forward.
|
||
|
||
### 13.2 Professional Watch Mode
|
||
|
||
Run this after a full reboot when you want the machine to keep checking until the whole stack is ready:
|
||
|
||
```bash
|
||
bash scripts/reboot-recovery/full-stack-cold-start-check.sh \
|
||
--watch \
|
||
--interval 60 \
|
||
--max-attempts 30 \
|
||
--send-alert-test
|
||
```
|
||
|
||
This is the standard next-reboot release command. It checks every 60 seconds for up to 30 attempts and exits only when the stack is `GREEN` or the last attempt remains degraded/blocked.
|
||
|
||
Use `--send-alert-test` for final release because the alert chain is not proven until the AWOOOI Alertmanager webhook accepts a real POST. Without `--send-alert-test`, the script intentionally leaves a warning so operators do not falsely mark alerting as complete.
|
||
|
||
### 13.3 Persistent Read-Only Monitor
|
||
|
||
After recovery, host 110 should run the same gate as a node-exporter textfile monitor:
|
||
|
||
```bash
|
||
bash scripts/reboot-recovery/install-cold-start-monitor-110.sh
|
||
```
|
||
|
||
This installs two scripts under `/home/wooo/scripts/`, adds a marked user-cron block, and writes:
|
||
|
||
- `/home/wooo/node_exporter_textfiles/cold_start_recovery.prom`
|
||
- `/home/wooo/reboot-recovery/cold-start-last.log`
|
||
|
||
The cron path uses `--monitor-read-only`, so it does not POST Alertmanager smoke events every 10 minutes. It converts the cold-start gate into Prometheus metrics:
|
||
|
||
- `awoooi_cold_start_monitor_up`
|
||
- `awoooi_cold_start_pass_gates`
|
||
- `awoooi_cold_start_warn_gates`
|
||
- `awoooi_cold_start_blocked_gates`
|
||
- `awoooi_cold_start_last_run_timestamp`
|
||
- `awoooi_cold_start_last_green_timestamp`
|
||
- `awoooi_cold_start_last_result{result="green|degraded|blocked|check_failed"}`
|
||
|
||
Prometheus rules in `ops/monitoring/alerts-unified.yml` alert when the monitor is missing, stale, blocked, degraded, or has not been green for more than 6 hours.
|
||
|
||
### 13.4 Script-To-SOP Coverage Map
|
||
|
||
| Script gate | SOP coverage | Blocks |
|
||
|-------------|--------------|--------|
|
||
| `P0-NETWORK` | host reachability, ARP, SSH | every later phase |
|
||
| `P0-188-DATA` | PostgreSQL, Redis, momo, SignOz | K3s, AWOOOI API, momo public site |
|
||
| `P0-110-REGISTRY-OBSERVABILITY` | Harbor, Gitea, Prometheus, Alertmanager, Sentry, runner quotas | image pulls, CD, alert rules, runners |
|
||
| `P1-K3S` | 120/121 K3s, VIP, node readiness, pod health | workload and webhook health |
|
||
| `P2-WORKLOAD-ALERTCHAIN` | AWOOOI API/Web, Alertmanager webhook | AI auto-remediation and alert confidence |
|
||
| `P2-PUBLIC-ROUTES` | external AWOOOI and momo URLs | external release |
|
||
| `P2-SCHEDULES` | cron, CronJobs, backups, textfile exporters, DR drill | final done criteria |
|
||
|
||
### 13.5 Next-Reboot Operator Contract
|
||
|
||
1. Run the watch command above.
|
||
2. If it stops at `BLOCKED`, repair the first blocked gate and rerun watch mode.
|
||
3. If it stops at `WARN`, do not release runner/CD/AI full execution; clear or explicitly accept each warning.
|
||
4. Release high-load services only after `GREEN` and load/core stays below `1.0` for 15 minutes.
|
||
5. Record the final output summary and any manual repair in `docs/LOGBOOK.md`.
|
||
|
||
### 13.6 2026-05-29 補充：188 Public Gateway 與備份告警
|
||
|
||
`aiops.wooo.work` 的 188 public gateway 不可再指向單一 `192.168.0.120:31234/31235`。120 失聯時這會讓 public route 直接 502。正式 baseline 必須走 K3s VIP：
|
||
|
||
```nginx
|
||
location /api/ {
|
||
proxy_pass http://192.168.0.125:32334/api/;
|
||
}
|
||
|
||
location /api/v1/ws {
|
||
proxy_pass http://192.168.0.125:32334/api/v1/ws;
|
||
}
|
||
|
||
location / {
|
||
proxy_pass http://192.168.0.125:32335;
|
||
}
|
||
```
|
||
|
||
變更來源必須是 `infra/ansible/roles/nginx/templates/188-all-sites.conf.j2`，再用 `infra/ansible/playbooks/nginx-sync.yml` 收斂；禁止只改 188 live 檔而不回寫 Ansible baseline。
|
||
|
||
備份告警有兩層，缺一不可：
|
||
|
||
- `ops/monitoring/alerts-unified.yml` 是 repo canonical。
|
||
- 110 live `/home/wooo/monitoring/alerts.yml` 與 `/home/wooo/monitoring/alerts-unified.canonical.yml` 必須一致，否則 `prometheus-rule-drift-guard` 可能把規則拉回舊版。
|
||
|
||
重啟後必查：
|
||
|
||
```bash
|
||
curl -s http://127.0.0.1:9090/api/v1/rules \
|
||
| python3 -c 'import json,sys; d=json.load(sys.stdin); names=[r.get("name") for g in d["data"]["groups"] for r in g["rules"]]; print([n for n in ["BackupAggregateRunFailed","BackupConfigCapturePartial","BackupOffsiteCopyStale","BackupCredentialEscrowEvidenceMissing","ColdStartRecoveryBlocked"] if n not in names])'
|
||
|
||
cat /home/wooo/node_exporter_textfiles/prometheus_rule_drift_guard.prom
|
||
```
|
||
|
||
若 120 尚未恢復，`BackupConfigCapturePartial{target="120-k3s-host-configs"}` 與 cold-start blocked 是正確訊號，不可消音。120 恢復後再重跑：
|
||
|
||
```bash
|
||
/backup/scripts/backup-configs.sh
|
||
/backup/scripts/backup-all.sh
|
||
/backup/scripts/sync-offsite-backups.sh --mode sync
|
||
/backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color
|
||
```
|
||
|
||
### 13.7 2026-05-29 補充：momo PostgreSQL Index 與資料同步
|
||
|
||
`mo.wooo.work` 不能只看 `/health` 或首頁 200。重啟或 fsck 後，PostgreSQL index 可能讓匯入流程表面完成，但 `daily_sales_snapshot` 未同步到 `realtime_sales_monthly`。本次症狀：
|
||
|
||
- `daily_sales_snapshot` 已有 2026-05-01 到 2026-05-28 的 17,353 筆。
|
||
- `realtime_sales_monthly` 同日期範圍為 0 筆。
|
||
- momo-scheduler log 出現 PostgreSQL 內部錯誤 `posting list tuple ... cannot be split`。
|
||
|
||
標準處理順序：
|
||
|
||
```bash
|
||
# 188 / momo-db，只重建索引，不刪資料
|
||
docker exec -i momo-db bash -lc 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -v ON_ERROR_STOP=1' <<'SQL'
|
||
REINDEX TABLE CONCURRENTLY public.realtime_sales_monthly;
|
||
SQL
|
||
```
|
||
|
||
重建索引後，才可針對缺漏日期做 idempotent 補同步。正式作法必須先確認 `realtime_sales_monthly` 該日期範圍筆數，若非 0，需先保存查詢結果並確認是否重跑同範圍同步；不可整表 truncate、不可整庫 restore。補同步後至少驗證：
|
||
|
||
```sql
|
||
SELECT count(*), min(snapshot_date::date), max(snapshot_date::date)
|
||
FROM daily_sales_snapshot
|
||
WHERE snapshot_date::date BETWEEN DATE '2026-05-01' AND DATE '2026-05-28';
|
||
|
||
SELECT count(*), min("日期"::date), max("日期"::date)
|
||
FROM realtime_sales_monthly
|
||
WHERE "日期"::date BETWEEN DATE '2026-05-01' AND DATE '2026-05-28';
|
||
```
|
||
|
||
兩張表同日期範圍筆數與日期上下界必須一致。完成後清除 momo 應用 cache：
|
||
|
||
```bash
|
||
docker exec momo-pro-system python -c 'from services.cache_service import clear_all_cache; clear_all_cache(); print("cache_cleared")'
|
||
```
|
||
|
||
---
|
||
|
||
## 14. 主機開機、關機、重啟 SOP
|
||
|
||
本節是每次 110 / 120 / 121 / 188 相關電源操作的標準程序。112 是 Kali，只保留 read-only evidence，不納入本輪恢復或例行重啟釋出。
|
||
|
||
### 14.1 共同紅線
|
||
|
||
| 類型 | 禁止事項 | 正確處理 |
|
||
|------|----------|----------|
|
||
| 120 offline | 不可消音 `ColdStartHost120Unreachable`、`ColdStartRecoveryBlocked` 或 120 config backup alert | 保留紅燈，直到 console/VM recovery 後重跑完整 chain |
|
||
| Filesystem | 不可對已掛載 root filesystem 做 online `fsck` | 只在 console/rescue/initramfs 狀態下離線修復 |
|
||
| Backup | 不可用單項 backup 成功宣稱 aggregate backup green | 以 `backup-all`、offsite verifier、cold-start scorecard 三者共同判定 |
|
||
| Credential | 不可把密碼、token、private key 寫進 repo、LOGBOOK 或聊天 | 只寫 non-secret evidence marker / vault reference |
|
||
| Stateful data | 不可 truncate、DROP、整庫 restore 或整批刪 volume | 先保存證據，優先 `REINDEX TABLE CONCURRENTLY` / clean-clone / idempotent resync |
|
||
| Automation | 不可在 P0/P1 未綠時釋出 runner/CD/AI full execution | observe-only，runner/CD 最後釋出 |
|
||
|
||
### 14.2 關機前 SOP
|
||
|
||
目標是保留證據、停止高負載來源、讓 stateful service 乾淨落地。
|
||
|
||
1. 宣告維護窗口，建立 `docs/LOGBOOK.md` 重啟紀錄草稿。
|
||
2. 跑 preflight snapshot：
|
||
|
||
```bash
|
||
/home/wooo/scripts/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1
|
||
/backup/scripts/backup-status.sh --no-notify
|
||
/backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color
|
||
```
|
||
|
||
3. 保存 host reboot evidence：
|
||
|
||
```bash
|
||
for h in 110 120 121; do
|
||
ssh wooo@192.168.0.$h 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20; systemctl --failed --no-pager' || true
|
||
done
|
||
ssh ollama@192.168.0.188 'hostname; date; uptime; who -b; last -x reboot shutdown | head -20; systemctl --failed --no-pager' || true
|
||
```
|
||
|
||
4. 暫停高負載與自動化釋出：
|
||
|
||
| 順序 | 對象 | 操作原則 |
|
||
|------|------|----------|
|
||
| 1 | Gitea / actions runners | 停止新 job；不要在 build 中途硬關，先讓可完成 job 結束或人工取消 |
|
||
| 2 | AI auto-remediation | 切 observe-only；禁止自動 restart stateful services |
|
||
| 3 | momo crawler / scheduler / heavy batch | 暫停會啟動 Chrome、批次匯入或大量 DB 寫入的工作 |
|
||
| 4 | Sentry/Snuba/ClickHouse heavy consumers | 確認沒有 restart storm；必要時 controlled stop |
|
||
| 5 | K3s workload | 優先 drain / cordon 可達節點；不可在 120 已不可達時假裝 drain 完成 |
|
||
|
||
5. 全機關機順序：
|
||
|
||
```text
|
||
1. runner/CD and high-load batch
|
||
2. AI auto-remediation execution
|
||
3. AWOOOI workload layer
|
||
4. 121 K3s agent side
|
||
5. 120 K3s server side
|
||
6. 110 registry / observability, after evidence and backup status are captured
|
||
7. 188 data layer last
|
||
8. network / UPS / hypervisor last, if applicable
|
||
```
|
||
|
||
188 必須最後關，因為 PostgreSQL / Redis / momo DB / K3s datastore 是其他層的共同依賴。
|
||
|
||
### 14.3 開機 SOP
|
||
|
||
開機順序固定走 dependency chain，不追最吵的 alert。
|
||
|
||
```text
|
||
1. Physical network: switch, NIC, ARP, SSH
|
||
2. 188 data layer: PostgreSQL, Redis, Docker, momo DB, SignOz dependencies
|
||
3. 110 registry / observability: Harbor, Gitea, Prometheus, Alertmanager, Sentry
|
||
4. 120 K3s server / VIP path
|
||
5. 121 K3s agent / failover path
|
||
6. AWOOOI API/Web workload
|
||
7. Public routes and Alertmanager E2E
|
||
8. Backups, cron, CronJobs, textfile exporters
|
||
9. momo scheduler / crawlers and high-load consumers
|
||
10. runners/CD
|
||
11. AI auto-remediation limited execution
|
||
```
|
||
|
||
開機後每一層都要有 live evidence。最小驗收命令：
|
||
|
||
```bash
|
||
for h in 110 120 121 188; do
|
||
ping -c 2 -W 2 192.168.0.$h >/dev/null && echo "PING_OK 192.168.0.$h" || echo "PING_FAIL 192.168.0.$h"
|
||
nc -G 3 -z 192.168.0.$h 22 && echo "SSH_OK 192.168.0.$h" || echo "SSH_FAIL 192.168.0.$h"
|
||
done
|
||
|
||
ssh ollama@192.168.0.188 'systemctl is-active docker postgresql@14-main redis-server nginx || true; pg_isready -h localhost -p 5432 || true; docker ps --format "{{.Names}}\t{{.Status}}" | head -80'
|
||
ssh wooo@192.168.0.110 'systemctl is-active docker cron || true; curl -s --max-time 5 http://127.0.0.1:9090/-/ready || true; curl -s --max-time 5 http://127.0.0.1:9093/-/healthy || true'
|
||
ssh wooo@192.168.0.121 'sudo kubectl get nodes -o wide; sudo kubectl get pods -A | grep -v -E "Running|Completed" || true'
|
||
|
||
/backup/scripts/backup-status.sh --no-notify
|
||
/backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color
|
||
/home/wooo/scripts/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1
|
||
```
|
||
|
||
### 14.4 單主機重啟 SOP
|
||
|
||
| Host | 重啟前條件 | 重啟後必查 | 完成條件 |
|
||
|------|------------|------------|----------|
|
||
| 110 | 不在 `backup-all` / rclone / verify window；runner job 已停止或人工取消；188 healthy | Docker, Harbor, Gitea, Prometheus, Alertmanager, Sentry, cron, textfile exporters, `/backup/scripts/backup-status.sh --no-notify` | 110 services green；backup status 沒有新增 stale / failed；runner/CD 最後釋出 |
|
||
| 120 | 必須是 console-first 維護；若可達，先 cordon/drain；若不可達，不宣稱 drain 成功 | power/VM/NIC/boot/initramfs/fsck state, SSH, `kubectl get nodes`, `SchedulingDisabled` 清除狀態 | 120 ping/SSH OK；`mon Ready`；backup configs/all/offsite/verify/cold-start chain 重跑 |
|
||
| 121 | 120 / 188 healthy；可達時先 cordon/drain | `k3s-agent` 或 live role、VIP 狀態、`kubectl get nodes`, pod placement | `mon1 Ready`；VIP / NodePort 路徑正常；workload 無新增 failed pods |
|
||
| 188 | 110 backup status 已保存；停止或延後 momo heavy import；確認無 DB restore / migration | PostgreSQL, Redis, Docker, momo DB parity, SignOz/ClickHouse, cron, backup freshness | DB accepting；momo parity 綠；188 backup jobs fresh；高負載服務最後釋出 |
|
||
|
||
### 14.4.1 110 重啟後恢復指揮卡
|
||
|
||
110 是 registry / observability / backup center。重啟後先看 host 與核心端口，不要第一時間重啟 Docker daemon。
|
||
|
||
| 順序 | 檢查 | 成功基準 | 失敗處理 |
|
||
|------|------|----------|----------|
|
||
| 1 | `systemctl is-system-running` / failed units / Swap | `running`、failed `0` 或可解釋、Swap 未持續增加 | 先分辨 stale unit、現役 service、storage/network 問題 |
|
||
| 2 | Docker daemon | `systemctl is-active docker=active` | 若 Docker `activating`，先看 journal；不要連續 restart/kill |
|
||
| 3 | Harbor / registry | local `/v2/` 回 `200/401`，public registry 未登入 `401` | 只針對失效 upstream 做最小修復；避免 daemon restart |
|
||
| 4 | Gitea / runners | Gitea 200/302；runner 最後釋出 | runner job 不可在 P0/P1 未綠時搶資源 |
|
||
| 5 | Prometheus / Alertmanager | `/-/ready`、`/-/healthy` OK；required alerts visible | 若告警缺失，先修 rules/drift guard，再談自動化 |
|
||
| 6 | Sentry / Langfuse / Stock / public tools | public 2xx/3xx；container 非 restart loop | 只修明確故障服務；不要 compose 全套重建 |
|
||
| 7 | backup / offsite | `backup-status --no-notify`、offsite verifier | 120 不可達時 Configs 紅燈保留 |
|
||
|
||
110 post-reboot 最小命令：
|
||
|
||
```bash
|
||
ssh wooo@192.168.0.110 '
|
||
date; uptime; systemctl is-system-running || true; systemctl --failed --no-pager --plain || true
|
||
free -h; swapon --show
|
||
systemctl is-active docker cron || true
|
||
curl -s -o /dev/null -w "harbor_v2=%{http_code}\n" --max-time 5 http://127.0.0.1:5000/v2/ || true
|
||
curl -s -o /dev/null -w "gitea=%{http_code}\n" --max-time 5 http://127.0.0.1:3001/ || true
|
||
curl -s --max-time 5 http://127.0.0.1:9090/-/ready || true
|
||
curl -s --max-time 5 http://127.0.0.1:9093/-/healthy || true
|
||
docker ps --format "{{.Names}}\t{{.Status}}" | head -120
|
||
'
|
||
```
|
||
|
||
2026-06-12 補充：`stockplatform-shared-ui-monitor.timer` 指向不存在的 legacy path 時，可停用 stale timer 解除 host failed unit；但正式 source-of-truth 必須後續清理，不能把 `reset-failed` 當修復。
|
||
|
||
### 14.4.2 188 重啟後恢復指揮卡
|
||
|
||
188 是資料與 AI/Web 依賴主機。它恢復前，不釋出 K3s、AWOOOI API、momo heavy import 或 AI auto-remediation。
|
||
|
||
| 順序 | 檢查 | 成功基準 |
|
||
|------|------|----------|
|
||
| 1 | PostgreSQL | `pg_isready` accepting，無 checkpoint / WAL panic |
|
||
| 2 | Redis | `PONG` |
|
||
| 3 | Docker / containerd | active；momo-db / signoz / openclaw / litellm 非 restart loop |
|
||
| 4 | momo DB parity | `daily_sales_snapshot` 與 `realtime_sales_monthly` 目前月份筆數與日期上下界一致 |
|
||
| 5 | SignOz / monitoring bridge | HTTP 200；ClickHouse 不在修復風暴 |
|
||
| 6 | momo scheduler | container healthy，recent activity pattern > 0；heavy import 等 DB green 後釋出 |
|
||
| 7 | backup freshness | 188 backup textfile / 110 backup-from-188 freshness OK |
|
||
|
||
188 post-reboot 不可用「首頁 200」取代 DB parity；若出現 `posting list tuple ... cannot be split`，只走 `REINDEX TABLE CONCURRENTLY public.realtime_sales_monthly;`，不可 truncate 或整庫 restore。
|
||
|
||
### 14.4.3 120 恢復指揮卡
|
||
|
||
120 目前是 console-first blocker。它不可達時，遠端只能做證據收集，不能假裝修復。
|
||
|
||
| 狀態 | 判定 | 正確動作 |
|
||
|------|------|----------|
|
||
| ping / SSH / ARP 全失敗 | host / VM / network 層未知 | 到 hypervisor / console 確認 power、NIC、boot screen |
|
||
| initramfs / fsck prompt | filesystem repair gate | 依 `120-fsck-maintenance-checklist.sh` 離線處理 |
|
||
| SSH 恢復但 K3s NotReady | K3s / runtime 層 | 先看 `journalctl -u k3s`、containerd、188 PostgreSQL，再解除 cordon |
|
||
| node Ready 但 SchedulingDisabled | 調度狀態未解除 | 確認健康後 `kubectl uncordon mon`，再看 workload |
|
||
|
||
120 恢復後不得只看 `kubectl get nodes`。必須強制補跑：
|
||
|
||
```bash
|
||
/backup/scripts/backup-configs.sh
|
||
/backup/scripts/backup-all.sh
|
||
/backup/scripts/sync-offsite-backups.sh --mode sync
|
||
/backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color
|
||
/home/wooo/scripts/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1
|
||
```
|
||
|
||
### 14.4.4 121 重啟後恢復指揮卡
|
||
|
||
121 是 K3s failover / secondary control-plane path。它重啟後的核心是「不要讓 mon1 Ready 掩蓋 mon 不可達」。
|
||
|
||
| 檢查 | 成功基準 | 注意 |
|
||
|------|----------|------|
|
||
| SSH / systemd | host ready、failed units 可解釋 | 121 green 不代表 120 green |
|
||
| K3s role | `kubectl get nodes -o wide` 可讀 | 若只剩 `mon1 Ready`，仍是 degraded cluster |
|
||
| VIP / NodePort | VIP / public routes 通 | 必須確認 route 走 `192.168.0.125:32334/32335` |
|
||
| Cron / DR drill | cron present、DR drill 未被誤停 | schedule green 是 cold-start done criteria 的一部分 |
|
||
|
||
若 121 重啟後看到 `mon1 Ready` 但 `mon NotReady,SchedulingDisabled`，結論是「121 恢復，cluster 仍 degraded」，不可把 121 正常誤報成 K3s 全綠。
|
||
|
||
### 14.5 每次重啟紀錄格式
|
||
|
||
每次開機、關機、重啟都要在 `docs/LOGBOOK.md` 追加紀錄，並把必要狀態同步到本 SOP 或 workplan。
|
||
|
||
```text
|
||
## YYYY-MM-DD | Host reboot / shutdown / startup record
|
||
|
||
Scope:
|
||
- Hosts:
|
||
- Operation: shutdown / startup / reboot / recovery
|
||
- SOP version used:
|
||
- Operator:
|
||
- Maintenance window:
|
||
|
||
Pre-check:
|
||
- Cold-start scorecard:
|
||
- Backup status:
|
||
- Offsite verifier:
|
||
- Public routes:
|
||
- momo DB parity:
|
||
- Alertmanager rules / E2E:
|
||
- Credential escrow:
|
||
|
||
Execution:
|
||
- Start time:
|
||
- End time:
|
||
- Commands / console actions:
|
||
- Services paused:
|
||
- Services released:
|
||
|
||
Result:
|
||
- 110:
|
||
- 120:
|
||
- 121:
|
||
- 188:
|
||
- Cold-start scorecard after:
|
||
- Backup status after:
|
||
- Offsite verifier after:
|
||
- DB parity after:
|
||
- Alerts after:
|
||
|
||
Difference versus previous reboot:
|
||
- Faster:
|
||
- Slower:
|
||
- New blocker:
|
||
- Repeated blocker:
|
||
- False positive / detector tuning:
|
||
- SOP change required: yes/no
|
||
|
||
SOP update:
|
||
- Previous version:
|
||
- New version:
|
||
- Change reason:
|
||
- Files updated:
|
||
```
|
||
|
||
### 14.6 SOP 版本比較與改版規則
|
||
|
||
每次重啟後必須比較上一次紀錄，不只寫「已恢復」。
|
||
|
||
| 比較項 | 判定方式 |
|
||
|--------|----------|
|
||
| Time to SSH | 從 power-on 到各 host SSH OK |
|
||
| Time to K3s Ready | 從 120/121 boot 到 nodes Ready |
|
||
| Time to public routes | 從 K3s Ready 到 public 2xx/3xx |
|
||
| Time to backup green | 從 110 ready 到 backup status / offsite verifier green |
|
||
| Persistent blockers | 連續兩次以上出現即列入 SOP hard gate |
|
||
| False positives | 例如 momo scheduler detector WARN；要寫清楚直接證據與調整方向 |
|
||
| Procedure drift | live cron、Ansible template、script path 與 SOP 不一致時，先修 canonical，再修 SOP |
|
||
|
||
改版規則：
|
||
|
||
- 只更新 live baseline 或百分比：不升版，只更新日期與 evidence。
|
||
- 新增、刪除或改變操作順序：升 minor version，例如 `v1.4` -> `v1.5`。
|
||
- 牽涉破壞性操作、資料修復策略或人為批准邊界：升 major-ready review，先經人工批准。
|
||
|
||
### 14.7 2026-06-06 重啟紀錄比較錨點
|
||
|
||
2026-06-06 沒有執行新重啟；本次是 live recovery check。它仍要作為下一次重啟比較基準：
|
||
|
||
| 項目 | 2026-06-06 baseline |
|
||
|------|----------------------|
|
||
| Overall | `65% BLOCKED` |
|
||
| Cold-start | `PASS=71 WARN=3 BLOCKED=3` |
|
||
| Remaining hard blocker | 120 ping / SSH / K3s read-only check |
|
||
| Backup aggregate | `failed=1`, Configs only, due 120 config capture |
|
||
| Backup freshness | 110 and 188 fresh, no stale jobs |
|
||
| Offsite | 13 repos latest-only green |
|
||
| Escrow | 5 markers missing |
|
||
| momo scheduler | direct healthy; 15:03 scorecard no longer emits scheduler WARN |
|
||
|
||
### 14.8 2026-06-12 重啟後比較錨點
|
||
|
||
2026-06-12 110 被非計畫重啟後，本 SOP v1.5 的新比較錨點如下：
|
||
|
||
| 項目 | 2026-06-12 post-reboot baseline |
|
||
|------|----------------------------------|
|
||
| 110 host | `systemd running`，failed units `0`，Swap `0B/7.8GiB` |
|
||
| 110 service recovery | Harbor / Gitea / Prometheus / Alertmanager / Sentry / Stock / public tools reachable |
|
||
| Cold-start | `PASS=72 WARN=2 BLOCKED=3` |
|
||
| Remaining hard blocker | 120 ping / SSH / K3s read-only check |
|
||
| WARN | 120-driven backup aggregate/config component and 120 K3s schedule check |
|
||
| Backup freshness | 110 `13/13 fresh failed=1`，188 `2/2 fresh failed=0`，stale none |
|
||
| Offsite | 13 repos latest-only green，`REMOTE_LATEST_ONLY_OK=1`，`VERIFY_OK=1` |
|
||
| Alerts | Prometheus and Alertmanager expose all five required backup/cold-start/escrow alerts |
|
||
| momo scheduler | scorecard reads `SCHEDULER_RECENT_ACTIVITY 1070` after detector fix |
|
||
| SOP change | v1.5 adds startup judgment layers, GO/NO-GO tree, host recovery cards, and timeline checks |
|
||
|
||
### 14.9 2026-06-13 CD 後恢復比較錨點
|
||
|
||
2026-06-13 不是主機重啟，而是用來驗證「120/121 workload balancing + CD known_hosts guardrail」是否能承受下一次正常部署的比較錨點。
|
||
|
||
| 項目 | 2026-06-14 03:10 baseline |
|
||
|------|----------------------------|
|
||
| Gitea / ArgoCD | Gitea main `8868c025`，deploy marker `7b034b58`，ArgoCD revision `8868c025`，sync `Synced`，health `Degraded` |
|
||
| K3s image readback | API/Web/Worker/CronJob image tag `26b67d11f7b7de4f9c9d95c01bb1dacf4000e887` |
|
||
| K3s placement | API/Web verified split across `mon` / `mon1` after the latest deploy marker；Worker single replica healthy |
|
||
| Cold-start | `PASS=81 WARN=2 BLOCKED=0` |
|
||
| Public routes | Scorecard verifies awoooi API/Web, momo, gitea, harbor, registry, sentry, signoz, stock, langfuse, bitan, aiops over TLS |
|
||
| Backup | `backup-status`: 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5`，last aggregate `2026-06-14 02:40:22` |
|
||
| Offsite | textfile `remote_verify_ok=1`、`full_verify_fresh=1`，13 repos each `snapshot_count=1` |
|
||
| SSH trust | Global `known_hosts` retained 120 / 188 entries after CD; deploy-specific trust moved to `deploy_known_hosts` |
|
||
| Remaining non-service debt | `km-vectorize-29689620` official Job failed with `BackoffLimitExceeded`; failed Pod/log was deleted before inspection; credential escrow missing count remains `5`; 110 has `fwupd` failed units |
|
||
| SOP change | v1.10 changes the first-screen declaration from full green back to degraded, records official `km-vectorize` failure evidence, and verifies live `restartPolicy: Never` / `FallbackToLogsOnError` evidence retention for the next official run |
|
||
|
||
### 14.10 2026-06-14 110 failed-unit 清理比較錨點
|
||
|
||
2026-06-14 08:24 的變更不是主機重啟，而是把 110 非核心 `fwupd` failed-unit 噪音從 cold-start 判定中收斂。這個錨點的用途是避免未來把 firmware metadata refresh failure 誤判成 AWOOOI runtime 失敗，同時保留 rollback。
|
||
|
||
| 項目 | 2026-06-14 08:24 baseline |
|
||
|------|----------------------------|
|
||
| 110 failed units | `systemctl --failed` 回 `0 loaded units listed` |
|
||
| fwupd policy | `fwupd-refresh.timer` 為 `disabled / inactive`，原因是非核心 firmware metadata refresh 失敗不應阻擋 AWOOOI service recovery |
|
||
| Rollback | 若需要恢復 firmware metadata refresh timer，執行 `sudo systemctl enable --now fwupd-refresh.timer` 後重跑 cold-start |
|
||
| Cold-start | `PASS=82 WARN=1 BLOCKED=0` |
|
||
| Remaining WARN | 只剩 K8s failed Job `km-vectorize-29689620`；等待下一次官方 03:00 排程成功或保留 failed Pod/log |
|
||
| Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` |
|
||
| Credential escrow | 仍缺 5 個 non-secret evidence marker；不可用 placeholder 或 secret 清紅燈 |
|
||
| SOP change | v1.11 把 110 failed-unit gate 從 `GREEN_WITH_FWUPD_WARNING` 改成 `GREEN_WITH_FWUPD_TIMER_DISABLED`，並把完成宣告上限固定為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` |
|
||
|
||
### 14.11 2026-06-14 post-CD recovery readback
|
||
|
||
2026-06-14 08:40 的變更不是主機重啟，而是確認 latest CD deploy marker 沒有讓重啟恢復狀態倒退。這個錨點用來比較「治理 / 前端 / API CD 後，cold-start SOP 是否仍成立」。
|
||
|
||
| 項目 | 2026-06-14 08:40 post-CD baseline |
|
||
|------|------------------------------------|
|
||
| Gitea / ArgoCD | Gitea main `18b867c3`，ArgoCD revision `18b867c3`，sync `Synced`，health `Degraded` |
|
||
| K3s image readback | API/Web/Worker/CronJob image tag `e0a6d339669fc635357d36ea94215df25e652fa9` |
|
||
| CronJob readback | `km-vectorize` has `KM_PROJECT_ID=awoooi`、`restartPolicy: Never`、`terminationMessagePolicy: FallbackToLogsOnError`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z` |
|
||
| K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon` |
|
||
| Cold-start | `PASS=82 WARN=1 BLOCKED=0` |
|
||
| Public routes | Scorecard verifies awoooi API/Web, momo, gitea, harbor, registry, sentry, signoz, stock, langfuse, bitan, aiops over TLS |
|
||
| Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` |
|
||
| 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` |
|
||
| Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` |
|
||
| SOP change | v1.12 records the post-CD no-regression readback and keeps the declaration ceiling at `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` |
|
||
|
||
### 14.12 2026-06-14 P2-135 deploy 後 recovery readback
|
||
|
||
2026-06-14 09:27 的變更不是主機重啟，而是確認 P2-135 deploy 與正式驗證後，reboot recovery baseline 沒有倒退。這個錨點也記錄 stockplatform-v2 rollout warmup 期間短暫 `502` 的判定方式：直接重查 route / TLS，並重跑完整 cold-start；只有重跑仍失敗才升級成 persistent public route blocker。
|
||
|
||
| 項目 | 2026-06-14 09:27 post-P2-135 baseline |
|
||
|------|----------------------------------------|
|
||
| Gitea / ArgoCD | Gitea main `5bad267e`，ArgoCD revision `5bad267e`，sync `Synced`，health `Degraded` |
|
||
| K3s image readback | API/Web/Worker/CronJob image tag `280e0fbef0d5dccb10f1efe2cc18cf423544254e` |
|
||
| CronJob readback | `km-vectorize` has `KM_PROJECT_ID=awoooi`、`restartPolicy: Never`、`terminationMessagePolicy: FallbackToLogsOnError`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z` |
|
||
| K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon1` |
|
||
| First cold-start | 09:26 first run saw `stock.wooo.work` `502` while stockplatform-v2 containers were less than one minute old; direct route and TLS recheck returned `200` |
|
||
| Final cold-start | 09:27 rerun returned `PASS=82 WARN=1 BLOCKED=0` |
|
||
| Public routes | Final scorecard verifies awoooi API/Web, momo, gitea, harbor, registry, sentry, signoz, stock, langfuse, bitan, aiops over TLS |
|
||
| Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` |
|
||
| 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` |
|
||
| Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` |
|
||
| SOP change | v1.13 records the P2-135 post-deploy no-regression readback and keeps the declaration ceiling at `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` |
|
||
|
||
### 14.13 2026-06-14 P2-136 / AI Agent 活動正式部署後 recovery readback
|
||
|
||
2026-06-14 09:56 的變更不是主機重啟，而是確認 P2-136 / AI Agent 活動正式部署後，reboot recovery baseline 仍沒有倒退。這個錨點特別記錄 deploy marker、ArgoCD revision、live image 與 cold-start scorecard 必須一起看，避免只看 `gitea/main` 或 CD 成功就誤報 full-stack green。
|
||
|
||
| 項目 | 2026-06-14 09:56 post-P2-136 baseline |
|
||
|------|----------------------------------------|
|
||
| Gitea / ArgoCD | 本 recovery commit 前最新文件 head `a0fe7741`；runtime deploy marker `60a0415c chore(cd): deploy a3de0ff [skip ci]`，ArgoCD revision `60a0415c`，sync `Synced`，health `Degraded` |
|
||
| K3s image readback | API/Web/Worker/CronJob image tag `a3de0ffb8275b6838604b6dff87cd978b8e91122` |
|
||
| CronJob readback | `km-vectorize` has `KM_PROJECT_ID=awoooi`、`restartPolicy: Never`、`terminationMessagePolicy: FallbackToLogsOnError`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；failed Job `km-vectorize-29689620` remains retained |
|
||
| K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon1` |
|
||
| Cold-start | 09:56 returned `PASS=82 WARN=1 BLOCKED=0` |
|
||
| Public routes | Final scorecard verifies awoooi API/Web, momo, gitea, harbor, registry, sentry, signoz, stock, langfuse, bitan, aiops over TLS |
|
||
| Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` |
|
||
| 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` |
|
||
| Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` |
|
||
| SOP change | v1.14 records the P2-136 / AI Agent 活動正式部署後 no-regression readback and keeps the declaration ceiling at `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` |
|
||
|
||
### 14.14 2026-06-14 P2-137 / CI smoke timeout 修正後 recovery readback
|
||
|
||
2026-06-14 10:40 的變更不是主機重啟，而是確認 P2-137 正式部署與 BusyBox `timeout` smoke 修正後，reboot recovery baseline 仍沒有倒退。這個錨點只記錄 recovery / cold-start readback，不重複 P2-137 正式驗證內容。
|
||
|
||
| 項目 | 2026-06-14 10:40 post-P2-137 baseline |
|
||
|------|----------------------------------------|
|
||
| Gitea / ArgoCD | 本 recovery commit 前最新文件 head `50d4f2ba`；runtime deploy marker `d023f5d7 chore(cd): deploy f737f27 [skip ci]`，ArgoCD revision `50d4f2ba`，sync `Synced`，health `Degraded` |
|
||
| K3s image readback | API/Web/Worker/CronJob image tag `f737f278dc14372ff1fb15b124b1370c20e1bb99` |
|
||
| CronJob readback | `km-vectorize` has `KM_PROJECT_ID=awoooi`、`restartPolicy: Never`、`terminationMessagePolicy: FallbackToLogsOnError`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；failed Job `km-vectorize-29689620` remains retained |
|
||
| K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon` |
|
||
| Cold-start | 10:40 returned `PASS=82 WARN=1 BLOCKED=0` |
|
||
| Public routes | Final scorecard verifies awoooi API/Web, momo, gitea, harbor, registry, sentry, signoz, stock, langfuse, bitan, aiops over TLS |
|
||
| Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` |
|
||
| 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` |
|
||
| Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` |
|
||
| SOP change | v1.15 記錄 P2-137 / CI smoke timeout 修正後 no-regression readback，並維持宣告上限為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` |
|
||
|
||
### 14.15 2026-06-14 P2-143 owner response 預檢後 recovery readback
|
||
|
||
2026-06-14 15:00 的變更不是主機重啟，而是確認 P2-143 owner response 預檢與拒收邊界正式部署後，reboot recovery baseline 仍沒有倒退。這個錨點只記錄 recovery / cold-start readback，不重複 P2-142 / P2-143 正式驗證內容，也不把 owner response preflight 視為 runtime 授權。
|
||
|
||
| 項目 | 2026-06-14 15:00 post-P2-143 baseline |
|
||
|------|----------------------------------------|
|
||
| Gitea / ArgoCD | 最新文件基準 `b09eb1c6 docs(ai): 校準 P2-143 正式驗證紀錄`；runtime deploy marker `667d6329 chore(cd): deploy 755b0a8 [skip ci]`；ArgoCD revision `4abf0c0f750254d3c7137eae049abdfd99630f5f`，sync `Synced`，health `Degraded` |
|
||
| K3s image readback | API/Web/Worker/CronJob image tag `755b0a8d3038df2c52dee280067863d92db1eda5` |
|
||
| CronJob readback | `km-vectorize` schedule `0 3 * * *`、`timeZone=Asia/Taipei`、`suspend=false`、`failedJobsHistoryLimit=3`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；failed Job `km-vectorize-29689620` 仍保留，但目前沒有可讀的 failed Pod / log |
|
||
| K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon` |
|
||
| Cold-start | 15:00 returned `PASS=82 WARN=1 BLOCKED=0` |
|
||
| Public routes | 最終 scorecard 已驗證 awoooi API/Web、momo、gitea、harbor、registry、sentry、signoz、stock、langfuse、bitan、aiops 皆可透過 TLS 回應 |
|
||
| Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` |
|
||
| 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` |
|
||
| P2-143 API boundary | Production endpoint 回 current `P2-143`、next `P2-144`、completion `100`，且 reviewer / Gateway queue、Telegram、Bot API、result capture、learning、PlayBook trust、production write、secret read、destructive operation 全部維持 `0 / false` |
|
||
| Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` |
|
||
| SOP change | v1.16 記錄 P2-143 owner response 預檢後 no-regression readback，並維持宣告上限為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` |
|
||
|
||
### 14.16 2026-06-14 P2-144 owner response 回讀後 recovery readback
|
||
|
||
2026-06-14 15:58 的變更不是主機重啟，而是確認 P2-144 owner response 回讀狀態與後續 deploy marker `180a6543` 正式部署後，reboot recovery baseline 仍沒有倒退。這個錨點只記錄 recovery / cold-start readback，不重複 P2-144 正式驗證內容，也不把 owner response readback 視為 runtime 授權、正式收件或 owner acceptance。
|
||
|
||
| 項目 | 2026-06-14 15:58 post-P2-144 baseline |
|
||
|------|----------------------------------------|
|
||
| Gitea / ArgoCD | `gitea/main` 已前進至 `180a6543 chore(cd): deploy fef94df [skip ci]`；ArgoCD source revision `180a6543eaf26dd6b345d45114316926056a965a`，sync `Synced`，health `Degraded` |
|
||
| K3s image readback | API/Web/Worker/CronJob image tag `fef94df877c5438f9f34ddbcace8ad8112a141ef` |
|
||
| CronJob readback | `km-vectorize` schedule `0 3 * * *`、`timeZone=Asia/Taipei`、`suspend=false`、`failedJobsHistoryLimit=3`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；failed Job `km-vectorize-29689620` 仍保留，但目前沒有可讀的 failed Pod / log |
|
||
| K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon1` |
|
||
| Cold-start | 15:58 returned `PASS=82 WARN=1 BLOCKED=0` |
|
||
| Public routes | 最終 scorecard 已驗證 awoooi API/Web、momo、gitea、harbor、registry、sentry、signoz、stock、langfuse、bitan、aiops 皆可透過 TLS 回應 |
|
||
| Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` |
|
||
| 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` |
|
||
| P2-144 API boundary | Production endpoint 回 current `P2-144`、next `P2-145`、completion `100`，且 owner response received / accepted / rejected、reviewer / Gateway queue、Telegram、Bot API、result capture、learning、PlayBook trust、production write、secret read、destructive operation 全部維持 `0 / false` |
|
||
| Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` |
|
||
| SOP change | v1.17 記錄 P2-144 owner response 回讀後 no-regression readback，並維持宣告上限為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` |
|
||
|
||
### 14.17 2026-06-14 P2-145 owner response 驗收門檻後 recovery readback
|
||
|
||
2026-06-14 16:29 的變更不是主機重啟，而是確認 P2-145 owner response 驗收門檻正式部署後，reboot recovery baseline 仍沒有倒退。這個錨點只記錄 recovery / cold-start readback，不重複 P2-145 正式驗證內容，也不把 acceptance gate 視為 owner response received / accepted、runtime 授權或正式寫入。
|
||
|
||
| 項目 | 2026-06-14 16:29 post-P2-145 baseline |
|
||
|------|----------------------------------------|
|
||
| Gitea / ArgoCD | 最新文件基準 `06fe0a8f docs(logbook): 記錄 P2-145 正式驗證 [skip ci]`；runtime deploy marker `36fbfc6b chore(cd): deploy 386dbd0 [skip ci]`；ArgoCD source revision `06fe0a8f14167824fea512f942d2569431bbcbc8`，sync `Synced`，health `Degraded` |
|
||
| K3s image readback | API/Web/Worker/CronJob image tag `386dbd078ef63401d9736048463f4ef5326442d9` |
|
||
| CronJob readback | `km-vectorize` schedule `0 3 * * *`、`timeZone=Asia/Taipei`、`suspend=false`、`failedJobsHistoryLimit=3`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；failed Job `km-vectorize-29689620` 仍為 `Failed 0/1` |
|
||
| K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon` |
|
||
| Cold-start | 16:29 returned `PASS=82 WARN=1 BLOCKED=0` |
|
||
| Public routes | 最終 scorecard 已驗證 awoooi API/Web、momo、gitea、harbor、registry、sentry、signoz、stock、langfuse、bitan、aiops 皆可透過 TLS 回應 |
|
||
| Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` |
|
||
| 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` |
|
||
| P2-145 API boundary | Production endpoint 回 current `P2-145`、next `P2-146`、completion `100`，且 owner response received / accepted / rejected、reviewer / Gateway queue、Telegram、Bot API、result capture、learning、PlayBook trust、production write、secret read、destructive operation 全部維持 `0 / false` |
|
||
| Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` |
|
||
| SOP change | v1.18 記錄 P2-145 owner response 驗收門檻後 no-regression readback，並維持宣告上限為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` |
|
||
|
||
### 14.18 2026-06-14 IwoooS P0 配置控管優先序後 recovery readback
|
||
|
||
2026-06-14 17:04 的變更不是主機重啟，而是確認 IwoooS P0 配置控管優先序正式部署後，reboot recovery baseline 仍沒有倒退。這個錨點只記錄 recovery / cold-start readback，不重複 P0 配置控管正式驗證內容，也不把前台看板可見視為 Nginx reload、DNS / TLS probe、certbot renew、workflow / secret 修改、public route change 或 runtime gate。
|
||
|
||
| 項目 | 2026-06-14 17:04 post-IwoooS-P0-config baseline |
|
||
|------|--------------------------------------------------|
|
||
| Gitea / ArgoCD | 最新文件基準 `af62ec1f docs(iwooos): 記錄 P0 配置控管正式驗證 [skip ci]`；runtime deploy marker `ed651a98 chore(cd): deploy e992af8 [skip ci]`；ArgoCD source revision `af62ec1fe72b3e84e179d80e788e5a5902bdaf27`，sync `Synced`，health `Degraded` |
|
||
| K3s image readback | API/Web/Worker/CronJob image tag `e992af89955f8aae40a383b2f2e2f645445a690d` |
|
||
| CronJob readback | `km-vectorize` schedule `0 3 * * *`、`timeZone=Asia/Taipei`、`suspend=false`、`failedJobsHistoryLimit=3`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；failed Job `km-vectorize-29689620` 仍為 `Failed 0/1` |
|
||
| K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon1` |
|
||
| Cold-start | 17:04 returned `PASS=82 WARN=1 BLOCKED=0` |
|
||
| Public routes | 最終 scorecard 已驗證 awoooi API/Web、momo、gitea、harbor、registry、sentry、signoz、stock、langfuse、bitan、aiops 皆可透過 TLS 回應；IwoooS route `/zh-TW/iwooos` 額外 readback 回 `200` |
|
||
| Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` |
|
||
| 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` |
|
||
| IwoooS boundary | P0 配置控管優先序已可見，但 live evidence received、runtime gate、Nginx live config、DNS / TLS probe、certbot renew、workflow / secret 修改、public route change、production write 仍不得從本 readback 推定為已授權 |
|
||
| Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` |
|
||
| SOP change | v1.19 記錄 IwoooS P0 配置控管優先序後 no-regression readback，並維持宣告上限為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` |
|
||
|
||
### 14.19 2026-06-14 高價值配置 Owner Packet 前台同步後 recovery readback
|
||
|
||
2026-06-14 18:15 的變更不是主機重啟，而是確認高價值配置 Owner Packet 前台同步正式部署後，reboot recovery baseline 仍沒有倒退。這個錨點只記錄 recovery / cold-start readback，不重複 Owner Packet 前台正式驗證、posture projection 或 intake preflight 內容，也不把前台草案可見視為 request sent、owner response received / accepted、runtime gate、Nginx reload、DNS / TLS probe、certbot renew、workflow / secret 修改、host write、active scan 或 production write。
|
||
|
||
| 項目 | 2026-06-14 18:15 post-owner-packet-frontend baseline |
|
||
|------|-------------------------------------------------------|
|
||
| Gitea / ArgoCD | 最新 repo 文件基準 `0a4766dd docs(security): 新增高價值配置 owner request 草稿包 [skip ci]`；runtime deploy marker `16c6b983 chore(cd): deploy e999c16 [skip ci]`；feature commit `e999c16b fix(iwooos): 同步高價值配置 owner packet 前台`；ArgoCD source revision `0a4766ddc94b0690824ce3deba5c6b9a69764f94`，sync `Synced`，health `Degraded` |
|
||
| K3s image readback | API/Web/Worker/CronJob image tag `e999c16b3435f197b78fe2adfeec1c4faa6c4675` |
|
||
| CronJob readback | `km-vectorize` schedule `0 3 * * *`、`timeZone=Asia/Taipei`、`suspend=false`、`failedJobsHistoryLimit=3`、`lastScheduleTime=2026-06-13T19:00:00Z`、`lastSuccessfulTime=2026-06-04T11:00:37Z`；failed Job `km-vectorize-29689620` 仍為 `Failed 0/1` |
|
||
| K3s placement | API pods split `mon` / `mon1`，Web pods split `mon` / `mon1`，Worker single replica on `mon` |
|
||
| Cold-start | 18:15 returned `PASS=82 WARN=1 BLOCKED=0` |
|
||
| Public routes | 最終 scorecard 已驗證 awoooi API/Web、momo、gitea、harbor、registry、sentry、signoz、stock、langfuse、bitan、aiops 皆可透過 TLS 回應；IwoooS route `/zh-TW/iwooos` 與 AwoooP route `/zh-TW/awooop` 額外 readback 皆回 `200` |
|
||
| Backup | 110 `13/13 fresh failed=0`，188 `2/2 fresh failed=0`，`core_blockers=0`，`escrow_missing=5` |
|
||
| 110 host | `systemctl --failed` 回 `0 loaded units listed`；`fwupd-refresh.timer` 維持 `disabled / inactive` |
|
||
| Owner Packet boundary | Owner Packet 前台數字已可見，但 request sent、owner response received / accepted / rejected、reviewer queue write、live evidence、runtime gate、Nginx live config、DNS / TLS probe、certbot renew、workflow / secret 修改、host write、active scan、production write 仍不得從本 readback 推定為已授權 |
|
||
| Remaining gate | `km-vectorize-29689620` official Job 仍 failed；Credential escrow missing count 仍 `5` |
|
||
| SOP change | v1.20 記錄高價值配置 Owner Packet 前台同步後 no-regression readback，並維持宣告上限為 `SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED` |
|
||
|
||
### 14.20 重啟後時間軸驗證
|
||
|
||
每次重啟後照時間軸推進，不要等到最後才一次判定。
|
||
|
||
| 時間點 | 目標 | 必跑證據 | 可以宣稱 |
|
||
|--------|------|----------|----------|
|
||
| `T+0` | power / VM / console 已開始 | console / hypervisor / UPS / operator note | maintenance started |
|
||
| `T+5m` | LAN / SSH 回復 | ping、ARP、SSH port、`who -b` | `HOST_BOOTED` |
|
||
| `T+15m` | 主機基礎服務回復 | `systemctl is-system-running`、failed units、Docker / PostgreSQL / Redis / K3s role checks | `HOST_READY` |
|
||
| `T+30m` | 核心服務回復 | 188 DB、110 Harbor/Gitea/Prom/AM、K3s nodes、AWOOOI API/Web、public routes | `SERVICE_READY` for scoped hosts |
|
||
| `T+45m` | 排程與資料一致性 | backup status、offsite verifier、momo DB parity、CronJobs、alert visibility | service recovery confidence |
|
||
| `T+60m` | 釋出高負載與自動化 | cold-start scorecard、load/core、runner guardrails、AI observe-only gate | release runner/CD only if gates allow |
|
||
|
||
若任一時間點卡住，記錄卡在哪個 gate，不要跳到下一層。連續兩次重啟都卡同一 gate，必須回寫 §16 Known Drift 或 workplan。
|
||
|
||
---
|
||
|
||
## 15. Done Criteria
|
||
|
||
All must be true:
|
||
|
||
- Four hosts reachable by SSH.
|
||
- 188 PostgreSQL and Redis healthy.
|
||
- 110 Harbor, Gitea, Prometheus, Alertmanager healthy.
|
||
- 120/121 K3s nodes Ready.
|
||
- VIP `192.168.0.125` present.
|
||
- AWOOOI API and Web reachable through NodePort/VIP.
|
||
- Alertmanager E2E webhook succeeds.
|
||
- cron/CronJob schedules are active, unsuspended, and verified.
|
||
- momo `daily_sales_snapshot` 與 `realtime_sales_monthly` 在最新匯入日期範圍內筆數一致。
|
||
- Sentry and SignOz are either healthy or explicitly in controlled backlog recovery.
|
||
- High-load batch services are capped or delayed.
|
||
- Runners are guarded and released last.
|
||
- AI auto-remediation is not in full execution mode until all gates are green.
|
||
- 110 cold-start textfile monitor is fresh, and Prometheus has the cold-start alert rules loaded.
|
||
- 110 global `/home/wooo/.ssh/known_hosts` still contains verified 120 / 188 entries after any CD run; deploy jobs use `/home/wooo/.ssh/deploy_known_hosts` only.
|
||
|
||
### 15.1 可宣稱狀態
|
||
|
||
| 可宣稱文字 | 必要條件 |
|
||
|------------|----------|
|
||
| `110 host recovered` | 110 `HOST_READY`，failed units `0` 或全部可解釋，核心端口與 cron / backup status 已查 |
|
||
| `public core services recovered` | public routes/TLS 2xx/3xx，AWOOOI API health、Harbor/Gitea/Stock/Sentry/SignOz/Langfuse/Bitan smoke OK |
|
||
| `backup/offsite current` | `backup-status --no-notify` 無 stale，offsite verifier `VERIFY_OK=1`，且任何 failed component 有明確 owner |
|
||
| `service recovery with known blocker` | cold-start `BLOCKED` 只剩已知 blocker，例如 120；告警保持可見 |
|
||
| `full-stack green` | §15 全部成立，cold-start `WARN=0 BLOCKED=0` |
|
||
| `DR complete` | full-stack green 且 credential escrow missing count 為 0 |
|
||
|
||
---
|
||
|
||
## 16. Known Drift To Fix After Recovery
|
||
|
||
這些項目必須在事故後整理，不要在 P0 恢復中途順手大改。
|
||
|
||
- `SERVICE-ENDPOINTS.md` still has old Prometheus/Alertmanager locations.
|
||
- Audit older docs for direct node webhook targets; current main path should be VIP `192.168.0.125:32334`.
|
||
- OpenClaw `8088` vs `8089` must be live-confirmed and normalized.
|
||
- 188 compose paths drift between `/home/ollama/*` and Ansible `/opt/*`.
|
||
- 110 runner docs still mention Docker runner in places; live startup prefers host `gitea-act-runner-host.service`.
|
||
- `scripts/setup-runner-watchdog.sh` conflicts with the 2026-05-05 runner watchdog disablement guardrail.
|
||
- `grist.wooo.work` / `registry.wooo.work` public HTTP/HTTPS currently route to `aiops.wooo.work`; their old 110 certbot renewal configs are disabled until public routing is corrected or DNS-01 renewal is configured.
|
||
- `stockplatform-shared-ui-monitor.timer` / service source-of-truth 仍需清理或重建；2026-06-12 只停用 stale timer 以解除 host degraded。
|
||
- 111 local Ollama fallback 目前不可達；production provider 由 GCP-A / GCP-B 承接，但 111 恢復應另列 AI provider resilience 工作。
|
||
- 本 SOP v1.5 新增內容已用繁體中文補強；舊章節仍有英文段落，後續 runbook hygiene 應分批翻譯，不要在事故 P0 中混入大規模格式重排。
|