wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 3 Packages Projects Releases Wiki Activity

fix(k8s): retain km-vectorize failed pod evidence
Some checks failed
Code Review / ai-code-review (push) Successful in 13s
Details
CD Pipeline / tests (push) Successful in 1m27s
Details
CD Pipeline / post-deploy-checks (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Your Name
2026-06-14 03:15:15 +08:00
parent c82f320b97
commit 8868c0255d
4 changed files with 61 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md
View File

@@ -11,13 +11,13 @@
| Area | Status | Completion | Evidence |
|------|--------|------------|----------|
| Overall recovery readiness | SERVICE_GREEN_WORKLOAD_BALANCED_DR_ESCROW_BLOCKED | 96% | 2026-06-13 14:16 final cold-start scorecard is `PASS=83 WARN=0 BLOCKED=0`; 120/121 K3s are both `Ready control-plane`, backup core blockers remain `0`, public routes/TLS/momo DB/schedules/Alertmanager are green, deploy marker `a520c32d` put API/Web/Worker image `e897c8bf` live, API/Web remain live-verified split across 120 / 121, and CD no longer clobbers global `known_hosts`. 14:16 escrow status still has `escrow_missing=5`; DR remains blocked by five missing credential escrow evidence markers. ArgoCD `km-vectorize` is tracked separately as governance health debt until its official scheduled Job refreshes `lastSuccessfulTime`. |
| Overall recovery readiness | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | 94% | 2026-06-14 03:11 cold-start scorecard is `PASS=81 WARN=2 BLOCKED=0`; 120/121 K3s are both `Ready control-plane`, backup core blockers remain `0`, public route/API smoke remains green, deploy marker `7b034b58` put API/Web/Worker image `26b67d11` live, and API/Web remain live-verified split across 120 / 121. The 03:00 official `km-vectorize-29689620` Job failed with `BackoffLimitExceeded`, and 110 has `fwupd` failed units, so full cold-start cannot be declared green. DR remains blocked by five missing credential escrow evidence markers. |
| P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; host is reachable, root is mounted `rw`, failed units `0`, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 92% | 2026-06-13 14:16 `backup-status` shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-13 02:34:06`; 13:10 escrow report shows `SCRIPT_MISSING_COUNT=0`, `OFFSITE_CONFIGURED=1`, `RCLONE_CONFIGURED=1`, `ESCROW_MISSING_COUNT=5`, `PASS=8 WARN=5 BLOCKED=0`. Owner request package is ready; actual marker write remains blocked on real non-secret evidence IDs. |
| P2 service / data truth | VERIFIED_WORKLOAD_BALANCED | 100% | 2026-06-13 14:16 cold-start is green; public routes/TLS are green, VIP API/Web are reachable, momo current-month parity remains covered by the scorecard, schedules/services are green. API/Web both keep 120 / 121 split placement after latest ArgoCD revision `a520c32d`, with live API/Web/Worker image `e897c8bf`. |
| P3 docs / automation contracts | DONE_WITH_VALIDATION_GAP | 100% | Workplan, SOP v1.9, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, T+0/T+60 timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, and `km-vectorize` remediation tracking are updated; Ansible syntax check is unavailable on this workstation. |
| P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 92% | 2026-06-14 03:11 `backup-status` shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-14 02:40:22`. Owner request package is ready; actual marker write remains blocked on real non-secret evidence IDs. |
| P2 service / data truth | VERIFIED_WORKLOAD_BALANCED_WITH_CRON_WARN | 98% | 2026-06-14 03:11 cold-start is degraded by warnings only; public route/API smoke is green, VIP API/Web are reachable, momo current-month parity remains covered by the scorecard, schedules/services are mostly green. API/Web both keep 120 / 121 split placement after latest ArgoCD revision `c82f320b`, with live API/Web/Worker image `26b67d11`; the exception is failed `km-vectorize-29689620`. |
| P3 docs / automation contracts | DONE_WITH_VALIDATION_GAP | 100% | Workplan, SOP v1.10, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, T+0/T+60 timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, and `km-vectorize` remediation tracking are updated; Ansible syntax check is unavailable on this workstation. |
Full cold-start may be declared green only for the latest verified evidence set. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
Full cold-start may be declared green only for the latest verified evidence set. As of 2026-06-14 03:11, the latest evidence set is degraded, not green. Do not declare DR scorecard complete while credential escrow evidence remains blocked.
2026-06-13 01:26 refresh: full cold-start is again green for the current evidence set. AWOOOI API/Web workload balancing survived the next normal CD deploy: Gitea main `e4a349bc`, ArgoCD revision `e4a349bc`, images from `414413a5`, API/Web split across `mon` / `mon1`, and global `known_hosts` retained 120 / 188 after CD fix `80e6ec1a`. Do not declare DR complete while credential escrow is missing. `km-vectorize` remediation is `90%`: schedule/label fix is live, and the remaining gate is the next official 03:00 CronJob success readback.
@@ -62,6 +62,7 @@ Full cold-start may be declared green only for the latest verified evidence set.
| 2026-06-13 security mirror production image closeout | LIVE_VERIFIED | Gitea main `64ea2444` records the Web rebuild trigger. Deploy marker `2cc02f1c chore(cd): deploy 6cf8d3c [skip ci]` put Web image `6cf8d3ca` live; ArgoCD source revision later advanced to `64ea2444` while Web image correctly remains `6cf8d3ca` because `64ea2444` is docs/changelog only. Public `/zh-TW/governance` and `/en/governance` return `200`, API health is `healthy`, `security-mirror-progress-guard.py` passes, and 14:10 cold-start is `PASS=83 WARN=0 BLOCKED=0`. |
| 2026-06-13 final post-trigger deploy closeout | LIVE_VERIFIED | Deploy marker `834ccdba chore(cd): deploy bf86017 [skip ci]` put API/Web/Worker image `bf860177` live. ArgoCD revision `834ccdba` is `Synced / Degraded` only by `km-vectorize`; routes `/zh-TW/governance` and `/en/governance` return `200`, API health is `healthy`, source guards pass, backup status has `core_blockers=0` and `escrow_missing=5`, and 14:13 cold-start is `PASS=83 WARN=0 BLOCKED=0`. |
| 2026-06-13 final goal audit refresh | SERVICE_GREEN_REMAINING_GATES_EXPLICIT | Clean worktree rebased onto `a520c32d` and reran source guards successfully; live ArgoCD tracks revision `a520c32d` with API/Web/Worker image `e897c8bf`, health `Degraded` only by `km-vectorize`; `km-vectorize` schedule remains `0 3 * * *`, `timeZone=Asia/Taipei`, `failedJobsHistoryLimit=3`, and no failed Job is currently retained. Public `/zh-TW/governance`, `/en/governance`, and `/api/v1/health` are green; backup core blockers remain `0`, `escrow_missing=5`; 14:16 cold-start is `PASS=83 WARN=0 BLOCKED=0`. Remaining gates: five credential escrow markers and next official 03:00 `km-vectorize` success readback. |
| 2026-06-14 `km-vectorize` official run follow-up | DEGRADED_EVIDENCE_RETENTION_PATCHED | 03:00 official `km-vectorize-29689620` ran from CronJob and failed with `BackoffLimitExceeded`; ArgoCD revision `c82f320b` remains `Synced / Degraded`. Job is retained, but failed Pod `km-vectorize-29689620-nwpqz` was deleted before logs could be read, so root cause remains unproven. GitOps candidate changes `restartPolicy: Never` and adds `terminationMessagePolicy: FallbackToLogsOnError` so the next official failure should retain Pod/log evidence. Backup core remains green, `escrow_missing=5`, and 03:11 cold-start is `PASS=81 WARN=2 BLOCKED=0`. |
---
@@ -134,7 +135,7 @@ Next: <single next action>
| P1-011 | DONE | 100 | Confirm 2026-06-12 backup convergence | 18:55 live check confirms the post-120 aggregate held: no stale jobs, no configured/missing script jobs, no failed components, offsite fresh, and only credential escrow remains as DR warning. | Keep escrow as explicit red gate. | `stale110=none`, `stale188=none`, `failed=0`, `config_failed=0`, `core_blockers=0`. |
| P1-012 | DONE | 100 | Audit credential escrow marker write safety | 2026-06-12 15:02 `mark-credential-escrow-verified.sh --status` reports all five allowed items missing; `offsite-escrow-evidence-report.sh --no-color` reports rclone/offsite configured and `ESCROW_MISSING_COUNT=5`; repo search found only runbooks/placeholders/rules, not real evidence IDs. | Write markers only after a real non-secret evidence ID exists for each item; never write placeholder or secret. | The marker blocker is narrowed to missing external evidence IDs, not missing script/config/offsite readiness. |
| P1-014 | DONE | 100 | Publish credential escrow owner request package | 2026-06-13 13:10 live report confirms `SCRIPT_MISSING_COUNT=0`, `OFFSITE_CONFIGURED=1`, `RCLONE_CONFIGURED=1`, `ESCROW_MISSING_COUNT=5`, `PASS=8 WARN=5 BLOCKED=0`. New owner request package defines allowed evidence-id types, forbidden secret values, safe dry-run flow, write flow, and closeout gates. | Dispatch to the credential owners without collecting secret values; keep marker write gated until owner gives real non-secret evidence IDs. | `docs/security/CREDENTIAL-ESCROW-EVIDENCE-OWNER-REQUEST.md` and snapshot exist and validate. |
| P1-013 | IN_PROGRESS | 92 | Remediate `km-vectorize` CronJob health debt | ArgoCD Degraded is isolated to `CronJob/km-vectorize`: `lastSuccessfulTime` is stale even though retained 6/2-6/4 Jobs completed. The schedule is now correct (`0 3` with `timeZone: Asia/Taipei`), but the 2026-06-13 03:00 run did not update `lastSuccessfulTime` and no failed Job was retained because `failedJobsHistoryLimit=0`. This hid the failure evidence and slowed triage. | Apply GitOps `failedJobsHistoryLimit=3`, verify ArgoCD sync, then verify the next 03:00 official CronJob either succeeds or leaves inspectable failure evidence. | `lastSuccessfulTime` is after the manifest sync, the official scheduled Job is `Complete`, and ArgoCD `awoooi-prod` health is `Healthy`; if it fails, the failed Job/Pod/log remains available. |
| P1-013 | IN_PROGRESS | 94 | Remediate `km-vectorize` CronJob health debt | ArgoCD Degraded is isolated to `CronJob/km-vectorize`. The schedule is correct (`0 3` with `timeZone: Asia/Taipei`) and `failedJobsHistoryLimit=3` retained the 2026-06-14 failed Job, but `restartPolicy: OnFailure` still allowed the failed Pod/log to be deleted before inspection. The 03:00 official Job `km-vectorize-29689620` failed with `BackoffLimitExceeded`; root cause remains unproven without logs. | Apply GitOps evidence-retention patch: `restartPolicy: Never` plus `terminationMessagePolicy: FallbackToLogsOnError`, verify ArgoCD sync, then verify the next 03:00 official CronJob either succeeds or leaves inspectable failed Pod/log evidence. Do not manual-run or patch live. | `lastSuccessfulTime` is after the manifest sync, the official scheduled Job is `Complete`, and ArgoCD `awoooi-prod` health is `Healthy`; if it fails, the failed Job/Pod/log remains available for read-only triage. |
---