From 32a1d9012fc6602b69027a0923d6d1bd1ac5ce3e Mon Sep 17 00:00:00 2001 From: Your Name Date: Sun, 14 Jun 2026 08:22:18 +0800 Subject: [PATCH] docs(ops): record km-vectorize project context live [skip ci] --- docs/LOGBOOK.md | 5 ++++- docs/runbooks/FULL-STACK-COLD-START-SOP.md | 6 +++--- ...26-06-04-reboot-cold-start-backup-recovery-workplan.md | 8 ++++---- 3 files changed, 11 insertions(+), 8 deletions(-) diff --git a/docs/LOGBOOK.md b/docs/LOGBOOK.md index 33ff43531..9a51f3475 100644 --- a/docs/LOGBOOK.md +++ b/docs/LOGBOOK.md @@ -11,9 +11,12 @@ - Targeted pytest：`DATABASE_URL=postgresql+asyncpg://test:test@127.0.0.1:5432/test pytest apps/api/tests/test_cron_km_vectorize.py apps/api/tests/test_db_context_guard.py -q` → `7 passed`。 - `kubectl kustomize k8s/awoooi-prod` 渲染確認 `KM_PROJECT_ID=awoooi`、`restartPolicy: Never`、`terminationMessagePolicy: FallbackToLogsOnError`。 - YAML parse 與 `git diff --check` 通過。 +- Deploy marker：`ec03f0b7 chore(cd): deploy 8ddb80d [skip ci]`。 +- Live readback：ArgoCD revision `ec03f0b75908762300a420bbd27fca3e271cd5e5`，`Synced / Degraded`；API / Worker / `km-vectorize` CronJob image 均為 `8ddb80d63d2dd7d0406f4c9c3899d539f18e1613`。 +- Live CronJob readback：`KM_PROJECT_ID=awoooi`、`restartPolicy: Never`、`terminationMessagePolicy: FallbackToLogsOnError`。 **邊界：** -- 這是 root-cause candidate 修正，不是完成證明；仍必須等下一次官方 03:00 `km-vectorize` 成功更新 `lastSuccessfulTime`，或失敗時留下 Pod/log 證據再繼續修。 +- 這是 root-cause candidate 修正且已部署，不是完成證明；仍必須等下一次官方 03:00 `km-vectorize` 成功更新 `lastSuccessfulTime`，或失敗時留下 Pod/log 證據再繼續修。 - 不手動建立 Job、不 patch live、不刪 failed Job、不偽造 credential escrow evidence。 ## 2026-06-14｜P2-133 Final release candidate readback 完成與正式驗證 diff --git a/docs/runbooks/FULL-STACK-COLD-START-SOP.md b/docs/runbooks/FULL-STACK-COLD-START-SOP.md index 08e4f12f3..08827991f 100644 --- a/docs/runbooks/FULL-STACK-COLD-START-SOP.md +++ b/docs/runbooks/FULL-STACK-COLD-START-SOP.md @@ -29,8 +29,8 @@ | Cold-start scorecard | 03:11 read-only scorecard after official `km-vectorize` run: `PASS=81 WARN=2 BLOCKED=0` | `DEGRADED_NO_BLOCKERS` | | momo DB parity | `4571|4571|2026-06-01|2026-06-07|2026-06-01|2026-06-07` | `GREEN` | | momo scheduler | container healthy; scorecard reads `SCHEDULER_RECENT_ACTIVITY 1136`; detector widened and deployed to 110 | `GREEN` | -| ArgoCD app health | Latest live app remains `Synced / Degraded` because official Job `km-vectorize-29689620` failed with `BackoffLimitExceeded`. Root-cause candidate is missing `X-Project-ID` on the internal `/api/v1/knowledge/embed-all` call; GitOps candidate now adds `KM_PROJECT_ID=awoooi` and sends `X-Project-ID`. Existing live evidence retention remains `restartPolicy: Never` and `terminationMessagePolicy: FallbackToLogsOnError`. | `TENANT_CONTEXT_FIX_WAITING_OFFICIAL_RUN` | -| Workload balancing | Live API/Web/Worker image is `26b67d11`; API/Web pods remain ready, Worker single replica remains healthy | `GREEN` | +| ArgoCD app health | Latest live app revision `ec03f0b7` remains `Synced / Degraded` because official Job `km-vectorize-29689620` failed with `BackoffLimitExceeded`. Root-cause candidate is missing `X-Project-ID` on the internal `/api/v1/knowledge/embed-all` call; deployed image `8ddb80d6` now sends `X-Project-ID` and live CronJob has `KM_PROJECT_ID=awoooi`, `restartPolicy: Never`, and `terminationMessagePolicy: FallbackToLogsOnError`. | `TENANT_CONTEXT_FIX_LIVE_WAITING_OFFICIAL_RUN` | +| Workload balancing | Live API/Web/Worker image is `8ddb80d6`; API/Web pods remain ready, Worker single replica remains healthy | `GREEN` | | Credential escrow | 5 non-secret evidence markers missing | `BLOCKED` | Release rule: @@ -52,7 +52,7 @@ GO for "AWOOOI core workload balanced"; topology spread is in Gitea main / ArgoC NO-GO for "full cold-start green" until 110 failed units are resolved/accepted and `km-vectorize` failed Job is cleared by an official successful run. NO-GO for "ArgoCD fully healthy" until `km-vectorize` updates `lastSuccessfulTime` after an official scheduled Job, not a manual `UnexpectedJob`. NO-GO for any CD workflow that writes deploy host keys into `/home/wooo/.ssh/known_hosts`; deploy jobs must use an isolated `UserKnownHostsFile=/home/wooo/.ssh/deploy_known_hosts`. -Current allowed wording: "core service and backup are available; cold-start is degraded by `km-vectorize` official Job failure and 110 fwupd failed units; DR complete still blocked by credential escrow; `km-vectorize` failed Job is retained, live CronJob keeps failed Pod/log evidence, and tenant context fix is waiting for the next official run." +Current allowed wording: "core service and backup are available; cold-start is degraded by `km-vectorize` official Job failure and 110 fwupd failed units; DR complete still blocked by credential escrow; `km-vectorize` failed Job is retained, live CronJob keeps failed Pod/log evidence, and deployed tenant context fix is waiting for the next official run." ``` After any future 120 recovery, rerun this exact chain from 110: diff --git a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md index abe7f7382..36d748ab4 100644 --- a/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md +++ b/docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md @@ -11,10 +11,10 @@ | Area | Status | Completion | Evidence | |------|--------|------------|----------| -| Overall recovery readiness | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | 95% | 2026-06-14 03:11 cold-start scorecard is `PASS=81 WARN=2 BLOCKED=0`; 120/121 K3s are both `Ready control-plane`, backup core blockers remain `0`, public route/API smoke remains green, deploy marker `7b034b58` put API/Web/Worker image `26b67d11` live, and API/Web remain live-verified split across 120 / 121. The 03:00 official `km-vectorize-29689620` Job failed with `BackoffLimitExceeded`, and 110 has `fwupd` failed units, so full cold-start cannot be declared green. Evidence retention is live; root-cause candidate fix adds `KM_PROJECT_ID=awoooi` / `X-Project-ID` for fail-closed RLS. DR remains blocked by five missing credential escrow evidence markers. | +| Overall recovery readiness | SERVICE_AVAILABLE_KM_VECTORIZE_FAILED_DR_ESCROW_BLOCKED | 96% | 2026-06-14 03:11 cold-start scorecard is `PASS=81 WARN=2 BLOCKED=0`; 120/121 K3s are both `Ready control-plane`, backup core blockers remain `0`, public route/API smoke remains green, deploy marker `ec03f0b7` put API/Web/Worker/CronJob image `8ddb80d6` live, and API/Web remain live-verified split across 120 / 121. The 03:00 official `km-vectorize-29689620` Job failed with `BackoffLimitExceeded`, and 110 has `fwupd` failed units, so full cold-start cannot be declared green. Evidence retention and tenant-context fix are live; DR remains blocked by five missing credential escrow evidence markers. | | P0 host / K3s recovery | DONE | 100% | 120 booted after console fsck at `2026-06-12 15:13`; host is reachable, root is mounted `rw`, failed units `0`, `mon` and `mon1` are both `Ready control-plane`, and cold-start P0/P1 checks are green. | | P1 backup / alert / escrow | BLOCKED_DR_ESCROW | 92% | 2026-06-14 03:11 `backup-status` shows 110 `13/13 fresh failed=0`, 188 `2/2 fresh failed=0`, `core_blockers=0`, `escrow_missing=5`, last aggregate `2026-06-14 02:40:22`. Owner request package is ready; actual marker write remains blocked on real non-secret evidence IDs. | -| P2 service / data truth | VERIFIED_WORKLOAD_BALANCED_WITH_CRON_WARN | 98% | 2026-06-14 03:11 cold-start is degraded by warnings only; public route/API smoke is green, VIP API/Web are reachable, momo current-month parity remains covered by the scorecard, schedules/services are mostly green. API/Web both keep 120 / 121 split placement after latest ArgoCD revision `c82f320b`, with live API/Web/Worker image `26b67d11`; the exception is failed `km-vectorize-29689620`. | +| P2 service / data truth | VERIFIED_WORKLOAD_BALANCED_WITH_CRON_WARN | 98% | 2026-06-14 03:11 cold-start is degraded by warnings only; public route/API smoke is green, VIP API/Web are reachable, momo current-month parity remains covered by the scorecard, schedules/services are mostly green. API/Web both keep 120 / 121 split placement after latest ArgoCD revision `ec03f0b7`, with live API/Web/Worker image `8ddb80d6`; the exception is failed `km-vectorize-29689620`. | | P3 docs / automation contracts | DONE_WITH_VALIDATION_GAP | 100% | Workplan, SOP v1.10, BACKUP-STATUS, LOGBOOK, 120 console/fsck recovery, Gitea backup stale-dump hardening, reboot ledger/version-comparison SOP, escrow evidence audit, 188 nginx Ansible baseline, 110 cold-start detector script, startup judgment layers, GO/NO-GO tree, host recovery cards, T+0/T+60 timeline checks, host role / load-balancing assessment, CD `known_hosts` guardrail, and `km-vectorize` remediation tracking are updated; Ansible syntax check is unavailable on this workstation. | Full cold-start may be declared green only for the latest verified evidence set. As of 2026-06-14 03:11, the latest evidence set is degraded, not green. Do not declare DR scorecard complete while credential escrow evidence remains blocked. @@ -63,7 +63,7 @@ Full cold-start may be declared green only for the latest verified evidence set. | 2026-06-13 final post-trigger deploy closeout | LIVE_VERIFIED | Deploy marker `834ccdba chore(cd): deploy bf86017 [skip ci]` put API/Web/Worker image `bf860177` live. ArgoCD revision `834ccdba` is `Synced / Degraded` only by `km-vectorize`; routes `/zh-TW/governance` and `/en/governance` return `200`, API health is `healthy`, source guards pass, backup status has `core_blockers=0` and `escrow_missing=5`, and 14:13 cold-start is `PASS=83 WARN=0 BLOCKED=0`. | | 2026-06-13 final goal audit refresh | SERVICE_GREEN_REMAINING_GATES_EXPLICIT | Clean worktree rebased onto `a520c32d` and reran source guards successfully; live ArgoCD tracks revision `a520c32d` with API/Web/Worker image `e897c8bf`, health `Degraded` only by `km-vectorize`; `km-vectorize` schedule remains `0 3 * * *`, `timeZone=Asia/Taipei`, `failedJobsHistoryLimit=3`, and no failed Job is currently retained. Public `/zh-TW/governance`, `/en/governance`, and `/api/v1/health` are green; backup core blockers remain `0`, `escrow_missing=5`; 14:16 cold-start is `PASS=83 WARN=0 BLOCKED=0`. Remaining gates: five credential escrow markers and next official 03:00 `km-vectorize` success readback. | | 2026-06-14 `km-vectorize` official run follow-up | DEGRADED_EVIDENCE_RETENTION_LIVE | 03:00 official `km-vectorize-29689620` ran from CronJob and failed with `BackoffLimitExceeded`; ArgoCD later auto-synced revision `8868c025` and remains `Synced / Degraded`. Job is retained, but failed Pod `km-vectorize-29689620-nwpqz` was deleted before logs could be read, so root cause remains unproven for this run. Live CronJob is now `restartPolicy: Never` plus `terminationMessagePolicy: FallbackToLogsOnError`, so the next official failure should retain Pod/log evidence. Backup core remains green, `escrow_missing=5`, and 03:11 cold-start is `PASS=81 WARN=2 BLOCKED=0`. | -| 2026-06-14 `km-vectorize` tenant context follow-up | ROOT_CAUSE_CANDIDATE_PATCHED | Source audit shows `cron_km_vectorize.py` calls `/api/v1/knowledge/embed-all` without project context, while API middleware and `get_db_context()` require `X-Project-ID` / tenant context for fail-closed RLS. API logs show matching `db_context_missing` / `Missing tenant context` patterns. GitOps candidate adds `KM_PROJECT_ID=awoooi` and sends `X-Project-ID`; targeted pytest `7 passed`, kustomize renders the env/header support path, and no manual Job was created. Completion still waits for the next official 03:00 success or retained failed Pod/log. | +| 2026-06-14 `km-vectorize` tenant context follow-up | ROOT_CAUSE_CANDIDATE_LIVE | Source audit shows `cron_km_vectorize.py` calls `/api/v1/knowledge/embed-all` without project context, while API middleware and `get_db_context()` require `X-Project-ID` / tenant context for fail-closed RLS. API logs show matching `db_context_missing` / `Missing tenant context` patterns. Deploy marker `ec03f0b7` put image `8ddb80d6` live; CronJob now has `KM_PROJECT_ID=awoooi`, script sends `X-Project-ID`, targeted pytest `7 passed`, and no manual Job was created. Completion still waits for the next official 03:00 success or retained failed Pod/log. | --- @@ -136,7 +136,7 @@ Next: | P1-011 | DONE | 100 | Confirm 2026-06-12 backup convergence | 18:55 live check confirms the post-120 aggregate held: no stale jobs, no configured/missing script jobs, no failed components, offsite fresh, and only credential escrow remains as DR warning. | Keep escrow as explicit red gate. | `stale110=none`, `stale188=none`, `failed=0`, `config_failed=0`, `core_blockers=0`. | | P1-012 | DONE | 100 | Audit credential escrow marker write safety | 2026-06-12 15:02 `mark-credential-escrow-verified.sh --status` reports all five allowed items missing; `offsite-escrow-evidence-report.sh --no-color` reports rclone/offsite configured and `ESCROW_MISSING_COUNT=5`; repo search found only runbooks/placeholders/rules, not real evidence IDs. | Write markers only after a real non-secret evidence ID exists for each item; never write placeholder or secret. | The marker blocker is narrowed to missing external evidence IDs, not missing script/config/offsite readiness. | | P1-014 | DONE | 100 | Publish credential escrow owner request package | 2026-06-13 13:10 live report confirms `SCRIPT_MISSING_COUNT=0`, `OFFSITE_CONFIGURED=1`, `RCLONE_CONFIGURED=1`, `ESCROW_MISSING_COUNT=5`, `PASS=8 WARN=5 BLOCKED=0`. New owner request package defines allowed evidence-id types, forbidden secret values, safe dry-run flow, write flow, and closeout gates. | Dispatch to the credential owners without collecting secret values; keep marker write gated until owner gives real non-secret evidence IDs. | `docs/security/CREDENTIAL-ESCROW-EVIDENCE-OWNER-REQUEST.md` and snapshot exist and validate. | -| P1-013 | IN_PROGRESS | 96 | Remediate `km-vectorize` CronJob health debt | ArgoCD Degraded is isolated to `CronJob/km-vectorize`. The schedule is correct (`0 3` with `timeZone: Asia/Taipei`), evidence retention is live, and source/log audit found a strong root-cause candidate: the internal CronJob called `/api/v1/knowledge/embed-all` without `X-Project-ID`, while `get_db_context()` now fail-closes without project context. GitOps candidate adds `KM_PROJECT_ID=awoooi` and script header `X-Project-ID`. | Push the tenant-context patch, verify ArgoCD sync, then verify the next 03:00 official CronJob either succeeds or leaves inspectable failed Pod/log evidence. Do not manual-run or patch live. | `lastSuccessfulTime` is after the manifest sync, the official scheduled Job is `Complete`, and ArgoCD `awoooi-prod` health is `Healthy`; if it fails, the failed Job/Pod/log remains available for read-only triage. | +| P1-013 | IN_PROGRESS | 97 | Remediate `km-vectorize` CronJob health debt | ArgoCD Degraded is isolated to `CronJob/km-vectorize`. The schedule is correct (`0 3` with `timeZone: Asia/Taipei`), evidence retention is live, and source/log audit found a strong root-cause candidate: the internal CronJob called `/api/v1/knowledge/embed-all` without `X-Project-ID`, while `get_db_context()` now fail-closes without project context. Deploy marker `ec03f0b7` put tenant-context patch image `8ddb80d6` live and CronJob has `KM_PROJECT_ID=awoooi`. | Verify the next 03:00 official CronJob either succeeds or leaves inspectable failed Pod/log evidence. Do not manual-run or patch live. | `lastSuccessfulTime` is after the manifest sync, the official scheduled Job is `Complete`, and ArgoCD `awoooi-prod` health is `Healthy`; if it fails, the failed Job/Pod/log remains available for read-only triage. | ---