wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 3 Packages Projects Releases Wiki Activity

feat(governance): add agent market automation surfaces
Some checks failed
Ansible Lint / lint (push) Successful in 35s
Details
CD Pipeline / tests (push) Failing after 13s
Details
CD Pipeline / build-and-deploy (push) Has been skipped
Details
CD Pipeline / post-deploy-checks (push) Has been skipped
Details
Code Review / ai-code-review (push) Failing after 11s
Details

Browse Source
This commit is contained in:
Your Name
2026-06-04 21:40:12 +08:00
parent b9bd5e3ba8
commit cfb866d055
334 changed files with 62000 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files

237
docs/workplans/2026-06-04-reboot-cold-start-backup-recovery-workplan.md Normal file
View File

@@ -0,0 +1,237 @@
# 2026-06-04 Reboot / Cold-Start / Backup Recovery Workplan
> Owner: SRE / DevOps commander
> Timezone: Asia/Taipei
> Baseline: 2026-06-04 15:00 live read-only checks. Do not reuse the 2026-05-29 baseline without rerunning checks.
> Scope: 110 / 120 / 121 / 188. 112 is Kali and is intentionally excluded from this recovery wave.
---
## 1. Current Verdict
| Area | Status | Completion | Evidence |
|------|--------|------------|----------|
| Overall recovery readiness | BLOCKED | 62% | Documentation, route, backup freshness, alert, nginx baseline, momo live failure contract, and 120 console handoff diagnosis advanced; latest scorecard improved to `PASS=71 WARN=3 BLOCKED=3`, but release remains blocked by 120 and credential escrow. |
| P0 host / K3s recovery | BLOCKED | 36% | 120 ping failed, SSH failed, ARP incomplete from local/110/121/188 views; K3s `mon` lease stopped renewing on 2026-05-22 02:48:36 +08 and remains `NotReady,SchedulingDisabled`. |
| P1 backup / alert / escrow | BLOCKED | 74% | Cron, rclone offsite, latest-only, live alert rules, backup freshness, 188 backup exporter contract, and scorecard schedule checks are verified; credential escrow markers are 5/5 missing and aggregate backup remains red until 120 config capture recovers. |
| P2 service / data truth | VERIFIED | 88% | Public routes and momo current-month parity are green; momo live 188 code now fails monthly-sync jobs correctly and containers were reloaded healthy. Next real Drive import still needs archive-movement observation. |
| P3 docs / automation contracts | DONE | 96% | Workplan, SOP, BACKUP-STATUS, LOGBOOK, 120 console handoff, and 188 nginx Ansible baseline are updated; Ansible syntax check is unavailable on this workstation. |
Do not declare "full cold-start green" or "DR scorecard complete" while P0 and credential escrow are blocked.
---
## 2. Live Check Evidence, 2026-06-04
| Target | Live result | Notes |
|--------|-------------|-------|
| 192.168.0.110 | ping OK, SSH port OK | Boot `2026-05-06 12:12`; load was elevated around `10.54 7.42 6.28`; cron and Docker active. |
| 192.168.0.120 | ping failed, SSH port failed | ARP incomplete; K3s node `mon` remains `NotReady,SchedulingDisabled`. |
| 192.168.0.121 | ping OK, SSH port OK | Boot `2026-05-22 02:30`; `sudo kubectl get nodes` shows `mon1 Ready`. |
| 192.168.0.188 | ping OK, SSH port OK | Boot `2026-05-06 12:07`; Docker/PostgreSQL/Redis/nginx active; momo containers healthy. |
| Cold-start scorecard | BLOCKED | 18:55 read-only rerun: `PASS=71 WARN=3 BLOCKED=3`; hard blocks remain 120 reachability / SSH / 120 K3s read-only check. |
| Public routes | OK ingress only | `awoooi`, `aiops`, `mo`, `gitea`, `harbor`, `registry`, `sentry`, `signoz`, `stock`, `langfuse`, `bitan` returned 2xx/3xx. |
| momo DB current-month parity | OK | Scorecard reports `2215|2215|2026-06-01|2026-06-04|2026-06-01|2026-06-04`; snapshot and realtime tables match row count and date bounds. |
| 110 daily backup cron | OK | `02:00 backup-all`, `03:00 rclone sync`, `06:05 backup-status`, `07:20 full offsite verify`. |
| Backup freshness | OK with remaining aggregate blocker | Manual refresh cleared `stale110=none`, `stale188=none`, `configured_missing_188=0`; remaining `core_blockers=1` is the 120-driven aggregate/config capture failure. |
| Google Drive latest-only | OK | 2026-06-04 07:20 verifier: 13 repos, each `remote snapshots=1`, `REMOTE_LATEST_ONLY_OK=1`. |
| Live Prometheus alert rules | OK | All five required alerts found live: `BackupConfigCapturePartial`, `BackupAggregateRunFailed`, `BackupCredentialEscrowEvidenceMissing`, `ColdStartRecoveryBlocked`, `ColdStartHost120Unreachable`. |
| Credential escrow | BLOCKED | Missing markers: `break_glass_admin_credentials`, `dns_registrar_recovery`, `oauth_ai_provider_recovery`, `offsite_provider_credentials`, `restic_repository_password`. |
| Config backup capture | BLOCKED until 120 returns | `awoooi_backup_config_capture_ok{target="120-k3s-host-configs"} 0`; critical failed count `1`. |
| Live 110 script sync | OK | Six recovery/check scripts exist under `/home/wooo/scripts/` with May 29 timestamps. |
| Gitea commit evidence | VERIFIED | Gitea `main` at `0260ec89...` contains `ae7b39d9 fix(ops): harden reboot recovery and backup alerts`. |
| 188 nginx Ansible baseline | DONE | Template now pins `aiops.wooo.work` to VIP `192.168.0.125:32334/32335`, contains no `192.168.0.120`, and live smoke returned `https://aiops.wooo.work/` 307 plus `/api/v1/health` 200. |
| 120 failure-domain triage | BLOCKED | 19:02 checks from local/110/121/188 all fail to reach 120; 121 reports `Destination Host Unreachable`; K3s node lease renew stopped at `2026-05-21T18:48:36Z`; `120-fsck-maintenance-checklist.sh --no-color` returns `PASS=2 WARN=2 BLOCKED=3`, `MAINTENANCE REQUIRED`. |
---
## 3. Progress Update Contract
Every phase update must change both status and percentage in this file.
| State | Meaning |
|-------|---------|
| NOT_STARTED | Listed but no live evidence gathered in this session. |
| IN_PROGRESS | Actively being checked or fixed. |
| BLOCKED | A live red gate prevents completion. Do not downgrade or silence the alert. |
| WAITING_HOST_120 | Action is intentionally held until 120 is reachable. |
| VERIFIED | Live evidence proves the item. |
| DONE | Fix is implemented, verified, and documented. |
Completion is weighted by release risk:
| Priority | Weight |
|----------|--------|
| P0 | 45% |
| P1 | 25% |
| P2 | 20% |
| P3 | 10% |
For every push forward, update:
```text
YYYY-MM-DD HH:MM Asia/Taipei
Phase: P0/P1/P2/P3
Before: <old percent>
After: <new percent>
Evidence: <command/file/snapshot>
Blocked: <yes/no and why>
Next: <single next action>
```
---
## 4. P0 Must-Do Gates
| ID | Status | % | Work item | Fine analysis | Next action | Done criteria |
|----|--------|---:|-----------|---------------|-------------|---------------|
| P0-001 | VERIFIED | 100 | Rerun four-host reachability | 110/121/188 are reachable; 120 is still hard down. This confirms the 2026-05-29 blocker is still real on 2026-06-04. | Keep evidence in LOGBOOK/runbook. | Host reachability table recorded with date/time. |
| P0-002 | BLOCKED | 20 | Recover 192.168.0.120 | 120 fails ping/SSH and is ARP incomplete from all checked LAN perspectives. K3s still records `mon` as `NotReady,SchedulingDisabled`; node lease stopped at `2026-05-22 02:48:36 +08`. This blocks full cold-start. | Use physical/VM console path; if filesystem corruption appears, follow `120-fsck-maintenance-checklist.sh`; no online fsck. | 120 ping/SSH OK, node `Ready`, no filesystem error events. |
| P0-003 | WAITING_HOST_120 | 0 | Rerun `/backup/scripts/backup-configs.sh` | Current config capture failed exactly at `120-k3s-host-configs`. Running before 120 returns will preserve the red result, not fix it. | Run immediately after 120 returns. | `awoooi_backup_config_capture_critical_failed_count=0`. |
| P0-004 | WAITING_HOST_120 | 0 | Rerun `/backup/scripts/backup-all.sh` | Cold-start check reports latest aggregate/config backup had failed components. 120 must be reachable before this can be green. | Run after P0-003. | Aggregate backup exits 0; backup health failed count 0. |
| P0-005 | WAITING_HOST_120 | 0 | Rerun `/backup/scripts/sync-offsite-backups.sh --mode sync` | Offsite is currently fresh and latest-only, but the post-120 backup must be mirrored after local backup is green. | Run after P0-004. | New rclone last-success marker after local backup timestamp. |
| P0-006 | WAITING_HOST_120 | 0 | Rerun `/backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color` | Today 07:20 verifier is green; after P0 backup rerun, remote latest-only must be re-proven. | Run after P0-005. | `REMOTE_LATEST_ONLY_OK=1`, all 13 repos `snapshots=1`. |
| P0-007 | BLOCKED | 71 | Rerun full cold-start scorecard | Latest read-only scorecard improved to `PASS=71 WARN=3 BLOCKED=3`, but the remaining hard blockers are all 120-centered. | Rerun after P0-006. | `BLOCKED=0`; if WARN remains, each WARN must have owner and downgrade reason. |
| P0-008 | DONE | 100 | Narrow 120 failure domain and prepare console handoff | 110 and 188 see no route / no ping; 121 reports destination host unreachable; local ARP is incomplete. Kubernetes retained only stale node/lease data and cannot read current 120 host/filesystem state. No BMC/IPMI/WOL inventory was found in the repo. | Physical/VM console must verify power state, NIC attachment, boot screen, initramfs/fsck state, and then restore SSH. | Handoff evidence is recorded; no remote-only fix path remains before console access. |
---
## 5. P1 Backup And Alert Gates
| ID | Status | % | Work item | Fine analysis | Next action | Done criteria |
|----|--------|---:|-----------|---------------|-------------|---------------|
| P1-001 | VERIFIED | 100 | Confirm 110 backup schedule | Live crontab has `02:00 backup-all`, `03:00 rclone gated sync`, `06:05 backup-status`, `07:20 full offsite verify`. | Update `BACKUP-STATUS.md`. | Schedule documented and matches live crontab. |
| P1-002 | VERIFIED | 100 | Confirm success-noise policy | Daily status is once at 06:05; normal backup success is not a Telegram spam path. | Keep failure-only escalation in backup docs. | Docs say failures escalate; daily status is summary only. |
| P1-003 | VERIFIED | 100 | Confirm Google Drive latest-only | 2026-06-04 verifier shows 13 repos with exactly one remote snapshot each. | Record evidence in backup status. | `REMOTE_LATEST_ONLY_OK=1`. |
| P1-004 | VERIFIED | 100 | Confirm required alerts exist | Live Prometheus rules include all five required backup/cold-start alerts. | Keep in scorecard. | All five alert names FOUND live. |
| P1-005 | BLOCKED | 0 | Fill credential escrow evidence markers | Five markers are missing. This is a DR scorecard blocker, not a service outage. Secrets must not enter repo or chat. | Human verifies vault/offline escrow, then writes non-secret evidence IDs using `/backup/scripts/mark-credential-escrow-verified.sh`. | `awoooi_backup_dr_credential_escrow_missing_count=0`. |
| P1-006 | WAITING_HOST_120 | 55 | Fix backup health failed component | Stale job freshness is fixed; the remaining failed component is 120 config capture, reflected by `backup_all failed=1` and `core_blockers=1`. | Tie to P0-003/P0-004. | `failed_count=0`, `config_failed=0`. |
| P1-007 | DONE | 100 | Refresh stale backup jobs | `backup-status --no-notify` initially reported `stale110=awoooi_db` and `stale188=momo_pg_daily`. Manual AWOOOI high-frequency DB backup and 188 momo PostgreSQL backup cleared both stale markers. | Keep normal cron cadence. | `stale110=none`, `stale188=none`, 110 `13/13 fresh`, 188 `2/2 fresh`. |
| P1-008 | DONE | 100 | Align 188 momo backup cron/exporter contract | 188 backup exporter expected `/home/ollama/bin/momo-pg-backup.sh`; crontab still pointed to the old app-side script. Crontab was backed up and updated to the host-owned controller script. | Keep backup controller path in future deploy docs. | `configured_missing_188=0`, `awoooi_backup_job_configured{host="188",job="momo_pg_daily"} 1`. |
---
## 6. P2 Service And Data Gates
| ID | Status | % | Work item | Fine analysis | Next action | Done criteria |
|----|--------|---:|-----------|---------------|-------------|---------------|
| P2-001 | VERIFIED | 100 | Public route smoke | All listed domains returned 2xx/3xx over HTTPS. This proves ingress/TLS only, not app correctness. | Keep as one row in scorecard. | Public route table updated after each reboot. |
| P2-002 | VERIFIED | 100 | momo latest/current-month parity | Latest current-month scorecard check: both tables have 2215 rows and matching bounds from `2026-06-01` through `2026-06-04`. Earlier latest snapshot `2026-06-02` parity also matched 404/404. | Keep daily check in cold-start SOP. | Latest snapshot/current-month row count and bounds match. |
| P2-003 | VERIFIED | 95 | Fix momo job semantics | `/Users/ogt/momo-pro-system/services/import_service.py` and live `/home/ollama/momo-pro/services/import_service.py` now mark monthly sync failure as `failed`, write `drive_file_movable=false`, return `False`, emit a failure alert path, and make auto-import aggregate failures as `success=false`. Live 188 backup: `services/import_service.py.bak.20260604-152827`; live hash after patch: `3fc45671986fa4cc155119f588bc1ebefd272927730052e42e2b9eb4352b2586`. | Watch the next real Google Drive import and confirm no file moves unless both tables sync; keep canonical source-control reconciliation open as a separate supply-chain task. | Live isolated temp-DB/real-Excel test passes; containers reloaded healthy; Telegram token/chat markers are present without exposing secrets; latest DB parity remains 404/404. |
| P2-004 | DONE | 100 | PostgreSQL index corruption runbook path | SOP v1.2 now states `posting list tuple ... cannot be split` is an index repair incident. | Use only concurrent reindex if the error returns. | No truncate, no whole DB restore; `REINDEX TABLE CONCURRENTLY public.realtime_sales_monthly;` and idempotent resync evidence recorded. |
| P2-005 | VERIFIED | 90 | Do not rely on route 200 only | We now have route + DB + backup + schedule + alert + cold-start scorecard evidence. P0/P1 blockers remain outside route health. | Keep this cross-surface checklist mandatory after every reboot. | Each reboot record has route, DB, backup, schedules, alert, scorecard rows. |
---
## 7. P3 Documentation And Automation
| ID | Status | % | Work item | Fine analysis | Next action | Done criteria |
|----|--------|---:|-----------|---------------|-------------|---------------|
| P3-001 | VERIFIED | 100 | Confirm hardening commit | Gitea `main` currently points to `0260ec89...`; `git merge-base --is-ancestor ae7b39d9 0260ec89...` returned true. | Keep evidence in LOGBOOK. | Gitea main contains `ae7b39d9 fix(ops): harden reboot recovery and backup alerts`. |
| P3-002 | VERIFIED | 100 | Confirm live 110 scripts | All six required scripts exist under `/home/wooo/scripts/`. | Record in LOGBOOK. | Script paths and timestamps recorded. |
| P3-003 | DONE | 100 | Reconcile 188 nginx Ansible baseline | Live 188 already routes `aiops.wooo.work` through VIP; the Ansible template now matches that route and has no 120 upstream for aiops. Content guard passed; `ansible-playbook` is not installed locally, so syntax-check could not be run here. | Run Ansible syntax/apply validation from the normal Ansible environment before the next route apply. | Template and live config agree; no 120 upstream for aiops. |
| P3-004 | DONE | 100 | Update `docs/LOGBOOK.md` | Live blocker and new docs are recorded. | Keep this entry updated after each recovery phase. | LOGBOOK has current recovery status and next actions. |
| P3-005 | DONE | 100 | Update cold-start SOP | SOP now includes start, shutdown, reboot, record, comparison, and 120 blocker handling. | Increment SOP version after each process change. | SOP has controlled power-operation sections and ledger template. |
| P3-006 | DONE | 100 | Update backup status | Backup status now reflects current cron, rclone latest-only, failure-only alert posture, and escrow blocker. | Refresh after 120 backup rerun. | Backup status no longer claims noisy success Telegram notifications. |
---
## 8. Required 120 Recovery Sequence
Do this only after physical/VM console access confirms 120 is powered on, attached to the LAN, and either booted or repairable.
```bash
# 0. Console-side checks first; do not do these through an online mounted root filesystem.
# - power / VM state
# - NIC connected to the 192.168.0.x LAN
# - boot screen / initramfs / rescue state
# - if root FS repair is required: fsck -f /dev/mapper/ubuntu--vg-ubuntu--lv from console/rescue only
# 1. After SSH returns, run read-only 120 maintenance readiness
bash scripts/reboot-recovery/120-fsck-maintenance-checklist.sh --no-color
# 2. After 120 is reachable and stable, on 110
/backup/scripts/backup-configs.sh
/backup/scripts/backup-all.sh
/backup/scripts/sync-offsite-backups.sh --mode sync
/backup/scripts/verify-offsite-full-sync.sh --write-textfile --no-color
# 3. Final cold-start scorecard
/home/wooo/scripts/full-stack-cold-start-check.sh --monitor-read-only --no-color --watch --interval 1 --max-attempts 1
```
Do not run `truncate`, whole DB restore, force-push, DROP, or online root filesystem `fsck` as part of this flow.
---
## 9. Progress Updates
```text
2026-06-04 15:23 Asia/Taipei
Phase: P3
Before: 78%
After: 95%
Evidence: infra/ansible/roles/nginx/templates/188-all-sites.conf.j2 now contains aiops VIP upstreams 192.168.0.125:32334/32335; live smoke aiops / -> 307 and /api/v1/health -> 200; content guard passed.
Blocked: no for route baseline; ansible-playbook is unavailable on this workstation, so syntax-check remains delegated to the normal Ansible environment before next apply.
Next: run Ansible syntax/apply validation from the Ansible host before changing 188 nginx live config.
```
```text
2026-06-04 15:23 Asia/Taipei
Phase: P2
Before: 52%
After: 66%
Evidence: /Users/ogt/momo-pro-system/services/import_service.py updated; /Users/ogt/momo-pro-system/tests/test_daily_sales_monthly_sync_failure.py added; targeted pytest passed with temp SQLite and real Excel input.
Blocked: yes. Live 188 uses /home/ollama/momo-pro bind-mounted code, while momo/ewoooc canonical source remains unresolved.
Next: reconcile canonical source/deploy path, apply the same monthly-sync failure contract to live, then run controlled live auto-import failure-path verification.
```
```text
2026-06-04 15:34 Asia/Taipei
Phase: P2
Before: 66%
After: 86%
Evidence: live /home/ollama/momo-pro/services/import_service.py patched from backup services/import_service.py.bak.20260604-152827; live hash 3fc45671986fa4cc155119f588bc1ebefd272927730052e42e2b9eb4352b2586; container isolated temp-DB/real-Excel contract test passed; momo-scheduler and momo-pro-system restarted and healthy; mo.wooo.work /health 200; latest DB parity daily=404 and monthly=404 for 2026-06-02.
Blocked: no for momo failure contract. Overall remains blocked by 120 reachability and credential escrow.
Next: observe the next real Google Drive import and keep canonical momo/ewoooc source-control reconciliation as a separate supply-chain item.
```
```text
2026-06-04 15:50 Asia/Taipei
Phase: P1
Before: 58%
After: 72%
Evidence: /backup/scripts/backup-status.sh --no-notify initially showed stale110=awoooi_db, stale188=momo_pg_daily, configured_missing_188=1; manual 188 momo PostgreSQL backup completed and kept latest-only; manual 110 backup-awoooi-frequent completed with restic snapshot 7440d75f; 188 crontab now points momo_pg_daily to /home/ollama/bin/momo-pg-backup.sh; final backup-status shows stale110=none, stale188=none, configured_missing_188=0, core_blockers=1, escrow_missing=5.
Blocked: yes. 120 config capture still keeps aggregate backup red, and five credential escrow evidence markers are still missing.
Next: after 120 returns, rerun backup-configs, backup-all, offsite sync, full offsite verify, then cold-start scorecard; separately fill escrow only with real non-secret evidence IDs.
```
```text
2026-06-04 18:55 Asia/Taipei
Phase: P0/P1/P2
Before: Overall 60%, P1 72%, P2 86%
After: Overall 61%, P1 74%, P2 88%
Evidence: local ping to 192.168.0.120 still 0/3, SSH 22 timed out, ARP incomplete; 121 kubectl still shows mon NotReady,SchedulingDisabled and mon1 Ready; 110 backup-status --no-notify shows stale110=none, stale188=none, configured_missing_188=0, core_blockers=1, escrow_missing=5; cold-start scorecard now reports PASS=71 WARN=3 BLOCKED=3 and momo monthly parity 2215/2215 for 2026-06-01 through 2026-06-04.
Blocked: yes. The three hard blocks are still 120 ping, 120 SSH, and 120 K3s read-only check; escrow remains missing 5 evidence markers.
Next: wait for physical/console recovery of 120, then run the required backup-configs / backup-all / offsite sync / full verify / cold-start sequence.
```
```text
2026-06-04 19:02 Asia/Taipei
Phase: P0/P3
Before: Overall 61%, P0 35%, P3 95%
After: Overall 62%, P0 36%, P3 96%
Evidence: local/110/121/188 all failed to reach 192.168.0.120; 121 returned Destination Host Unreachable; kubectl describe node mon shows LastHeartbeatTime 2026-05-22 02:44:13 +08, Ready Unknown since 2026-05-22 02:49:48 +08, and kube-node-lease renewTime 2026-05-22 02:48:36 +08; 120-fsck-maintenance-checklist.sh --no-color returned PASS=2 WARN=2 BLOCKED=3 and MAINTENANCE REQUIRED; repo search found no BMC/IPMI/WOL inventory for 120.
Blocked: yes. 120 requires physical or VM console recovery before backup-configs, backup-all, offsite sync, and full cold-start can be made green.
Next: use console to verify 120 power/NIC/boot/initramfs state, perform offline fsck only if needed, then restore SSH and run the required recovery sequence.
```
---
## 10. Completion Claims That Are Not Allowed Yet
- Do not claim every reboot is guaranteed green. 120 is still down.
- Do not silence 120 alerts. They are correct red lights.
- Do not claim DR scorecard complete. Credential escrow markers are missing.
- Do not claim public-route success is system success. Route checks must be paired with DB, backup, schedules, Alertmanager, and cold-start scorecard evidence.
- Do not claim the next real Google Drive import has succeeded until the post-import row counts/date bounds and Drive archive movement are rechecked.