awoooi/HOST-REBOOT-AI-AUTOMATION-SOP.md

# Host Reboot AI Automation SOP

Last updated: 2026-07-09

## P0 Goal

After host 99 or any VM host is rebooted, the automation must detect the reboot, restore VM reachability, restore critical services, and prove public sites are no longer returning 502.

Target recovery bar:

- Detect 99 / 110 / 112 / 120 / 121 / 188 reboot or down state.
- Ensure VMware Workstation on 99 is running and required VMs are powered on.
- Restore Harbor / registry on 110 before forcing K3s workloads to repull images.
- Restore AWOOOI production pods on 120.
- Verify public HTTP readback for AWOOOI, StockPlatform, FIFA, Gitea, and Harbor.
- Emit operator-visible evidence and Telegram alerts for down, recovering, recovered, and degraded states.

## 2026-07-09 Incident Findings To Automate

The reboot incident was not a generic 502 problem. The confirmed chain was:

1. 110 rebooted and Docker/Harbor did not fully recover.
2. `harbor-core` entered a failed/restarting state because Harbor Redis / Docker DNS dependency was not clean.
3. K3s on 120 could not pull AWOOOI images from `192.168.0.110:5000`.
4. AWOOOI prod pods stayed in `ImagePullBackOff`.
5. Public nginx returned 502 because upstream pods were unavailable.
6. Restarting / rebooting 110 cleared the Docker / systemd / Harbor dependency state.
7. Once `harbor-core` became healthy, 120 pods repulled images and AWOOOI recovered.

This failure mode must be treated as a first-class automated recovery path.

## Recovery Priority Order

P0. Control-plane and VM reachability

- 99 Windows host reachable.
- VMware Workstation process running on 99.
- VMs 110, 112, 120, 121, and 188 powered on.
- Hosts 110, 112, 120, 121, and 188 respond to ICMP or SSH.

P1. 110 Harbor and registry dependency chain

- Docker engine active.
- Harbor Redis running.
- `harbor-core` running and healthy.
- Harbor registry / registryctl running and healthy.
- `harbor.wooo.work` returns HTTP 200 after redirect.
- Internal registry pull path from K3s nodes is reachable.

P2. 120 K3s production workloads

- `awoooi-prod` deployments ready:
  - `awoooi-api` 1/1
  - `awoooi-web` 2/2
  - `awoooi-worker` 1/1
  - `awoooi-auto-repair-canary` 1/1
- No critical pods stuck in `ImagePullBackOff`, `ErrImagePull`, `CrashLoopBackOff`, or `Pending`.
- If image pull failures remain after Harbor is healthy, delete only the failed pods and let controllers recreate them.

P3. Public user-visible surfaces

- `awoooi.wooo.work` returns HTTP 200 for web.
- `awoooi.wooo.work/api/v1/health` returns JSON and not 502.
- `stock.wooo.work` returns HTTP 200.
- `2026fifa.wooo.work` returns HTTP 200.
- `gitea.wooo.work` returns HTTP 200.
- `harbor.wooo.work` returns HTTP 200.

P4. Degraded provider follow-up

- AWOOOI health can be `degraded` even when 502 is fixed.
- Treat AI provider health separately from user-visible 502 restoration.
- Follow-up checks include `ollama_gcp_a`, `ollama_gcp_b`, and `ollama_local`.
- Do not block public-site 502 recovery on non-primary AI provider cooldown unless the app is user-impacting.

## Automated Decision Tree

1. Detect reboot / outage.
   - Ping 110, 112, 120, 121, 188.
   - SSH to 110 and 120 if keys are available.
   - Query VMware running VM list on 99 if `vmrun.exe` is available.

2. If a VM is not reachable.
   - Check VMware VM state.
   - Start the VM from 99.
   - Wait for guest ping / SSH.
   - Alert Telegram: `host_recovering`.

3. If AWOOOI public route is 502.
   - Check 120 `awoooi-prod` deployments.
   - If pods are `ImagePullBackOff`, check 110 Harbor first.
   - Do not repeatedly delete pods before Harbor is healthy.

4. If 110 Harbor is unhealthy.
   - Check Docker active state.
   - Check `redis`, `harbor-core`, `registry`, and `registryctl`.
   - Restart Harbor Redis and `harbor-core` first.
   - If Docker/systemd/cgroup operations hang, use controlled 110 guest reboot through VMware.
   - After 110 returns, wait for Harbor core healthy before touching K3s pods.

5. If Harbor is healthy and pods are still in image pull failure.
   - Delete only failed AWOOOI pods in `awoooi-prod`.
   - Wait for deployments to become ready.
   - Read back public HTTP.

6. If public sites are HTTP 200 but AWOOOI health is degraded.
   - Record degraded providers.
   - Alert `service_degraded`.
   - Continue provider-specific remediation separately.

## 99 Windows AI Agent Control Node

99 must become the automation entrypoint for reboot recovery.

Required capabilities:

- Run a local AI Agent service on Windows 99 with least-privilege operator permissions.
- Execute the reboot scorecard on a schedule and on boot.
- Control VMware Workstation VM startup through `vmrun.exe`.
- Use SSH keys to 110 and 120 for service-level checks and controlled repair.
- Send Telegram alerts for down, recovering, recovered, degraded, failed, and controlled-remediation-result states.
- Do not stop at alerting. Every warning/critical alert must either trigger an allowlisted remediation, record that no remediation matched, or record that the action was skipped by cooldown/min-severity.
- Do not send start/complete/info noise to Telegram by default. Human-facing alerts must be deduped and action-oriented.
- Persist scorecard evidence to a local non-secret log directory.
- Never store raw passwords, tokens, cookies, sessions, or `.env` in the repo.
- Accept operator commands through the 99 desktop submitter, Telegram `/agent99 ...`, and local queue files without depending on Codex quota.
- Accept SRE war-room alert JSON at `C:\Wooo\Agent99\alerts\incoming`, resolve each alert into an allowlisted mode, and either run controlled remediation or record why no remediation matched.

Candidate implementation layers:

- P0: Windows Task Scheduler runs `host-reboot-scorecard.ps1` at startup and every minute for the first 15 minutes after 99 boot.
- P1: Install a local Windows service wrapper for the scorecard loop.
- P2: Add an AI Agent runtime that consumes scorecard JSON and chooses approved recovery actions.
- P3: Add a UI/console dashboard on 99 for operator evidence and manual override.

Current command and alert loops:

- `Wooo-Agent99-Control-Loop`: processes `C:\Wooo\Agent99\queue`, updates the dashboard, and writes execution evidence.
- `Wooo-Agent99-Telegram-Inbox`: reads Telegram group commands with Agent99 prefixes and submits allowlisted queue work.
- `Wooo-Agent99-SRE-Alert-Inbox`: reads `C:\Wooo\Agent99\alerts\incoming`, converts actionable alerts into queue work, and triggers the control loop.
- `Wooo-Agent99-SRE-Alert-Relay`: runs the 99-side HTTP relay for sanitized AWOOI Alertmanager JSON and writes accepted alerts into `C:\Wooo\Agent99\alerts\incoming`.
- `Wooo-Agent99-Performance-Guard`: runs every minute and performs allowlisted load shedding before escalating unresolved performance alerts.
- `Wooo-Agent99-Self-Health`: verifies the scheduled tasks and evidence freshness for the control plane itself.

AwoooI SRE Telegram command loop:

- Operators and project AI Agents can issue commands in the SRE Telegram group with `/agent99 ...`.
- Agent99 records these as `source=telegram-sre-war-room`.
- Completed automated repairs send `operator_command_result` back to Telegram.
- Results include source, instruction, mode, exit code, evidence path, problem count, and KM path.
- The command path must not depend on Codex quota.
- The command path must not depend only on the default `TELEGRAM_CHAT_ID`, because that can point to a private chat. Agent99 also accepts command updates from Telegram chats whose title matches `AwoooI SRE|AWOOOI SRE|SRE`.
- When the command comes from a matched Telegram group, the completion report is sent back to that same Telegram origin; stored evidence redacts the raw chat id as `telegram-origin`.
- Non-command monitoring alerts in the SRE Telegram group are auto-ingested when they look like Alertmanager/FIRING/critical/warning/502/CPU/disk/Harbor/K3s/backup/host-down/provider-freshness alerts.
- Auto-ingested monitoring alerts use `source=telegram-sre-war-room-monitoring` and enter the same SRE alert inbox / queue / recovery / Telegram completion / KM-RAG path.

Alert-to-action routing:

- Public 502 / route down / website down -> `Recover`.
- Host down / reboot detected / VM not running -> `Recover`.
- CPU, memory, disk, or load pressure -> `Perf` with controlled apply.
- Harbor or registry unhealthy -> `HarborRepair`.
- AWOOOI K3s, pod, deployment, or image pull failure -> `AwoooRepair`.
- Backup freshness, snapshot, restore drill, or cron issue -> `BackupCheck`.
- Provider freshness, last-seen, ingestion, raw signal, or no-false-green risk -> `ProviderFreshness`.
- Unknown actionable alert -> `Status` evidence first.

Producer-side Agent99 bridge:

- AWOOI API Alertmanager webhook schedules an Agent99 bridge after normalization and fingerprinting.
- Preferred transport: when `AGENT99_SRE_ALERT_RELAY_URL` is configured, the bridge POSTs sanitized alert JSON to the 99 relay. Production URL is `http://192.168.0.99:8787/agent99/sre-alert/`.
- Fallback transport: when `AGENT99_SRE_ALERT_INBOX_PATH` is configured and relay URL is empty, the bridge writes sanitized alert JSON directly to the Agent99 SRE inbox contract.
- If relay/path is empty or unavailable, the bridge fails open and logs; it must not cause Alertmanager retries.
- Relay safety: only LAN/local source prefixes are accepted by default; optional `AGENT99_SRE_RELAY_TOKEN` / `AGENT99_SRE_ALERT_RELAY_TOKEN` values stay in env/Secret and are never stored in repo or evidence.
- Provider freshness bridge output must require provider window, last seen, direct reference, and verifier readback before dashboard green is trusted.

Approved controlled actions for the 99 agent:

- Start a missing VM.
- Ping and SSH readback.
- Restart Harbor Redis / Harbor core after evidence of Harbor token failure.
- Delete K3s pods stuck in image pull failure only after Harbor is healthy.
- Run public route readback.
- Send Telegram status.

Actions still requiring explicit break-glass:

- Reading or exfiltrating secrets.
- Destructive database operations.
- Firewall cutover.
- Remote delete / prune / force push.
- Production provider route cutover with cost or billing impact.

## Evidence Required Before Calling Recovery Complete

Recovery is complete only when all of the following are captured:

- Five VM hosts have reachability evidence.
- 110 Harbor core is healthy.
- 120 AWOOOI deployments are ready.
- Public routes return non-502, preferably HTTP 200.
- Any degraded health components are listed separately with owner / next action.

## Current 2026-07-09 Receipts

- 110 `harbor-core`: recovered to `Up ... (healthy)`.
- 120 `awoooi-prod`:
  - `awoooi-api`: 1/1
  - `awoooi-web`: 2/2
  - `awoooi-worker`: 1/1
  - `awoooi-auto-repair-canary`: 1/1
- Public readback:
  - `awoooi.wooo.work`: HTTP 200 after redirect.
  - `stock.wooo.work`: HTTP 200 after redirect.
  - `2026fifa.wooo.work`: HTTP 200 after redirect.
  - `gitea.wooo.work`: HTTP 200 after redirect.
  - `harbor.wooo.work`: HTTP 200 after redirect.
- Host reachability:
  - 110, 112, 120, 121, 188 responded to ping or SSH.
- Remaining degraded signal:
  - AWOOOI health reported degraded AI provider signals including `ollama_gcp_b` timeout and `ollama_local` connection refused / unavailable.

## Agent99 Installation Receipt

2026-07-09 16:49 Asia/Taipei:

- 99 host: `WOOO-SUPER`.
- Agent root: `C:\Wooo\Agent99`.
- VMware CLI: `C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe`.
- Live config: `C:\Wooo\Agent99\config\agent99.config.json`.
- Evidence directory: `C:\Wooo\Agent99\evidence`.
- Scheduled tasks:
  - `Wooo-Agent99-Heartbeat`: `LastTaskResult=0`, latest status evidence `agent99-Status-20260709-164845.json`.
  - `Wooo-Agent99-Startup-Recovery`: `LastTaskResult=0`, latest recovery evidence `agent99-Recover-20260709-164937.json`.
- Startup recovery readback:
  - VM selector: 110, 188, 112, 120, 121 all `host_already_reachable`.
  - Harbor: core, Redis, and registry all healthy.
  - AWOOOI: deploy ready, no image pull failure, no crash failure.
  - Public routes: `awoooi.wooo.work`, `stock.wooo.work`, `2026fifa.wooo.work`, `gitea.wooo.work`, and `harbor.wooo.work` all HTTP 200 / non-502.

## Agent99 Performance Guard Receipt

2026-07-09 17:15 Asia/Taipei:

- Scheduled task: `Wooo-Agent99-Performance-Guard`.
- Trigger interval: every 1 minute.
- Last task result: `0`.
- Latest evidence: `C:\Wooo\Agent99\evidence\agent99-Perf-20260709-171502.json`.
- Host performance readback:
  - 110: OK.
  - 112: OK.
  - 120: OK.
  - 121: OK.
  - 188: OK.
- Controlled load shedding:
  - Only allowlisted actions can run.
  - Current 110 runner action requires `load_per_core_critical`.
  - Memory/disk pressure alerts do not execute the runner pause action.
- Telegram audit:
  - Agent99 emits Telegram attempts for all operation start/complete events, warnings, critical events, and load-shed actions.
  - Agent99 supports local env aliases and an existing-token relay.
  - Current relay: 110 `stockplatform-v2-api-1`, using that container's existing `TELEGRAM_BOT_TOKEN` / `TELEGRAM_CHAT_ID`.
  - 2026-07-09 17:27 verification: `Wooo-Agent99-Performance-Guard` `LastTaskResult=0`; latest evidence `agent99-Perf-20260709-172712.json`; `tgSent=2`; `relayOk=True`.
  - Token and chat id values were not printed, copied into repo files, or written to evidence.

## Agent99 Command And SRE Alert Integration Receipt

2026-07-09 23:03 Asia/Taipei:

- Telegram command inbox is deployed and scheduled:
  - Script: `C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1`
  - Task: `Wooo-Agent99-Telegram-Inbox`
  - Latest evidence: `C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260709-230031.json`
  - Result: `ok=True`, `updatesSeen=0`, `processedCount=0`
  - Evidence check: no Python code, no token text.
- SRE war-room alert inbox is deployed and scheduled:
  - Script: `C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1`
  - Task: `Wooo-Agent99-SRE-Alert-Inbox`
  - Incoming path: `C:\Wooo\Agent99\alerts\incoming`
  - Latest evidence: `C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260709-230226.json`
- Synthetic 502 alert test:
  - Alert id: `synthetic-public-502-20260709-230107`
  - Alert source: `awoooi-sre-war-room`
  - Alert kind: `public_route_502`
  - Resolved mode: `Recover`
  - Queue processor evidence: `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-230108.json`
  - Processed count: `1`
  - Mode: `Recover`
  - Exit code: `0`
  - Telegram relay: `sent=True`, `relayOk=True`
- Recovery verifier from the synthetic alert:
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-Recover-20260709-230108.json`
  - Public route failures: `0`
  - Host reachability failures: `0`
  - Harbor core healthy: `True`
  - AWOOOI deploy ready: `True`
- Agent99 self-health after integration:
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-230223.json`
  - Severity: `ok`
  - Task failures: `0`
  - Evidence critical: `0`
  - Evidence warning: `0`
  - Telegram inbox task: OK
  - SRE alert inbox task: OK
  - Dashboard queue pending: `0`
  - Dashboard alert pending: `0`

## Agent99 Problem Count / KM / RAG Receipt

2026-07-09 23:12 Asia/Taipei:

- Synthetic AwoooI SRE Telegram command:
  - instruction: `檢查所有網站是否恢復正常，若502就自動修復`
  - source: `telegram-sre-war-room`
  - request id: `request-20260709-231157-d53f53d8`
  - resolved mode: `Recover`
  - control exit code: `0`
- Execution evidence:
  - `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-231158.json`
  - processed source: `telegram-sre-war-room`
  - processed mode: `Recover`
  - processed ok: `True`
  - processed exit code: `0`
  - Telegram relay sent: `1`
  - Telegram evidence contains no Python code.
- Problem counters:
  - path: `C:\Wooo\Agent99\state\problem-counts.json`
  - total events: `1`
  - total resolved: `1`
  - total failed: `0`
  - problem key: `telegram-sre-war-room_agent99_operator_instruction`
  - occurrences: `1`
  - resolved: `1`
  - failed: `0`
- Knowledge outputs:
  - full event ledger: `C:\Wooo\Agent99\playbooks\agent99-incident-records.jsonl`
  - KM summary: `C:\Wooo\Agent99\playbooks\agent99-incident-km.md`
  - RAG import feed: `C:\Wooo\Agent99\playbooks\rag-import\agent99-incidents.jsonl`
- Dashboard:
  - problem events: `1`
  - resolved: `1`
  - failed: `0`
  - queue pending: `0`
  - alert pending: `0`
- SRE Telegram group routing:
  - Default relay chat readback was private chat `OoO`, not the SRE group.
  - Live Agent99 config now includes `allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE`.
  - Telegram inbox readback after the change: `ok=True`, `updatesSeen=0`, `processedCount=0`.
  - Next real `/agent99 ...` message from the SRE group will be accepted by title match and will reply back to the same Telegram origin.

## Agent99 SRE Telegram Monitoring Alert Auto-Ingest Receipt

2026-07-09 23:32 Asia/Taipei:

- Runtime behavior:
  - SRE Telegram group monitoring alerts are now consumed automatically.
  - Prefix `/agent99` is no longer required for monitoring alerts.
  - Auto-ingest is controlled by `autoIngestMonitoringAlerts=True`.
  - Group matching uses `allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE`.
- Loop prevention:
  - Agent99's own `Agent99 Alert` / `operator_command_result` messages are ignored by monitoring auto-ingest.
  - Operator command result Telegram reports bypass dedupe so every completed repair is reported.
  - Provider freshness signals are not passive dashboard health; they create a read-only freshness candidate and require provider window, last seen, direct reference, and verifier readback.
  - Provider freshness guardrail: do not switch provider, call paid model, edit env, or reload.
- Synthetic monitoring alert:
  - `Alertmanager FIRING critical public_route_502 service=awoooi host=192.168.0.120 HTTP 502`
  - no `/agent99` prefix.
- Evidence chain:
  - Telegram inbox: `C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260709-233230.json`
    - `inboxOk=True`
    - `alertedCount=1`
  - SRE alert inbox: `C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260709-233231.json`
    - queued count: `1`
    - failed count: `0`
  - Control loop: `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-233232.json`
    - alert source: `telegram-sre-war-room-monitoring`
    - alert kind: `public_route_502`
    - mode: `Recover`
    - processed ok: `True`
    - exit code: `0`
    - Telegram attempts: `1`
    - Telegram sent: `1`
  - Recovery: `C:\Wooo\Agent99\evidence\agent99-Recover-20260709-233055.json`
- Problem counters:
  - total events before: `2`
  - total events after: `3`
  - total resolved after: `3`
  - total failed after: `0`
  - problem key: `agent99-sre-alert-inbox_awoooi_192.168.0.120_public_route_502`
  - problem occurrences: `2`

## OpenClaw Cold-Start Readback

2026-07-09 17:38 Asia/Taipei:

- Production OpenClaw URL: `http://192.168.0.188:8088`.
- 188 service: `clawbot.service` is `active`.
- Health endpoint: `http://192.168.0.188:8088/health`.
- Health readback: `status=healthy`, `version=6.0`, `service=ClawBot`, `environment=production`, `telegram_bot=connected`.
- Agent99 latest status evidence: `C:\Wooo\Agent99\evidence\agent99-Status-20260709-173813.json`.
- Agent99 `aiServices.openclaw`: `ok=True`, `status=healthy`.
- Telegram relay from the Agent99 status run: `tgSent=2`, `relayOk=2`.
- Note: `192.168.0.188:8089` is not the current production OpenClaw endpoint.

## Windows Update No-Surprise-Reboot Policy

2026-07-09 17:39 Asia/Taipei:

- Evidence: `C:\Wooo\Agent99\evidence\agent99-windows-update-policy-20260709-173933.json`.
- `NoAutoRebootWithLoggedOnUsers=1`.
- `AlwaysAutoRebootAtScheduledTime=0`.
- `AUOptions=3`.
- `ActiveHoursStart=0`, `ActiveHoursEnd=23`.
- `gpupdateOk=True`.

## Agent99 Self-Health Receipt

2026-07-09 17:52 Asia/Taipei:

- Scheduled task: `Wooo-Agent99-Self-Health`.
- Trigger interval: every 5 minutes.
- Task Scheduler trigger verification:
  - `LastTaskResult=0`.
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-175218.json`.
  - `evidenceAfterTrigger=True`.
- Self-health checks:
  - `Wooo-Agent99-Startup-Recovery`: OK.
  - `Wooo-Agent99-Heartbeat`: OK.
  - `Wooo-Agent99-Performance-Guard`: OK.
  - `Wooo-Agent99-Self-Health`: OK.
  - Evidence freshness: performance, status, self-health, and startup recovery OK.
  - Latest performance evidence: all five hosts OK.
  - Telegram delivery: `tgSent=3`, `relayOk=3`.
  - Updated rule: a quiet interval with no alert attempts is OK; only failed alert attempts are Telegram critical.

## 110 Load Pressure Incident Readback

2026-07-09 18:07 Asia/Taipei:

- Symptom: 110 host load stayed above warning after all hosts rebooted.
- Initial findings:
  - `gitea` and `stockplatform-v2-postgres-1` were the highest regular service containers.
  - `/backup/scripts/backup-awoooi.sh` was running a remote `pg_dump` from 188 and local gzip compression.
  - A random container `mystifying_napier` was running `alpine "tar czf - /data"` with no restart policy and no labels, consuming CPU and writing compressed binary output into Docker logs.
- Controlled actions taken:
  - Backup pipeline PIDs were reduced to nice 15 / ionice best-effort 7.
  - `mystifying_napier` was stopped and left as `Dead`; it was not deleted, so evidence remains available.
- Agent99 automation added:
  - `stop-110-runaway-alpine-tar-data`: stops allowlisted `alpine` containers running `tar czf - /data` when 110 is at warning or critical load.
  - `renice-110-awoooi-backup-pressure`: lowers priority for `backup-awoooi.sh`, `gzip`, and remote `pg_dump` pipeline PIDs when 110 is at warning or critical load.
  - `pause-110-ci-runner-pressure`: still only runs at critical load with `load_per_core_critical`; runner is not touched for backup, disk, memory, or warning-only pressure.
  - Warning-level top CPU capture is enabled with a 10-second timeout to avoid diagnostics becoming a new load source.
- Verification:
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-Perf-20260709-180704.json`.
  - 110 load-per-core: `0.67`, severity `ok`.
  - 112, 120, 121, and 188: severity `ok`.
  - `perfCritical=0`, `perfWarning=0`.
  - Load-shedding actions skipped because host severity was below threshold.
  - Telegram relay: `tgSent=2`, `relayOk=2`.

## Agent99 Backup Health Receipt

2026-07-09 19:07 Asia/Taipei:

- Scheduled task: `Wooo-Agent99-Backup-Health`.
- Trigger interval: every 15 minutes.
- Task Scheduler trigger verification:
  - `LastTaskResult=0`.
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260709-190617.json`.
  - `evidenceAfterTrigger=True`.
- BackupCheck coverage:
  - Snapshot freshness targets: `awoooi`, `gitea`, `harbor`, `momo`, `langfuse`, `monitoring`, `signoz`, `clawbot`, `open-webui`, `sentry`, `ai-artifacts`, `configs`, and `public-routes`.
  - File/status checks: `backup.log`, `backup-status.log`, legacy `status.log`, legacy `cron.log`, integrity check status, restore drill status, and configs last-status JSON.
  - Cron contract patterns: `backup-all.sh`, `backup-awoooi-frequent.sh`, `backup-health-textfile-exporter.py`, `check-backup-integrity.sh`, `sync-offsite-backups.sh --mode status`, and `backup-status.sh`.
- Latest result:
  - `backupHealth.severity=ok`.
  - `targetCritical=0`, `targetWarning=0`.
  - `fileCritical=0`, `fileWarning=0`.
  - `cronCritical=0`, `cronWarning=0`.
  - `targetCount=13`, `fileCheckCount=7`, `cronPatternCount=6`.
  - Telegram relay: `tgSent=2`, `relayOk=2`.
- Self-health cross-check:
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-190714.json`.
  - `Wooo-Agent99-Backup-Health`: OK, `LastTaskResult=0`.
  - Backup evidence freshness: OK, latest file `agent99-BackupCheck-20260709-190617.json`, age `0.5` minutes, max age `30` minutes.

## Alert-to-Remediation / Evidence Sanitization Readback

2026-07-09 22:11 Asia/Taipei:

- Root cause fixed for unreadable Telegram relay messages:
  - Relay sender argument index corrected from `sys.argv[1]` to `sys.argv[2]`.
  - Telegram now receives formatted Agent99 alert text, not embedded Python relay code.
- Noise control:
  - `alertAllOperations=false`.
  - `alertInfoEvents=false`.
  - `dedupeMinutes=30`.
  - `agent_complete` remains evidence-only and does not alert by default.
- Alert-to-action rule:
  - `Perf -ControlledApply` suppresses immediate performance spam, performs allowlisted remediation first, then alerts `load_shed_result`.
  - If no remediation applies, Agent99 alerts the unresolved performance issue.
  - 188 disk pressure is now allowlisted through `prune-188-docker-build-cache-disk-pressure`.
  - The 188 action uses `timeoutSeconds=240` plus a post-apply verifier; `postVerifyMaxDiskUsedPercent=89`.
  - Timeout alone is not final truth; post-verifier success can mark remediation effective.
- Self-health false critical fixed:
  - A quiet interval with no recent alerts is OK.
  - Telegram health is critical only when recent alert attempts exist and all delivery attempts fail.
  - Telegram delivery health now uses a 60-minute lookback across recent evidence files.
  - Transient performance warnings are no longer promoted to self-health critical; only missing or critical host performance evidence is critical.
- Backup monitor false critical fixed:
  - Backup snapshot readback now supports `snapshotReadTimeoutSeconds`.
  - 99 live config sets `snapshotReadTimeoutSeconds=90`.
  - Harbor backup was verified as present and fresh after a timeout false critical; latest metadata was from 2026-07-09 17:48 Asia/Taipei and within the 1800-minute SLA.
- Evidence security fixed:
  - Harbor readback now stores selected health fields only: name, running state, health state.
  - Raw `docker inspect` output is not stored.
  - Existing Status evidence on 99 was sanitized: 77 files checked; `inspectOutputNotRedactedCount=0`; `unsafeHarborFieldCount=0`.
- Latest recovery proof:
  - Status evidence: `C:\Wooo\Agent99\evidence\agent99-Status-20260709-221147.json`.
  - Host reachability: 110, 112, 120, 121, and 188 all OK.
  - Public routes: `awoooi.wooo.work`, `stock.wooo.work`, `2026fifa.wooo.work`, `gitea.wooo.work`, and `harbor.wooo.work` all HTTP 200 / non-502.
  - Performance: `perfCritical=0`, `perfWarning=0`; 188 disk used 87%; 110 load-per-core 0.85.
  - Harbor: core, Redis, and registry healthy with redacted evidence.
  - AWOOOI: `deployReady=true`, `imagePullFailure=false`, `crashFailure=false`.
  - Backup evidence after timeout fix: `C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260709-221759.json`, `backupHealth.severity=ok`, all target/file/cron critical and warning counts are 0.
  - Final self-health evidence: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-222236.json`, `selfHealth.severity=ok`, `taskFailures=0`, `performanceSeverity=ok`, `telegramSeverity=ok`.

## 99 Visible Agent Control Center

2026-07-09 22:33 Asia/Taipei:

- Agent99 is visible and controllable from the 99 Windows desktop.
- New scheduled task:
  - `Wooo-Agent99-Control-Loop`.
  - Runs every 1 minute.
  - Updates dashboard and processes queue commands.
- Desktop shortcuts:
  - `C:\Users\Public\Desktop\Agent99 Control Center.lnk`.
  - `C:\Users\Public\Desktop\Agent99 Queue.lnk`.
  - `C:\Users\Public\Desktop\Agent99 Evidence.lnk`.
  - `D:\Desktop\Agent99 Control Center.lnk`.
  - `D:\Desktop\Agent99 Queue.lnk`.
  - `D:\Desktop\Agent99 Evidence.lnk`.
- Dashboard:
  - HTML: `C:\Wooo\Agent99\dashboard\index.html`.
  - JSON: `C:\Wooo\Agent99\dashboard\agent99-control-center.latest.json`.
  - Latest dashboard timestamp readback: `2026-07-09T22:33:26+08:00`.
  - Queue readback: `pending=0`, `processed=2`, `failed=0`.
- Queue command contract:
  - Add a JSON command to `C:\Wooo\Agent99\queue`.
  - `ControlTick` processes only allowlisted modes.
  - Results are written to `C:\Wooo\Agent99\queue\processed` or `C:\Wooo\Agent99\queue\failed`.
  - Operator command results are sent to Telegram.
- Verification:
  - Command ID: `codex-visible-publicsmoke-20260709-223308`.
  - Mode: `PublicSmoke`.
  - Result: `ok=true`, `exitCode=0`.
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-223309.json`.
  - Telegram: `operator_command_result`, `severity=info`, `sent=true`, `relayOk=true`.
  - Self-health: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-223340.json`, `selfHealth.severity=ok`, `Wooo-Agent99-Control-Loop=ok`.
  - Latest performance: `C:\Wooo\Agent99\evidence\agent99-Perf-20260709-223419.json`, `perfCritical=0`, `perfWarning=0`.

## Operator Instruction Intake

2026-07-09 22:41 Asia/Taipei:

- Agent99 accepts text instructions on 99 through:
  - `C:\Wooo\Agent99\bin\agent99-submit-request.ps1`.
  - `C:\Wooo\Agent99\agent99-submit-request.cmd`.
  - `Agent99 Submit Request.lnk` on the desktop.
- Flow:
  - instruction -> request record -> queue command -> `ControlTick` -> evidence -> Telegram result.
- Current allowlisted request router:
  - websites / routes / 502 -> `PublicSmoke`.
  - restore / reboot / repair -> `Recover`.
  - VM / VMware -> `StartVMs`.
  - CPU / load / memory / disk / performance -> `Perf`.
  - backup / snapshot / cron -> `BackupCheck`.
  - self health / agent health -> `SelfCheck`.
  - Harbor / registry -> `HarborRepair`.
  - AWOOOI / K3s / pods -> `AwoooRepair`.
- Verification:
  - Chinese instruction submitted from Codex to 99: check whether all websites are normal.
  - Request ID: `request-20260709-224133-0dc045fe`.
  - Resolved mode: `PublicSmoke`.
  - Processed result: `ok=true`, `exitCode=0`.
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-224134.json`.
  - Public routes: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502.
  - Telegram: `operator_command_result`, `severity=info`, `sent=true`, `relayOk=true`.