awoooi/HOST-REBOOT-AI-AUTOMATION-SOP.md

# Host Reboot AI Automation SOP

Last updated: 2026-07-11

Canonical enterprise scope and current work order:

- `docs/workplans/2026-07-10-agent99-enterprise-ai-automation-master-plan.md`
- `docs/operations/agent99-enterprise-ai-automation-work-items.snapshot.json`

This SOP remains the detailed reboot procedure. It cannot mark recovery complete from command success, route HTTP 200, or a Telegram delivery receipt. Completion requires the reboot state machine, all required host/service/product/data gates, mode-specific verifiers and AWOOOI/AwoooP callback readback.

## P0 Goal

After host 99 or any VM host is rebooted, the automation must detect the reboot, restore VM reachability, restore critical services, and prove public sites are no longer returning 502.

Target recovery bar:

- Detect 99 / 110 / 112 / 120 / 121 / 188 reboot or down state.
- Ensure VMware Workstation on 99 is running and required VMs are powered on.
- Restore Harbor / registry on 110 before forcing K3s workloads to repull images.
- Restore AWOOOI production pods on 120.
- Verify public HTTP readback for AWOOOI, StockPlatform, FIFA, Gitea, and Harbor.
- Emit operator-visible evidence and Telegram alerts for down, recovering, recovered, and degraded states.

## 2026-07-09 Incident Findings To Automate

The reboot incident was not a generic 502 problem. The confirmed chain was:

1. 110 rebooted and Docker/Harbor did not fully recover.
2. `harbor-core` entered a failed/restarting state because Harbor Redis / Docker DNS dependency was not clean.
3. K3s on 120 could not pull AWOOOI images from `192.168.0.110:5000`.
4. AWOOOI prod pods stayed in `ImagePullBackOff`.
5. Public nginx returned 502 because upstream pods were unavailable.
6. Restarting / rebooting 110 cleared the Docker / systemd / Harbor dependency state.
7. Once `harbor-core` became healthy, 120 pods repulled images and AWOOOI recovered.

This failure mode must be treated as a first-class automated recovery path.

## Recovery Priority Order

P0. Control-plane and VM reachability

- 99 Windows host reachable.
- VMware Workstation process running on 99.
- VMs 110, 112, 120, 121, and 188 powered on.
- Hosts 110, 112, 120, 121, and 188 respond to ICMP or SSH.

P1. 110 Harbor and registry dependency chain

- Docker engine active.
- Harbor Redis running.
- `harbor-core` running and healthy.
- Harbor registry / registryctl running and healthy.
- `harbor.wooo.work` returns HTTP 200 after redirect.
- Internal registry pull path from K3s nodes is reachable.

P2. 120 K3s production workloads

- `awoooi-prod` deployments ready:
  - `awoooi-api` 1/1
  - `awoooi-web` 2/2
  - `awoooi-worker` 1/1
  - `awoooi-auto-repair-canary` 1/1
- No critical pods stuck in `ImagePullBackOff`, `ErrImagePull`, `CrashLoopBackOff`, or `Pending`.
- If image pull failures remain after Harbor is healthy, delete only the failed pods and let controllers recreate them.

P3. Public user-visible surfaces

- `awoooi.wooo.work` returns HTTP 200 for web.
- `awoooi.wooo.work/api/v1/health` returns JSON and not 502.
- `stock.wooo.work` returns HTTP 200.
- `2026fifa.wooo.work` returns HTTP 200.
- `gitea.wooo.work` returns HTTP 200.
- `harbor.wooo.work` returns HTTP 200.

P4. Degraded provider follow-up

- AWOOOI health can be `degraded` even when 502 is fixed.
- Treat AI provider health separately from user-visible 502 restoration.
- Follow-up checks include `ollama_gcp_a`, `ollama_gcp_b`, and `ollama_local`.
- Do not block public-site 502 recovery on non-primary AI provider cooldown unless the app is user-impacting.

## Automated Decision Tree

1. Detect reboot / outage.
   - Ping 110, 112, 120, 121, 188.
   - SSH to 110 and 120 if keys are available.
   - Query VMware running VM list on 99 if `vmrun.exe` is available.

2. If a VM is not reachable.
   - Check VMware VM state.
   - Start the VM from 99.
   - Wait for guest ping / SSH.
   - Alert Telegram: `host_recovering`.

3. If AWOOOI public route is 502.
   - Check 120 `awoooi-prod` deployments.
   - If pods are `ImagePullBackOff`, check 110 Harbor first.
   - Do not repeatedly delete pods before Harbor is healthy.

4. If 110 Harbor is unhealthy.
   - Check Docker active state.
   - Check `redis`, `harbor-core`, `registry`, and `registryctl`.
   - Restart Harbor Redis and `harbor-core` first.
   - If Docker/systemd/cgroup operations hang, use controlled 110 guest reboot through VMware.
   - After 110 returns, wait for Harbor core healthy before touching K3s pods.

5. If Harbor is healthy and pods are still in image pull failure.
   - Delete only failed AWOOOI pods in `awoooi-prod`.
   - Wait for deployments to become ready.
   - Read back public HTTP.

6. If public sites are HTTP 200 but AWOOOI health is degraded.
   - Record degraded providers.
   - Alert `service_degraded`.
   - Continue provider-specific remediation separately.

## 99 Windows AI Agent Control Node

99 must become the automation entrypoint for reboot recovery.

Required capabilities:

- Run a local AI Agent service on Windows 99 with least-privilege operator permissions.
- Execute the reboot scorecard on a schedule and on boot.
- Control VMware Workstation VM startup through `vmrun.exe`.
- Use SSH keys to 110 and 120 for service-level checks and controlled repair.
- Send Telegram alerts for down, recovering, recovered, degraded, failed, and controlled-remediation-result states.
- Do not stop at alerting. Every warning/critical alert must either trigger an allowlisted remediation, record that no remediation matched, or record that the action was skipped by cooldown/min-severity.
- Do not send start/complete/info noise to Telegram by default. Human-facing alerts must be deduped and action-oriented.
- Persist scorecard evidence to a local non-secret log directory.
- Never store raw passwords, tokens, cookies, sessions, or `.env` in the repo.
- Accept operator commands through the 99 desktop submitter, Telegram `/agent99 ...`, and local queue files without depending on Codex quota.
- Accept SRE war-room alert JSON at `C:\Wooo\Agent99\alerts\incoming`, resolve each alert into an allowlisted mode, and either run controlled remediation or record why no remediation matched.

Candidate implementation layers:

- P0: Windows Task Scheduler runs `host-reboot-scorecard.ps1` at startup and every minute for the first 15 minutes after 99 boot.
- P1: Install a local Windows service wrapper for the scorecard loop.
- P2: Add an AI Agent runtime that consumes scorecard JSON and chooses approved recovery actions.
- P3: Add a UI/console dashboard on 99 for operator evidence and manual override.

Current command and alert loops:

- `Wooo-Agent99-Control-Loop`: processes `C:\Wooo\Agent99\queue`, updates the dashboard, and writes execution evidence.
- `Wooo-Agent99-Telegram-Inbox`: reads Telegram group commands with Agent99 prefixes and submits allowlisted queue work.
- `Wooo-Agent99-SRE-Alert-Inbox`: reads `C:\Wooo\Agent99\alerts\incoming`, converts actionable alerts into queue work, and triggers the control loop.
- `Wooo-Agent99-SRE-Alert-Relay`: runs the 99-side HTTP relay for sanitized AWOOI Alertmanager JSON and writes accepted alerts into `C:\Wooo\Agent99\alerts\incoming`.
- `Wooo-Agent99-Performance-Guard`: runs every minute and performs allowlisted load shedding before escalating unresolved performance alerts.
- `Wooo-Agent99-Self-Health`: verifies the scheduled tasks and evidence freshness for the control plane itself.

AwoooI SRE Telegram command loop:

- Operators and project AI Agents can issue commands in the SRE Telegram group with `/agent99 ...`.
- Agent99 records these as `source=telegram-sre-war-room`.
- Failed or escalated automated repairs send `operator_command_result` back to Telegram.
- Successful automated repairs are evidence-first: they write queue, evidence, problem count, KM, and RAG receipts, then suppress success-only Telegram noise.
- Failure results include source, failure instruction, mode, exit code, evidence path, and problem count.
- The command path must not depend on Codex quota.
- The command path must not depend only on the default `TELEGRAM_CHAT_ID`, because that can point to a private chat. Agent99 also accepts command updates from Telegram chats whose title matches `AwoooI SRE|AWOOOI SRE|SRE`.
- When the command comes from a matched Telegram group, the completion report is sent back to that same Telegram origin; stored evidence redacts the raw chat id as `telegram-origin`.
- Non-command monitoring alerts in the SRE Telegram group are auto-ingested when they look like Alertmanager/FIRING/critical/warning/502/CPU/disk/Harbor/K3s/backup/host-down/provider-freshness alerts.
- Auto-ingested monitoring alerts use `source=telegram-sre-war-room-monitoring` and enter the same SRE alert inbox / queue / recovery / Telegram completion / KM-RAG path.

Alert-to-action routing:

- Public 502 / route down / website down -> `Recover`.
- Host down / reboot detected / VM not running -> `Recover`.
- CPU, memory, disk, or load pressure -> `Perf` with controlled apply.
- Harbor or registry unhealthy -> `HarborRepair`.
- AWOOOI K3s, pod, deployment, or image pull failure -> `AwoooRepair`.
- Backup freshness, snapshot, restore drill, or cron issue -> `BackupCheck`.
- Provider freshness, last-seen, ingestion, raw signal, or no-false-green risk -> `ProviderFreshness`.
- Unknown actionable alert -> `Status` evidence first.

Producer-side Agent99 bridge:

- AWOOI API Alertmanager webhook schedules an Agent99 bridge after normalization and fingerprinting.
- Preferred transport: when `AGENT99_SRE_ALERT_RELAY_URL` is configured, the bridge POSTs sanitized alert JSON to the 99 relay. Production URL is `http://192.168.0.99:8787/agent99/sre-alert/`.
- Fallback transport: when `AGENT99_SRE_ALERT_INBOX_PATH` is configured and relay URL is empty, the bridge writes sanitized alert JSON directly to the Agent99 SRE inbox contract.
- If relay/path is empty or unavailable, the bridge fails open and logs; it must not cause Alertmanager retries.
- Relay safety: only LAN/local source prefixes are accepted by default; optional `AGENT99_SRE_RELAY_TOKEN` / `AGENT99_SRE_ALERT_RELAY_TOKEN` values stay in env/Secret and are never stored in repo or evidence.
- Provider freshness bridge output must run direct readback, then require provider window, last seen, direct reference, and verifier readback before dashboard green is trusted.
- Provider freshness no-false-green rule: route 200 or dashboard up is not enough. The Agent99 verifier must read `https://awoooi.wooo.work/api/v1/platform/events/dossier/recurrence?project_id=awoooi&limit=20` and record `providerWindowMinutes`, `lastSeen`, `directReference`, and `verifierReadback`.

2026-07-11 P0-002 source receipt, runtime deployment pending:

- `ACTION REQUIRED` cards for `cold-start-gate` now enter the same Agent99 ingress as Alertmanager and route to `Recover` with the reboot/cold-start scorecard verifier.
- Failed `AWOOOI CI/CD` cards enter Agent99 as `cicd_failure` in `Status` mode. Agent99 records evidence but waits for the next successful Gitea CD/deploy/runtime receipt before source-event closure.
- The producer stores a sanitized relay acceptance receipt in the outbound source envelope before Telegram delivery; Telegram remains a human lifecycle surface, not the only queue or database.
- Cold-start candidates use the Agent99 execution route and never synthesize `kubectl rollout restart deployment/cold-start-gate`.
- A relay receipt with `accepted=false` releases the fingerprint single-flight lock and returns failed. This prevents a transport-level response from permanently suppressing an unprocessed incident.
- Verification completed in source: 249 focused post-rebase tests and 262 expanded all-Telegram plus Agent99/RepairCandidate tests passed after integrating the Telegram single-owner mainline; the repository static governance suite also passed. This does not prove Windows 99 is running this revision; runtime manifest and exact do-not-page replay are still mandatory.

Approved controlled actions for the 99 agent:

- Start a missing VM.
- Ping and SSH readback.
- Restart Harbor Redis / Harbor core after evidence of Harbor token failure.
- Delete K3s pods stuck in image pull failure only after Harbor is healthy.
- Run public route readback.
- Send Telegram status.

Actions still requiring explicit break-glass:

- Reading or exfiltrating secrets.
- Destructive database operations.
- Firewall cutover.
- Remote delete / prune / force push.
- Production provider route cutover with cost or billing impact.

## Evidence Required Before Calling Recovery Complete

Recovery is complete only when all of the following are captured:

- Agent99 live preflight shows runtime manifest matched, tasks `9/9`, relay listening, active queues clear and every required evidence row `fresh=true`, `parsed=true`, `contentHealthy=true`, `usable=true`.
- The production-principal Recover receipt contains all seven phases: `vm_start`, `host_reachability`, `host112_guest_readiness`, `harbor_registry`, `k3s_workloads`, `public_routes`, and `full_sop_scorecard`.
- Telegram has one lifecycle receipt for the run: either a real `sent + visualSent` recovery timeline or an explicit same-state `dedupe_window` suppression backed by a previous sent visual receipt.
- For alert-triggered recovery, the identical callback/run/trace is durably readable from AWOOOI operation history and the AwoooP run projection. Telegram success alone never closes the incident.

- Five VM hosts have reachability evidence.
- 110 Harbor core is healthy.
- 120 AWOOOI deployments are ready.
- Public routes return non-502, preferably HTTP 200.
- Any degraded health components are listed separately with owner / next action.

## Current 2026-07-09 Receipts

- 110 `harbor-core`: recovered to `Up ... (healthy)`.
- 120 `awoooi-prod`:
  - `awoooi-api`: 1/1
  - `awoooi-web`: 2/2
  - `awoooi-worker`: 1/1
  - `awoooi-auto-repair-canary`: 1/1
- Public readback:
  - `awoooi.wooo.work`: HTTP 200 after redirect.
  - `stock.wooo.work`: HTTP 200 after redirect.
  - `2026fifa.wooo.work`: HTTP 200 after redirect.
  - `gitea.wooo.work`: HTTP 200 after redirect.
  - `harbor.wooo.work`: HTTP 200 after redirect.
- Host reachability:
  - 110, 112, 120, 121, 188 responded to ping or SSH.
- Remaining degraded signal:
  - AWOOOI health reported degraded AI provider signals including `ollama_gcp_b` timeout and `ollama_local` connection refused / unavailable.

## Agent99 Installation Receipt

2026-07-09 16:49 Asia/Taipei:

- 99 host: `WOOO-SUPER`.
- Agent root: `C:\Wooo\Agent99`.
- VMware CLI: `C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe`.
- Live config: `C:\Wooo\Agent99\config\agent99.config.json`.
- Evidence directory: `C:\Wooo\Agent99\evidence`.
- Scheduled tasks:
  - `Wooo-Agent99-Heartbeat`: `LastTaskResult=0`, latest status evidence `agent99-Status-20260709-164845.json`.
  - `Wooo-Agent99-Startup-Recovery`: `LastTaskResult=0`, latest recovery evidence `agent99-Recover-20260709-164937.json`.
- Startup recovery readback:
  - VM selector: 110, 188, 112, 120, 121 all `host_already_reachable`.
  - Harbor: core, Redis, and registry all healthy.
  - AWOOOI: deploy ready, no image pull failure, no crash failure.
  - Public routes: `awoooi.wooo.work`, `stock.wooo.work`, `2026fifa.wooo.work`, `gitea.wooo.work`, and `harbor.wooo.work` all HTTP 200 / non-502.

## Agent99 Performance Guard Receipt

2026-07-09 17:15 Asia/Taipei:

- Scheduled task: `Wooo-Agent99-Performance-Guard`.
- Trigger interval: every 1 minute.
- Last task result: `0`.
- Latest evidence: `C:\Wooo\Agent99\evidence\agent99-Perf-20260709-171502.json`.
- Host performance readback:
  - 110: OK.
  - 112: OK.
  - 120: OK.
  - 121: OK.
  - 188: OK.
- Controlled load shedding:
  - Only allowlisted actions can run.
  - Current 110 runner action requires `load_per_core_critical`.
  - Memory/disk pressure alerts do not execute the runner pause action.
- Telegram audit:
  - Agent99 emits Telegram attempts for warnings, critical events, failed or escalated operator commands, and load-shed actions.
  - Successful operator commands are not Telegram noise; they are recorded in evidence, problem counts, KM, and RAG.
  - Agent99 supports local env aliases and an existing-token relay.
  - Current relay: 110 `stockplatform-v2-api-1`, using that container's existing `TELEGRAM_BOT_TOKEN` / `TELEGRAM_CHAT_ID`.
  - 2026-07-09 17:27 verification: `Wooo-Agent99-Performance-Guard` `LastTaskResult=0`; latest evidence `agent99-Perf-20260709-172712.json`; `tgSent=2`; `relayOk=True`.
  - Token and chat id values were not printed, copied into repo files, or written to evidence.

## Agent99 Command And SRE Alert Integration Receipt

2026-07-09 23:03 Asia/Taipei:

- Telegram command inbox is deployed and scheduled:
  - Script: `C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1`
  - Task: `Wooo-Agent99-Telegram-Inbox`
  - Latest evidence: `C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260709-230031.json`
  - Result: `ok=True`, `updatesSeen=0`, `processedCount=0`
  - Evidence check: no Python code, no token text.
- SRE war-room alert inbox is deployed and scheduled:
  - Script: `C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1`
  - Task: `Wooo-Agent99-SRE-Alert-Inbox`
  - Incoming path: `C:\Wooo\Agent99\alerts\incoming`
  - Latest evidence: `C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260709-230226.json`
- Synthetic 502 alert test:
  - Alert id: `synthetic-public-502-20260709-230107`
  - Alert source: `awoooi-sre-war-room`
  - Alert kind: `public_route_502`
  - Resolved mode: `Recover`
  - Queue processor evidence: `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-230108.json`
  - Processed count: `1`
  - Mode: `Recover`
  - Exit code: `0`
  - Telegram relay: `sent=True`, `relayOk=True`
- Recovery verifier from the synthetic alert:
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-Recover-20260709-230108.json`
  - Public route failures: `0`
  - Host reachability failures: `0`
  - Harbor core healthy: `True`
  - AWOOOI deploy ready: `True`
- Agent99 self-health after integration:
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-230223.json`
  - Severity: `ok`
  - Task failures: `0`
  - Evidence critical: `0`
  - Evidence warning: `0`
  - Telegram inbox task: OK
  - SRE alert inbox task: OK
  - Dashboard queue pending: `0`
  - Dashboard alert pending: `0`

## Agent99 Problem Count / KM / RAG Receipt

2026-07-09 23:12 Asia/Taipei:

- Synthetic AwoooI SRE Telegram command:
  - instruction: `檢查所有網站是否恢復正常，若502就自動修復`
  - source: `telegram-sre-war-room`
  - request id: `request-20260709-231157-d53f53d8`
  - resolved mode: `Recover`
  - control exit code: `0`
- Execution evidence:
  - `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-231158.json`
  - processed source: `telegram-sre-war-room`
  - processed mode: `Recover`
  - processed ok: `True`
  - processed exit code: `0`
  - Telegram relay sent: `1`
  - Telegram evidence contains no Python code.
- Problem counters:
  - path: `C:\Wooo\Agent99\state\problem-counts.json`
  - total events: `1`
  - total resolved: `1`
  - total failed: `0`
  - problem key: `telegram-sre-war-room_agent99_operator_instruction`
  - occurrences: `1`
  - resolved: `1`
  - failed: `0`
- Knowledge outputs:
  - full event ledger: `C:\Wooo\Agent99\playbooks\agent99-incident-records.jsonl`
  - KM summary: `C:\Wooo\Agent99\playbooks\agent99-incident-km.md`
  - RAG import feed: `C:\Wooo\Agent99\playbooks\rag-import\agent99-incidents.jsonl`
- Dashboard:
  - problem events: `1`
  - resolved: `1`
  - failed: `0`
  - queue pending: `0`
  - alert pending: `0`
- SRE Telegram group routing:
  - Default relay chat readback was private chat `OoO`, not the SRE group.
  - Live Agent99 config now includes `allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE`.
  - Telegram inbox readback after the change: `ok=True`, `updatesSeen=0`, `processedCount=0`.
  - Next real `/agent99 ...` message from the SRE group will be accepted by title match and will reply back to the same Telegram origin.

## Agent99 SRE Telegram Monitoring Alert Auto-Ingest Receipt

2026-07-09 23:32 Asia/Taipei:

- Runtime behavior:
  - SRE Telegram group monitoring alerts are now consumed automatically.
  - Prefix `/agent99` is no longer required for monitoring alerts.
  - Auto-ingest is controlled by `autoIngestMonitoringAlerts=True`.
  - Group matching uses `allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE`.
- Loop prevention:
  - Agent99's own `Agent99 Alert` / `operator_command_result` messages are ignored by monitoring auto-ingest.
  - Successful operator command results are suppressed from Telegram and kept in evidence/KM/RAG; failed or escalated command results remain alertable.
  - Provider freshness signals are not passive dashboard health; they create a read-only freshness candidate, run direct AWOOI readback, and require provider window, last seen, direct reference, and verifier readback.
  - Provider freshness guardrail: do not switch provider, call paid model, edit env, or reload.
- Synthetic monitoring alert:
  - `Alertmanager FIRING critical public_route_502 service=awoooi host=192.168.0.120 HTTP 502`
  - no `/agent99` prefix.
- Evidence chain:
  - Telegram inbox: `C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260709-233230.json`
    - `inboxOk=True`
    - `alertedCount=1`
  - SRE alert inbox: `C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260709-233231.json`
    - queued count: `1`
    - failed count: `0`
  - Control loop: `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-233232.json`
    - alert source: `telegram-sre-war-room-monitoring`
    - alert kind: `public_route_502`
    - mode: `Recover`
    - processed ok: `True`
    - exit code: `0`
    - Telegram attempts: `1`
    - Telegram sent: `1`
  - Recovery: `C:\Wooo\Agent99\evidence\agent99-Recover-20260709-233055.json`
- Problem counters:
  - total events before: `2`
  - total events after: `3`
  - total resolved after: `3`
  - total failed after: `0`
  - problem key: `agent99-sre-alert-inbox_awoooi_192.168.0.120_public_route_502`
  - problem occurrences: `2`

## Agent99 Alert Noise And Provider Freshness Readback Receipt

2026-07-10 10:00 Asia/Taipei:

- Runtime patch deployed on 99:
  - Script: `C:\Wooo\Agent99\bin\agent99-control-plane.ps1`.
  - Parser readback: `parse_ok`.
  - Source commit: `0a7a0115 fix(agent99): reduce alert noise and verify freshness`.
- Provider freshness behavior:
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-ProviderFreshness-20260710-100028.json`.
  - Status: `direct_readback_ok_no_source_events`.
  - Missing fields after automatic readback: `lastSeen` only.
  - `providerWindowMinutes=120`.
  - `directReference=https://awoooi.wooo.work/api/v1/platform/events/dossier/recurrence?project_id=awoooi&limit=20`.
  - `verifierReadback.ok=True`, `httpStatus=200`, `sourceEventTotal=0`.
- Telegram noise control:
  - Evidence log: `C:\Wooo\Agent99\evidence\agent99-ProviderFreshness-20260710-100028.log`.
  - Same provider-freshness state was suppressed: `telegram_dedupe_skip type=provider_freshness_triage severity=warning ageMinutes=5.83 cooldown=30`.
  - Evidence log: `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260710-100027.log`.
  - Successful operator command was evidence-only: `telegram_operator_success_suppressed type=operator_command_result mode=ProviderFreshness id=operator-suppress-smoke-20260710-100026`.
  - Control evidence: `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260710-100027.json`, `processedCount=1`, `pendingCount=0`, `telegram=[]`.
- Scheduled entrypoints verified on 99:
  - `Wooo-Agent99-Control-Loop`.
  - `Wooo-Agent99-SRE-Alert-Inbox`.
  - `Wooo-Agent99-Telegram-Inbox`.
  - `Wooo-Agent99-SRE-Alert-Relay` is running.
- Operational rule:
  - Do not treat repeated success receipts as user-facing alerts.
  - Send user-facing Telegram only for warnings, criticals, unresolved blockers, failed/escalated operator commands, and controlled-remediation actions.
  - Keep successful automation in evidence, problem counters, KM, and RAG so repeated incidents trend down without flooding the SRE group.

## Agent99 Telegram Alert UX And Noise Freeze Work Items

2026-07-10 10:41 Asia/Taipei:

- Operator requirement:
  - AwoooI SRE Telegram alerts must be human-readable, professionally formatted, and action-oriented.
  - Important incident updates should include visual context when available: dashboard image, public-route screenshot, service card, or evidence summary image.
  - Do not send raw machine fields, empty metrics, repeated success receipts, or text-only alert floods.
- Immediate runtime noise freeze applied on 99:
  - Disabled scheduled task: `Wooo-Agent99-Performance-Guard`.
  - Disabled scheduled task: `Wooo-Agent99-Backup-Health`.
  - Disabled scheduled task: `Wooo-Agent99-Self-Health`.
  - Ended running task instance: `Wooo-Agent99-Performance-Guard`.
  - Kept alert intake active: `Wooo-Agent99-SRE-Alert-Relay`, `Wooo-Agent99-SRE-Alert-Inbox`, `Wooo-Agent99-Telegram-Inbox`, and `Wooo-Agent99-Control-Loop`.
- 2026-07-10 10:47 runtime readback:
  - `Wooo-Agent99-Performance-Guard`: `Disabled`, last result `0`.
  - `Wooo-Agent99-Backup-Health`: `Disabled`, last result `0`.
  - `Wooo-Agent99-Self-Health`: `Disabled`, last result `0`.
  - `Wooo-Agent99-SRE-Alert-Relay`: `Running`.
  - `Wooo-Agent99-SRE-Alert-Inbox`: `Ready`, last result `0`.
  - `Wooo-Agent99-Telegram-Inbox`: `Running`.
  - `Wooo-Agent99-Control-Loop`: `Ready`, last result `0`.
  - `SecurityTriage` smoke evidence: `C:\Wooo\Agent99\evidence\agent99-SecurityTriage-20260710-104456.json`.
  - `SecurityTriage` smoke result: `security_triage_no_recent_alert_body_seen`, `hitCount=0`, `telegramCount=0`.
- P0 work items added:
  - P0-0: keep noisy sensor schedules paused until their alerts have dedupe, clear reason text, and verifier-backed fields.
  - P0-1: replace raw Agent99 text alerts with professional Telegram incident cards: title, severity, affected service, user impact, Agent99 action, verifier result, evidence path, repeated count, next action.
  - P0-2: add visual attachment support for high-impact alerts: public URL screenshots, dashboard screenshots, and rendered evidence summary images.
  - P0-3: suppress alerts with empty metric fields such as blank load/core, memory, disk, age, or maxAge unless a verifier explains why readback failed.
  - P0-4: route application config alerts such as Bitan `E_AI_MISSING_KEY` to evidence-first `Status`; do not read or print secret values.
  - P0-5: route IWOOOS/Wazuh/SIEM alerts to `SecurityTriage`; collect source log reference, affected host/service, blast radius, repeated count, and allowlisted next action without arbitrary shell or secret exposure.
  - P0-6: route Gitea repository missing alerts to evidence-first `Status` until a dedicated Gitea recovery playbook is verified.
  - P0-7: run live TG-to-Agent99 end-to-end tests for each alert family before re-enabling the paused sensor schedules.
  - P0-8: all Agent99 Telegram alert messages must include Traditional Chinese user-facing labels and human-readable context; professional terms such as `readback_failed`, `SecurityTriage`, `directReadback`, `KM/RAG`, and provider names may remain English when translation would reduce precision.
  - P0-9: Telegram inbox must not force `controlledApply=true` on every monitoring alert. `controlledApply` is resolved by SRE mode: remediation modes may run controlled apply, evidence-first modes such as `ProviderFreshness`, `BackupCheck`, `SecurityTriage`, and `Status` must remain non-apply unless explicitly allowlisted.
  - P0-10: Telegram delivery must support actual image/photo delivery through the relay path, not only text plus a local evidence image.
  - P0-11: paused Agent99 sensors must pass a no-TG/no-remediation sensor restore gate before schedules are re-enabled. The gate must prove Performance, BackupHealth, and SelfHealth are eligible and must write readback evidence with `sentTelegram=false`, `controlledApply=false`, and `ranRemediation=false`.
  - P0-12: slow SSH readback after reboot must not be collapsed into "host down". Performance and backup metadata checks must record `elapsedMs`, use configurable timeouts, and run a slow retry on timeout before raising `perf_readback_failed` or backup readback failure.
  - P0-13: first-run evidence refresh after a noise freeze must use controlled `-SuppressAlerts` for Perf/Backup/SelfCheck so stale evidence is refreshed without Telegram noise; schedules are then re-enabled without suppression.
  - P0-14: synthetic or `TEST_DO_NOT_PAGE` monitoring tests must still run the full Telegram -> SRE inbox -> ControlTick -> mode evidence path, but `suppressTelegram=true` and `SuppressReason=do_not_page` must propagate end to end so validation never pages the SRE Telegram group.
  - P0-15: real SRE alert completion must send a human-readable Traditional Chinese automation result card back to Telegram with action taken, result, evidence, problem count, and KM path. Ordinary successful operator commands remain suppressed unless explicitly configured.
- 2026-07-10 10:54 visual card verifier:
  - Runtime script deployed: `C:\Wooo\Agent99\bin\agent99-control-plane.ps1`.
  - Backup before deploy: `C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-105439`.
  - Parser readback: `parse_ok`.
  - Self-test evidence: `C:\Wooo\Agent99\evidence\agent99-TelegramCardSelfTest-20260710-105440.json`.
  - Visual card: `C:\Wooo\Agent99\evidence\telegram-cards\agent99-card-performance_warning-20260710-105440.png`.
  - Result: `ok=True`, `visualExists=True`, `sentTelegram=False`, `messageFormat=incident_card_v2`.
  - Local visual inspection: PNG is `1040x620`, non-blank, formatted as an incident card.
- 2026-07-10 11:00 Telegram inbox classifier verifier:
  - Runtime script deployed: `C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1`.
  - Backup before deploy: `C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1.bak-20260710-110003`.
  - Parser readback: `parse_ok`.
  - Self-test evidence: `C:\Wooo\Agent99\evidence\agent99-TelegramInboxSelfTest-20260710-110004.json`.
  - Result: `ok=True`, `caseCount=8`, `passedCount=8`, `failedCount=0`, `wroteAlerts=False`, `sentTelegram=False`.
  - Covered inputs: Agent99 echo suppression, provider freshness, host performance readback failure, backup health, Bitan AI config missing key, IWOOOS/Wazuh/SIEM security, Gitea repository missing, and public 502.
  - Severity correction: `readback_failed`, `missing_key`, locale guard, and AI support config failures stay `warning`; true `critical`, `P0`, `emergency`, `fatal`, or `down` signals remain `critical`.
- 2026-07-10 11:08 bilingual Traditional Chinese card verifier:
  - Runtime script deployed with UTF-8 BOM to avoid Windows PowerShell 5 mojibake: `C:\Wooo\Agent99\bin\agent99-control-plane.ps1`.
  - Backup before deploy: `C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-110842`.
  - Parser readback: `parse_ok`.
  - Self-test evidence: `C:\Wooo\Agent99\evidence\agent99-TelegramCardSelfTest-20260710-110842.json`.
  - Visual card: `C:\Wooo\Agent99\evidence\telegram-cards\agent99-card-performance_warning-20260710-110842.png`.
  - Result: `ok=True`, `visualExists=True`, `sentTelegram=False`, `messageFormat=incident_card_zh_tw_v3`.
  - UTF-8 readback confirmed local preview lines include: `Agent99 事件卡 / Incident Card`, `嚴重度 Severity`, `事件 Event`, `原因 Reason`, `影響 Impact`, `Agent99 動作 Action`, `目前狀態 Current`, and `時間 Time`.
  - Local visual inspection: PNG is `1040x620`, non-blank, and Traditional Chinese text renders correctly.
- 2026-07-10 11:17 synthetic TG-to-SRE e2e verifier:
  - First run found a real bug: Telegram inbox wrote `controlledApply=True` for a `ProviderFreshness` alert.
  - Unsafe test queue was quarantined before ControlTick execution: `C:\Wooo\Agent99\queue\failed\quarantined-e2e-controlled-apply-bug-sre-alert-20260710-111547-face168a.json`.
  - Runtime fix deployed: `C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1`.
  - Backup before deploy: `C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1.bak-20260710-111651`.
  - Classifier self-test: `ok=True`, `caseCount=8`, `passedCount=8`, `failedCount=0`.
  - Post-fix synthetic e2e evidence:
    - Telegram inbox evidence: `C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260710-111728.json`.
    - SRE inbox evidence: `C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260710-111729.json`.
    - Queue result: `mode=ProviderFreshness`, `controlledApply=False`, `wroteTelegram=False`, `ranControlTick=False`.
    - Test queue quarantined after verification: `C:\Wooo\Agent99\queue\failed\quarantined-e2e-after-verify-sre-alert-20260710-111729-b2712df3.json`.
- 2026-07-10 11:25 Telegram relay photo delivery verifier:
  - Runtime script deployed with relay `sendPhoto` support: `C:\Wooo\Agent99\bin\agent99-control-plane.ps1`.
  - Backup before deploy: `C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-112511`.
  - Self-test evidence: `C:\Wooo\Agent99\evidence\agent99-TelegramDeliverySelfTest-20260710-112512.json`.
  - Result: `ok=True`, `sentTelegram=True`, `visualSent=True`, `relay.photoSent=True`, `relay.output=photo_sent_ok`, `messageFormat=incident_card_zh_tw_v3`.
  - Delivery mode: Windows 99 generated the PNG, copied it to host 110, copied it into relay container `stockplatform-v2-api-1`, and sent it through Telegram `sendPhoto`; text-only fallback was not used.
- 2026-07-10 12:01 sensor restore gate verifier:
  - Runtime script deployed: `C:\Wooo\Agent99\bin\agent99-control-plane.ps1`.
  - Backups before deploy:
    - `C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-114000`.
    - `C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-114519`.
    - `C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-114816`.
    - `C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-115331`.
    - `C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-115644`.
  - New verifier mode: `-SelfTestSensorGate`.
  - New controlled refresh switch: `-SuppressAlerts`.
  - SSH readback hardening:
    - Performance readback default timeout raised to `45s`.
    - Performance timeout gets slow retry with default `90s`.
    - Backup file and crontab metadata readback default timeout raised to `45s`.
    - SSH evidence now includes `elapsedMs`; performance rows include `readTimeoutSeconds`, `slowReadTimeoutSeconds`, and `readbackSlowRetry`.
  - False-negative findings fixed during rollout:
    - 110 performance readback was timing out at the previous `25s` boundary even though SSH succeeded.
    - Backup file metadata checks were timing out at the previous `15s` boundary.
    - Backup crontab readback was timing out at the previous `12s` boundary.
  - Suppressed refresh evidence:
    - Perf refresh: `C:\Wooo\Agent99\evidence\agent99-refresh-perf-20260710-115704.txt`, exit code `0`.
    - Backup refresh: `C:\Wooo\Agent99\evidence\agent99-refresh-backup-20260710-115754.txt`, exit code `0`.
    - Self refresh: `C:\Wooo\Agent99\evidence\agent99-refresh-self-20260710-115858.txt`, exit code `0`.
  - Final sensor gate evidence: `C:\Wooo\Agent99\evidence\agent99-SensorGateSelfTest-20260710-115927.json`.
  - Final gate result:
    - `ok=True`.
    - `readyToEnableAll=True`.
    - `sentTelegram=False`.
    - `controlledApply=False`.
    - `ranRemediation=False`.
    - `performanceGuard=eligible`.
    - `backupHealth=eligible`.
    - `selfHealth=eligible`.
  - Final host performance readback:
    - `192.168.0.110`: `ok`, `loadPerCore=0.44`, `memAvailPct=59.93`, `diskUsedPct=83`, `elapsedMs=25834`.
    - `192.168.0.112`: `ok`, `loadPerCore=0.01`, `memAvailPct=53.09`, `diskUsedPct=53`, `elapsedMs=345`.
    - `192.168.0.120`: `ok`, `loadPerCore=0.07`, `memAvailPct=90.01`, `diskUsedPct=47`, `elapsedMs=709`.
    - `192.168.0.121`: `ok`, `loadPerCore=0.02`, `memAvailPct=92.75`, `diskUsedPct=27`, `elapsedMs=1247`.
    - `192.168.0.188`: `ok`, `loadPerCore=0.26`, `memAvailPct=83.77`, `diskUsedPct=88`, `elapsedMs=974`.
  - Final backup readback:
    - `backupSeverity=ok`.
    - `targetCritical=0`, `targetWarning=0`.
    - `fileCritical=0`, `fileWarning=0`.
    - `cronCritical=0`, `cronWarning=0`.
    - Coverage: `13` snapshot targets, `7` file checks, `6` cron patterns.
  - Final self-health readback:
    - `selfSeverity=ok`.
    - `taskFailures=0`.
    - `evidenceCritical=0`, `evidenceWarning=0`.
    - `performanceSeverity=ok`, `telegramSeverity=ok`.
  - Restored schedules:
    - `Wooo-Agent99-Performance-Guard`: enabled, latest Perf evidence `C:\Wooo\Agent99\evidence\agent99-Perf-20260710-120117.json`, `latestPerfTelegramSent=0`, `latestPerfCritical=0`, `latestPerfWarning=0`.
    - `Wooo-Agent99-Backup-Health`: enabled.
    - `Wooo-Agent99-Self-Health`: enabled.
  - 2026-07-10 12:09 post-restore scheduled-cycle readback:
    - `Wooo-Agent99-Performance-Guard`: `Running`, last run `2026-07-10T12:09:16+08:00`, latest evidence `C:\Wooo\Agent99\evidence\agent99-Perf-20260710-120829.json`.
    - `Wooo-Agent99-Backup-Health`: `Ready`, last run `2026-07-10T12:07:17+08:00`, latest evidence `C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260710-120718.json`.
    - `Wooo-Agent99-Self-Health`: `Ready`, last run `2026-07-10T12:06:17+08:00`, latest evidence `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260710-120618.json`.
    - `Wooo-Agent99-SRE-Alert-Inbox`: `Ready`, last run `2026-07-10T12:09:17+08:00`, latest evidence `C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260710-120917.json`.
    - `Wooo-Agent99-Telegram-Inbox`: `Running`, last run `2026-07-10T12:09:17+08:00`, latest evidence `C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260710-120818.json`.
    - `Wooo-Agent99-Control-Loop`: `Ready`, last run `2026-07-10T12:09:17+08:00`, latest evidence `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260710-120918.json`.
    - Scheduled Perf result: `critical=0`, `warning=0`, `sent=0`.
    - Scheduled Backup result: `severity=ok`, `targetCritical=0`, `targetWarning=0`, `fileCritical=0`, `fileWarning=0`, `cronCritical=0`, `cronWarning=0`, `sent=0`.
    - Scheduled SelfHealth result: `severity=ok`, `taskFailures=0`, `evidenceCritical=0`, `evidenceWarning=0`, `performanceSeverity=ok`, `telegramSeverity=ok`, `sent=0`.
    - Latest host performance after scheduled cycle:
      - `192.168.0.110`: `ok`, `loadPerCore=0.48`, `diskUsedPct=83`, `elapsedMs=22915`.
      - `192.168.0.112`: `ok`, `loadPerCore=0.01`, `diskUsedPct=53`, `elapsedMs=339`.
      - `192.168.0.120`: `ok`, `loadPerCore=0.05`, `diskUsedPct=47`, `elapsedMs=699`.
      - `192.168.0.121`: `ok`, `loadPerCore=0.02`, `diskUsedPct=27`, `elapsedMs=1183`.
      - `192.168.0.188`: `ok`, `loadPerCore=0.09`, `diskUsedPct=88`, `elapsedMs=881`.
  - 2026-07-10 12:12 Telegram inbox idle readback verifier:
    - Runtime script deployed: `C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1`.
    - Backup before deploy: `C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1.bak-20260710-121231`.
    - Parser readback: `parse_ok`.
    - Fix: Telegram relay readback now records `timeoutSeconds`, `elapsedMs`, and `slowRetry`; empty update polling is `ok=True` with `reason=no_updates`, not a false failure.
    - Self-test run: `C:\Wooo\Agent99\evidence\agent99-telegram-inbox-selftest-run-20260710-121249.txt`, exit code `0`.
    - Real inbox run evidence: `C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260710-121252.json`.
    - Real inbox result: `ok=True`, `reason=no_updates`, `updatesSeen=0`, `processedCount=0`, `alertedCount=0`, `ignoredCount=0`, `sentTelegram=0`.
    - Relay readback: host `192.168.0.110`, container `stockplatform-v2-api-1`, `ok=True`, `exitCode=0`, `elapsedMs=2176`, `timeoutSeconds=60`, `slowRetry=False`.
    - Scheduled inbox readback: `Wooo-Agent99-Telegram-Inbox` last run `2026-07-10T12:14:17+08:00`, last task result `0`, latest evidence `C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260710-121418.json`.
    - Scheduled inbox result: `ok=True`, `reason=no_updates`, `updatesSeen=0`, `processedCount=0`, `alertedCount=0`, `sentTelegram=0`.
    - Scheduled relay readback: `ok=True`, `exitCode=0`, `elapsedMs=35062`, `timeoutSeconds=60`, `slowRetry=False`.
  - 2026-07-10 12:31 synthetic do-not-page E2E verifier:
    - Runtime scripts deployed:
      - `C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1`, backup `C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1.bak-20260710-122314`.
      - `C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1`, backups `C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1.bak-20260710-122315` and `C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1.bak-20260710-123108`.
      - `C:\Wooo\Agent99\bin\agent99-control-plane.ps1`, backups `C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-122315`, `C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-122800`, and `C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-123108`.
      - `C:\Wooo\Agent99\agent99-run.ps1`, backups `C:\Wooo\Agent99\agent99-run.ps1.bak-20260710-122315` and `C:\Wooo\Agent99\agent99-run.ps1.bak-20260710-122800`.
    - Parser readback: all deployed PowerShell scripts and launcher parsed OK before write.
    - Safety fix: `suppressTelegram` now uses an explicit bool parser so string `"false"` does not become truthy in Windows PowerShell.
    - Self-tests:
      - Telegram inbox classifier: `ok=True`, `caseCount=9`, `passedCount=9`, `failedCount=0`; added `synthetic provider freshness do not page`.
      - SRE alert routing: `C:\Wooo\Agent99\evidence\agent99-SreAlertRoutingSelfTest-20260710-123126.json`, `ok=True`, `caseCount=13`, `passedCount=13`, `failedCount=0`; added `provider freshness do not page`.
    - E2E alert id: `telegram-monitoring-1783686686`.
    - Telegram inbox evidence: `C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260710-123126.json`, `synthetic=True`, `alertedCount=1`, `suppressTelegram=True`.
    - SRE inbox evidence: `C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260710-123126.json`, `queuedCount=1`, `controlExitCode=0`, queue `suppressTelegram=True`.
    - ControlTick evidence: `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260710-123127.json`, `processedCount=1`, `pendingCount=0`, command `ok=True`, `exitCode=0`, operator result `sent=False`, `suppressed=True`, `reason=do_not_page`.
    - ProviderFreshness evidence: `C:\Wooo\Agent99\evidence\agent99-ProviderFreshness-20260710-123128.json`, `status=candidate_created_waiting_direct_readback`, candidate `provider-freshness-20260710-123129`.
    - ProviderFreshness Telegram evidence: both `agent_start` and `provider_freshness_triage` have `sent=False`, `suppressed=True`, `reason=do_not_page`.
    - Problem/KM writeback: `problemKey=agent99-sre-alert-inbox_unknown_provider_freshness_signal`, `occurrences=6`, `resolved=5`, `failed=1`, `lastStatus=resolved`, KM path `C:\Wooo\Agent99\playbooks\agent99-incident-km.md`.
  - 2026-07-10 12:42 SRE completion card v4 verifier:
    - Runtime script deployed: `C:\Wooo\Agent99\bin\agent99-control-plane.ps1`.
    - Backups before deploy: `C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-124012` and `C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-124137`.
    - Runtime bootstrap deployed: `C:\Wooo\Agent99\bin\agent99-bootstrap.ps1`, backup `C:\Wooo\Agent99\bin\agent99-bootstrap.ps1.bak-20260710-124425`.
    - Runtime config readback: `C:\Wooo\Agent99\config\agent99.config.json` has `telegram.alertSreCommandResults=True`; config backup `C:\Wooo\Agent99\config\agent99.config.json.bak-20260710-124351`.
    - New behavior: real `source=agent99-sre-alert-inbox` command results with `alertId` are allowed to send completion cards even when ordinary successful operator commands are suppressed.
    - Guard: `suppressTelegram=true` or `TEST_DO_NOT_PAGE` still suppresses the entire path with `reason=do_not_page`.
    - Telegram card self-test evidence: `C:\Wooo\Agent99\evidence\agent99-TelegramCardSelfTest-20260710-124202.json`.
    - Self-test result: `ok=True`, `messageFormat=incident_card_zh_tw_v4`, `operatorTextOk=True`, `operatorDecisionOk=True`, `operatorVisualExists=True`, `sentTelegram=False`.
    - Operator completion card image: `C:\Wooo\Agent99\evidence\telegram-cards\agent99-card-operator_command_result-20260710-124202.png`.
    - Do-not-page E2E alert id: `telegram-monitoring-1783687343`.
    - Do-not-page E2E evidence:
      - Telegram inbox: `C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260710-124224.json`, `alertedCount=1`, `suppressTelegram=True`.
      - SRE inbox: `C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260710-124224.json`, `queuedCount=1`, `controlExitCode=0`.
      - ControlTick: `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260710-124225.json`, `processedCount=1`, `pendingCount=0`, operator result `sent=False`, `suppressed=True`, `reason=do_not_page`.
      - ProviderFreshness: `C:\Wooo\Agent99\evidence\agent99-ProviderFreshness-20260710-124226.json`, `status=direct_readback_ok_no_source_events`, candidate `provider-freshness-20260710-124226`.
    - Problem/KM writeback after v4 E2E: `problemKey=agent99-sre-alert-inbox_unknown_provider_freshness_signal`, `occurrences=7`, `resolved=6`, `failed=1`, `lastStatus=resolved`.

## Agent99 Monitoring Alert Routing Self-Test Receipt

2026-07-10 10:17 Asia/Taipei:

- Runtime patch deployed on 99:
  - Script: `C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1`.
  - New verifier mode: `-SelfTest`.
  - Parser readback: `parse_ok`.
- Self-test evidence:
  - `C:\Wooo\Agent99\evidence\agent99-SreAlertRoutingSelfTest-20260710-101747.json`.
  - `ok=True`.
  - `caseCount=9`.
  - `passedCount=9`.
  - `failedCount=0`.
- Expanded self-test evidence:
  - `C:\Wooo\Agent99\evidence\agent99-SreAlertRoutingSelfTest-20260710-104455.json`.
  - `ok=True`.
  - `caseCount=12`.
  - `passedCount=12`.
  - `failedCount=0`.
  - Added routes: IWOOOS/Wazuh/SIEM security alert -> `SecurityTriage`; Gitea repository missing -> `Status`; Bitan AI support missing key / locale guard -> `Status`.
- Covered alert routes:
  - Public route 502 -> `Recover`, `controlledApply=True`.
  - Host down / reboot / VM not running -> `Recover`, `controlledApply=True`.
  - CPU / memory / disk / performance pressure -> `Perf`, `controlledApply=True`.
  - Backup freshness / snapshot / cron -> `BackupCheck`, `controlledApply=False`.
  - Harbor / registry -> `HarborRepair`, `controlledApply=True`.
  - K3s / pod / deployment / image pull failure -> `AwoooRepair`, `controlledApply=True`.
  - Provider freshness / last seen / ingestion / no-false-green -> `ProviderFreshness`, `controlledApply=False`.
  - Unknown warning -> `Status`, `controlledApply=False`.
  - Info heartbeat noise -> ignored.
- Defect found and fixed during self-test:
  - `backup freshness` was incorrectly routed to `ProviderFreshness` because the generic word `freshness` matched too early.
  - Fix: route backup/snapshot/cron before provider freshness and restrict provider freshness to provider-specific patterns such as `provider_freshness_signal`, `source_provider_freshness_triage`, `provider window`, `last_seen`, and ingestion.

## OpenClaw Cold-Start Readback

2026-07-09 17:38 Asia/Taipei:

- Production OpenClaw URL: `http://192.168.0.188:8088`.
- 188 service: `clawbot.service` is `active`.
- Health endpoint: `http://192.168.0.188:8088/health`.
- Health readback: `status=healthy`, `version=6.0`, `service=ClawBot`, `environment=production`, `telegram_bot=connected`.
- Agent99 latest status evidence: `C:\Wooo\Agent99\evidence\agent99-Status-20260709-173813.json`.
- Agent99 `aiServices.openclaw`: `ok=True`, `status=healthy`.
- Telegram relay from the Agent99 status run: `tgSent=2`, `relayOk=2`.
- Note: `192.168.0.188:8089` is not the current production OpenClaw endpoint.

## Windows Update No-Surprise-Reboot Policy

2026-07-09 17:39 Asia/Taipei:

- Evidence: `C:\Wooo\Agent99\evidence\agent99-windows-update-policy-20260709-173933.json`.
- `NoAutoRebootWithLoggedOnUsers=1`.
- `AlwaysAutoRebootAtScheduledTime=0`.
- `AUOptions=3`.
- `ActiveHoursStart=0`, `ActiveHoursEnd=23`.
- `gpupdateOk=True`.

## Agent99 Self-Health Receipt

2026-07-09 17:52 Asia/Taipei:

- Scheduled task: `Wooo-Agent99-Self-Health`.
- Trigger interval: every 5 minutes.
- Task Scheduler trigger verification:
  - `LastTaskResult=0`.
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-175218.json`.
  - `evidenceAfterTrigger=True`.
- Self-health checks:
  - `Wooo-Agent99-Startup-Recovery`: OK.
  - `Wooo-Agent99-Heartbeat`: OK.
  - `Wooo-Agent99-Performance-Guard`: OK.
  - `Wooo-Agent99-Self-Health`: OK.
  - Evidence freshness: performance, status, self-health, and startup recovery OK.
  - Latest performance evidence: all five hosts OK.
  - Telegram delivery: `tgSent=3`, `relayOk=3`.
  - Updated rule: a quiet interval with no alert attempts is OK; only failed alert attempts are Telegram critical.

## 110 Load Pressure Incident Readback

2026-07-09 18:07 Asia/Taipei:

- Symptom: 110 host load stayed above warning after all hosts rebooted.
- Initial findings:
  - `gitea` and `stockplatform-v2-postgres-1` were the highest regular service containers.
  - `/backup/scripts/backup-awoooi.sh` was running a remote `pg_dump` from 188 and local gzip compression.
  - A random container `mystifying_napier` was running `alpine "tar czf - /data"` with no restart policy and no labels, consuming CPU and writing compressed binary output into Docker logs.
- Controlled actions taken:
  - Backup pipeline PIDs were reduced to nice 15 / ionice best-effort 7.
  - `mystifying_napier` was stopped and left as `Dead`; it was not deleted, so evidence remains available.
- Agent99 automation added:
  - `stop-110-runaway-alpine-tar-data`: stops allowlisted `alpine` containers running `tar czf - /data` when 110 is at warning or critical load.
  - `renice-110-awoooi-backup-pressure`: lowers priority for `backup-awoooi.sh`, `gzip`, and remote `pg_dump` pipeline PIDs when 110 is at warning or critical load.
  - `pause-110-ci-runner-pressure`: still only runs at critical load with `load_per_core_critical`; runner is not touched for backup, disk, memory, or warning-only pressure.
  - Warning-level top CPU capture is enabled with a 10-second timeout to avoid diagnostics becoming a new load source.
- Verification:
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-Perf-20260709-180704.json`.
  - 110 load-per-core: `0.67`, severity `ok`.
  - 112, 120, 121, and 188: severity `ok`.
  - `perfCritical=0`, `perfWarning=0`.
  - Load-shedding actions skipped because host severity was below threshold.
  - Telegram relay: `tgSent=2`, `relayOk=2`.

## Agent99 Backup Health Receipt

2026-07-09 19:07 Asia/Taipei:

- Scheduled task: `Wooo-Agent99-Backup-Health`.
- Trigger interval: every 15 minutes.
- Task Scheduler trigger verification:
  - `LastTaskResult=0`.
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260709-190617.json`.
  - `evidenceAfterTrigger=True`.
- BackupCheck coverage:
  - Snapshot freshness targets: `awoooi`, `gitea`, `harbor`, `momo`, `langfuse`, `monitoring`, `signoz`, `clawbot`, `open-webui`, `sentry`, `ai-artifacts`, `configs`, and `public-routes`.
  - File/status checks: `backup.log`, `backup-status.log`, legacy `status.log`, legacy `cron.log`, integrity check status, restore drill status, and configs last-status JSON.
  - Cron contract patterns: `backup-all.sh`, `backup-awoooi-frequent.sh`, `backup-health-textfile-exporter.py`, `check-backup-integrity.sh`, `sync-offsite-backups.sh --mode status`, and `backup-status.sh`.
- Latest result:
  - `backupHealth.severity=ok`.
  - `targetCritical=0`, `targetWarning=0`.
  - `fileCritical=0`, `fileWarning=0`.
  - `cronCritical=0`, `cronWarning=0`.
  - `targetCount=13`, `fileCheckCount=7`, `cronPatternCount=6`.
  - Telegram relay: `tgSent=2`, `relayOk=2`.
- Self-health cross-check:
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-190714.json`.
  - `Wooo-Agent99-Backup-Health`: OK, `LastTaskResult=0`.
  - Backup evidence freshness: OK, latest file `agent99-BackupCheck-20260709-190617.json`, age `0.5` minutes, max age `30` minutes.

## Alert-to-Remediation / Evidence Sanitization Readback

2026-07-09 22:11 Asia/Taipei:

- Root cause fixed for unreadable Telegram relay messages:
  - Relay sender argument index corrected from `sys.argv[1]` to `sys.argv[2]`.
  - Telegram now receives formatted Agent99 alert text, not embedded Python relay code.
- Noise control:
  - `alertAllOperations=false`.
  - `alertInfoEvents=false`.
  - `dedupeMinutes=30`.
  - `agent_complete` remains evidence-only and does not alert by default.
- Alert-to-action rule:
  - `Perf -ControlledApply` suppresses immediate performance spam, performs allowlisted remediation first, then alerts `load_shed_result`.
  - If no remediation applies, Agent99 alerts the unresolved performance issue.
  - 188 disk pressure is now allowlisted through `prune-188-docker-build-cache-disk-pressure`.
  - The 188 action uses `timeoutSeconds=240` plus a post-apply verifier; `postVerifyMaxDiskUsedPercent=89`.
  - Timeout alone is not final truth; post-verifier success can mark remediation effective.
- Self-health false critical fixed:
  - A quiet interval with no recent alerts is OK.
  - Telegram health is critical only when recent alert attempts exist and all delivery attempts fail.
  - Telegram delivery health now uses a 60-minute lookback across recent evidence files.
  - Transient performance warnings are no longer promoted to self-health critical; only missing or critical host performance evidence is critical.
- Backup monitor false critical fixed:
  - Backup snapshot readback now supports `snapshotReadTimeoutSeconds`.
  - 99 live config sets `snapshotReadTimeoutSeconds=90`.
  - Harbor backup was verified as present and fresh after a timeout false critical; latest metadata was from 2026-07-09 17:48 Asia/Taipei and within the 1800-minute SLA.
- Evidence security fixed:
  - Harbor readback now stores selected health fields only: name, running state, health state.
  - Raw `docker inspect` output is not stored.
  - Existing Status evidence on 99 was sanitized: 77 files checked; `inspectOutputNotRedactedCount=0`; `unsafeHarborFieldCount=0`.
- Latest recovery proof:
  - Status evidence: `C:\Wooo\Agent99\evidence\agent99-Status-20260709-221147.json`.
  - Host reachability: 110, 112, 120, 121, and 188 all OK.
  - Public routes: `awoooi.wooo.work`, `stock.wooo.work`, `2026fifa.wooo.work`, `gitea.wooo.work`, and `harbor.wooo.work` all HTTP 200 / non-502.
  - Performance: `perfCritical=0`, `perfWarning=0`; 188 disk used 87%; 110 load-per-core 0.85.
  - Harbor: core, Redis, and registry healthy with redacted evidence.
  - AWOOOI: `deployReady=true`, `imagePullFailure=false`, `crashFailure=false`.
  - Backup evidence after timeout fix: `C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260709-221759.json`, `backupHealth.severity=ok`, all target/file/cron critical and warning counts are 0.
  - Final self-health evidence: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-222236.json`, `selfHealth.severity=ok`, `taskFailures=0`, `performanceSeverity=ok`, `telegramSeverity=ok`.

## 99 Visible Agent Control Center

2026-07-09 22:33 Asia/Taipei:

- Agent99 is visible and controllable from the 99 Windows desktop.
- New scheduled task:
  - `Wooo-Agent99-Control-Loop`.
  - Runs every 1 minute.
  - Updates dashboard and processes queue commands.
- Desktop shortcuts:
  - `C:\Users\Public\Desktop\Agent99 Control Center.lnk`.
  - `C:\Users\Public\Desktop\Agent99 Queue.lnk`.
  - `C:\Users\Public\Desktop\Agent99 Evidence.lnk`.
  - `D:\Desktop\Agent99 Control Center.lnk`.
  - `D:\Desktop\Agent99 Queue.lnk`.
  - `D:\Desktop\Agent99 Evidence.lnk`.
- Dashboard:
  - HTML: `C:\Wooo\Agent99\dashboard\index.html`.
  - JSON: `C:\Wooo\Agent99\dashboard\agent99-control-center.latest.json`.
  - Latest dashboard timestamp readback: `2026-07-09T22:33:26+08:00`.
  - Queue readback: `pending=0`, `processed=2`, `failed=0`.
- Queue command contract:
  - Add a JSON command to `C:\Wooo\Agent99\queue`.
  - `ControlTick` processes only allowlisted modes.
  - Results are written to `C:\Wooo\Agent99\queue\processed` or `C:\Wooo\Agent99\queue\failed`.
  - Operator command results are sent to Telegram.
- Verification:
  - Command ID: `codex-visible-publicsmoke-20260709-223308`.
  - Mode: `PublicSmoke`.
  - Result: `ok=true`, `exitCode=0`.
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-223309.json`.
  - Telegram: `operator_command_result`, `severity=info`, `sent=true`, `relayOk=true`.
  - Self-health: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-223340.json`, `selfHealth.severity=ok`, `Wooo-Agent99-Control-Loop=ok`.
  - Latest performance: `C:\Wooo\Agent99\evidence\agent99-Perf-20260709-223419.json`, `perfCritical=0`, `perfWarning=0`.

## Operator Instruction Intake

2026-07-09 22:41 Asia/Taipei:

- Agent99 accepts text instructions on 99 through:
  - `C:\Wooo\Agent99\bin\agent99-submit-request.ps1`.
  - `C:\Wooo\Agent99\agent99-submit-request.cmd`.
  - `Agent99 Submit Request.lnk` on the desktop.
- Flow:
  - instruction -> request record -> queue command -> `ControlTick` -> evidence -> Telegram result.
- Current allowlisted request router:
  - websites / routes / 502 -> `PublicSmoke`.
  - restore / reboot / repair -> `Recover`.
  - VM / VMware -> `StartVMs`.
  - CPU / load / memory / disk / performance -> `Perf`.
  - backup / snapshot / cron -> `BackupCheck`.
  - self health / agent health -> `SelfCheck`.
  - Harbor / registry -> `HarborRepair`.
  - AWOOOI / K3s / pods -> `AwoooRepair`.
- Verification:
  - Chinese instruction submitted from Codex to 99: check whether all websites are normal.
  - Request ID: `request-20260709-224133-0dc045fe`.
  - Resolved mode: `PublicSmoke`.
  - Processed result: `ok=true`, `exitCode=0`.
  - Evidence: `C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-224134.json`.
  - Public routes: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502.
  - Telegram: `operator_command_result`, `severity=info`, `sent=true`, `relayOk=true`.