wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity
Files
ca5ed596dc76dfae1d2b1412f4daec42e86aefa8
awoooi/HOST-REBOOT-AI-AUTOMATION-SOP.md
ogt 34e32f6ed6
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 1s
Details
CD Pipeline / cancel-stale-cd (push) Has been skipped
Details
CD Pipeline / tests (push) Successful in 2m27s
Details
CD Pipeline / build-and-deploy (push) Successful in 6m39s
Details
CD Pipeline / post-deploy-checks (push) Successful in 1m55s
Details
fix(agent99): dispatch actionable alerts with receipts
2026-07-11 10:30:29 +08:00

56 KiB
Raw Blame History

Host Reboot AI Automation SOP

Last updated: 2026-07-11

Canonical enterprise scope and current work order:

  • docs/workplans/2026-07-10-agent99-enterprise-ai-automation-master-plan.md
  • docs/operations/agent99-enterprise-ai-automation-work-items.snapshot.json

This SOP remains the detailed reboot procedure. It cannot mark recovery complete from command success, route HTTP 200, or a Telegram delivery receipt. Completion requires the reboot state machine, all required host/service/product/data gates, mode-specific verifiers and AWOOOI/AwoooP callback readback.

P0 Goal

After host 99 or any VM host is rebooted, the automation must detect the reboot, restore VM reachability, restore critical services, and prove public sites are no longer returning 502.

Target recovery bar:

  • Detect 99 / 110 / 112 / 120 / 121 / 188 reboot or down state.
  • Ensure VMware Workstation on 99 is running and required VMs are powered on.
  • Restore Harbor / registry on 110 before forcing K3s workloads to repull images.
  • Restore AWOOOI production pods on 120.
  • Verify public HTTP readback for AWOOOI, StockPlatform, FIFA, Gitea, and Harbor.
  • Emit operator-visible evidence and Telegram alerts for down, recovering, recovered, and degraded states.

2026-07-09 Incident Findings To Automate

The reboot incident was not a generic 502 problem. The confirmed chain was:

  1. 110 rebooted and Docker/Harbor did not fully recover.
  2. harbor-core entered a failed/restarting state because Harbor Redis / Docker DNS dependency was not clean.
  3. K3s on 120 could not pull AWOOOI images from 192.168.0.110:5000.
  4. AWOOOI prod pods stayed in ImagePullBackOff.
  5. Public nginx returned 502 because upstream pods were unavailable.
  6. Restarting / rebooting 110 cleared the Docker / systemd / Harbor dependency state.
  7. Once harbor-core became healthy, 120 pods repulled images and AWOOOI recovered.

This failure mode must be treated as a first-class automated recovery path.

Recovery Priority Order

P0. Control-plane and VM reachability

  • 99 Windows host reachable.
  • VMware Workstation process running on 99.
  • VMs 110, 112, 120, 121, and 188 powered on.
  • Hosts 110, 112, 120, 121, and 188 respond to ICMP or SSH.

P1. 110 Harbor and registry dependency chain

  • Docker engine active.
  • Harbor Redis running.
  • harbor-core running and healthy.
  • Harbor registry / registryctl running and healthy.
  • harbor.wooo.work returns HTTP 200 after redirect.
  • Internal registry pull path from K3s nodes is reachable.

P2. 120 K3s production workloads

  • awoooi-prod deployments ready:
    • awoooi-api 1/1
    • awoooi-web 2/2
    • awoooi-worker 1/1
    • awoooi-auto-repair-canary 1/1
  • No critical pods stuck in ImagePullBackOff, ErrImagePull, CrashLoopBackOff, or Pending.
  • If image pull failures remain after Harbor is healthy, delete only the failed pods and let controllers recreate them.

P3. Public user-visible surfaces

  • awoooi.wooo.work returns HTTP 200 for web.
  • awoooi.wooo.work/api/v1/health returns JSON and not 502.
  • stock.wooo.work returns HTTP 200.
  • 2026fifa.wooo.work returns HTTP 200.
  • gitea.wooo.work returns HTTP 200.
  • harbor.wooo.work returns HTTP 200.

P4. Degraded provider follow-up

  • AWOOOI health can be degraded even when 502 is fixed.
  • Treat AI provider health separately from user-visible 502 restoration.
  • Follow-up checks include ollama_gcp_a, ollama_gcp_b, and ollama_local.
  • Do not block public-site 502 recovery on non-primary AI provider cooldown unless the app is user-impacting.

Automated Decision Tree

  1. Detect reboot / outage.

    • Ping 110, 112, 120, 121, 188.
    • SSH to 110 and 120 if keys are available.
    • Query VMware running VM list on 99 if vmrun.exe is available.

  2. If a VM is not reachable.

    • Check VMware VM state.
    • Start the VM from 99.
    • Wait for guest ping / SSH.
    • Alert Telegram: host_recovering.

  3. If AWOOOI public route is 502.

    • Check 120 awoooi-prod deployments.
    • If pods are ImagePullBackOff, check 110 Harbor first.
    • Do not repeatedly delete pods before Harbor is healthy.

  4. If 110 Harbor is unhealthy.

    • Check Docker active state.
    • Check redis, harbor-core, registry, and registryctl.
    • Restart Harbor Redis and harbor-core first.
    • If Docker/systemd/cgroup operations hang, use controlled 110 guest reboot through VMware.
    • After 110 returns, wait for Harbor core healthy before touching K3s pods.

  5. If Harbor is healthy and pods are still in image pull failure.

    • Delete only failed AWOOOI pods in awoooi-prod.
    • Wait for deployments to become ready.
    • Read back public HTTP.

  6. If public sites are HTTP 200 but AWOOOI health is degraded.

    • Record degraded providers.
    • Alert service_degraded.
    • Continue provider-specific remediation separately.

99 Windows AI Agent Control Node

99 must become the automation entrypoint for reboot recovery.

Required capabilities:

  • Run a local AI Agent service on Windows 99 with least-privilege operator permissions.
  • Execute the reboot scorecard on a schedule and on boot.
  • Control VMware Workstation VM startup through vmrun.exe.
  • Use SSH keys to 110 and 120 for service-level checks and controlled repair.
  • Send Telegram alerts for down, recovering, recovered, degraded, failed, and controlled-remediation-result states.
  • Do not stop at alerting. Every warning/critical alert must either trigger an allowlisted remediation, record that no remediation matched, or record that the action was skipped by cooldown/min-severity.
  • Do not send start/complete/info noise to Telegram by default. Human-facing alerts must be deduped and action-oriented.
  • Persist scorecard evidence to a local non-secret log directory.
  • Never store raw passwords, tokens, cookies, sessions, or .env in the repo.
  • Accept operator commands through the 99 desktop submitter, Telegram /agent99 ..., and local queue files without depending on Codex quota.
  • Accept SRE war-room alert JSON at C:\Wooo\Agent99\alerts\incoming, resolve each alert into an allowlisted mode, and either run controlled remediation or record why no remediation matched.

Candidate implementation layers:

  • P0: Windows Task Scheduler runs host-reboot-scorecard.ps1 at startup and every minute for the first 15 minutes after 99 boot.
  • P1: Install a local Windows service wrapper for the scorecard loop.
  • P2: Add an AI Agent runtime that consumes scorecard JSON and chooses approved recovery actions.
  • P3: Add a UI/console dashboard on 99 for operator evidence and manual override.

Current command and alert loops:

  • Wooo-Agent99-Control-Loop: processes C:\Wooo\Agent99\queue, updates the dashboard, and writes execution evidence.
  • Wooo-Agent99-Telegram-Inbox: reads Telegram group commands with Agent99 prefixes and submits allowlisted queue work.
  • Wooo-Agent99-SRE-Alert-Inbox: reads C:\Wooo\Agent99\alerts\incoming, converts actionable alerts into queue work, and triggers the control loop.
  • Wooo-Agent99-SRE-Alert-Relay: runs the 99-side HTTP relay for sanitized AWOOI Alertmanager JSON and writes accepted alerts into C:\Wooo\Agent99\alerts\incoming.
  • Wooo-Agent99-Performance-Guard: runs every minute and performs allowlisted load shedding before escalating unresolved performance alerts.
  • Wooo-Agent99-Self-Health: verifies the scheduled tasks and evidence freshness for the control plane itself.

AwoooI SRE Telegram command loop:

  • Operators and project AI Agents can issue commands in the SRE Telegram group with /agent99 ....
  • Agent99 records these as source=telegram-sre-war-room.
  • Failed or escalated automated repairs send operator_command_result back to Telegram.
  • Successful automated repairs are evidence-first: they write queue, evidence, problem count, KM, and RAG receipts, then suppress success-only Telegram noise.
  • Failure results include source, failure instruction, mode, exit code, evidence path, and problem count.
  • The command path must not depend on Codex quota.
  • The command path must not depend only on the default TELEGRAM_CHAT_ID, because that can point to a private chat. Agent99 also accepts command updates from Telegram chats whose title matches AwoooI SRE|AWOOOI SRE|SRE.
  • When the command comes from a matched Telegram group, the completion report is sent back to that same Telegram origin; stored evidence redacts the raw chat id as telegram-origin.
  • Non-command monitoring alerts in the SRE Telegram group are auto-ingested when they look like Alertmanager/FIRING/critical/warning/502/CPU/disk/Harbor/K3s/backup/host-down/provider-freshness alerts.
  • Auto-ingested monitoring alerts use source=telegram-sre-war-room-monitoring and enter the same SRE alert inbox / queue / recovery / Telegram completion / KM-RAG path.

Alert-to-action routing:

  • Public 502 / route down / website down -> Recover.
  • Host down / reboot detected / VM not running -> Recover.
  • CPU, memory, disk, or load pressure -> Perf with controlled apply.
  • Harbor or registry unhealthy -> HarborRepair.
  • AWOOOI K3s, pod, deployment, or image pull failure -> AwoooRepair.
  • Backup freshness, snapshot, restore drill, or cron issue -> BackupCheck.
  • Provider freshness, last-seen, ingestion, raw signal, or no-false-green risk -> ProviderFreshness.
  • Unknown actionable alert -> Status evidence first.

Producer-side Agent99 bridge:

  • AWOOI API Alertmanager webhook schedules an Agent99 bridge after normalization and fingerprinting.
  • Preferred transport: when AGENT99_SRE_ALERT_RELAY_URL is configured, the bridge POSTs sanitized alert JSON to the 99 relay. Production URL is http://192.168.0.99:8787/agent99/sre-alert/.
  • Fallback transport: when AGENT99_SRE_ALERT_INBOX_PATH is configured and relay URL is empty, the bridge writes sanitized alert JSON directly to the Agent99 SRE inbox contract.
  • If relay/path is empty or unavailable, the bridge fails open and logs; it must not cause Alertmanager retries.
  • Relay safety: only LAN/local source prefixes are accepted by default; optional AGENT99_SRE_RELAY_TOKEN / AGENT99_SRE_ALERT_RELAY_TOKEN values stay in env/Secret and are never stored in repo or evidence.
  • Provider freshness bridge output must run direct readback, then require provider window, last seen, direct reference, and verifier readback before dashboard green is trusted.
  • Provider freshness no-false-green rule: route 200 or dashboard up is not enough. The Agent99 verifier must read https://awoooi.wooo.work/api/v1/platform/events/dossier/recurrence?project_id=awoooi&limit=20 and record providerWindowMinutes, lastSeen, directReference, and verifierReadback.

2026-07-11 P0-002 source receipt, runtime deployment pending:

  • ACTION REQUIRED cards for cold-start-gate now enter the same Agent99 ingress as Alertmanager and route to Recover with the reboot/cold-start scorecard verifier.
  • Failed AWOOOI CI/CD cards enter Agent99 as cicd_failure in Status mode. Agent99 records evidence but waits for the next successful Gitea CD/deploy/runtime receipt before source-event closure.
  • The producer stores a sanitized relay acceptance receipt in the outbound source envelope before Telegram delivery; Telegram remains a human lifecycle surface, not the only queue or database.
  • Cold-start candidates use the Agent99 execution route and never synthesize kubectl rollout restart deployment/cold-start-gate.
  • A relay receipt with accepted=false releases the fingerprint single-flight lock and returns failed. This prevents a transport-level response from permanently suppressing an unprocessed incident.
  • Verification completed in source: 249 focused post-rebase tests and 262 expanded all-Telegram plus Agent99/RepairCandidate tests passed after integrating the Telegram single-owner mainline; the repository static governance suite also passed. This does not prove Windows 99 is running this revision; runtime manifest and exact do-not-page replay are still mandatory.

Approved controlled actions for the 99 agent:

  • Start a missing VM.
  • Ping and SSH readback.
  • Restart Harbor Redis / Harbor core after evidence of Harbor token failure.
  • Delete K3s pods stuck in image pull failure only after Harbor is healthy.
  • Run public route readback.
  • Send Telegram status.

Actions still requiring explicit break-glass:

  • Reading or exfiltrating secrets.
  • Destructive database operations.
  • Firewall cutover.
  • Remote delete / prune / force push.
  • Production provider route cutover with cost or billing impact.

Evidence Required Before Calling Recovery Complete

Recovery is complete only when all of the following are captured:

  • Five VM hosts have reachability evidence.
  • 110 Harbor core is healthy.
  • 120 AWOOOI deployments are ready.
  • Public routes return non-502, preferably HTTP 200.
  • Any degraded health components are listed separately with owner / next action.

Current 2026-07-09 Receipts

  • 110 harbor-core: recovered to Up ... (healthy).
  • 120 awoooi-prod:
    • awoooi-api: 1/1
    • awoooi-web: 2/2
    • awoooi-worker: 1/1
    • awoooi-auto-repair-canary: 1/1
  • Public readback:
    • awoooi.wooo.work: HTTP 200 after redirect.
    • stock.wooo.work: HTTP 200 after redirect.
    • 2026fifa.wooo.work: HTTP 200 after redirect.
    • gitea.wooo.work: HTTP 200 after redirect.
    • harbor.wooo.work: HTTP 200 after redirect.
  • Host reachability:
    • 110, 112, 120, 121, 188 responded to ping or SSH.
  • Remaining degraded signal:
    • AWOOOI health reported degraded AI provider signals including ollama_gcp_b timeout and ollama_local connection refused / unavailable.

Agent99 Installation Receipt

2026-07-09 16:49 Asia/Taipei:

  • 99 host: WOOO-SUPER.
  • Agent root: C:\Wooo\Agent99.
  • VMware CLI: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe.
  • Live config: C:\Wooo\Agent99\config\agent99.config.json.
  • Evidence directory: C:\Wooo\Agent99\evidence.
  • Scheduled tasks:
    • Wooo-Agent99-Heartbeat: LastTaskResult=0, latest status evidence agent99-Status-20260709-164845.json.
    • Wooo-Agent99-Startup-Recovery: LastTaskResult=0, latest recovery evidence agent99-Recover-20260709-164937.json.
  • Startup recovery readback:
    • VM selector: 110, 188, 112, 120, 121 all host_already_reachable.
    • Harbor: core, Redis, and registry all healthy.
    • AWOOOI: deploy ready, no image pull failure, no crash failure.
    • Public routes: awoooi.wooo.work, stock.wooo.work, 2026fifa.wooo.work, gitea.wooo.work, and harbor.wooo.work all HTTP 200 / non-502.

Agent99 Performance Guard Receipt

2026-07-09 17:15 Asia/Taipei:

  • Scheduled task: Wooo-Agent99-Performance-Guard.
  • Trigger interval: every 1 minute.
  • Last task result: 0.
  • Latest evidence: C:\Wooo\Agent99\evidence\agent99-Perf-20260709-171502.json.
  • Host performance readback:
    • 110: OK.
    • 112: OK.
    • 120: OK.
    • 121: OK.
    • 188: OK.
  • Controlled load shedding:
    • Only allowlisted actions can run.
    • Current 110 runner action requires load_per_core_critical.
    • Memory/disk pressure alerts do not execute the runner pause action.
  • Telegram audit:
    • Agent99 emits Telegram attempts for warnings, critical events, failed or escalated operator commands, and load-shed actions.
    • Successful operator commands are not Telegram noise; they are recorded in evidence, problem counts, KM, and RAG.
    • Agent99 supports local env aliases and an existing-token relay.
    • Current relay: 110 stockplatform-v2-api-1, using that container's existing TELEGRAM_BOT_TOKEN / TELEGRAM_CHAT_ID.
    • 2026-07-09 17:27 verification: Wooo-Agent99-Performance-Guard LastTaskResult=0; latest evidence agent99-Perf-20260709-172712.json; tgSent=2; relayOk=True.
    • Token and chat id values were not printed, copied into repo files, or written to evidence.

Agent99 Command And SRE Alert Integration Receipt

2026-07-09 23:03 Asia/Taipei:

  • Telegram command inbox is deployed and scheduled:
    • Script: C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1
    • Task: Wooo-Agent99-Telegram-Inbox
    • Latest evidence: C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260709-230031.json
    • Result: ok=True, updatesSeen=0, processedCount=0
    • Evidence check: no Python code, no token text.
  • SRE war-room alert inbox is deployed and scheduled:
    • Script: C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1
    • Task: Wooo-Agent99-SRE-Alert-Inbox
    • Incoming path: C:\Wooo\Agent99\alerts\incoming
    • Latest evidence: C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260709-230226.json
  • Synthetic 502 alert test:
    • Alert id: synthetic-public-502-20260709-230107
    • Alert source: awoooi-sre-war-room
    • Alert kind: public_route_502
    • Resolved mode: Recover
    • Queue processor evidence: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-230108.json
    • Processed count: 1
    • Mode: Recover
    • Exit code: 0
    • Telegram relay: sent=True, relayOk=True
  • Recovery verifier from the synthetic alert:
    • Evidence: C:\Wooo\Agent99\evidence\agent99-Recover-20260709-230108.json
    • Public route failures: 0
    • Host reachability failures: 0
    • Harbor core healthy: True
    • AWOOOI deploy ready: True
  • Agent99 self-health after integration:
    • Evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-230223.json
    • Severity: ok
    • Task failures: 0
    • Evidence critical: 0
    • Evidence warning: 0
    • Telegram inbox task: OK
    • SRE alert inbox task: OK
    • Dashboard queue pending: 0
    • Dashboard alert pending: 0

Agent99 Problem Count / KM / RAG Receipt

2026-07-09 23:12 Asia/Taipei:

  • Synthetic AwoooI SRE Telegram command:
    • instruction: 檢查所有網站是否恢復正常，若502就自動修復
    • source: telegram-sre-war-room
    • request id: request-20260709-231157-d53f53d8
    • resolved mode: Recover
    • control exit code: 0
  • Execution evidence:
    • C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-231158.json
    • processed source: telegram-sre-war-room
    • processed mode: Recover
    • processed ok: True
    • processed exit code: 0
    • Telegram relay sent: 1
    • Telegram evidence contains no Python code.
  • Problem counters:
    • path: C:\Wooo\Agent99\state\problem-counts.json
    • total events: 1
    • total resolved: 1
    • total failed: 0
    • problem key: telegram-sre-war-room_agent99_operator_instruction
    • occurrences: 1
    • resolved: 1
    • failed: 0
  • Knowledge outputs:
    • full event ledger: C:\Wooo\Agent99\playbooks\agent99-incident-records.jsonl
    • KM summary: C:\Wooo\Agent99\playbooks\agent99-incident-km.md
    • RAG import feed: C:\Wooo\Agent99\playbooks\rag-import\agent99-incidents.jsonl
  • Dashboard:
    • problem events: 1
    • resolved: 1
    • failed: 0
    • queue pending: 0
    • alert pending: 0
  • SRE Telegram group routing:
    • Default relay chat readback was private chat OoO, not the SRE group.
    • Live Agent99 config now includes allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE.
    • Telegram inbox readback after the change: ok=True, updatesSeen=0, processedCount=0.
    • Next real /agent99 ... message from the SRE group will be accepted by title match and will reply back to the same Telegram origin.

Agent99 SRE Telegram Monitoring Alert Auto-Ingest Receipt

2026-07-09 23:32 Asia/Taipei:

  • Runtime behavior:
    • SRE Telegram group monitoring alerts are now consumed automatically.
    • Prefix /agent99 is no longer required for monitoring alerts.
    • Auto-ingest is controlled by autoIngestMonitoringAlerts=True.
    • Group matching uses allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE.
  • Loop prevention:
    • Agent99's own Agent99 Alert / operator_command_result messages are ignored by monitoring auto-ingest.
    • Successful operator command results are suppressed from Telegram and kept in evidence/KM/RAG; failed or escalated command results remain alertable.
    • Provider freshness signals are not passive dashboard health; they create a read-only freshness candidate, run direct AWOOI readback, and require provider window, last seen, direct reference, and verifier readback.
    • Provider freshness guardrail: do not switch provider, call paid model, edit env, or reload.
  • Synthetic monitoring alert:
    • Alertmanager FIRING critical public_route_502 service=awoooi host=192.168.0.120 HTTP 502
    • no /agent99 prefix.
  • Evidence chain:
    • Telegram inbox: C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260709-233230.json
      • inboxOk=True
      • alertedCount=1
    • SRE alert inbox: C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260709-233231.json
      • queued count: 1
      • failed count: 0
    • Control loop: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-233232.json
      • alert source: telegram-sre-war-room-monitoring
      • alert kind: public_route_502
      • mode: Recover
      • processed ok: True
      • exit code: 0
      • Telegram attempts: 1
      • Telegram sent: 1
    • Recovery: C:\Wooo\Agent99\evidence\agent99-Recover-20260709-233055.json
  • Problem counters:
    • total events before: 2
    • total events after: 3
    • total resolved after: 3
    • total failed after: 0
    • problem key: agent99-sre-alert-inbox_awoooi_192.168.0.120_public_route_502
    • problem occurrences: 2

Agent99 Alert Noise And Provider Freshness Readback Receipt

2026-07-10 10:00 Asia/Taipei:

  • Runtime patch deployed on 99:
    • Script: C:\Wooo\Agent99\bin\agent99-control-plane.ps1.
    • Parser readback: parse_ok.
    • Source commit: 0a7a0115 fix(agent99): reduce alert noise and verify freshness.
  • Provider freshness behavior:
    • Evidence: C:\Wooo\Agent99\evidence\agent99-ProviderFreshness-20260710-100028.json.
    • Status: direct_readback_ok_no_source_events.
    • Missing fields after automatic readback: lastSeen only.
    • providerWindowMinutes=120.
    • directReference=https://awoooi.wooo.work/api/v1/platform/events/dossier/recurrence?project_id=awoooi&limit=20.
    • verifierReadback.ok=True, httpStatus=200, sourceEventTotal=0.
  • Telegram noise control:
    • Evidence log: C:\Wooo\Agent99\evidence\agent99-ProviderFreshness-20260710-100028.log.
    • Same provider-freshness state was suppressed: telegram_dedupe_skip type=provider_freshness_triage severity=warning ageMinutes=5.83 cooldown=30.
    • Evidence log: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260710-100027.log.
    • Successful operator command was evidence-only: telegram_operator_success_suppressed type=operator_command_result mode=ProviderFreshness id=operator-suppress-smoke-20260710-100026.
    • Control evidence: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260710-100027.json, processedCount=1, pendingCount=0, telegram=[].
  • Scheduled entrypoints verified on 99:
    • Wooo-Agent99-Control-Loop.
    • Wooo-Agent99-SRE-Alert-Inbox.
    • Wooo-Agent99-Telegram-Inbox.
    • Wooo-Agent99-SRE-Alert-Relay is running.
  • Operational rule:
    • Do not treat repeated success receipts as user-facing alerts.
    • Send user-facing Telegram only for warnings, criticals, unresolved blockers, failed/escalated operator commands, and controlled-remediation actions.
    • Keep successful automation in evidence, problem counters, KM, and RAG so repeated incidents trend down without flooding the SRE group.

Agent99 Telegram Alert UX And Noise Freeze Work Items

2026-07-10 10:41 Asia/Taipei:

  • Operator requirement:
    • AwoooI SRE Telegram alerts must be human-readable, professionally formatted, and action-oriented.
    • Important incident updates should include visual context when available: dashboard image, public-route screenshot, service card, or evidence summary image.
    • Do not send raw machine fields, empty metrics, repeated success receipts, or text-only alert floods.
  • Immediate runtime noise freeze applied on 99:
    • Disabled scheduled task: Wooo-Agent99-Performance-Guard.
    • Disabled scheduled task: Wooo-Agent99-Backup-Health.
    • Disabled scheduled task: Wooo-Agent99-Self-Health.
    • Ended running task instance: Wooo-Agent99-Performance-Guard.
    • Kept alert intake active: Wooo-Agent99-SRE-Alert-Relay, Wooo-Agent99-SRE-Alert-Inbox, Wooo-Agent99-Telegram-Inbox, and Wooo-Agent99-Control-Loop.
  • 2026-07-10 10:47 runtime readback:
    • Wooo-Agent99-Performance-Guard: Disabled, last result 0.
    • Wooo-Agent99-Backup-Health: Disabled, last result 0.
    • Wooo-Agent99-Self-Health: Disabled, last result 0.
    • Wooo-Agent99-SRE-Alert-Relay: Running.
    • Wooo-Agent99-SRE-Alert-Inbox: Ready, last result 0.
    • Wooo-Agent99-Telegram-Inbox: Running.
    • Wooo-Agent99-Control-Loop: Ready, last result 0.
    • SecurityTriage smoke evidence: C:\Wooo\Agent99\evidence\agent99-SecurityTriage-20260710-104456.json.
    • SecurityTriage smoke result: security_triage_no_recent_alert_body_seen, hitCount=0, telegramCount=0.
  • P0 work items added:
    • P0-0: keep noisy sensor schedules paused until their alerts have dedupe, clear reason text, and verifier-backed fields.
    • P0-1: replace raw Agent99 text alerts with professional Telegram incident cards: title, severity, affected service, user impact, Agent99 action, verifier result, evidence path, repeated count, next action.
    • P0-2: add visual attachment support for high-impact alerts: public URL screenshots, dashboard screenshots, and rendered evidence summary images.
    • P0-3: suppress alerts with empty metric fields such as blank load/core, memory, disk, age, or maxAge unless a verifier explains why readback failed.
    • P0-4: route application config alerts such as Bitan E_AI_MISSING_KEY to evidence-first Status; do not read or print secret values.
    • P0-5: route IWOOOS/Wazuh/SIEM alerts to SecurityTriage; collect source log reference, affected host/service, blast radius, repeated count, and allowlisted next action without arbitrary shell or secret exposure.
    • P0-6: route Gitea repository missing alerts to evidence-first Status until a dedicated Gitea recovery playbook is verified.
    • P0-7: run live TG-to-Agent99 end-to-end tests for each alert family before re-enabling the paused sensor schedules.
    • P0-8: all Agent99 Telegram alert messages must include Traditional Chinese user-facing labels and human-readable context; professional terms such as readback_failed, SecurityTriage, directReadback, KM/RAG, and provider names may remain English when translation would reduce precision.
    • P0-9: Telegram inbox must not force controlledApply=true on every monitoring alert. controlledApply is resolved by SRE mode: remediation modes may run controlled apply, evidence-first modes such as ProviderFreshness, BackupCheck, SecurityTriage, and Status must remain non-apply unless explicitly allowlisted.
    • P0-10: Telegram delivery must support actual image/photo delivery through the relay path, not only text plus a local evidence image.
    • P0-11: paused Agent99 sensors must pass a no-TG/no-remediation sensor restore gate before schedules are re-enabled. The gate must prove Performance, BackupHealth, and SelfHealth are eligible and must write readback evidence with sentTelegram=false, controlledApply=false, and ranRemediation=false.
    • P0-12: slow SSH readback after reboot must not be collapsed into "host down". Performance and backup metadata checks must record elapsedMs, use configurable timeouts, and run a slow retry on timeout before raising perf_readback_failed or backup readback failure.
    • P0-13: first-run evidence refresh after a noise freeze must use controlled -SuppressAlerts for Perf/Backup/SelfCheck so stale evidence is refreshed without Telegram noise; schedules are then re-enabled without suppression.
    • P0-14: synthetic or TEST_DO_NOT_PAGE monitoring tests must still run the full Telegram -> SRE inbox -> ControlTick -> mode evidence path, but suppressTelegram=true and SuppressReason=do_not_page must propagate end to end so validation never pages the SRE Telegram group.
    • P0-15: real SRE alert completion must send a human-readable Traditional Chinese automation result card back to Telegram with action taken, result, evidence, problem count, and KM path. Ordinary successful operator commands remain suppressed unless explicitly configured.
  • 2026-07-10 10:54 visual card verifier:
    • Runtime script deployed: C:\Wooo\Agent99\bin\agent99-control-plane.ps1.
    • Backup before deploy: C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-105439.
    • Parser readback: parse_ok.
    • Self-test evidence: C:\Wooo\Agent99\evidence\agent99-TelegramCardSelfTest-20260710-105440.json.
    • Visual card: C:\Wooo\Agent99\evidence\telegram-cards\agent99-card-performance_warning-20260710-105440.png.
    • Result: ok=True, visualExists=True, sentTelegram=False, messageFormat=incident_card_v2.
    • Local visual inspection: PNG is 1040x620, non-blank, formatted as an incident card.
  • 2026-07-10 11:00 Telegram inbox classifier verifier:
    • Runtime script deployed: C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1.
    • Backup before deploy: C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1.bak-20260710-110003.
    • Parser readback: parse_ok.
    • Self-test evidence: C:\Wooo\Agent99\evidence\agent99-TelegramInboxSelfTest-20260710-110004.json.
    • Result: ok=True, caseCount=8, passedCount=8, failedCount=0, wroteAlerts=False, sentTelegram=False.
    • Covered inputs: Agent99 echo suppression, provider freshness, host performance readback failure, backup health, Bitan AI config missing key, IWOOOS/Wazuh/SIEM security, Gitea repository missing, and public 502.
    • Severity correction: readback_failed, missing_key, locale guard, and AI support config failures stay warning; true critical, P0, emergency, fatal, or down signals remain critical.
  • 2026-07-10 11:08 bilingual Traditional Chinese card verifier:
    • Runtime script deployed with UTF-8 BOM to avoid Windows PowerShell 5 mojibake: C:\Wooo\Agent99\bin\agent99-control-plane.ps1.
    • Backup before deploy: C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-110842.
    • Parser readback: parse_ok.
    • Self-test evidence: C:\Wooo\Agent99\evidence\agent99-TelegramCardSelfTest-20260710-110842.json.
    • Visual card: C:\Wooo\Agent99\evidence\telegram-cards\agent99-card-performance_warning-20260710-110842.png.
    • Result: ok=True, visualExists=True, sentTelegram=False, messageFormat=incident_card_zh_tw_v3.
    • UTF-8 readback confirmed local preview lines include: Agent99 事件卡 / Incident Card, 嚴重度 Severity, 事件 Event, 原因 Reason, 影響 Impact, Agent99 動作 Action, 目前狀態 Current, and 時間 Time.
    • Local visual inspection: PNG is 1040x620, non-blank, and Traditional Chinese text renders correctly.
  • 2026-07-10 11:17 synthetic TG-to-SRE e2e verifier:
    • First run found a real bug: Telegram inbox wrote controlledApply=True for a ProviderFreshness alert.
    • Unsafe test queue was quarantined before ControlTick execution: C:\Wooo\Agent99\queue\failed\quarantined-e2e-controlled-apply-bug-sre-alert-20260710-111547-face168a.json.
    • Runtime fix deployed: C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1.
    • Backup before deploy: C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1.bak-20260710-111651.
    • Classifier self-test: ok=True, caseCount=8, passedCount=8, failedCount=0.
    • Post-fix synthetic e2e evidence:
      • Telegram inbox evidence: C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260710-111728.json.
      • SRE inbox evidence: C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260710-111729.json.
      • Queue result: mode=ProviderFreshness, controlledApply=False, wroteTelegram=False, ranControlTick=False.
      • Test queue quarantined after verification: C:\Wooo\Agent99\queue\failed\quarantined-e2e-after-verify-sre-alert-20260710-111729-b2712df3.json.
  • 2026-07-10 11:25 Telegram relay photo delivery verifier:
    • Runtime script deployed with relay sendPhoto support: C:\Wooo\Agent99\bin\agent99-control-plane.ps1.
    • Backup before deploy: C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-112511.
    • Self-test evidence: C:\Wooo\Agent99\evidence\agent99-TelegramDeliverySelfTest-20260710-112512.json.
    • Result: ok=True, sentTelegram=True, visualSent=True, relay.photoSent=True, relay.output=photo_sent_ok, messageFormat=incident_card_zh_tw_v3.
    • Delivery mode: Windows 99 generated the PNG, copied it to host 110, copied it into relay container stockplatform-v2-api-1, and sent it through Telegram sendPhoto; text-only fallback was not used.
  • 2026-07-10 12:01 sensor restore gate verifier:
    • Runtime script deployed: C:\Wooo\Agent99\bin\agent99-control-plane.ps1.
    • Backups before deploy:
      • C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-114000.
      • C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-114519.
      • C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-114816.
      • C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-115331.
      • C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-115644.
    • New verifier mode: -SelfTestSensorGate.
    • New controlled refresh switch: -SuppressAlerts.
    • SSH readback hardening:
      • Performance readback default timeout raised to 45s.
      • Performance timeout gets slow retry with default 90s.
      • Backup file and crontab metadata readback default timeout raised to 45s.
      • SSH evidence now includes elapsedMs; performance rows include readTimeoutSeconds, slowReadTimeoutSeconds, and readbackSlowRetry.
    • False-negative findings fixed during rollout:
      • 110 performance readback was timing out at the previous 25s boundary even though SSH succeeded.
      • Backup file metadata checks were timing out at the previous 15s boundary.
      • Backup crontab readback was timing out at the previous 12s boundary.
    • Suppressed refresh evidence:
      • Perf refresh: C:\Wooo\Agent99\evidence\agent99-refresh-perf-20260710-115704.txt, exit code 0.
      • Backup refresh: C:\Wooo\Agent99\evidence\agent99-refresh-backup-20260710-115754.txt, exit code 0.
      • Self refresh: C:\Wooo\Agent99\evidence\agent99-refresh-self-20260710-115858.txt, exit code 0.
    • Final sensor gate evidence: C:\Wooo\Agent99\evidence\agent99-SensorGateSelfTest-20260710-115927.json.
    • Final gate result:
      • ok=True.
      • readyToEnableAll=True.
      • sentTelegram=False.
      • controlledApply=False.
      • ranRemediation=False.
      • performanceGuard=eligible.
      • backupHealth=eligible.
      • selfHealth=eligible.
    • Final host performance readback:
      • 192.168.0.110: ok, loadPerCore=0.44, memAvailPct=59.93, diskUsedPct=83, elapsedMs=25834.
      • 192.168.0.112: ok, loadPerCore=0.01, memAvailPct=53.09, diskUsedPct=53, elapsedMs=345.
      • 192.168.0.120: ok, loadPerCore=0.07, memAvailPct=90.01, diskUsedPct=47, elapsedMs=709.
      • 192.168.0.121: ok, loadPerCore=0.02, memAvailPct=92.75, diskUsedPct=27, elapsedMs=1247.
      • 192.168.0.188: ok, loadPerCore=0.26, memAvailPct=83.77, diskUsedPct=88, elapsedMs=974.
    • Final backup readback:
      • backupSeverity=ok.
      • targetCritical=0, targetWarning=0.
      • fileCritical=0, fileWarning=0.
      • cronCritical=0, cronWarning=0.
      • Coverage: 13 snapshot targets, 7 file checks, 6 cron patterns.
    • Final self-health readback:
      • selfSeverity=ok.
      • taskFailures=0.
      • evidenceCritical=0, evidenceWarning=0.
      • performanceSeverity=ok, telegramSeverity=ok.
    • Restored schedules:
      • Wooo-Agent99-Performance-Guard: enabled, latest Perf evidence C:\Wooo\Agent99\evidence\agent99-Perf-20260710-120117.json, latestPerfTelegramSent=0, latestPerfCritical=0, latestPerfWarning=0.
      • Wooo-Agent99-Backup-Health: enabled.
      • Wooo-Agent99-Self-Health: enabled.
    • 2026-07-10 12:09 post-restore scheduled-cycle readback:
      • Wooo-Agent99-Performance-Guard: Running, last run 2026-07-10T12:09:16+08:00, latest evidence C:\Wooo\Agent99\evidence\agent99-Perf-20260710-120829.json.
      • Wooo-Agent99-Backup-Health: Ready, last run 2026-07-10T12:07:17+08:00, latest evidence C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260710-120718.json.
      • Wooo-Agent99-Self-Health: Ready, last run 2026-07-10T12:06:17+08:00, latest evidence C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260710-120618.json.
      • Wooo-Agent99-SRE-Alert-Inbox: Ready, last run 2026-07-10T12:09:17+08:00, latest evidence C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260710-120917.json.
      • Wooo-Agent99-Telegram-Inbox: Running, last run 2026-07-10T12:09:17+08:00, latest evidence C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260710-120818.json.
      • Wooo-Agent99-Control-Loop: Ready, last run 2026-07-10T12:09:17+08:00, latest evidence C:\Wooo\Agent99\evidence\agent99-ControlTick-20260710-120918.json.
      • Scheduled Perf result: critical=0, warning=0, sent=0.
      • Scheduled Backup result: severity=ok, targetCritical=0, targetWarning=0, fileCritical=0, fileWarning=0, cronCritical=0, cronWarning=0, sent=0.
      • Scheduled SelfHealth result: severity=ok, taskFailures=0, evidenceCritical=0, evidenceWarning=0, performanceSeverity=ok, telegramSeverity=ok, sent=0.
      • Latest host performance after scheduled cycle:
        • 192.168.0.110: ok, loadPerCore=0.48, diskUsedPct=83, elapsedMs=22915.
        • 192.168.0.112: ok, loadPerCore=0.01, diskUsedPct=53, elapsedMs=339.
        • 192.168.0.120: ok, loadPerCore=0.05, diskUsedPct=47, elapsedMs=699.
        • 192.168.0.121: ok, loadPerCore=0.02, diskUsedPct=27, elapsedMs=1183.
        • 192.168.0.188: ok, loadPerCore=0.09, diskUsedPct=88, elapsedMs=881.
    • 2026-07-10 12:12 Telegram inbox idle readback verifier:
      • Runtime script deployed: C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1.
      • Backup before deploy: C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1.bak-20260710-121231.
      • Parser readback: parse_ok.
      • Fix: Telegram relay readback now records timeoutSeconds, elapsedMs, and slowRetry; empty update polling is ok=True with reason=no_updates, not a false failure.
      • Self-test run: C:\Wooo\Agent99\evidence\agent99-telegram-inbox-selftest-run-20260710-121249.txt, exit code 0.
      • Real inbox run evidence: C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260710-121252.json.
      • Real inbox result: ok=True, reason=no_updates, updatesSeen=0, processedCount=0, alertedCount=0, ignoredCount=0, sentTelegram=0.
      • Relay readback: host 192.168.0.110, container stockplatform-v2-api-1, ok=True, exitCode=0, elapsedMs=2176, timeoutSeconds=60, slowRetry=False.
      • Scheduled inbox readback: Wooo-Agent99-Telegram-Inbox last run 2026-07-10T12:14:17+08:00, last task result 0, latest evidence C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260710-121418.json.
      • Scheduled inbox result: ok=True, reason=no_updates, updatesSeen=0, processedCount=0, alertedCount=0, sentTelegram=0.
      • Scheduled relay readback: ok=True, exitCode=0, elapsedMs=35062, timeoutSeconds=60, slowRetry=False.
    • 2026-07-10 12:31 synthetic do-not-page E2E verifier:
      • Runtime scripts deployed:
        • C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1, backup C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1.bak-20260710-122314.
        • C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1, backups C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1.bak-20260710-122315 and C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1.bak-20260710-123108.
        • C:\Wooo\Agent99\bin\agent99-control-plane.ps1, backups C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-122315, C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-122800, and C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-123108.
        • C:\Wooo\Agent99\agent99-run.ps1, backups C:\Wooo\Agent99\agent99-run.ps1.bak-20260710-122315 and C:\Wooo\Agent99\agent99-run.ps1.bak-20260710-122800.
      • Parser readback: all deployed PowerShell scripts and launcher parsed OK before write.
      • Safety fix: suppressTelegram now uses an explicit bool parser so string "false" does not become truthy in Windows PowerShell.
      • Self-tests:
        • Telegram inbox classifier: ok=True, caseCount=9, passedCount=9, failedCount=0; added synthetic provider freshness do not page.
        • SRE alert routing: C:\Wooo\Agent99\evidence\agent99-SreAlertRoutingSelfTest-20260710-123126.json, ok=True, caseCount=13, passedCount=13, failedCount=0; added provider freshness do not page.
      • E2E alert id: telegram-monitoring-1783686686.
      • Telegram inbox evidence: C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260710-123126.json, synthetic=True, alertedCount=1, suppressTelegram=True.
      • SRE inbox evidence: C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260710-123126.json, queuedCount=1, controlExitCode=0, queue suppressTelegram=True.
      • ControlTick evidence: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260710-123127.json, processedCount=1, pendingCount=0, command ok=True, exitCode=0, operator result sent=False, suppressed=True, reason=do_not_page.
      • ProviderFreshness evidence: C:\Wooo\Agent99\evidence\agent99-ProviderFreshness-20260710-123128.json, status=candidate_created_waiting_direct_readback, candidate provider-freshness-20260710-123129.
      • ProviderFreshness Telegram evidence: both agent_start and provider_freshness_triage have sent=False, suppressed=True, reason=do_not_page.
      • Problem/KM writeback: problemKey=agent99-sre-alert-inbox_unknown_provider_freshness_signal, occurrences=6, resolved=5, failed=1, lastStatus=resolved, KM path C:\Wooo\Agent99\playbooks\agent99-incident-km.md.
    • 2026-07-10 12:42 SRE completion card v4 verifier:
      • Runtime script deployed: C:\Wooo\Agent99\bin\agent99-control-plane.ps1.
      • Backups before deploy: C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-124012 and C:\Wooo\Agent99\bin\agent99-control-plane.ps1.bak-20260710-124137.
      • Runtime bootstrap deployed: C:\Wooo\Agent99\bin\agent99-bootstrap.ps1, backup C:\Wooo\Agent99\bin\agent99-bootstrap.ps1.bak-20260710-124425.
      • Runtime config readback: C:\Wooo\Agent99\config\agent99.config.json has telegram.alertSreCommandResults=True; config backup C:\Wooo\Agent99\config\agent99.config.json.bak-20260710-124351.
      • New behavior: real source=agent99-sre-alert-inbox command results with alertId are allowed to send completion cards even when ordinary successful operator commands are suppressed.
      • Guard: suppressTelegram=true or TEST_DO_NOT_PAGE still suppresses the entire path with reason=do_not_page.
      • Telegram card self-test evidence: C:\Wooo\Agent99\evidence\agent99-TelegramCardSelfTest-20260710-124202.json.
      • Self-test result: ok=True, messageFormat=incident_card_zh_tw_v4, operatorTextOk=True, operatorDecisionOk=True, operatorVisualExists=True, sentTelegram=False.
      • Operator completion card image: C:\Wooo\Agent99\evidence\telegram-cards\agent99-card-operator_command_result-20260710-124202.png.
      • Do-not-page E2E alert id: telegram-monitoring-1783687343.
      • Do-not-page E2E evidence:
        • Telegram inbox: C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260710-124224.json, alertedCount=1, suppressTelegram=True.
        • SRE inbox: C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260710-124224.json, queuedCount=1, controlExitCode=0.
        • ControlTick: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260710-124225.json, processedCount=1, pendingCount=0, operator result sent=False, suppressed=True, reason=do_not_page.
        • ProviderFreshness: C:\Wooo\Agent99\evidence\agent99-ProviderFreshness-20260710-124226.json, status=direct_readback_ok_no_source_events, candidate provider-freshness-20260710-124226.
      • Problem/KM writeback after v4 E2E: problemKey=agent99-sre-alert-inbox_unknown_provider_freshness_signal, occurrences=7, resolved=6, failed=1, lastStatus=resolved.

Agent99 Monitoring Alert Routing Self-Test Receipt

2026-07-10 10:17 Asia/Taipei:

  • Runtime patch deployed on 99:
    • Script: C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1.
    • New verifier mode: -SelfTest.
    • Parser readback: parse_ok.
  • Self-test evidence:
    • C:\Wooo\Agent99\evidence\agent99-SreAlertRoutingSelfTest-20260710-101747.json.
    • ok=True.
    • caseCount=9.
    • passedCount=9.
    • failedCount=0.
  • Expanded self-test evidence:
    • C:\Wooo\Agent99\evidence\agent99-SreAlertRoutingSelfTest-20260710-104455.json.
    • ok=True.
    • caseCount=12.
    • passedCount=12.
    • failedCount=0.
    • Added routes: IWOOOS/Wazuh/SIEM security alert -> SecurityTriage; Gitea repository missing -> Status; Bitan AI support missing key / locale guard -> Status.
  • Covered alert routes:
    • Public route 502 -> Recover, controlledApply=True.
    • Host down / reboot / VM not running -> Recover, controlledApply=True.
    • CPU / memory / disk / performance pressure -> Perf, controlledApply=True.
    • Backup freshness / snapshot / cron -> BackupCheck, controlledApply=False.
    • Harbor / registry -> HarborRepair, controlledApply=True.
    • K3s / pod / deployment / image pull failure -> AwoooRepair, controlledApply=True.
    • Provider freshness / last seen / ingestion / no-false-green -> ProviderFreshness, controlledApply=False.
    • Unknown warning -> Status, controlledApply=False.
    • Info heartbeat noise -> ignored.
  • Defect found and fixed during self-test:
    • backup freshness was incorrectly routed to ProviderFreshness because the generic word freshness matched too early.
    • Fix: route backup/snapshot/cron before provider freshness and restrict provider freshness to provider-specific patterns such as provider_freshness_signal, source_provider_freshness_triage, provider window, last_seen, and ingestion.

OpenClaw Cold-Start Readback

2026-07-09 17:38 Asia/Taipei:

  • Production OpenClaw URL: http://192.168.0.188:8088.
  • 188 service: clawbot.service is active.
  • Health endpoint: http://192.168.0.188:8088/health.
  • Health readback: status=healthy, version=6.0, service=ClawBot, environment=production, telegram_bot=connected.
  • Agent99 latest status evidence: C:\Wooo\Agent99\evidence\agent99-Status-20260709-173813.json.
  • Agent99 aiServices.openclaw: ok=True, status=healthy.
  • Telegram relay from the Agent99 status run: tgSent=2, relayOk=2.
  • Note: 192.168.0.188:8089 is not the current production OpenClaw endpoint.

Windows Update No-Surprise-Reboot Policy

2026-07-09 17:39 Asia/Taipei:

  • Evidence: C:\Wooo\Agent99\evidence\agent99-windows-update-policy-20260709-173933.json.
  • NoAutoRebootWithLoggedOnUsers=1.
  • AlwaysAutoRebootAtScheduledTime=0.
  • AUOptions=3.
  • ActiveHoursStart=0, ActiveHoursEnd=23.
  • gpupdateOk=True.

Agent99 Self-Health Receipt

2026-07-09 17:52 Asia/Taipei:

  • Scheduled task: Wooo-Agent99-Self-Health.
  • Trigger interval: every 5 minutes.
  • Task Scheduler trigger verification:
    • LastTaskResult=0.
    • Evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-175218.json.
    • evidenceAfterTrigger=True.
  • Self-health checks:
    • Wooo-Agent99-Startup-Recovery: OK.
    • Wooo-Agent99-Heartbeat: OK.
    • Wooo-Agent99-Performance-Guard: OK.
    • Wooo-Agent99-Self-Health: OK.
    • Evidence freshness: performance, status, self-health, and startup recovery OK.
    • Latest performance evidence: all five hosts OK.
    • Telegram delivery: tgSent=3, relayOk=3.
    • Updated rule: a quiet interval with no alert attempts is OK; only failed alert attempts are Telegram critical.

110 Load Pressure Incident Readback

2026-07-09 18:07 Asia/Taipei:

  • Symptom: 110 host load stayed above warning after all hosts rebooted.
  • Initial findings:
    • gitea and stockplatform-v2-postgres-1 were the highest regular service containers.
    • /backup/scripts/backup-awoooi.sh was running a remote pg_dump from 188 and local gzip compression.
    • A random container mystifying_napier was running alpine "tar czf - /data" with no restart policy and no labels, consuming CPU and writing compressed binary output into Docker logs.
  • Controlled actions taken:
    • Backup pipeline PIDs were reduced to nice 15 / ionice best-effort 7.
    • mystifying_napier was stopped and left as Dead; it was not deleted, so evidence remains available.
  • Agent99 automation added:
    • stop-110-runaway-alpine-tar-data: stops allowlisted alpine containers running tar czf - /data when 110 is at warning or critical load.
    • renice-110-awoooi-backup-pressure: lowers priority for backup-awoooi.sh, gzip, and remote pg_dump pipeline PIDs when 110 is at warning or critical load.
    • pause-110-ci-runner-pressure: still only runs at critical load with load_per_core_critical; runner is not touched for backup, disk, memory, or warning-only pressure.
    • Warning-level top CPU capture is enabled with a 10-second timeout to avoid diagnostics becoming a new load source.
  • Verification:
    • Evidence: C:\Wooo\Agent99\evidence\agent99-Perf-20260709-180704.json.
    • 110 load-per-core: 0.67, severity ok.
    • 112, 120, 121, and 188: severity ok.
    • perfCritical=0, perfWarning=0.
    • Load-shedding actions skipped because host severity was below threshold.
    • Telegram relay: tgSent=2, relayOk=2.

Agent99 Backup Health Receipt

2026-07-09 19:07 Asia/Taipei:

  • Scheduled task: Wooo-Agent99-Backup-Health.
  • Trigger interval: every 15 minutes.
  • Task Scheduler trigger verification:
    • LastTaskResult=0.
    • Evidence: C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260709-190617.json.
    • evidenceAfterTrigger=True.
  • BackupCheck coverage:
    • Snapshot freshness targets: awoooi, gitea, harbor, momo, langfuse, monitoring, signoz, clawbot, open-webui, sentry, ai-artifacts, configs, and public-routes.
    • File/status checks: backup.log, backup-status.log, legacy status.log, legacy cron.log, integrity check status, restore drill status, and configs last-status JSON.
    • Cron contract patterns: backup-all.sh, backup-awoooi-frequent.sh, backup-health-textfile-exporter.py, check-backup-integrity.sh, sync-offsite-backups.sh --mode status, and backup-status.sh.
  • Latest result:
    • backupHealth.severity=ok.
    • targetCritical=0, targetWarning=0.
    • fileCritical=0, fileWarning=0.
    • cronCritical=0, cronWarning=0.
    • targetCount=13, fileCheckCount=7, cronPatternCount=6.
    • Telegram relay: tgSent=2, relayOk=2.
  • Self-health cross-check:
    • Evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-190714.json.
    • Wooo-Agent99-Backup-Health: OK, LastTaskResult=0.
    • Backup evidence freshness: OK, latest file agent99-BackupCheck-20260709-190617.json, age 0.5 minutes, max age 30 minutes.

Alert-to-Remediation / Evidence Sanitization Readback

2026-07-09 22:11 Asia/Taipei:

  • Root cause fixed for unreadable Telegram relay messages:
    • Relay sender argument index corrected from sys.argv[1] to sys.argv[2].
    • Telegram now receives formatted Agent99 alert text, not embedded Python relay code.
  • Noise control:
    • alertAllOperations=false.
    • alertInfoEvents=false.
    • dedupeMinutes=30.
    • agent_complete remains evidence-only and does not alert by default.
  • Alert-to-action rule:
    • Perf -ControlledApply suppresses immediate performance spam, performs allowlisted remediation first, then alerts load_shed_result.
    • If no remediation applies, Agent99 alerts the unresolved performance issue.
    • 188 disk pressure is now allowlisted through prune-188-docker-build-cache-disk-pressure.
    • The 188 action uses timeoutSeconds=240 plus a post-apply verifier; postVerifyMaxDiskUsedPercent=89.
    • Timeout alone is not final truth; post-verifier success can mark remediation effective.
  • Self-health false critical fixed:
    • A quiet interval with no recent alerts is OK.
    • Telegram health is critical only when recent alert attempts exist and all delivery attempts fail.
    • Telegram delivery health now uses a 60-minute lookback across recent evidence files.
    • Transient performance warnings are no longer promoted to self-health critical; only missing or critical host performance evidence is critical.
  • Backup monitor false critical fixed:
    • Backup snapshot readback now supports snapshotReadTimeoutSeconds.
    • 99 live config sets snapshotReadTimeoutSeconds=90.
    • Harbor backup was verified as present and fresh after a timeout false critical; latest metadata was from 2026-07-09 17:48 Asia/Taipei and within the 1800-minute SLA.
  • Evidence security fixed:
    • Harbor readback now stores selected health fields only: name, running state, health state.
    • Raw docker inspect output is not stored.
    • Existing Status evidence on 99 was sanitized: 77 files checked; inspectOutputNotRedactedCount=0; unsafeHarborFieldCount=0.
  • Latest recovery proof:
    • Status evidence: C:\Wooo\Agent99\evidence\agent99-Status-20260709-221147.json.
    • Host reachability: 110, 112, 120, 121, and 188 all OK.
    • Public routes: awoooi.wooo.work, stock.wooo.work, 2026fifa.wooo.work, gitea.wooo.work, and harbor.wooo.work all HTTP 200 / non-502.
    • Performance: perfCritical=0, perfWarning=0; 188 disk used 87%; 110 load-per-core 0.85.
    • Harbor: core, Redis, and registry healthy with redacted evidence.
    • AWOOOI: deployReady=true, imagePullFailure=false, crashFailure=false.
    • Backup evidence after timeout fix: C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260709-221759.json, backupHealth.severity=ok, all target/file/cron critical and warning counts are 0.
    • Final self-health evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-222236.json, selfHealth.severity=ok, taskFailures=0, performanceSeverity=ok, telegramSeverity=ok.

99 Visible Agent Control Center

2026-07-09 22:33 Asia/Taipei:

  • Agent99 is visible and controllable from the 99 Windows desktop.
  • New scheduled task:
    • Wooo-Agent99-Control-Loop.
    • Runs every 1 minute.
    • Updates dashboard and processes queue commands.
  • Desktop shortcuts:
    • C:\Users\Public\Desktop\Agent99 Control Center.lnk.
    • C:\Users\Public\Desktop\Agent99 Queue.lnk.
    • C:\Users\Public\Desktop\Agent99 Evidence.lnk.
    • D:\Desktop\Agent99 Control Center.lnk.
    • D:\Desktop\Agent99 Queue.lnk.
    • D:\Desktop\Agent99 Evidence.lnk.
  • Dashboard:
    • HTML: C:\Wooo\Agent99\dashboard\index.html.
    • JSON: C:\Wooo\Agent99\dashboard\agent99-control-center.latest.json.
    • Latest dashboard timestamp readback: 2026-07-09T22:33:26+08:00.
    • Queue readback: pending=0, processed=2, failed=0.
  • Queue command contract:
    • Add a JSON command to C:\Wooo\Agent99\queue.
    • ControlTick processes only allowlisted modes.
    • Results are written to C:\Wooo\Agent99\queue\processed or C:\Wooo\Agent99\queue\failed.
    • Operator command results are sent to Telegram.
  • Verification:
    • Command ID: codex-visible-publicsmoke-20260709-223308.
    • Mode: PublicSmoke.
    • Result: ok=true, exitCode=0.
    • Evidence: C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-223309.json.
    • Telegram: operator_command_result, severity=info, sent=true, relayOk=true.
    • Self-health: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-223340.json, selfHealth.severity=ok, Wooo-Agent99-Control-Loop=ok.
    • Latest performance: C:\Wooo\Agent99\evidence\agent99-Perf-20260709-223419.json, perfCritical=0, perfWarning=0.

Operator Instruction Intake

2026-07-09 22:41 Asia/Taipei:

  • Agent99 accepts text instructions on 99 through:
    • C:\Wooo\Agent99\bin\agent99-submit-request.ps1.
    • C:\Wooo\Agent99\agent99-submit-request.cmd.
    • Agent99 Submit Request.lnk on the desktop.
  • Flow:
    • instruction -> request record -> queue command -> ControlTick -> evidence -> Telegram result.
  • Current allowlisted request router:
    • websites / routes / 502 -> PublicSmoke.
    • restore / reboot / repair -> Recover.
    • VM / VMware -> StartVMs.
    • CPU / load / memory / disk / performance -> Perf.
    • backup / snapshot / cron -> BackupCheck.
    • self health / agent health -> SelfCheck.
    • Harbor / registry -> HarborRepair.
    • AWOOOI / K3s / pods -> AwoooRepair.
  • Verification:
    • Chinese instruction submitted from Codex to 99: check whether all websites are normal.
    • Request ID: request-20260709-224133-0dc045fe.
    • Resolved mode: PublicSmoke.
    • Processed result: ok=true, exitCode=0.
    • Evidence: C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-224134.json.
    • Public routes: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502.
    • Telegram: operator_command_result, severity=info, sent=true, relayOk=true.
Reference in New Issue View Git Blame Copy Permalink