wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity
Files
ea6787d368d7008d403e8a51389a4e7f7e9ebfcc
awoooi/HOST-REBOOT-AI-AUTOMATION-SOP.md

32 KiB
Raw Blame History

Host Reboot AI Automation SOP

Last updated: 2026-07-09

P0 Goal

After host 99 or any VM host is rebooted, the automation must detect the reboot, restore VM reachability, restore critical services, and prove public sites are no longer returning 502.

Target recovery bar:

  • Detect 99 / 110 / 112 / 120 / 121 / 188 reboot or down state.
  • Ensure VMware Workstation on 99 is running and required VMs are powered on.
  • Restore Harbor / registry on 110 before forcing K3s workloads to repull images.
  • Restore AWOOOI production pods on 120.
  • Verify public HTTP readback for AWOOOI, StockPlatform, FIFA, Gitea, and Harbor.
  • Emit operator-visible evidence and Telegram alerts for down, recovering, recovered, and degraded states.

2026-07-09 Incident Findings To Automate

The reboot incident was not a generic 502 problem. The confirmed chain was:

  1. 110 rebooted and Docker/Harbor did not fully recover.
  2. harbor-core entered a failed/restarting state because Harbor Redis / Docker DNS dependency was not clean.
  3. K3s on 120 could not pull AWOOOI images from 192.168.0.110:5000.
  4. AWOOOI prod pods stayed in ImagePullBackOff.
  5. Public nginx returned 502 because upstream pods were unavailable.
  6. Restarting / rebooting 110 cleared the Docker / systemd / Harbor dependency state.
  7. Once harbor-core became healthy, 120 pods repulled images and AWOOOI recovered.

This failure mode must be treated as a first-class automated recovery path.

Recovery Priority Order

P0. Control-plane and VM reachability

  • 99 Windows host reachable.
  • VMware Workstation process running on 99.
  • VMs 110, 112, 120, 121, and 188 powered on.
  • Hosts 110, 112, 120, 121, and 188 respond to ICMP or SSH.

P1. 110 Harbor and registry dependency chain

  • Docker engine active.
  • Harbor Redis running.
  • harbor-core running and healthy.
  • Harbor registry / registryctl running and healthy.
  • harbor.wooo.work returns HTTP 200 after redirect.
  • Internal registry pull path from K3s nodes is reachable.

P2. 120 K3s production workloads

  • awoooi-prod deployments ready:
    • awoooi-api 1/1
    • awoooi-web 2/2
    • awoooi-worker 1/1
    • awoooi-auto-repair-canary 1/1
  • No critical pods stuck in ImagePullBackOff, ErrImagePull, CrashLoopBackOff, or Pending.
  • If image pull failures remain after Harbor is healthy, delete only the failed pods and let controllers recreate them.

P3. Public user-visible surfaces

  • awoooi.wooo.work returns HTTP 200 for web.
  • awoooi.wooo.work/api/v1/health returns JSON and not 502.
  • stock.wooo.work returns HTTP 200.
  • 2026fifa.wooo.work returns HTTP 200.
  • gitea.wooo.work returns HTTP 200.
  • harbor.wooo.work returns HTTP 200.

P4. Degraded provider follow-up

  • AWOOOI health can be degraded even when 502 is fixed.
  • Treat AI provider health separately from user-visible 502 restoration.
  • Follow-up checks include ollama_gcp_a, ollama_gcp_b, and ollama_local.
  • Do not block public-site 502 recovery on non-primary AI provider cooldown unless the app is user-impacting.

Automated Decision Tree

  1. Detect reboot / outage.

    • Ping 110, 112, 120, 121, 188.
    • SSH to 110 and 120 if keys are available.
    • Query VMware running VM list on 99 if vmrun.exe is available.

  2. If a VM is not reachable.

    • Check VMware VM state.
    • Start the VM from 99.
    • Wait for guest ping / SSH.
    • Alert Telegram: host_recovering.

  3. If AWOOOI public route is 502.

    • Check 120 awoooi-prod deployments.
    • If pods are ImagePullBackOff, check 110 Harbor first.
    • Do not repeatedly delete pods before Harbor is healthy.

  4. If 110 Harbor is unhealthy.

    • Check Docker active state.
    • Check redis, harbor-core, registry, and registryctl.
    • Restart Harbor Redis and harbor-core first.
    • If Docker/systemd/cgroup operations hang, use controlled 110 guest reboot through VMware.
    • After 110 returns, wait for Harbor core healthy before touching K3s pods.

  5. If Harbor is healthy and pods are still in image pull failure.

    • Delete only failed AWOOOI pods in awoooi-prod.
    • Wait for deployments to become ready.
    • Read back public HTTP.

  6. If public sites are HTTP 200 but AWOOOI health is degraded.

    • Record degraded providers.
    • Alert service_degraded.
    • Continue provider-specific remediation separately.

99 Windows AI Agent Control Node

99 must become the automation entrypoint for reboot recovery.

Required capabilities:

  • Run a local AI Agent service on Windows 99 with least-privilege operator permissions.
  • Execute the reboot scorecard on a schedule and on boot.
  • Control VMware Workstation VM startup through vmrun.exe.
  • Use SSH keys to 110 and 120 for service-level checks and controlled repair.
  • Send Telegram alerts for down, recovering, recovered, degraded, failed, and controlled-remediation-result states.
  • Do not stop at alerting. Every warning/critical alert must either trigger an allowlisted remediation, record that no remediation matched, or record that the action was skipped by cooldown/min-severity.
  • Do not send start/complete/info noise to Telegram by default. Human-facing alerts must be deduped and action-oriented.
  • Persist scorecard evidence to a local non-secret log directory.
  • Never store raw passwords, tokens, cookies, sessions, or .env in the repo.
  • Accept operator commands through the 99 desktop submitter, Telegram /agent99 ..., and local queue files without depending on Codex quota.
  • Accept SRE war-room alert JSON at C:\Wooo\Agent99\alerts\incoming, resolve each alert into an allowlisted mode, and either run controlled remediation or record why no remediation matched.

Candidate implementation layers:

  • P0: Windows Task Scheduler runs host-reboot-scorecard.ps1 at startup and every minute for the first 15 minutes after 99 boot.
  • P1: Install a local Windows service wrapper for the scorecard loop.
  • P2: Add an AI Agent runtime that consumes scorecard JSON and chooses approved recovery actions.
  • P3: Add a UI/console dashboard on 99 for operator evidence and manual override.

Current command and alert loops:

  • Wooo-Agent99-Control-Loop: processes C:\Wooo\Agent99\queue, updates the dashboard, and writes execution evidence.
  • Wooo-Agent99-Telegram-Inbox: reads Telegram group commands with Agent99 prefixes and submits allowlisted queue work.
  • Wooo-Agent99-SRE-Alert-Inbox: reads C:\Wooo\Agent99\alerts\incoming, converts actionable alerts into queue work, and triggers the control loop.
  • Wooo-Agent99-SRE-Alert-Relay: runs the 99-side HTTP relay for sanitized AWOOI Alertmanager JSON and writes accepted alerts into C:\Wooo\Agent99\alerts\incoming.
  • Wooo-Agent99-Performance-Guard: runs every minute and performs allowlisted load shedding before escalating unresolved performance alerts.
  • Wooo-Agent99-Self-Health: verifies the scheduled tasks and evidence freshness for the control plane itself.

AwoooI SRE Telegram command loop:

  • Operators and project AI Agents can issue commands in the SRE Telegram group with /agent99 ....
  • Agent99 records these as source=telegram-sre-war-room.
  • Failed or escalated automated repairs send operator_command_result back to Telegram.
  • Successful automated repairs are evidence-first: they write queue, evidence, problem count, KM, and RAG receipts, then suppress success-only Telegram noise.
  • Failure results include source, failure instruction, mode, exit code, evidence path, and problem count.
  • The command path must not depend on Codex quota.
  • The command path must not depend only on the default TELEGRAM_CHAT_ID, because that can point to a private chat. Agent99 also accepts command updates from Telegram chats whose title matches AwoooI SRE|AWOOOI SRE|SRE.
  • When the command comes from a matched Telegram group, the completion report is sent back to that same Telegram origin; stored evidence redacts the raw chat id as telegram-origin.
  • Non-command monitoring alerts in the SRE Telegram group are auto-ingested when they look like Alertmanager/FIRING/critical/warning/502/CPU/disk/Harbor/K3s/backup/host-down/provider-freshness alerts.
  • Auto-ingested monitoring alerts use source=telegram-sre-war-room-monitoring and enter the same SRE alert inbox / queue / recovery / Telegram completion / KM-RAG path.

Alert-to-action routing:

  • Public 502 / route down / website down -> Recover.
  • Host down / reboot detected / VM not running -> Recover.
  • CPU, memory, disk, or load pressure -> Perf with controlled apply.
  • Harbor or registry unhealthy -> HarborRepair.
  • AWOOOI K3s, pod, deployment, or image pull failure -> AwoooRepair.
  • Backup freshness, snapshot, restore drill, or cron issue -> BackupCheck.
  • Provider freshness, last-seen, ingestion, raw signal, or no-false-green risk -> ProviderFreshness.
  • Unknown actionable alert -> Status evidence first.

Producer-side Agent99 bridge:

  • AWOOI API Alertmanager webhook schedules an Agent99 bridge after normalization and fingerprinting.
  • Preferred transport: when AGENT99_SRE_ALERT_RELAY_URL is configured, the bridge POSTs sanitized alert JSON to the 99 relay. Production URL is http://192.168.0.99:8787/agent99/sre-alert/.
  • Fallback transport: when AGENT99_SRE_ALERT_INBOX_PATH is configured and relay URL is empty, the bridge writes sanitized alert JSON directly to the Agent99 SRE inbox contract.
  • If relay/path is empty or unavailable, the bridge fails open and logs; it must not cause Alertmanager retries.
  • Relay safety: only LAN/local source prefixes are accepted by default; optional AGENT99_SRE_RELAY_TOKEN / AGENT99_SRE_ALERT_RELAY_TOKEN values stay in env/Secret and are never stored in repo or evidence.
  • Provider freshness bridge output must run direct readback, then require provider window, last seen, direct reference, and verifier readback before dashboard green is trusted.
  • Provider freshness no-false-green rule: route 200 or dashboard up is not enough. The Agent99 verifier must read https://awoooi.wooo.work/api/v1/platform/events/dossier/recurrence?project_id=awoooi&limit=20 and record providerWindowMinutes, lastSeen, directReference, and verifierReadback.

Approved controlled actions for the 99 agent:

  • Start a missing VM.
  • Ping and SSH readback.
  • Restart Harbor Redis / Harbor core after evidence of Harbor token failure.
  • Delete K3s pods stuck in image pull failure only after Harbor is healthy.
  • Run public route readback.
  • Send Telegram status.

Actions still requiring explicit break-glass:

  • Reading or exfiltrating secrets.
  • Destructive database operations.
  • Firewall cutover.
  • Remote delete / prune / force push.
  • Production provider route cutover with cost or billing impact.

Evidence Required Before Calling Recovery Complete

Recovery is complete only when all of the following are captured:

  • Five VM hosts have reachability evidence.
  • 110 Harbor core is healthy.
  • 120 AWOOOI deployments are ready.
  • Public routes return non-502, preferably HTTP 200.
  • Any degraded health components are listed separately with owner / next action.

Current 2026-07-09 Receipts

  • 110 harbor-core: recovered to Up ... (healthy).
  • 120 awoooi-prod:
    • awoooi-api: 1/1
    • awoooi-web: 2/2
    • awoooi-worker: 1/1
    • awoooi-auto-repair-canary: 1/1
  • Public readback:
    • awoooi.wooo.work: HTTP 200 after redirect.
    • stock.wooo.work: HTTP 200 after redirect.
    • 2026fifa.wooo.work: HTTP 200 after redirect.
    • gitea.wooo.work: HTTP 200 after redirect.
    • harbor.wooo.work: HTTP 200 after redirect.
  • Host reachability:
    • 110, 112, 120, 121, 188 responded to ping or SSH.
  • Remaining degraded signal:
    • AWOOOI health reported degraded AI provider signals including ollama_gcp_b timeout and ollama_local connection refused / unavailable.

Agent99 Installation Receipt

2026-07-09 16:49 Asia/Taipei:

  • 99 host: WOOO-SUPER.
  • Agent root: C:\Wooo\Agent99.
  • VMware CLI: C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe.
  • Live config: C:\Wooo\Agent99\config\agent99.config.json.
  • Evidence directory: C:\Wooo\Agent99\evidence.
  • Scheduled tasks:
    • Wooo-Agent99-Heartbeat: LastTaskResult=0, latest status evidence agent99-Status-20260709-164845.json.
    • Wooo-Agent99-Startup-Recovery: LastTaskResult=0, latest recovery evidence agent99-Recover-20260709-164937.json.
  • Startup recovery readback:
    • VM selector: 110, 188, 112, 120, 121 all host_already_reachable.
    • Harbor: core, Redis, and registry all healthy.
    • AWOOOI: deploy ready, no image pull failure, no crash failure.
    • Public routes: awoooi.wooo.work, stock.wooo.work, 2026fifa.wooo.work, gitea.wooo.work, and harbor.wooo.work all HTTP 200 / non-502.

Agent99 Performance Guard Receipt

2026-07-09 17:15 Asia/Taipei:

  • Scheduled task: Wooo-Agent99-Performance-Guard.
  • Trigger interval: every 1 minute.
  • Last task result: 0.
  • Latest evidence: C:\Wooo\Agent99\evidence\agent99-Perf-20260709-171502.json.
  • Host performance readback:
    • 110: OK.
    • 112: OK.
    • 120: OK.
    • 121: OK.
    • 188: OK.
  • Controlled load shedding:
    • Only allowlisted actions can run.
    • Current 110 runner action requires load_per_core_critical.
    • Memory/disk pressure alerts do not execute the runner pause action.
  • Telegram audit:
    • Agent99 emits Telegram attempts for warnings, critical events, failed or escalated operator commands, and load-shed actions.
    • Successful operator commands are not Telegram noise; they are recorded in evidence, problem counts, KM, and RAG.
    • Agent99 supports local env aliases and an existing-token relay.
    • Current relay: 110 stockplatform-v2-api-1, using that container's existing TELEGRAM_BOT_TOKEN / TELEGRAM_CHAT_ID.
    • 2026-07-09 17:27 verification: Wooo-Agent99-Performance-Guard LastTaskResult=0; latest evidence agent99-Perf-20260709-172712.json; tgSent=2; relayOk=True.
    • Token and chat id values were not printed, copied into repo files, or written to evidence.

Agent99 Command And SRE Alert Integration Receipt

2026-07-09 23:03 Asia/Taipei:

  • Telegram command inbox is deployed and scheduled:
    • Script: C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1
    • Task: Wooo-Agent99-Telegram-Inbox
    • Latest evidence: C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260709-230031.json
    • Result: ok=True, updatesSeen=0, processedCount=0
    • Evidence check: no Python code, no token text.
  • SRE war-room alert inbox is deployed and scheduled:
    • Script: C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1
    • Task: Wooo-Agent99-SRE-Alert-Inbox
    • Incoming path: C:\Wooo\Agent99\alerts\incoming
    • Latest evidence: C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260709-230226.json
  • Synthetic 502 alert test:
    • Alert id: synthetic-public-502-20260709-230107
    • Alert source: awoooi-sre-war-room
    • Alert kind: public_route_502
    • Resolved mode: Recover
    • Queue processor evidence: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-230108.json
    • Processed count: 1
    • Mode: Recover
    • Exit code: 0
    • Telegram relay: sent=True, relayOk=True
  • Recovery verifier from the synthetic alert:
    • Evidence: C:\Wooo\Agent99\evidence\agent99-Recover-20260709-230108.json
    • Public route failures: 0
    • Host reachability failures: 0
    • Harbor core healthy: True
    • AWOOOI deploy ready: True
  • Agent99 self-health after integration:
    • Evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-230223.json
    • Severity: ok
    • Task failures: 0
    • Evidence critical: 0
    • Evidence warning: 0
    • Telegram inbox task: OK
    • SRE alert inbox task: OK
    • Dashboard queue pending: 0
    • Dashboard alert pending: 0

Agent99 Problem Count / KM / RAG Receipt

2026-07-09 23:12 Asia/Taipei:

  • Synthetic AwoooI SRE Telegram command:
    • instruction: 檢查所有網站是否恢復正常，若502就自動修復
    • source: telegram-sre-war-room
    • request id: request-20260709-231157-d53f53d8
    • resolved mode: Recover
    • control exit code: 0
  • Execution evidence:
    • C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-231158.json
    • processed source: telegram-sre-war-room
    • processed mode: Recover
    • processed ok: True
    • processed exit code: 0
    • Telegram relay sent: 1
    • Telegram evidence contains no Python code.
  • Problem counters:
    • path: C:\Wooo\Agent99\state\problem-counts.json
    • total events: 1
    • total resolved: 1
    • total failed: 0
    • problem key: telegram-sre-war-room_agent99_operator_instruction
    • occurrences: 1
    • resolved: 1
    • failed: 0
  • Knowledge outputs:
    • full event ledger: C:\Wooo\Agent99\playbooks\agent99-incident-records.jsonl
    • KM summary: C:\Wooo\Agent99\playbooks\agent99-incident-km.md
    • RAG import feed: C:\Wooo\Agent99\playbooks\rag-import\agent99-incidents.jsonl
  • Dashboard:
    • problem events: 1
    • resolved: 1
    • failed: 0
    • queue pending: 0
    • alert pending: 0
  • SRE Telegram group routing:
    • Default relay chat readback was private chat OoO, not the SRE group.
    • Live Agent99 config now includes allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE.
    • Telegram inbox readback after the change: ok=True, updatesSeen=0, processedCount=0.
    • Next real /agent99 ... message from the SRE group will be accepted by title match and will reply back to the same Telegram origin.

Agent99 SRE Telegram Monitoring Alert Auto-Ingest Receipt

2026-07-09 23:32 Asia/Taipei:

  • Runtime behavior:
    • SRE Telegram group monitoring alerts are now consumed automatically.
    • Prefix /agent99 is no longer required for monitoring alerts.
    • Auto-ingest is controlled by autoIngestMonitoringAlerts=True.
    • Group matching uses allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE.
  • Loop prevention:
    • Agent99's own Agent99 Alert / operator_command_result messages are ignored by monitoring auto-ingest.
    • Successful operator command results are suppressed from Telegram and kept in evidence/KM/RAG; failed or escalated command results remain alertable.
    • Provider freshness signals are not passive dashboard health; they create a read-only freshness candidate, run direct AWOOI readback, and require provider window, last seen, direct reference, and verifier readback.
    • Provider freshness guardrail: do not switch provider, call paid model, edit env, or reload.
  • Synthetic monitoring alert:
    • Alertmanager FIRING critical public_route_502 service=awoooi host=192.168.0.120 HTTP 502
    • no /agent99 prefix.
  • Evidence chain:
    • Telegram inbox: C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260709-233230.json
      • inboxOk=True
      • alertedCount=1
    • SRE alert inbox: C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260709-233231.json
      • queued count: 1
      • failed count: 0
    • Control loop: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-233232.json
      • alert source: telegram-sre-war-room-monitoring
      • alert kind: public_route_502
      • mode: Recover
      • processed ok: True
      • exit code: 0
      • Telegram attempts: 1
      • Telegram sent: 1
    • Recovery: C:\Wooo\Agent99\evidence\agent99-Recover-20260709-233055.json
  • Problem counters:
    • total events before: 2
    • total events after: 3
    • total resolved after: 3
    • total failed after: 0
    • problem key: agent99-sre-alert-inbox_awoooi_192.168.0.120_public_route_502
    • problem occurrences: 2

Agent99 Alert Noise And Provider Freshness Readback Receipt

2026-07-10 10:00 Asia/Taipei:

  • Runtime patch deployed on 99:
    • Script: C:\Wooo\Agent99\bin\agent99-control-plane.ps1.
    • Parser readback: parse_ok.
    • Source commit: 0a7a0115 fix(agent99): reduce alert noise and verify freshness.
  • Provider freshness behavior:
    • Evidence: C:\Wooo\Agent99\evidence\agent99-ProviderFreshness-20260710-100028.json.
    • Status: direct_readback_ok_no_source_events.
    • Missing fields after automatic readback: lastSeen only.
    • providerWindowMinutes=120.
    • directReference=https://awoooi.wooo.work/api/v1/platform/events/dossier/recurrence?project_id=awoooi&limit=20.
    • verifierReadback.ok=True, httpStatus=200, sourceEventTotal=0.
  • Telegram noise control:
    • Evidence log: C:\Wooo\Agent99\evidence\agent99-ProviderFreshness-20260710-100028.log.
    • Same provider-freshness state was suppressed: telegram_dedupe_skip type=provider_freshness_triage severity=warning ageMinutes=5.83 cooldown=30.
    • Evidence log: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260710-100027.log.
    • Successful operator command was evidence-only: telegram_operator_success_suppressed type=operator_command_result mode=ProviderFreshness id=operator-suppress-smoke-20260710-100026.
    • Control evidence: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260710-100027.json, processedCount=1, pendingCount=0, telegram=[].
  • Scheduled entrypoints verified on 99:
    • Wooo-Agent99-Control-Loop.
    • Wooo-Agent99-SRE-Alert-Inbox.
    • Wooo-Agent99-Telegram-Inbox.
    • Wooo-Agent99-SRE-Alert-Relay is running.
  • Operational rule:
    • Do not treat repeated success receipts as user-facing alerts.
    • Send user-facing Telegram only for warnings, criticals, unresolved blockers, failed/escalated operator commands, and controlled-remediation actions.
    • Keep successful automation in evidence, problem counters, KM, and RAG so repeated incidents trend down without flooding the SRE group.

OpenClaw Cold-Start Readback

2026-07-09 17:38 Asia/Taipei:

  • Production OpenClaw URL: http://192.168.0.188:8088.
  • 188 service: clawbot.service is active.
  • Health endpoint: http://192.168.0.188:8088/health.
  • Health readback: status=healthy, version=6.0, service=ClawBot, environment=production, telegram_bot=connected.
  • Agent99 latest status evidence: C:\Wooo\Agent99\evidence\agent99-Status-20260709-173813.json.
  • Agent99 aiServices.openclaw: ok=True, status=healthy.
  • Telegram relay from the Agent99 status run: tgSent=2, relayOk=2.
  • Note: 192.168.0.188:8089 is not the current production OpenClaw endpoint.

Windows Update No-Surprise-Reboot Policy

2026-07-09 17:39 Asia/Taipei:

  • Evidence: C:\Wooo\Agent99\evidence\agent99-windows-update-policy-20260709-173933.json.
  • NoAutoRebootWithLoggedOnUsers=1.
  • AlwaysAutoRebootAtScheduledTime=0.
  • AUOptions=3.
  • ActiveHoursStart=0, ActiveHoursEnd=23.
  • gpupdateOk=True.

Agent99 Self-Health Receipt

2026-07-09 17:52 Asia/Taipei:

  • Scheduled task: Wooo-Agent99-Self-Health.
  • Trigger interval: every 5 minutes.
  • Task Scheduler trigger verification:
    • LastTaskResult=0.
    • Evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-175218.json.
    • evidenceAfterTrigger=True.
  • Self-health checks:
    • Wooo-Agent99-Startup-Recovery: OK.
    • Wooo-Agent99-Heartbeat: OK.
    • Wooo-Agent99-Performance-Guard: OK.
    • Wooo-Agent99-Self-Health: OK.
    • Evidence freshness: performance, status, self-health, and startup recovery OK.
    • Latest performance evidence: all five hosts OK.
    • Telegram delivery: tgSent=3, relayOk=3.
    • Updated rule: a quiet interval with no alert attempts is OK; only failed alert attempts are Telegram critical.

110 Load Pressure Incident Readback

2026-07-09 18:07 Asia/Taipei:

  • Symptom: 110 host load stayed above warning after all hosts rebooted.
  • Initial findings:
    • gitea and stockplatform-v2-postgres-1 were the highest regular service containers.
    • /backup/scripts/backup-awoooi.sh was running a remote pg_dump from 188 and local gzip compression.
    • A random container mystifying_napier was running alpine "tar czf - /data" with no restart policy and no labels, consuming CPU and writing compressed binary output into Docker logs.
  • Controlled actions taken:
    • Backup pipeline PIDs were reduced to nice 15 / ionice best-effort 7.
    • mystifying_napier was stopped and left as Dead; it was not deleted, so evidence remains available.
  • Agent99 automation added:
    • stop-110-runaway-alpine-tar-data: stops allowlisted alpine containers running tar czf - /data when 110 is at warning or critical load.
    • renice-110-awoooi-backup-pressure: lowers priority for backup-awoooi.sh, gzip, and remote pg_dump pipeline PIDs when 110 is at warning or critical load.
    • pause-110-ci-runner-pressure: still only runs at critical load with load_per_core_critical; runner is not touched for backup, disk, memory, or warning-only pressure.
    • Warning-level top CPU capture is enabled with a 10-second timeout to avoid diagnostics becoming a new load source.
  • Verification:
    • Evidence: C:\Wooo\Agent99\evidence\agent99-Perf-20260709-180704.json.
    • 110 load-per-core: 0.67, severity ok.
    • 112, 120, 121, and 188: severity ok.
    • perfCritical=0, perfWarning=0.
    • Load-shedding actions skipped because host severity was below threshold.
    • Telegram relay: tgSent=2, relayOk=2.

Agent99 Backup Health Receipt

2026-07-09 19:07 Asia/Taipei:

  • Scheduled task: Wooo-Agent99-Backup-Health.
  • Trigger interval: every 15 minutes.
  • Task Scheduler trigger verification:
    • LastTaskResult=0.
    • Evidence: C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260709-190617.json.
    • evidenceAfterTrigger=True.
  • BackupCheck coverage:
    • Snapshot freshness targets: awoooi, gitea, harbor, momo, langfuse, monitoring, signoz, clawbot, open-webui, sentry, ai-artifacts, configs, and public-routes.
    • File/status checks: backup.log, backup-status.log, legacy status.log, legacy cron.log, integrity check status, restore drill status, and configs last-status JSON.
    • Cron contract patterns: backup-all.sh, backup-awoooi-frequent.sh, backup-health-textfile-exporter.py, check-backup-integrity.sh, sync-offsite-backups.sh --mode status, and backup-status.sh.
  • Latest result:
    • backupHealth.severity=ok.
    • targetCritical=0, targetWarning=0.
    • fileCritical=0, fileWarning=0.
    • cronCritical=0, cronWarning=0.
    • targetCount=13, fileCheckCount=7, cronPatternCount=6.
    • Telegram relay: tgSent=2, relayOk=2.
  • Self-health cross-check:
    • Evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-190714.json.
    • Wooo-Agent99-Backup-Health: OK, LastTaskResult=0.
    • Backup evidence freshness: OK, latest file agent99-BackupCheck-20260709-190617.json, age 0.5 minutes, max age 30 minutes.

Alert-to-Remediation / Evidence Sanitization Readback

2026-07-09 22:11 Asia/Taipei:

  • Root cause fixed for unreadable Telegram relay messages:
    • Relay sender argument index corrected from sys.argv[1] to sys.argv[2].
    • Telegram now receives formatted Agent99 alert text, not embedded Python relay code.
  • Noise control:
    • alertAllOperations=false.
    • alertInfoEvents=false.
    • dedupeMinutes=30.
    • agent_complete remains evidence-only and does not alert by default.
  • Alert-to-action rule:
    • Perf -ControlledApply suppresses immediate performance spam, performs allowlisted remediation first, then alerts load_shed_result.
    • If no remediation applies, Agent99 alerts the unresolved performance issue.
    • 188 disk pressure is now allowlisted through prune-188-docker-build-cache-disk-pressure.
    • The 188 action uses timeoutSeconds=240 plus a post-apply verifier; postVerifyMaxDiskUsedPercent=89.
    • Timeout alone is not final truth; post-verifier success can mark remediation effective.
  • Self-health false critical fixed:
    • A quiet interval with no recent alerts is OK.
    • Telegram health is critical only when recent alert attempts exist and all delivery attempts fail.
    • Telegram delivery health now uses a 60-minute lookback across recent evidence files.
    • Transient performance warnings are no longer promoted to self-health critical; only missing or critical host performance evidence is critical.
  • Backup monitor false critical fixed:
    • Backup snapshot readback now supports snapshotReadTimeoutSeconds.
    • 99 live config sets snapshotReadTimeoutSeconds=90.
    • Harbor backup was verified as present and fresh after a timeout false critical; latest metadata was from 2026-07-09 17:48 Asia/Taipei and within the 1800-minute SLA.
  • Evidence security fixed:
    • Harbor readback now stores selected health fields only: name, running state, health state.
    • Raw docker inspect output is not stored.
    • Existing Status evidence on 99 was sanitized: 77 files checked; inspectOutputNotRedactedCount=0; unsafeHarborFieldCount=0.
  • Latest recovery proof:
    • Status evidence: C:\Wooo\Agent99\evidence\agent99-Status-20260709-221147.json.
    • Host reachability: 110, 112, 120, 121, and 188 all OK.
    • Public routes: awoooi.wooo.work, stock.wooo.work, 2026fifa.wooo.work, gitea.wooo.work, and harbor.wooo.work all HTTP 200 / non-502.
    • Performance: perfCritical=0, perfWarning=0; 188 disk used 87%; 110 load-per-core 0.85.
    • Harbor: core, Redis, and registry healthy with redacted evidence.
    • AWOOOI: deployReady=true, imagePullFailure=false, crashFailure=false.
    • Backup evidence after timeout fix: C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260709-221759.json, backupHealth.severity=ok, all target/file/cron critical and warning counts are 0.
    • Final self-health evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-222236.json, selfHealth.severity=ok, taskFailures=0, performanceSeverity=ok, telegramSeverity=ok.

99 Visible Agent Control Center

2026-07-09 22:33 Asia/Taipei:

  • Agent99 is visible and controllable from the 99 Windows desktop.
  • New scheduled task:
    • Wooo-Agent99-Control-Loop.
    • Runs every 1 minute.
    • Updates dashboard and processes queue commands.
  • Desktop shortcuts:
    • C:\Users\Public\Desktop\Agent99 Control Center.lnk.
    • C:\Users\Public\Desktop\Agent99 Queue.lnk.
    • C:\Users\Public\Desktop\Agent99 Evidence.lnk.
    • D:\Desktop\Agent99 Control Center.lnk.
    • D:\Desktop\Agent99 Queue.lnk.
    • D:\Desktop\Agent99 Evidence.lnk.
  • Dashboard:
    • HTML: C:\Wooo\Agent99\dashboard\index.html.
    • JSON: C:\Wooo\Agent99\dashboard\agent99-control-center.latest.json.
    • Latest dashboard timestamp readback: 2026-07-09T22:33:26+08:00.
    • Queue readback: pending=0, processed=2, failed=0.
  • Queue command contract:
    • Add a JSON command to C:\Wooo\Agent99\queue.
    • ControlTick processes only allowlisted modes.
    • Results are written to C:\Wooo\Agent99\queue\processed or C:\Wooo\Agent99\queue\failed.
    • Operator command results are sent to Telegram.
  • Verification:
    • Command ID: codex-visible-publicsmoke-20260709-223308.
    • Mode: PublicSmoke.
    • Result: ok=true, exitCode=0.
    • Evidence: C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-223309.json.
    • Telegram: operator_command_result, severity=info, sent=true, relayOk=true.
    • Self-health: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-223340.json, selfHealth.severity=ok, Wooo-Agent99-Control-Loop=ok.
    • Latest performance: C:\Wooo\Agent99\evidence\agent99-Perf-20260709-223419.json, perfCritical=0, perfWarning=0.

Operator Instruction Intake

2026-07-09 22:41 Asia/Taipei:

  • Agent99 accepts text instructions on 99 through:
    • C:\Wooo\Agent99\bin\agent99-submit-request.ps1.
    • C:\Wooo\Agent99\agent99-submit-request.cmd.
    • Agent99 Submit Request.lnk on the desktop.
  • Flow:
    • instruction -> request record -> queue command -> ControlTick -> evidence -> Telegram result.
  • Current allowlisted request router:
    • websites / routes / 502 -> PublicSmoke.
    • restore / reboot / repair -> Recover.
    • VM / VMware -> StartVMs.
    • CPU / load / memory / disk / performance -> Perf.
    • backup / snapshot / cron -> BackupCheck.
    • self health / agent health -> SelfCheck.
    • Harbor / registry -> HarborRepair.
    • AWOOOI / K3s / pods -> AwoooRepair.
  • Verification:
    • Chinese instruction submitted from Codex to 99: check whether all websites are normal.
    • Request ID: request-20260709-224133-0dc045fe.
    • Resolved mode: PublicSmoke.
    • Processed result: ok=true, exitCode=0.
    • Evidence: C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-224134.json.
    • Public routes: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502.
    • Telegram: operator_command_result, severity=info, sent=true, relayOk=true.
Reference in New Issue View Git Blame Copy Permalink