wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity
Files
85cb7fba426ad193ed81fc4b389267f2a7e23607
awoooi/AGENT99-AI-WORKSTATION-SOP.md

34 KiB
Raw Blame History

Agent99 AI Workstation SOP

Last updated: 2026-07-10

Canonical enterprise scope and work order:

  • docs/workplans/2026-07-10-agent99-enterprise-ai-automation-master-plan.md
  • docs/operations/agent99-enterprise-ai-automation-work-items.snapshot.json

This SOP defines how Agent99 operates. The canonical work ledger defines what remains incomplete across all hosts, products, sites, services, tools and packages. A transport or command exitCode=0 never means an incident is resolved; the mode-specific verifier, source-event resolution and callback receipt are required.

Objective

192.168.0.99 is the AI Agent workstation and local operations control node.

Host 99 is not the only availability observer. An external watcher and maintenance surface must remain available when 99 or the entire local failure domain is down.

2026-07-10 P0-001 Outcome Truth Contract

  • exitCode=0 means only that the PowerShell transport completed.
  • A command is resolved only when its mode-specific post-condition verifier passes, its evidence is fresh and mode-matched, and any alert-source event has an explicit resolved readback.
  • Other legal outcomes are candidate_ready, verifying, degraded, blocked, and failed.
  • Alert-originated work without sourceEventResolved=true remains verifying, even when the repair command and post-condition verifier succeeded.
  • problem-counts.json v2 preserves the legacy totalResolved field but labels it as possibly transport-derived. New trustworthy closure is counted separately in totalVerifiedResolved and per-problem verifiedResolved.
  • Bootstrap writes state\runtime-manifest.json with source revision, source/runtime SHA-256, content match, file count and mismatch count. Self-health is critical when this manifest is missing, invalid, stale by hash, or incomplete.
  • All .ps1 runtime files are normalized to UTF-8 BOM by bootstrap so Windows PowerShell 5 cannot corrupt Traditional Chinese strings during parsing.
  • Staging verification on 99 must pass -SelfTestOutcomeContract and runtimeMatched=true before production Agent99 files are replaced.

It must act as:

  • Secretary: reminders, status summaries, evidence packets, operator updates.
  • Assistant: routine checks, file organization, browser/readback checks, alert routing.
  • Engineer: VM control, service recovery, deploy/readback verification, runbook execution.
  • Incident operator: reboot detection, recovery orchestration, Telegram alerting, SLA countdown.

The machine must be able to operate VMware, SSH to service hosts, run public HTTP readbacks, and produce evidence without depending on old chat context.

Agent99 Operating Modes

Observe mode:

  • Read host, VM, service, and website state.
  • Produce evidence only.
  • Default mode for scheduled checks.

Controlled apply mode:

  • Execute approved low-blast-radius actions.
  • Must write evidence before and after each action.
  • Requires explicit operator approval or scheduled startup recovery policy.

Break-glass mode:

  • Reserved for destructive, irreversible, secret-bearing, cost-bearing, or firewall/provider cutover actions.
  • Must not be hidden inside automatic tasks.

Required Local Capabilities On 99

Base runtime:

  • Windows PowerShell 5+ or PowerShell 7.
  • Windows OpenSSH client.
  • curl.exe.
  • VMware Workstation.
  • vmrun.exe.
  • Windows Task Scheduler.

Agent filesystem:

  • C:\Wooo\Agent99\bin
  • C:\Wooo\Agent99\config
  • C:\Wooo\Agent99\evidence
  • C:\Wooo\Agent99\logs
  • C:\Wooo\Agent99\queue
  • C:\Wooo\Agent99\requests
  • C:\Wooo\Agent99\alerts\incoming
  • C:\Wooo\Agent99\playbooks

Agent scripts:

  • agent99-bootstrap.ps1
  • agent99-control-plane.ps1
  • agent99-submit-request.ps1
  • agent99-telegram-inbox.ps1
  • agent99-sre-alert-inbox.ps1
  • host-reboot-scorecard.ps1
  • agent99.config.json

Command Ingress Without Codex Quota

Codex is not the runtime command plane. Agent99 must accept operator work through local 99 channels even when Codex is unavailable.

Supported command channels:

  • Local desktop: Agent99 Submit Request.lnk.
  • Local queue: write allowlisted command JSON to C:\Wooo\Agent99\queue.
  • AwoooI SRE Telegram group: send /agent99 ..., !agent99 ..., #agent99 ..., agent99 ..., or @agent99 ....
  • SRE war room alerts: write alert JSON to C:\Wooo\Agent99\alerts\incoming.
  • AwoooI SRE Telegram monitoring alerts: non-command messages that look like Alertmanager/FIRING/critical/warning/502/CPU/disk/Harbor/K3s/backup/host-down/provider-freshness alerts are auto-ingested.

Command processing requirements:

  • Every command resolves to an allowlisted mode only.
  • Website plus 502 plus recover/repair intent resolves to Recover, not passive smoke-only checking.
  • Alert-driven remediation uses controlled apply only for allowlisted modes such as Recover, HarborRepair, AwoooRepair, Perf, and LoadShed.
  • Every accepted command writes a request record, queue record, execution result, and evidence pointer.
  • Telegram messages must be human-readable: source, instruction, mode, result, and evidence path. Raw Python, raw stack trace, token, chat id, cookies, sessions, or .env must never be sent.
  • Telegram SRE group commands must be recorded with source=telegram-sre-war-room.
  • Telegram SRE monitoring alerts must be recorded with source=telegram-sre-war-room-monitoring.
  • Telegram SRE group matching uses allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE, so the command path is not limited to the default private chat id.
  • When a command originates from a matched Telegram group, completion replies are routed back to the same Telegram origin while evidence stores only telegram-origin, not the raw chat id.
  • After automated recovery completes, Agent99 must report the result back to Telegram so the project AI Agents and human operators know whether the abnormality is resolved.
  • Every queue execution must update problem counts, write a Playbook/KM record, and append a RAG-import JSONL line.
  • Operator command and remediation result Telegram reports are not deduped; each completed repair emits a completion report. Noisy warning/critical signals still use dedupe.

SRE Telegram command examples:

  • /agent99 檢查所有網站是否正常，若502就自動修復
  • /agent99 檢查110和188 CPU負載，必要時執行允許的降載
  • /agent99 檢查AWOOOI K3s與Harbor，若ImagePullBackOff就修復
  • /agent99 檢查備份健康狀態
  • /agent99 建立 provider freshness candidate，要求 provider window、last seen、direct ref 與 verifier readback

Monitoring alert auto-ingest examples:

  • Alertmanager FIRING critical public_route_502 service=awoooi host=192.168.0.120 HTTP 502
  • HostHighCpuLoad warning host=192.168.0.110
  • HarborRegistryDown critical host=192.168.0.110
  • KubePodCrashLooping critical namespace=awoooi-prod
  • provider_freshness_signal target=provider_freshness_signal freshness last_seen missing direct_ref required

Problem memory outputs:

  • Problem counters: C:\Wooo\Agent99\state\problem-counts.json.
  • Full event ledger: C:\Wooo\Agent99\playbooks\agent99-incident-records.jsonl.
  • Playbook/KM summary: C:\Wooo\Agent99\playbooks\agent99-incident-km.md.
  • RAG import feed: C:\Wooo\Agent99\playbooks\rag-import\agent99-incidents.jsonl.
  • Dashboard: C:\Wooo\Agent99\dashboard\index.html, section Problem Counts.

SRE War Room Alert Contract

AWOOOI SRE monitors and other AI Agent work windows can hand alerts to Agent99 by writing JSON files to:

C:\Wooo\Agent99\alerts\incoming

Minimum fields:

{
  "id": "alert-unique-id",
  "source": "awoooi-sre-war-room",
  "severity": "critical",
  "kind": "public_route_502",
  "service": "awoooi",
  "host": "192.168.0.120",
  "title": "AWOOOI public route returned 502",
  "message": "The public health route is returning 502.",
  "createdAt": "2026-07-09T22:50:00+08:00"
}

Routing rules:

  • 502, site_502, public_route_502, route_down, host_down, VM down, or reboot detected -> Recover.
  • CPU, memory, disk, performance pressure -> Perf with controlled apply.
  • Harbor or registry -> HarborRepair.
  • AWOOOI, K3s, image pull, pod, deployment -> AwoooRepair.
  • Backup, snapshot, restore drill, or cron -> BackupCheck.
  • Unknown actionable alert -> Status evidence first.

Alert processing evidence:

  • Inbox evidence: agent99-SreAlertInbox-*.json.
  • Queued request: C:\Wooo\Agent99\requests.
  • Execution evidence: agent99-Recover-*.json, agent99-Perf-*.json, or the resolved mode evidence file.

VM Control Requirements

Agent99 must be able to operate these VM hosts:

  • 110
  • 112
  • 120
  • 121
  • 188

Required actions:

  • Detect VM powered-off state.
  • Start missing VMs through vmrun.exe.
  • Detect VMware config lock symptoms.
  • Avoid deleting VM lock files unless VMware process state proves the VM is not running.
  • Prefer guest shutdown/restart first; use reset only when the guest is stuck and SLA requires recovery.

Engineering Recovery Requirements

Agent99 must support these engineering actions:

  • SSH readback to 110 and 120.
  • Harbor core / Redis / registry checks on 110.
  • K3s AWOOOI deployment and pod checks on 120.
  • Public route readback for user-visible websites.
  • Evidence capture after every action.
  • Telegram alert hooks.

Main Scheduled Automations

Startup recovery:

  • Trigger: 99 boot or operator logon.
  • Frequency: once at startup, then repeat during first recovery window.
  • Mode: controlled apply allowed for approved actions.
  • Purpose: start VMs, restore Harbor, restore K3s pods, verify public routes.

Heartbeat:

  • Trigger: every 5 minutes.
  • Mode: observe by default.
  • Purpose: detect host down, 502, service degraded, backup job missing, or stale freshness.

Daily evidence summary:

  • Trigger: daily.
  • Mode: observe.
  • Purpose: produce a short host/service/site/backup status packet.

Telegram Alert Classes

The agent must send alerts for:

  • host_down
  • host_reboot_detected
  • vm_not_running
  • vm_start_attempted
  • service_recovering
  • harbor_unhealthy
  • k3s_image_pull_failure
  • site_502
  • site_recovered
  • service_degraded
  • backup_missing
  • backup_failed
  • recovery_sla_risk
  • recovery_complete

Telegram credentials must be stored outside the repo, preferably in Windows Credential Manager or machine environment variables.

2026-07-09 Recovery Logic To Embed

When AWOOOI or other sites return 502:

  1. Do not only inspect nginx.
  2. Check 120 K3s workloads first.
  3. If AWOOOI pods are ImagePullBackOff, check 110 Harbor before deleting pods.
  4. If Harbor token endpoint returns 500 or pulls fail, inspect:
    • Docker engine.
    • Harbor Redis.
    • harbor-core.
    • registry.
    • registryctl.
  5. If harbor-core is not healthy:
    • Restart Redis and harbor-core.
    • If Docker/systemd hangs, execute controlled 110 guest restart via VMware.
  6. After harbor-core is healthy:
    • Delete only failed AWOOOI pods.
    • Wait for deployment readiness.
    • Verify public HTTP.

First Implementation Milestone

Milestone M1 is complete when:

  • Agent99 directory exists on 99.
  • Config file exists.
  • Startup and heartbeat scheduled tasks exist.
  • Scorecard can run in observe mode.
  • Recovery can run in controlled apply mode.
  • Evidence JSON is written.
  • No secrets are stored in repo files.

2026-07-09 M1 Receipt

M1 is operational on 99 as of 2026-07-09 16:49 Asia/Taipei.

  • C:\Wooo\Agent99 exists on WOOO-SUPER.
  • agent99-control-plane.ps1, host-reboot-scorecard.ps1, and task registration scripts are deployed.
  • agent99.config.json includes VMX selectors for 110, 188, 112, 120, and 121.
  • Wooo-Agent99-Heartbeat is registered and produced LastTaskResult=0.
  • Wooo-Agent99-Startup-Recovery is registered and produced LastTaskResult=0.
  • Latest heartbeat evidence showed Harbor healthy, AWOOOI deploy ready, and public routes non-502.
  • Latest recovery evidence showed all five configured VM hosts already reachable and no recovery action required.

Remaining M2 work:

  • Install Telegram credentials with environment variables or Windows Credential Manager, then verify real delivery.
  • Add backup-job freshness readback and alert classes.
  • Add Windows Update no-surprise-reboot policy verification.
  • Promote this into a long-running Windows service after the scheduled-task lane stays clean.

2026-07-09 M2 Performance Guard Receipt

Agent99 performance guard is operational on 99 as of 2026-07-09 17:15 Asia/Taipei.

  • Wooo-Agent99-Performance-Guard is registered in Windows Task Scheduler.
  • Trigger interval: every 1 minute.
  • Last manual verification: LastTaskResult=0.
  • Latest evidence: C:\Wooo\Agent99\evidence\agent99-Perf-20260709-171502.json.
  • Readback coverage:
    • 110: OK, load-per-core 0.35, memory available 62.93%, disk used 82%.
    • 112: OK, load-per-core 0.07, memory available 61.06%, disk used 47%.
    • 120: OK, load-per-core 0.05, memory available 91.76%, disk used 44%.
    • 121: OK, load-per-core 0.00, memory available 92.79%, disk used 27%.
    • 188: OK, load-per-core 0.08, memory available 90.19%, disk used 89%.
  • 99 SSH access was completed for:
    • 112 with user kali.
    • 188 with user ollama.
  • Load-shedding is controlled by allowlist only.
    • Current allowlisted action: pause vibework-dedicated-runner on 110 only when load_per_core_critical is present.
    • Disk or memory pressure only alerts; it does not trigger the CPU runner pause action.
  • Telegram audit is enabled in config and called for Agent99 start, warnings, critical events, load-shed attempts/results, and completion.
    • 99 local env aliases are supported for AGENT99_*, AWOOOI_TG_*, OPENCLAW_TG_*, TELEGRAM_*, and TG_*.
    • If 99 does not have local Telegram env values, Agent99 relays through the existing 110 stockplatform-v2-api-1 container and uses that container's existing TELEGRAM_BOT_TOKEN / TELEGRAM_CHAT_ID.
    • 2026-07-09 17:27 verification: Wooo-Agent99-Performance-Guard produced LastTaskResult=0, agent99-Perf-20260709-172712.json, tgSent=2, and relayOk=True.
    • No token or chat id is stored in repo files or Agent99 evidence.

2026-07-09 OpenClaw Receipt

OpenClaw is configured and now included in Agent99 evidence.

  • Source truth:
    • AWOOOI prod config: OPENCLAW_URL=http://192.168.0.188:8088.
    • 188 Ansible service target: OpenClaw on port 8088.
  • Runtime truth:
    • 192.168.0.188:8088/health returns status=healthy, version=6.0, service=ClawBot, environment=production, and telegram_bot=connected.
    • clawbot.service on 188 is active.
    • 192.168.0.188:8089 is not the current production endpoint.
  • Agent99 truth:
    • Latest status evidence: C:\Wooo\Agent99\evidence\agent99-Status-20260709-173813.json.
    • aiServices.openclaw: ok=True, status=healthy, url=http://192.168.0.188:8088/health.
    • Telegram relay evidence: tgSent=2, relayOk=2.

2026-07-09 M3 Command And Alert Ingress Receipt

Agent99 no longer depends on Codex as the only command channel.

  • Telegram command inbox deployed: C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1.
  • SRE alert inbox deployed: C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1.
  • Scheduled tasks registered:
    • Wooo-Agent99-Telegram-Inbox
    • Wooo-Agent99-SRE-Alert-Inbox
  • PowerShell parse readback: all five Agent99 scripts parsed successfully on 99.
  • Live config includes both scheduled tasks and freshness contracts.
  • Telegram inbox evidence: C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260709-230031.json.
    • ok=True
    • updatesSeen=0
    • processedCount=0
    • no Python code or token text in evidence.
  • SRE synthetic alert evidence: C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260709-230107.json.
    • input kind: public_route_502
    • source: awoooi-sre-war-room
    • resolved mode: Recover
    • queued count: 1
    • failed count: 0
    • controlExitCode=0
  • Recovery execution evidence: C:\Wooo\Agent99\evidence\agent99-Recover-20260709-230108.json.
    • public route failures: 0
    • host reachability failures: 0
    • Harbor core healthy: True
    • AWOOOI deploy ready: True
  • Queue processor evidence: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-230108.json.
    • processed count: 1
    • source: agent99-sre-alert-inbox
    • reason: sre_alert
    • mode: Recover
    • exit code: 0
    • Telegram relay: sent=True, relayOk=True
    • evidence contains no Python code.
  • Self-health evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-230223.json.
    • severity: ok
    • task failures: 0
    • evidence critical: 0
    • evidence warning: 0
    • Telegram inbox task: OK
    • SRE alert inbox task: OK
  • Dashboard JSON: C:\Wooo\Agent99\dashboard\agent99-control-center.latest.json.
    • queue pending: 0
    • alert pending: 0

2026-07-09 M4 SRE Telegram Command And Problem Memory Receipt

Agent99 now accepts AwoooI SRE Telegram-group commands as first-class work orders.

  • Telegram SRE command source mapping:
    • source: telegram-sre-war-room
    • request path: C:\Wooo\Agent99\requests
    • queue path: C:\Wooo\Agent99\queue
    • command result event: operator_command_result
  • Synthetic SRE Telegram command verification:
    • instruction: 檢查所有網站是否恢復正常，若502就自動修復
    • request id: request-20260709-231157-d53f53d8
    • resolved mode: Recover
    • control exit code: 0
    • evidence: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-231158.json
    • source: telegram-sre-war-room
    • processed mode: Recover
    • processed ok: True
    • processed exit code: 0
    • Telegram relay: sent 1
    • Telegram evidence contains no Python code.
  • Problem memory readback:
    • problem key: telegram-sre-war-room_agent99_operator_instruction
    • occurrences: 1
    • resolved: 1
    • failed: 0
    • stats path: C:\Wooo\Agent99\state\problem-counts.json
    • KM exists: True
    • RAG feed exists: True
    • dashboard problem events: 1
    • dashboard resolved: 1
    • dashboard failed: 0
  • Post-check:
    • Telegram inbox evidence: ok=True, updatesSeen=0, processedCount=0
    • Self-health evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-231255.json
    • self-health severity: ok
    • task failures: 0
    • queue pending: 0
    • alert pending: 0
  • SRE group routing hardening:
    • Existing relay default chat readback was a private chat, so group commands must not depend only on TELEGRAM_CHAT_ID.
    • Live config was updated with allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE.
    • Telegram inbox accepts matched group commands and carries a redacted telegram-origin reply route for completion reports.

2026-07-09 M5 SRE Telegram Monitoring Alert Auto-Ingest Receipt

Agent99 now ingests monitoring alerts from the AwoooI SRE Telegram group, not only /agent99 commands.

  • Runtime switch:
    • autoIngestMonitoringAlerts=True
    • allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE
  • Encoding hardening:
    • Runtime PowerShell scripts are ASCII-only.
    • Chinese alert keyword support uses Unicode codepoint checks instead of raw Chinese strings, avoiding Windows PowerShell 5 parser failures.
  • Synthetic SRE Telegram monitoring alert:
    • text: Alertmanager FIRING critical public_route_502 service=awoooi host=192.168.0.120 HTTP 502
    • no /agent99 prefix.
    • Telegram inbox evidence: C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260709-233230.json
    • inboxOk=True
    • alertedCount=1
  • SRE alert bridge evidence:
    • C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260709-233231.json
    • queued count: 1
    • failed count: 0
  • Control/recovery evidence:
    • C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-233232.json
    • source: telegram-sre-war-room-monitoring
    • kind: public_route_502
    • mode: Recover
    • processed ok: True
    • exit code: 0
    • Telegram attempts: 1
    • Telegram sent: 1
    • evidence contains no Python code.
  • Problem memory:
    • total events before: 2
    • total events after: 3
    • total resolved after: 3
    • total failed after: 0

2026-07-09 M6 Provider Freshness Signal Receipt

Provider freshness alerts must not be treated as passive route health.

  • Gap found:
    • Telegram Inbox and SRE Alert Inbox were running, but provider_freshness_signal was not present in Agent99 alerts, queue, or execution evidence after 2026-07-09 23:30 Asia/Taipei.
    • Only SelfCheck evidenceFreshness fields matched the word freshness, so the original alert was not processed as an actionable provider freshness signal.
  • Runtime behavior added:
    • Telegram auto-ingest recognizes provider_freshness_signal, source_provider_freshness_triage, raw_signal_normalized, controlled_runtime_candidate, freshness, last seen, last_seen, and ingestion.
    • SRE Alert Inbox routes freshness signals to ProviderFreshness.
    • ProviderFreshness creates a read-only candidate under C:\Wooo\Agent99\state\provider-freshness-candidates.
    • Candidate evidence must require provider window, last seen, direct reference, and verifier readback.
    • Guardrail remains: no provider switch, no paid model call, no env change, no reload.

2026-07-09 M7 AWOOI API To Agent99 Alert Bridge

Agent99 must not depend only on Telegram text polling for AwoooI SRE alerts.

  • Producer-side bridge:
    • AWOOI API POST /api/v1/webhooks/alertmanager now schedules a fail-open Agent99 SRE bridge after alert normalization and fingerprinting.
    • The bridge POSTs the same Agent99 alert contract to AGENT99_SRE_ALERT_RELAY_URL when configured.
    • If relay URL is empty, the bridge writes the contract expected by C:\Wooo\Agent99\alerts\incoming when AGENT99_SRE_ALERT_INBOX_PATH is configured.
    • Empty relay/path means bridge disabled; Alertmanager response must not fail or retry-storm because Agent99 inbox is unavailable.
  • Safety:
    • Payloads are sanitized before writing.
    • Labels with token, secret, password, authorization, cookie, session, or private-key names are redacted.
    • Provider freshness signals map to ProviderFreshness.
    • Route 502 maps to Recover; CPU/memory/disk/load maps to Perf; backup maps to BackupCheck; Harbor maps to HarborRepair; K3s/pod/image-pull maps to AwoooRepair.
  • Guardrail:
    • This bridge does not switch provider, call paid models, edit env, reload services, or read raw secrets.
    • problem key: agent99-sre-alert-inbox_awoooi_192.168.0.120_public_route_502
    • problem occurrences: 2

2026-07-10 M8 Agent99 Direct SRE Relay Contract

Telegram text polling is not enough for SLA-grade SRE automation.

  • 99-side relay:
    • Script: C:\Wooo\Agent99\bin\agent99-sre-alert-relay.ps1.
    • Scheduled task: Wooo-Agent99-SRE-Alert-Relay.
    • Default prefix: http://+:8787/agent99/sre-alert/.
    • Accepted sources: local and 192.168.0.* by default.
    • Optional token env: AGENT99_SRE_RELAY_TOKEN.
  • Production producer:
    • k8s/awoooi-prod/04-configmap.yaml sets AGENT99_SRE_ALERT_RELAY_URL=http://192.168.0.99:8787/agent99/sre-alert/.
    • AGENT99_SRE_ALERT_RELAY_TOKEN is optional and must come from Secret/env only.
    • API bridge uses relay first, then mounted file inbox fallback.
  • Runtime behavior:
    • Relay validates POST + JSON, writes into C:\Wooo\Agent99\alerts\incoming, and triggers agent99-sre-alert-inbox.ps1 -RunNow.
    • SRE Alert Inbox resolves mode and queues only allowlisted work.
    • Control Loop writes evidence, Telegram completion, problem counts, KM, and RAG import records.
    • Relay must not execute arbitrary shell, read secrets, switch providers, reload services, or perform destructive actions.

2026-07-09 Windows Update Policy Receipt

99 is configured to prevent surprise Windows Update reboots.

  • Evidence: C:\Wooo\Agent99\evidence\agent99-windows-update-policy-20260709-173933.json.
  • NoAutoRebootWithLoggedOnUsers=1.
  • AlwaysAutoRebootAtScheduledTime=0.
  • AUOptions=3.
  • ActiveHoursStart=0, ActiveHoursEnd=23.
  • gpupdateOk=True.

2026-07-09 M3 Self-Health Receipt

Agent99 now monitors its own automation loop.

  • Scheduled task: Wooo-Agent99-Self-Health.
  • Trigger interval: every 5 minutes.
  • Task Scheduler verification:
    • LastTaskResult=0.
    • Latest triggered evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-175218.json.
    • evidenceAfterTrigger=True.
  • Self-check coverage:
    • Startup recovery task, heartbeat task, performance guard task, and self-health task.
    • Evidence freshness for agent99-Perf-*.json, agent99-Status-*.json, agent99-SelfCheck-*.json, and agent99-Recover-*.json.
    • Latest performance evidence must include all five hosts: 110, 112, 120, 121, and 188.
    • Recent attempted Telegram delivery must be successful through direct env credentials or the 110 relay; a quiet interval with no alert attempts is OK and must not create a false critical self-check.
  • Latest result:
    • selfHealth.severity=ok.
    • taskFailures=0.
    • performanceSeverity=ok.
    • telegramSeverity=ok.
    • tgSent=3, relayOk=3.

2026-07-09 M4 110 Load Auto-Reduction Receipt

Agent99 now handles the 110 CPU/load pressure pattern found after reboot.

  • Incident findings:
    • The high-load source was not the runner.
    • Normal high consumers included Gitea and StockPlatform Postgres/API.
    • Backup load came from /backup/scripts/backup-awoooi.sh, gzip, and remote pg_dump.
    • A random alpine "tar czf - /data" container named mystifying_napier was consuming CPU and logging compressed binary output.
  • Manual controlled action applied once:
    • Backup PIDs were set to nice 15 / ionice best-effort 7.
    • mystifying_napier was stopped and retained as a dead container for evidence.
  • Automation added to performance.loadShedding.actions:
    • stop-110-runaway-alpine-tar-data: warning-or-critical action for allowlisted runaway alpine tar czf - /data containers.
    • renice-110-awoooi-backup-pressure: warning-or-critical action for backup/gzip/remote-pg-dump pressure.
    • pause-110-ci-runner-pressure: critical-only action for runner pressure, unchanged.
  • Verification:
    • Evidence: C:\Wooo\Agent99\evidence\agent99-Perf-20260709-180704.json.
    • 110: severity OK, load-per-core 0.67, memory available 62.02%, disk used 80%.
    • 112, 120, 121, and 188: severity OK.
    • perfCritical=0, perfWarning=0.
    • Telegram relay evidence: tgSent=2, relayOk=2.

2026-07-09 M5 Backup Health Receipt

Agent99 now monitors backup freshness and backup-job contracts.

  • Scheduled task: Wooo-Agent99-Backup-Health.
  • Trigger interval: every 15 minutes.
  • Task Scheduler verification:
    • LastTaskResult=0.
    • Latest triggered evidence: C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260709-190617.json.
    • evidenceAfterTrigger=True.
  • BackupCheck coverage:
    • 13 snapshot targets: awoooi, gitea, harbor, momo, langfuse, monitoring, signoz, clawbot, open-webui, sentry, ai-artifacts, configs, and public-routes.
    • 7 file/status checks: backup.log, backup-status.log, legacy status.log, legacy cron.log, integrity check status, restore drill status, and configs last-status JSON.
    • 6 cron contract patterns: backup-all.sh, backup-awoooi-frequent.sh, backup-health-textfile-exporter.py, check-backup-integrity.sh, sync-offsite-backups.sh --mode status, and backup-status.sh.
  • Latest result:
    • backupHealth.severity=ok.
    • targetCritical=0, targetWarning=0.
    • fileCritical=0, fileWarning=0.
    • cronCritical=0, cronWarning=0.
    • tgSent=2, relayOk=2.
  • Agent self-health now includes backup monitoring:
    • Evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-190714.json.
    • Wooo-Agent99-Backup-Health: OK, LastTaskResult=0.
    • Backup evidence freshness: OK, latest file agent99-BackupCheck-20260709-190617.json.

2026-07-09 M6 Alert-to-Remediation Receipt

Agent99 now treats alerts as action triggers, not as the final output.

  • Telegram relay bug fixed:
    • The relay sender now decodes the message argument from sys.argv[2], so Telegram receives the alert text instead of the Python relay source.
    • Info/start/complete events are suppressed by default: alertAllOperations=false, alertInfoEvents=false.
    • Duplicate warning/critical events are deduped for 30 minutes using stable event keys.
  • Controlled remediation behavior:
    • Perf -ControlledApply suppresses immediate performance spam, runs allowlisted load-shedding/remediation first, then alerts the controlled remediation result.
    • If no remediation applies, Agent99 still alerts the original warning/critical performance issue.
    • 188 disk pressure now has an allowlisted action: prune-188-docker-build-cache-disk-pressure.
    • The 188 prune action uses timeoutSeconds=240 and a post-apply verifier: postVerifyMaxDiskUsedPercent=89.
    • A command timeout is not reported as final critical when the post-verifier proves the host is back under the disk threshold.
  • False critical prevention:
    • Self-health no longer treats "no recent alerts were needed" as Telegram failure.
    • Telegram health is critical only when recent alert attempts exist and delivery failed.
    • Telegram delivery health uses a 60-minute lookback across recent evidence files, not only the newest file per evidence family.
    • Transient performance warnings are no longer promoted to self-health critical; only missing or critical host performance evidence is critical.
  • Backup monitor false critical prevention:
    • Backup snapshot readback now supports snapshotReadTimeoutSeconds.
    • 99 live config sets snapshotReadTimeoutSeconds=90.
    • Harbor backup timeout at 22:10 was verified as a readback timeout, not a missing backup: /backup/harbor/snapshots had latest metadata from 2026-07-09 17:48 Asia/Taipei and remained within the 1800-minute SLA.
  • Evidence hygiene:
    • Harbor status readback no longer stores raw docker inspect JSON.
    • Agent99 now records only selected Harbor health fields and sets inspectOutput=[redacted: raw docker inspect omitted].
    • Existing Status evidence on 99 was sanitized: 77 files checked; inspectOutputNotRedactedCount=0; unsafeHarborFieldCount=0.
  • Runtime readback after deployment:
    • Performance evidence: C:\Wooo\Agent99\evidence\agent99-Perf-20260709-220718.json.
    • Self-health evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-220829.json, selfHealth.severity=ok.
    • Status evidence: C:\Wooo\Agent99\evidence\agent99-Status-20260709-221147.json.
    • Host reachability: 110, 112, 120, 121, and 188 all ping OK.
    • Public routes: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502.
    • Harbor: core, Redis, and registry healthy with redacted evidence.
    • AWOOOI: deployReady=true, no image pull failure, no crash failure.
    • Backup evidence after timeout fix: C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260709-221759.json, backupHealth.severity=ok, all target/file/cron critical and warning counts are 0.
    • Final self-health evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-222236.json, selfHealth.severity=ok, taskFailures=0, performanceSeverity=ok, telegramSeverity=ok.

2026-07-09 M7 Visible Control Center Receipt

Agent99 is now visible and operable on the 99 Windows desktop instead of being only background scheduled tasks.

  • New mode:
    • ControlTick.
    • Runs the Agent99 visible control loop.
    • Updates dashboard HTML/JSON.
    • Processes allowlisted operator command queue files.
    • Writes evidence and alerts operator command results to Telegram.
  • Scheduled task:
    • Wooo-Agent99-Control-Loop.
    • Trigger interval: every 1 minute.
    • Self-health now checks this task.
  • Desktop shortcuts created:
    • C:\Users\Public\Desktop\Agent99 Control Center.lnk.
    • C:\Users\Public\Desktop\Agent99 Queue.lnk.
    • C:\Users\Public\Desktop\Agent99 Evidence.lnk.
    • D:\Desktop\Agent99 Control Center.lnk.
    • D:\Desktop\Agent99 Queue.lnk.
    • D:\Desktop\Agent99 Evidence.lnk.
  • Dashboard paths:
    • HTML: C:\Wooo\Agent99\dashboard\index.html.
    • JSON: C:\Wooo\Agent99\dashboard\agent99-control-center.latest.json.
  • Queue processor:
    • Input: C:\Wooo\Agent99\queue\*.json.
    • Processed output: C:\Wooo\Agent99\queue\processed\processed-<id>.json.
    • Failed output: C:\Wooo\Agent99\queue\failed\failed-<id>.json.
    • Allowed modes: Status, Recover, StartVMs, PublicSmoke, HarborRepair, AwoooRepair, Perf, LoadShed, SelfCheck, BackupCheck.
    • Operator command results are Telegram-alerted even when info/start/complete noise is suppressed.
  • Verification:
    • Control evidence: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-223309.json.
    • Processed command: codex-visible-publicsmoke-20260709-223308.
    • Processed result: ok=true, exitCode=0, durationSeconds=2.2.
    • Command evidence: C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-223309.json.
    • Public smoke: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502.
    • Telegram result: operator_command_result, severity=info, sent=true, relayOk=true.
    • Self-health evidence after control-loop install: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-223340.json, selfHealth.severity=ok, Wooo-Agent99-Control-Loop=ok.
    • Latest performance evidence: C:\Wooo\Agent99\evidence\agent99-Perf-20260709-223419.json, perfCritical=0, perfWarning=0.

2026-07-09 M8 Operator Instruction Intake Receipt

Agent99 now accepts operator text instructions and turns them into allowlisted queue commands.

  • New submit tool:
    • C:\Wooo\Agent99\bin\agent99-submit-request.ps1.
    • Interactive launcher: C:\Wooo\Agent99\agent99-submit-request.cmd.
    • Desktop shortcut: Agent99 Submit Request.lnk.
  • Request flow:
    • Operator enters one text instruction.
    • Agent99 writes the original request to C:\Wooo\Agent99\requests.
    • Agent99 resolves the request into an allowlisted mode.
    • Agent99 writes a queue command to C:\Wooo\Agent99\queue.
    • ControlTick processes the command, writes evidence, and sends Telegram operator_command_result.
  • Current allowlisted instruction targets:
    • website / route / 502 checks -> PublicSmoke.
    • reboot / restore / repair -> Recover.
    • VM / VMware start -> StartVMs.
    • CPU / load / memory / disk / performance -> Perf.
    • backup / snapshot / cron -> BackupCheck.
    • self / agent / health -> SelfCheck.
    • Harbor / registry -> HarborRepair.
    • AWOOOI / K3s / pod -> AwoooRepair.
  • Verification:
    • Test instruction: check all websites are normal equivalent in Traditional Chinese.
    • Request ID: request-20260709-224133-0dc045fe.
    • Resolved mode: PublicSmoke.
    • Processed result: ok=true, exitCode=0, durationSeconds=1.9.
    • Evidence: C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-224134.json.
    • Public smoke: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502.
    • Telegram result: operator_command_result, severity=info, sent=true, relayOk=true.
Reference in New Issue View Git Blame Copy Permalink