wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity
Files
b18a6c039e1837bdcbfc33e227103ff75eaa5574
awoooi/AGENT99-AI-WORKSTATION-SOP.md

43 KiB
Raw Blame History

Agent99 AI Workstation SOP

Last updated: 2026-07-11

Canonical enterprise scope and work order:

  • docs/workplans/2026-07-10-agent99-enterprise-ai-automation-master-plan.md
  • docs/operations/agent99-enterprise-ai-automation-work-items.snapshot.json

This SOP defines how Agent99 operates. The canonical work ledger defines what remains incomplete across all hosts, products, sites, services, tools and packages. A transport or command exitCode=0 never means an incident is resolved; the mode-specific verifier, source-event resolution and callback receipt are required.

Objective

192.168.0.99 is the AI Agent workstation and local operations control node.

Host 99 is not the only availability observer. An external watcher and maintenance surface must remain available when 99 or the entire local failure domain is down.

2026-07-10 P0-001 Outcome Truth Contract

  • exitCode=0 means only that the PowerShell transport completed.
  • A command is resolved only when its mode-specific post-condition verifier passes, its evidence is fresh and mode-matched, and any alert-source event has an explicit resolved readback.
  • Other legal outcomes are candidate_ready, verifying, degraded, blocked, and failed.
  • Alert-originated work without sourceEventResolved=true remains verifying, even when the repair command and post-condition verifier succeeded.
  • problem-counts.json v2 preserves the legacy totalResolved field but labels it as possibly transport-derived. New trustworthy closure is counted separately in totalVerifiedResolved and per-problem verifiedResolved.
  • Bootstrap writes state\runtime-manifest.json with source revision, source/runtime SHA-256, content match, file count and mismatch count. Self-health is critical when this manifest is missing, invalid, stale by hash, or incomplete.
  • All .ps1 runtime files are normalized to UTF-8 BOM by bootstrap so Windows PowerShell 5 cannot corrupt Traditional Chinese strings during parsing.
  • Staging verification on 99 must pass -SelfTestOutcomeContract and runtimeMatched=true before production Agent99 files are replaced.

2026-07-11 P0-002 Actionable Alert Dispatch Contract

This source candidate is test-green but not yet deployed to the Windows 99 runtime.

  • Direct Alertmanager events, ai_automation_alert_card_v1, ACTION REQUIRED incident cards, and failed AWOOOI CI/CD cards use one sanitized Agent99 dispatcher.
  • A cold-start or reboot gate is kind=host_recovery, suggestedMode=Recover; it must never become a fabricated Kubernetes deployment restart.
  • A failed CI/CD card is kind=cicd_failure, suggestedMode=Status; Agent99 collects status/evidence while the source event remains open until a later successful CD/deploy receipt resolves it.
  • Every dispatcher result writes a public-safe agent99_dispatch_receipt with alert id, kind, transport, HTTP status, accepted, and inbox_triggered. Raw response bodies, relay paths, tokens, chat ids, and workflow URLs are not stored.
  • HTTP 202 alone is not success. The bridge reports dispatched only when the relay body confirms ok=true; an unaccepted receipt releases the single-flight lock so the event can retry.
  • Stateful verifier lanes may use sourceEventResolutionPolicy=mode_verifier_can_resolve. External-source lanes use external_source_receipt_required and cannot self-close from a local command exit.
  • Source/test receipt: the focused post-rebase suite passed 249 tests, and the expanded all-Telegram plus Agent99/RepairCandidate suite passed 262 tests after integrating the Telegram single-owner mainline; YAML, shell/Python syntax, runner pressure, document secret, backup-alert, and recovery-scorecard checks passed. Windows 99 package deployment, runtime manifest reconciliation, and exact live card replay remain required before this P0 can close.

It must act as:

  • Secretary: reminders, status summaries, evidence packets, operator updates.
  • Assistant: routine checks, file organization, browser/readback checks, alert routing.
  • Engineer: VM control, service recovery, deploy/readback verification, runbook execution.
  • Incident operator: reboot detection, recovery orchestration, Telegram alerting, SLA countdown.

The machine must be able to operate VMware, SSH to service hosts, run public HTTP readbacks, and produce evidence without depending on old chat context.

Agent99 Operating Modes

Observe mode:

  • Read host, VM, service, and website state.
  • Produce evidence only.
  • Default mode for scheduled checks.

Controlled apply mode:

  • Execute approved low-blast-radius actions.
  • Must write evidence before and after each action.
  • Requires explicit operator approval or scheduled startup recovery policy.

Break-glass mode:

  • Reserved for destructive, irreversible, secret-bearing, cost-bearing, or firewall/provider cutover actions.
  • Must not be hidden inside automatic tasks.

Required Local Capabilities On 99

Base runtime:

  • Windows PowerShell 5+ or PowerShell 7.
  • Windows OpenSSH client.
  • curl.exe.
  • VMware Workstation.
  • vmrun.exe.
  • Windows Task Scheduler.

Agent filesystem:

  • C:\Wooo\Agent99\bin
  • C:\Wooo\Agent99\config
  • C:\Wooo\Agent99\evidence
  • C:\Wooo\Agent99\logs
  • C:\Wooo\Agent99\queue
  • C:\Wooo\Agent99\requests
  • C:\Wooo\Agent99\alerts\incoming
  • C:\Wooo\Agent99\playbooks

Agent scripts:

  • agent99-bootstrap.ps1
  • agent99-control-plane.ps1
  • agent99-submit-request.ps1
  • agent99-telegram-inbox.ps1
  • agent99-sre-alert-inbox.ps1
  • host-reboot-scorecard.ps1
  • agent99.config.json

Command Ingress Without Codex Quota

Codex is not the runtime command plane. Agent99 must accept operator work through local 99 channels even when Codex is unavailable.

Supported command channels:

  • Local desktop: Agent99 Submit Request.lnk.
  • Local queue: write allowlisted command JSON to C:\Wooo\Agent99\queue.
  • AwoooI SRE Telegram group: send /agent99 ..., !agent99 ..., #agent99 ..., agent99 ..., or @agent99 ....
  • SRE war room alerts: write alert JSON to C:\Wooo\Agent99\alerts\incoming.
  • AwoooI SRE Telegram monitoring alerts: non-command messages that look like Alertmanager/FIRING/critical/warning/502/CPU/disk/Harbor/K3s/backup/host-down/provider-freshness alerts are auto-ingested.

Command processing requirements:

  • Every command resolves to an allowlisted mode only.
  • Website plus 502 plus recover/repair intent resolves to Recover, not passive smoke-only checking.
  • Alert-driven remediation uses controlled apply only for allowlisted modes such as Recover, HarborRepair, AwoooRepair, Perf, and LoadShed.
  • Every accepted command writes a request record, queue record, execution result, and evidence pointer.
  • Telegram messages must be human-readable: source, instruction, mode, result, and evidence path. Raw Python, raw stack trace, token, chat id, cookies, sessions, or .env must never be sent.
  • Telegram SRE group commands must be recorded with source=telegram-sre-war-room.
  • Telegram SRE monitoring alerts must be recorded with source=telegram-sre-war-room-monitoring.
  • Telegram SRE group matching uses allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE, so the command path is not limited to the default private chat id.
  • When a command originates from a matched Telegram group, completion replies are routed back to the same Telegram origin while evidence stores only telegram-origin, not the raw chat id.
  • After automated recovery completes, Agent99 must report the result back to Telegram so the project AI Agents and human operators know whether the abnormality is resolved.
  • Every queue execution must update problem counts, write a Playbook/KM record, and append a RAG-import JSONL line.
  • Operator command and remediation result Telegram reports are not deduped; each completed repair emits a completion report. Noisy warning/critical signals still use dedupe.

SRE Telegram command examples:

  • /agent99 檢查所有網站是否正常，若502就自動修復
  • /agent99 檢查110和188 CPU負載，必要時執行允許的降載
  • /agent99 檢查AWOOOI K3s與Harbor，若ImagePullBackOff就修復
  • /agent99 檢查備份健康狀態
  • /agent99 建立 provider freshness candidate，要求 provider window、last seen、direct ref 與 verifier readback

Monitoring alert auto-ingest examples:

  • Alertmanager FIRING critical public_route_502 service=awoooi host=192.168.0.120 HTTP 502
  • HostHighCpuLoad warning host=192.168.0.110
  • HarborRegistryDown critical host=192.168.0.110
  • KubePodCrashLooping critical namespace=awoooi-prod
  • provider_freshness_signal target=provider_freshness_signal freshness last_seen missing direct_ref required

Problem memory outputs:

  • Problem counters: C:\Wooo\Agent99\state\problem-counts.json.
  • Full event ledger: C:\Wooo\Agent99\playbooks\agent99-incident-records.jsonl.
  • Playbook/KM summary: C:\Wooo\Agent99\playbooks\agent99-incident-km.md.
  • RAG import feed: C:\Wooo\Agent99\playbooks\rag-import\agent99-incidents.jsonl.
  • Dashboard: C:\Wooo\Agent99\dashboard\index.html, section Problem Counts.

SRE War Room Alert Contract

AWOOOI SRE monitors and other AI Agent work windows can hand alerts to Agent99 by writing JSON files to:

C:\Wooo\Agent99\alerts\incoming

Minimum fields:

{
  "id": "alert-unique-id",
  "source": "awoooi-sre-war-room",
  "severity": "critical",
  "kind": "public_route_502",
  "service": "awoooi",
  "host": "192.168.0.120",
  "title": "AWOOOI public route returned 502",
  "message": "The public health route is returning 502.",
  "createdAt": "2026-07-09T22:50:00+08:00"
}

Routing rules:

  • 502, site_502, public_route_502, route_down, host_down, VM down, or reboot detected -> Recover.
  • CPU, memory, disk, performance pressure -> Perf with controlled apply.
  • Harbor or registry -> HarborRepair.
  • AWOOOI, K3s, image pull, pod, deployment -> AwoooRepair.
  • Backup, snapshot, restore drill, or cron -> BackupCheck.
  • Unknown actionable alert -> Status evidence first.

Alert processing evidence:

  • Inbox evidence: agent99-SreAlertInbox-*.json.
  • Queued request: C:\Wooo\Agent99\requests.
  • Execution evidence: agent99-Recover-*.json, agent99-Perf-*.json, or the resolved mode evidence file.

VM Control Requirements

Agent99 must be able to operate these VM hosts:

  • 110
  • 112
  • 120
  • 121
  • 188

Required actions:

  • Detect VM powered-off state.
  • Start missing VMs through vmrun.exe.
  • Detect VMware config lock symptoms.
  • Avoid deleting VM lock files unless VMware process state proves the VM is not running.
  • Prefer guest shutdown/restart first; use reset only when the guest is stuck and SLA requires recovery.

Engineering Recovery Requirements

Agent99 must support these engineering actions:

  • SSH readback to 110 and 120.
  • Harbor core / Redis / registry checks on 110.
  • K3s AWOOOI deployment and pod checks on 120.
  • Public route readback for user-visible websites.
  • Evidence capture after every action.
  • Telegram alert hooks.

Main Scheduled Automations

Startup recovery:

  • Trigger: 99 boot or operator logon.
  • Frequency: once at startup, then repeat during first recovery window.
  • Mode: controlled apply allowed for approved actions.
  • Purpose: start VMs, restore Harbor, restore K3s pods, verify public routes.

Heartbeat:

  • Trigger: every 5 minutes.
  • Mode: observe by default.
  • Purpose: detect host down, 502, service degraded, backup job missing, or stale freshness.

Daily evidence summary:

  • Trigger: daily.
  • Mode: observe.
  • Purpose: produce a short host/service/site/backup status packet.

Telegram Alert Classes

The agent must send alerts for:

  • host_down
  • host_reboot_detected
  • vm_not_running
  • vm_start_attempted
  • service_recovering
  • harbor_unhealthy
  • k3s_image_pull_failure
  • site_502
  • site_recovered
  • service_degraded
  • backup_missing
  • backup_failed
  • recovery_sla_risk
  • recovery_complete

Telegram credentials must be stored outside the repo, preferably in Windows Credential Manager or machine environment variables.

2026-07-09 Recovery Logic To Embed

When AWOOOI or other sites return 502:

  1. Do not only inspect nginx.
  2. Check 120 K3s workloads first.
  3. If AWOOOI pods are ImagePullBackOff, check 110 Harbor before deleting pods.
  4. If Harbor token endpoint returns 500 or pulls fail, inspect:
    • Docker engine.
    • Harbor Redis.
    • harbor-core.
    • registry.
    • registryctl.
  5. If harbor-core is not healthy:
    • Restart Redis and harbor-core.
    • If Docker/systemd hangs, execute controlled 110 guest restart via VMware.
  6. After harbor-core is healthy:
    • Delete only failed AWOOOI pods.
    • Wait for deployment readiness.
    • Verify public HTTP.

First Implementation Milestone

Milestone M1 is complete when:

  • Agent99 directory exists on 99.
  • Config file exists.
  • Startup and heartbeat scheduled tasks exist.
  • Scorecard can run in observe mode.
  • Recovery can run in controlled apply mode.
  • Evidence JSON is written.
  • No secrets are stored in repo files.

2026-07-09 M1 Receipt

M1 is operational on 99 as of 2026-07-09 16:49 Asia/Taipei.

  • C:\Wooo\Agent99 exists on WOOO-SUPER.
  • agent99-control-plane.ps1, host-reboot-scorecard.ps1, and task registration scripts are deployed.
  • agent99.config.json includes VMX selectors for 110, 188, 112, 120, and 121.
  • Wooo-Agent99-Heartbeat is registered and produced LastTaskResult=0.
  • Wooo-Agent99-Startup-Recovery is registered and produced LastTaskResult=0.
  • Latest heartbeat evidence showed Harbor healthy, AWOOOI deploy ready, and public routes non-502.
  • Latest recovery evidence showed all five configured VM hosts already reachable and no recovery action required.

Remaining M2 work:

  • Install Telegram credentials with environment variables or Windows Credential Manager, then verify real delivery.
  • Add backup-job freshness readback and alert classes.
  • Add Windows Update no-surprise-reboot policy verification.
  • Promote this into a long-running Windows service after the scheduled-task lane stays clean.

2026-07-09 M2 Performance Guard Receipt

Agent99 performance guard is operational on 99 as of 2026-07-09 17:15 Asia/Taipei.

  • Wooo-Agent99-Performance-Guard is registered in Windows Task Scheduler.
  • Trigger interval: every 1 minute.
  • Last manual verification: LastTaskResult=0.
  • Latest evidence: C:\Wooo\Agent99\evidence\agent99-Perf-20260709-171502.json.
  • Readback coverage:
    • 110: OK, load-per-core 0.35, memory available 62.93%, disk used 82%.
    • 112: OK, load-per-core 0.07, memory available 61.06%, disk used 47%.
    • 120: OK, load-per-core 0.05, memory available 91.76%, disk used 44%.
    • 121: OK, load-per-core 0.00, memory available 92.79%, disk used 27%.
    • 188: OK, load-per-core 0.08, memory available 90.19%, disk used 89%.
  • 99 SSH access was completed for:
    • 112 with user kali.
    • 188 with user ollama.
  • Load-shedding is controlled by allowlist only.
    • Current allowlisted action: pause vibework-dedicated-runner on 110 only when load_per_core_critical is present.
    • Disk or memory pressure only alerts; it does not trigger the CPU runner pause action.
  • Telegram audit is enabled in config and called for Agent99 start, warnings, critical events, load-shed attempts/results, and completion.
    • 99 local env aliases are supported for AGENT99_*, AWOOOI_TG_*, OPENCLAW_TG_*, TELEGRAM_*, and TG_*.
    • If 99 does not have local Telegram env values, Agent99 relays through the existing 110 stockplatform-v2-api-1 container and uses that container's existing TELEGRAM_BOT_TOKEN / TELEGRAM_CHAT_ID.
    • 2026-07-09 17:27 verification: Wooo-Agent99-Performance-Guard produced LastTaskResult=0, agent99-Perf-20260709-172712.json, tgSent=2, and relayOk=True.
    • No token or chat id is stored in repo files or Agent99 evidence.

2026-07-09 OpenClaw Receipt

OpenClaw is configured and now included in Agent99 evidence.

  • Source truth:
    • AWOOOI prod config: OPENCLAW_URL=http://192.168.0.188:8088.
    • 188 Ansible service target: OpenClaw on port 8088.
  • Runtime truth:
    • 192.168.0.188:8088/health returns status=healthy, version=6.0, service=ClawBot, environment=production, and telegram_bot=connected.
    • clawbot.service on 188 is active.
    • 192.168.0.188:8089 is not the current production endpoint.
  • Agent99 truth:
    • Latest status evidence: C:\Wooo\Agent99\evidence\agent99-Status-20260709-173813.json.
    • aiServices.openclaw: ok=True, status=healthy, url=http://192.168.0.188:8088/health.
    • Telegram relay evidence: tgSent=2, relayOk=2.

2026-07-09 M3 Command And Alert Ingress Receipt

Agent99 no longer depends on Codex as the only command channel.

  • Telegram command inbox deployed: C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1.
  • SRE alert inbox deployed: C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1.
  • Scheduled tasks registered:
    • Wooo-Agent99-Telegram-Inbox
    • Wooo-Agent99-SRE-Alert-Inbox
  • PowerShell parse readback: all five Agent99 scripts parsed successfully on 99.
  • Live config includes both scheduled tasks and freshness contracts.
  • Telegram inbox evidence: C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260709-230031.json.
    • ok=True
    • updatesSeen=0
    • processedCount=0
    • no Python code or token text in evidence.
  • SRE synthetic alert evidence: C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260709-230107.json.
    • input kind: public_route_502
    • source: awoooi-sre-war-room
    • resolved mode: Recover
    • queued count: 1
    • failed count: 0
    • controlExitCode=0
  • Recovery execution evidence: C:\Wooo\Agent99\evidence\agent99-Recover-20260709-230108.json.
    • public route failures: 0
    • host reachability failures: 0
    • Harbor core healthy: True
    • AWOOOI deploy ready: True
  • Queue processor evidence: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-230108.json.
    • processed count: 1
    • source: agent99-sre-alert-inbox
    • reason: sre_alert
    • mode: Recover
    • exit code: 0
    • Telegram relay: sent=True, relayOk=True
    • evidence contains no Python code.
  • Self-health evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-230223.json.
    • severity: ok
    • task failures: 0
    • evidence critical: 0
    • evidence warning: 0
    • Telegram inbox task: OK
    • SRE alert inbox task: OK
  • Dashboard JSON: C:\Wooo\Agent99\dashboard\agent99-control-center.latest.json.
    • queue pending: 0
    • alert pending: 0

2026-07-09 M4 SRE Telegram Command And Problem Memory Receipt

Agent99 now accepts AwoooI SRE Telegram-group commands as first-class work orders.

  • Telegram SRE command source mapping:
    • source: telegram-sre-war-room
    • request path: C:\Wooo\Agent99\requests
    • queue path: C:\Wooo\Agent99\queue
    • command result event: operator_command_result
  • Synthetic SRE Telegram command verification:
    • instruction: 檢查所有網站是否恢復正常，若502就自動修復
    • request id: request-20260709-231157-d53f53d8
    • resolved mode: Recover
    • control exit code: 0
    • evidence: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-231158.json
    • source: telegram-sre-war-room
    • processed mode: Recover
    • processed ok: True
    • processed exit code: 0
    • Telegram relay: sent 1
    • Telegram evidence contains no Python code.
  • Problem memory readback:
    • problem key: telegram-sre-war-room_agent99_operator_instruction
    • occurrences: 1
    • resolved: 1
    • failed: 0
    • stats path: C:\Wooo\Agent99\state\problem-counts.json
    • KM exists: True
    • RAG feed exists: True
    • dashboard problem events: 1
    • dashboard resolved: 1
    • dashboard failed: 0
  • Post-check:
    • Telegram inbox evidence: ok=True, updatesSeen=0, processedCount=0
    • Self-health evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-231255.json
    • self-health severity: ok
    • task failures: 0
    • queue pending: 0
    • alert pending: 0
  • SRE group routing hardening:
    • Existing relay default chat readback was a private chat, so group commands must not depend only on TELEGRAM_CHAT_ID.
    • Live config was updated with allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE.
    • Telegram inbox accepts matched group commands and carries a redacted telegram-origin reply route for completion reports.

2026-07-09 M5 SRE Telegram Monitoring Alert Auto-Ingest Receipt

Agent99 now ingests monitoring alerts from the AwoooI SRE Telegram group, not only /agent99 commands.

  • Runtime switch:
    • autoIngestMonitoringAlerts=True
    • allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE
  • Encoding hardening:
    • Runtime PowerShell scripts are ASCII-only.
    • Chinese alert keyword support uses Unicode codepoint checks instead of raw Chinese strings, avoiding Windows PowerShell 5 parser failures.
  • Synthetic SRE Telegram monitoring alert:
    • text: Alertmanager FIRING critical public_route_502 service=awoooi host=192.168.0.120 HTTP 502
    • no /agent99 prefix.
    • Telegram inbox evidence: C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260709-233230.json
    • inboxOk=True
    • alertedCount=1
  • SRE alert bridge evidence:
    • C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260709-233231.json
    • queued count: 1
    • failed count: 0
  • Control/recovery evidence:
    • C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-233232.json
    • source: telegram-sre-war-room-monitoring
    • kind: public_route_502
    • mode: Recover
    • processed ok: True
    • exit code: 0
    • Telegram attempts: 1
    • Telegram sent: 1
    • evidence contains no Python code.
  • Problem memory:
    • total events before: 2
    • total events after: 3
    • total resolved after: 3
    • total failed after: 0

2026-07-09 M6 Provider Freshness Signal Receipt

Provider freshness alerts must not be treated as passive route health.

  • Gap found:
    • Telegram Inbox and SRE Alert Inbox were running, but provider_freshness_signal was not present in Agent99 alerts, queue, or execution evidence after 2026-07-09 23:30 Asia/Taipei.
    • Only SelfCheck evidenceFreshness fields matched the word freshness, so the original alert was not processed as an actionable provider freshness signal.
  • Runtime behavior added:
    • Telegram auto-ingest recognizes provider_freshness_signal, source_provider_freshness_triage, raw_signal_normalized, controlled_runtime_candidate, freshness, last seen, last_seen, and ingestion.
    • SRE Alert Inbox routes freshness signals to ProviderFreshness.
    • ProviderFreshness creates a read-only candidate under C:\Wooo\Agent99\state\provider-freshness-candidates.
    • Candidate evidence must require provider window, last seen, direct reference, and verifier readback.
    • Guardrail remains: no provider switch, no paid model call, no env change, no reload.

2026-07-09 M7 AWOOI API To Agent99 Alert Bridge

Agent99 must not depend only on Telegram text polling for AwoooI SRE alerts.

  • Producer-side bridge:
    • AWOOI API POST /api/v1/webhooks/alertmanager now schedules a fail-open Agent99 SRE bridge after alert normalization and fingerprinting.
    • The bridge POSTs the same Agent99 alert contract to AGENT99_SRE_ALERT_RELAY_URL when configured.
    • If relay URL is empty, the bridge writes the contract expected by C:\Wooo\Agent99\alerts\incoming when AGENT99_SRE_ALERT_INBOX_PATH is configured.
    • Empty relay/path means bridge disabled; Alertmanager response must not fail or retry-storm because Agent99 inbox is unavailable.
  • Safety:
    • Payloads are sanitized before writing.
    • Labels with token, secret, password, authorization, cookie, session, or private-key names are redacted.
    • Provider freshness signals map to ProviderFreshness.
    • Route 502 maps to Recover; CPU/memory/disk/load maps to Perf; backup maps to BackupCheck; Harbor maps to HarborRepair; K3s/pod/image-pull maps to AwoooRepair.
  • Guardrail:
    • This bridge does not switch provider, call paid models, edit env, reload services, or read raw secrets.
    • problem key: agent99-sre-alert-inbox_awoooi_192.168.0.120_public_route_502
    • problem occurrences: 2

2026-07-10 M8 Agent99 Direct SRE Relay Contract

Telegram text polling is not enough for SLA-grade SRE automation.

  • 99-side relay:
    • Script: C:\Wooo\Agent99\bin\agent99-sre-alert-relay.ps1.
    • Scheduled task: Wooo-Agent99-SRE-Alert-Relay.
    • Default prefix: http://+:8787/agent99/sre-alert/.
    • Accepted sources: local and 192.168.0.* by default.
    • Optional token env: AGENT99_SRE_RELAY_TOKEN.
  • Production producer:
    • k8s/awoooi-prod/04-configmap.yaml sets AGENT99_SRE_ALERT_RELAY_URL=http://192.168.0.99:8787/agent99/sre-alert/.
    • AGENT99_SRE_ALERT_RELAY_TOKEN is optional and must come from Secret/env only.
    • API bridge uses relay first, then mounted file inbox fallback.
  • Runtime behavior:
    • Relay validates POST + JSON, writes into C:\Wooo\Agent99\alerts\incoming, and triggers agent99-sre-alert-inbox.ps1 -RunNow.
    • SRE Alert Inbox resolves mode and queues only allowlisted work.
    • Control Loop writes evidence, Telegram completion, problem counts, KM, and RAG import records.
    • Relay must not execute arbitrary shell, read secrets, switch providers, reload services, or perform destructive actions.

2026-07-09 Windows Update Policy Receipt

99 is configured to prevent surprise Windows Update reboots.

  • Evidence: C:\Wooo\Agent99\evidence\agent99-windows-update-policy-20260709-173933.json.
  • NoAutoRebootWithLoggedOnUsers=1.
  • AlwaysAutoRebootAtScheduledTime=0.
  • AUOptions=3.
  • ActiveHoursStart=0, ActiveHoursEnd=23.
  • gpupdateOk=True.

2026-07-09 M3 Self-Health Receipt

Agent99 now monitors its own automation loop.

  • Scheduled task: Wooo-Agent99-Self-Health.
  • Trigger interval: every 5 minutes.
  • Task Scheduler verification:
    • LastTaskResult=0.
    • Latest triggered evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-175218.json.
    • evidenceAfterTrigger=True.
  • Self-check coverage:
    • Startup recovery task, heartbeat task, performance guard task, and self-health task.
    • Evidence freshness for agent99-Perf-*.json, agent99-Status-*.json, agent99-SelfCheck-*.json, and agent99-Recover-*.json.
    • Latest performance evidence must include all five hosts: 110, 112, 120, 121, and 188.
    • Recent attempted Telegram delivery must be successful through direct env credentials or the 110 relay; a quiet interval with no alert attempts is OK and must not create a false critical self-check.
  • Latest result:
    • selfHealth.severity=ok.
    • taskFailures=0.
    • performanceSeverity=ok.
    • telegramSeverity=ok.
    • tgSent=3, relayOk=3.

2026-07-09 M4 110 Load Auto-Reduction Receipt

Agent99 now handles the 110 CPU/load pressure pattern found after reboot.

  • Incident findings:
    • The high-load source was not the runner.
    • Normal high consumers included Gitea and StockPlatform Postgres/API.
    • Backup load came from /backup/scripts/backup-awoooi.sh, gzip, and remote pg_dump.
    • A random alpine "tar czf - /data" container named mystifying_napier was consuming CPU and logging compressed binary output.
  • Manual controlled action applied once:
    • Backup PIDs were set to nice 15 / ionice best-effort 7.
    • mystifying_napier was stopped and retained as a dead container for evidence.
  • Automation added to performance.loadShedding.actions:
    • stop-110-runaway-alpine-tar-data: warning-or-critical action for allowlisted runaway alpine tar czf - /data containers.
    • renice-110-awoooi-backup-pressure: warning-or-critical action for backup/gzip/remote-pg-dump pressure.
    • pause-110-ci-runner-pressure: critical-only action for runner pressure, unchanged.
  • Verification:
    • Evidence: C:\Wooo\Agent99\evidence\agent99-Perf-20260709-180704.json.
    • 110: severity OK, load-per-core 0.67, memory available 62.02%, disk used 80%.
    • 112, 120, 121, and 188: severity OK.
    • perfCritical=0, perfWarning=0.
    • Telegram relay evidence: tgSent=2, relayOk=2.

2026-07-09 M5 Backup Health Receipt

Agent99 now monitors backup freshness and backup-job contracts.

  • Scheduled task: Wooo-Agent99-Backup-Health.
  • Trigger interval: every 15 minutes.
  • Task Scheduler verification:
    • LastTaskResult=0.
    • Latest triggered evidence: C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260709-190617.json.
    • evidenceAfterTrigger=True.
  • BackupCheck coverage:
    • 13 snapshot targets: awoooi, gitea, harbor, momo, langfuse, monitoring, signoz, clawbot, open-webui, sentry, ai-artifacts, configs, and public-routes.
    • 7 file/status checks: backup.log, backup-status.log, legacy status.log, legacy cron.log, integrity check status, restore drill status, and configs last-status JSON.
    • 6 cron contract patterns: backup-all.sh, backup-awoooi-frequent.sh, backup-health-textfile-exporter.py, check-backup-integrity.sh, sync-offsite-backups.sh --mode status, and backup-status.sh.
  • Latest result:
    • backupHealth.severity=ok.
    • targetCritical=0, targetWarning=0.
    • fileCritical=0, fileWarning=0.
    • cronCritical=0, cronWarning=0.
    • tgSent=2, relayOk=2.
  • Agent self-health now includes backup monitoring:
    • Evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-190714.json.
    • Wooo-Agent99-Backup-Health: OK, LastTaskResult=0.
    • Backup evidence freshness: OK, latest file agent99-BackupCheck-20260709-190617.json.

2026-07-09 M6 Alert-to-Remediation Receipt

Agent99 now treats alerts as action triggers, not as the final output.

  • Telegram relay bug fixed:
    • The relay sender now decodes the message argument from sys.argv[2], so Telegram receives the alert text instead of the Python relay source.
    • Info/start/complete events are suppressed by default: alertAllOperations=false, alertInfoEvents=false.
    • Duplicate warning/critical events are deduped for 30 minutes using stable event keys.
  • Controlled remediation behavior:
    • Perf -ControlledApply suppresses immediate performance spam, runs allowlisted load-shedding/remediation first, then alerts the controlled remediation result.
    • If no remediation applies, Agent99 still alerts the original warning/critical performance issue.
    • 188 disk pressure now has an allowlisted action: prune-188-docker-build-cache-disk-pressure.
    • The 188 prune action uses timeoutSeconds=240 and a post-apply verifier: postVerifyMaxDiskUsedPercent=89.
    • A command timeout is not reported as final critical when the post-verifier proves the host is back under the disk threshold.
  • False critical prevention:
    • Self-health no longer treats "no recent alerts were needed" as Telegram failure.
    • Telegram health is critical only when recent alert attempts exist and delivery failed.
    • Telegram delivery health uses a 60-minute lookback across recent evidence files, not only the newest file per evidence family.
    • Transient performance warnings are no longer promoted to self-health critical; only missing or critical host performance evidence is critical.
  • Backup monitor false critical prevention:
    • Backup snapshot readback now supports snapshotReadTimeoutSeconds.
    • 99 live config sets snapshotReadTimeoutSeconds=90.
    • Harbor backup timeout at 22:10 was verified as a readback timeout, not a missing backup: /backup/harbor/snapshots had latest metadata from 2026-07-09 17:48 Asia/Taipei and remained within the 1800-minute SLA.
  • Evidence hygiene:
    • Harbor status readback no longer stores raw docker inspect JSON.
    • Agent99 now records only selected Harbor health fields and sets inspectOutput=[redacted: raw docker inspect omitted].
    • Existing Status evidence on 99 was sanitized: 77 files checked; inspectOutputNotRedactedCount=0; unsafeHarborFieldCount=0.
  • Runtime readback after deployment:
    • Performance evidence: C:\Wooo\Agent99\evidence\agent99-Perf-20260709-220718.json.
    • Self-health evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-220829.json, selfHealth.severity=ok.
    • Status evidence: C:\Wooo\Agent99\evidence\agent99-Status-20260709-221147.json.
    • Host reachability: 110, 112, 120, 121, and 188 all ping OK.
    • Public routes: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502.
    • Harbor: core, Redis, and registry healthy with redacted evidence.
    • AWOOOI: deployReady=true, no image pull failure, no crash failure.
    • Backup evidence after timeout fix: C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260709-221759.json, backupHealth.severity=ok, all target/file/cron critical and warning counts are 0.
    • Final self-health evidence: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-222236.json, selfHealth.severity=ok, taskFailures=0, performanceSeverity=ok, telegramSeverity=ok.

2026-07-09 M7 Visible Control Center Receipt

Agent99 is now visible and operable on the 99 Windows desktop instead of being only background scheduled tasks.

  • New mode:
    • ControlTick.
    • Runs the Agent99 visible control loop.
    • Updates dashboard HTML/JSON.
    • Processes allowlisted operator command queue files.
    • Writes evidence and alerts operator command results to Telegram.
  • Scheduled task:
    • Wooo-Agent99-Control-Loop.
    • Trigger interval: every 1 minute.
    • Self-health now checks this task.
  • Desktop shortcuts created:
    • C:\Users\Public\Desktop\Agent99 Control Center.lnk.
    • C:\Users\Public\Desktop\Agent99 Queue.lnk.
    • C:\Users\Public\Desktop\Agent99 Evidence.lnk.
    • D:\Desktop\Agent99 Control Center.lnk.
    • D:\Desktop\Agent99 Queue.lnk.
    • D:\Desktop\Agent99 Evidence.lnk.
  • Dashboard paths:
    • HTML: C:\Wooo\Agent99\dashboard\index.html.
    • JSON: C:\Wooo\Agent99\dashboard\agent99-control-center.latest.json.
  • Queue processor:
    • Input: C:\Wooo\Agent99\queue\*.json.
    • Processed output: C:\Wooo\Agent99\queue\processed\processed-<id>.json.
    • Failed output: C:\Wooo\Agent99\queue\failed\failed-<id>.json.
    • Allowed modes: Status, Recover, StartVMs, PublicSmoke, HarborRepair, AwoooRepair, Perf, LoadShed, SelfCheck, BackupCheck.
    • Operator command results are Telegram-alerted even when info/start/complete noise is suppressed.
  • Verification:
    • Control evidence: C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-223309.json.
    • Processed command: codex-visible-publicsmoke-20260709-223308.
    • Processed result: ok=true, exitCode=0, durationSeconds=2.2.
    • Command evidence: C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-223309.json.
    • Public smoke: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502.
    • Telegram result: operator_command_result, severity=info, sent=true, relayOk=true.
    • Self-health evidence after control-loop install: C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-223340.json, selfHealth.severity=ok, Wooo-Agent99-Control-Loop=ok.
    • Latest performance evidence: C:\Wooo\Agent99\evidence\agent99-Perf-20260709-223419.json, perfCritical=0, perfWarning=0.

2026-07-09 M8 Operator Instruction Intake Receipt

Agent99 now accepts operator text instructions and turns them into allowlisted queue commands.

  • New submit tool:
    • C:\Wooo\Agent99\bin\agent99-submit-request.ps1.
    • Interactive launcher: C:\Wooo\Agent99\agent99-submit-request.cmd.
    • Desktop shortcut: Agent99 Submit Request.lnk.
  • Request flow:
    • Operator enters one text instruction.
    • Agent99 writes the original request to C:\Wooo\Agent99\requests.
    • Agent99 resolves the request into an allowlisted mode.
    • Agent99 writes a queue command to C:\Wooo\Agent99\queue.
    • ControlTick processes the command, writes evidence, and sends Telegram operator_command_result.
  • Current allowlisted instruction targets:
    • website / route / 502 checks -> PublicSmoke.
    • reboot / restore / repair -> Recover.
    • VM / VMware start -> StartVMs.
    • CPU / load / memory / disk / performance -> Perf.
    • backup / snapshot / cron -> BackupCheck.
    • self / agent / health -> SelfCheck.
    • Harbor / registry -> HarborRepair.
    • AWOOOI / K3s / pod -> AwoooRepair.
  • Verification:
    • Test instruction: check all websites are normal equivalent in Traditional Chinese.
    • Request ID: request-20260709-224133-0dc045fe.
    • Resolved mode: PublicSmoke.
    • Processed result: ok=true, exitCode=0, durationSeconds=1.9.
    • Evidence: C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-224134.json.
    • Public smoke: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502.
    • Telegram result: operator_command_result, severity=info, sent=true, relayOk=true.

2026-07-10 M9 Source-Trusted Transport And Verified Remediation Receipt

Agent99 now uses a source-trusted deployment path, a dedicated bounded SSH transport, and verifier-authoritative remediation outcomes.

  • Source authority and deployment:
    • Gitea commit SHA is the source revision; a local scratch checkout or a Windows file timestamp is not deployment truth.
    • agent99-deploy.ps1 validates PowerShell parse, Agent99 contracts, synthetic checks, staging content, backup, atomic promotion, task registration, and rollback.
    • C:\Wooo\Agent99\state\runtime-manifest.json.sourceRevision must match the deployed Gitea SHA. A successful source test without this runtime readback is not a completed deployment.
  • Dedicated transport:
    • Private identity: C:\Wooo\Agent99\keys\agent99_ed25519; it remains on 99, is ACL-restricted, and is never copied into evidence, Telegram, logs, or the repository.
    • Public-key authorization is limited with from="192.168.0.99",restrict on 110 user wooo and 112 user kali.
    • Route map: 110 and 112 direct; 120, 121, and 188 prefer 110 as the jump host.
    • Live public-key fingerprint receipt: SHA256:4ccZZSe0buoWyivi2mfTMXE9LZfCmFBPr/evCzZ0nzA.
  • Automatic recovery:
    • Agent99 persists boot identity in C:\Wooo\Agent99\state\boot-state.json.
    • A changed boot observation queues one Recover request only; duplicate requests are suppressed by single-flight state and a 10-minute cooldown.
    • recovery_slo_result records the recovery window. It may claim success only after a real reboot drill proves all required hosts, services, routes, and freshness checks inside 10 minutes.
  • Verifier-authoritative remediation:
    • When an action configures postVerify, command success alone is insufficient. The verifier result is the effective action result.
    • A failed verifier creates only the 10-minute failure backoff; it cannot create a success cooldown.
    • policyVersion migration updates stale live policies and prevents an old command or marker from surviving a new source deployment.
  • Bounded SSH process lifecycle:
    • Every SSH call has a timeout; timeout cleanup terminates the SSH process tree and verifies that the client exited.
    • Every Agent99 process acquires the shared C:\Wooo\Agent99\state\ssh-transport.lock with an exclusive file handle before starting SSH. This serializes scheduled tasks and operator runs across Windows sessions; the OS releases the handle if a process exits.
    • Lock acquisition waits at most 90 seconds. Timeout evidence is ssh_transport_lock_timeout; it must not start another SSH client or be reported as a service outage.
    • Each performance row records transportSerialized and transportLockWaitMs, so lock behavior is verifier evidence rather than an implementation-only claim.
    • Production scheduled tasks run as the local WOOO S4U principal. An interactive Administrator run that cannot read keys\agent99_ed25519 is an identity mismatch, not host downtime; do not broaden the private-key ACL just to make an operator shell pass.
    • A long-running SRE alert relay remains healthy while Wooo-Agent99-SRE-Alert-Relay is Running/ok; stale process-start evidence is annotated as running_task_supersedes_stale_start_evidence and does not page.
    • Before controlled Perf or LoadShed, the stale-process guard only considers Agent99 allowlisted targets using BatchMode=yes and explicitly excludes -N, -L, -R, and -D tunnels.
    • A process is eligible after 15 minutes with a missing parent, or after the 60-minute hard limit. Evidence records selected process metadata but never the command line.
  • Live receipts:
    • Key provisioning: C:\Wooo\Agent99\evidence\agent99-ssh-key-provision-final.json.
    • False-success marker correction: C:\Wooo\Agent99\evidence\agent99-load-shed-marker-correction.json.
    • Controlled stale-client cleanup: C:\Wooo\Agent99\evidence\agent99-stale-ssh-cleanup-v2.json; 7 orphan clients stopped.
    • Performance readback: C:\Wooo\Agent99\evidence\agent99-Perf-20260710-224454.json; all five hosts OK, 188 disk reduced from 90% to 88%.
  • The deployment receipt and runtime manifest must contain the final deployed Gitea SHA; paths or source-test success alone are not accepted as runtime proof.
  • Remaining acceptance truth:
    • This receipt closes the transport stall, orphan SSH, 188 disk pressure, and false-green remediation defects.
    • It does not close the whole-host 10-minute cold-start objective until the next controlled reboot produces a fresh scorecard with all required gates green.

2026-07-11 Runtime Truth And Completion Callback Contract

  • scripts/reboot-recovery/agent99-live-preflight.ps1 now treats a required evidence file as usable only when it is fresh, JSON-parseable and content-healthy. A fresh timestamp with a failed safe-summary adapter is a blocker, not green.
  • SRE and Telegram inbox evidence use bounded shape adapters. The preflight reads counts, status and exit code only; it never reads raw Telegram updates, message text, token, chat id, .env, auth data, sessions or SQLite.
  • A successful Recover emits one recovery_slo_result lifecycle event even at info severity. The visual is a recovery timeline with scope, duration, run id and all seven phase verifiers; agent_complete remains no-alert to prevent duplicate noise.
  • Same-state Telegram suppression must append a dedupe_window attempt receipt. Silence without a receipt is invalid.
  • Production verifier: run Wooo-Agent99-Startup-Recovery under the registered WOOO S4U principal, then pipe scripts/reboot-recovery/agent99-recover-receipt-readback.ps1 to Windows PowerShell. Closure requires RECOVERY_VERIFIED=1 and NOTIFICATION_RECEIPTED=1; Administrator launcher output is not a substitute.
  • Canonical completion is not Telegram delivery. Agent99 posts a bounded agent99_completion_callback_v1 payload authenticated by the existing AGENT99_SRE_RELAY_TOKEN; raw logs, secrets and full evidence paths are rejected.
  • The API writes an idempotent Agent99 channel event, AWOOOI EXECUTION_COMPLETED/verified RESOLVED operation receipts and an AwoooP completed shadow run. Agent99 moves a callback from state\completion-callbacks\pending to processed only after the API returns the same callback/run/trace and durable_readback=true with operation receipts.
  • Callback failures retry in bounded batches after cooldown and never repeat the repair action. The callback source is not production-complete until Gitea CD, API runtime readback, Windows client deploy and one same-trace SRE replay all pass.
Reference in New Issue View Git Blame Copy Permalink