34e32f6ed6
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 1s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 2m27s
CD Pipeline / build-and-deploy (push) Successful in 6m39s
CD Pipeline / post-deploy-checks (push) Successful in 1m55s
764 lines
41 KiB
Markdown
764 lines
41 KiB
Markdown
# Agent99 AI Workstation SOP
|
||
|
||
Last updated: 2026-07-11
|
||
|
||
Canonical enterprise scope and work order:
|
||
|
||
- `docs/workplans/2026-07-10-agent99-enterprise-ai-automation-master-plan.md`
|
||
- `docs/operations/agent99-enterprise-ai-automation-work-items.snapshot.json`
|
||
|
||
This SOP defines how Agent99 operates. The canonical work ledger defines what remains incomplete across all hosts, products, sites, services, tools and packages. A transport or command `exitCode=0` never means an incident is resolved; the mode-specific verifier, source-event resolution and callback receipt are required.
|
||
|
||
## Objective
|
||
|
||
192.168.0.99 is the AI Agent workstation and local operations control node.
|
||
|
||
Host 99 is not the only availability observer. An external watcher and maintenance surface must remain available when 99 or the entire local failure domain is down.
|
||
|
||
## 2026-07-10 P0-001 Outcome Truth Contract
|
||
|
||
- `exitCode=0` means only that the PowerShell transport completed.
|
||
- A command is `resolved` only when its mode-specific post-condition verifier passes, its evidence is fresh and mode-matched, and any alert-source event has an explicit resolved readback.
|
||
- Other legal outcomes are `candidate_ready`, `verifying`, `degraded`, `blocked`, and `failed`.
|
||
- Alert-originated work without `sourceEventResolved=true` remains `verifying`, even when the repair command and post-condition verifier succeeded.
|
||
- `problem-counts.json` v2 preserves the legacy `totalResolved` field but labels it as possibly transport-derived. New trustworthy closure is counted separately in `totalVerifiedResolved` and per-problem `verifiedResolved`.
|
||
- Bootstrap writes `state\runtime-manifest.json` with source revision, source/runtime SHA-256, content match, file count and mismatch count. Self-health is critical when this manifest is missing, invalid, stale by hash, or incomplete.
|
||
- All `.ps1` runtime files are normalized to UTF-8 BOM by bootstrap so Windows PowerShell 5 cannot corrupt Traditional Chinese strings during parsing.
|
||
- Staging verification on 99 must pass `-SelfTestOutcomeContract` and `runtimeMatched=true` before production Agent99 files are replaced.
|
||
|
||
## 2026-07-11 P0-002 Actionable Alert Dispatch Contract
|
||
|
||
This source candidate is test-green but not yet deployed to the Windows 99 runtime.
|
||
|
||
- Direct Alertmanager events, `ai_automation_alert_card_v1`, `ACTION REQUIRED` incident cards, and failed `AWOOOI CI/CD` cards use one sanitized Agent99 dispatcher.
|
||
- A cold-start or reboot gate is `kind=host_recovery`, `suggestedMode=Recover`; it must never become a fabricated Kubernetes deployment restart.
|
||
- A failed CI/CD card is `kind=cicd_failure`, `suggestedMode=Status`; Agent99 collects status/evidence while the source event remains open until a later successful CD/deploy receipt resolves it.
|
||
- Every dispatcher result writes a public-safe `agent99_dispatch_receipt` with alert id, kind, transport, HTTP status, `accepted`, and `inbox_triggered`. Raw response bodies, relay paths, tokens, chat ids, and workflow URLs are not stored.
|
||
- `HTTP 202` alone is not success. The bridge reports `dispatched` only when the relay body confirms `ok=true`; an unaccepted receipt releases the single-flight lock so the event can retry.
|
||
- Stateful verifier lanes may use `sourceEventResolutionPolicy=mode_verifier_can_resolve`. External-source lanes use `external_source_receipt_required` and cannot self-close from a local command exit.
|
||
- Source/test receipt: the focused post-rebase suite passed 249 tests, and the expanded all-Telegram plus Agent99/RepairCandidate suite passed 262 tests after integrating the Telegram single-owner mainline; YAML, shell/Python syntax, runner pressure, document secret, backup-alert, and recovery-scorecard checks passed. Windows 99 package deployment, runtime manifest reconciliation, and exact live card replay remain required before this P0 can close.
|
||
|
||
It must act as:
|
||
|
||
- Secretary: reminders, status summaries, evidence packets, operator updates.
|
||
- Assistant: routine checks, file organization, browser/readback checks, alert routing.
|
||
- Engineer: VM control, service recovery, deploy/readback verification, runbook execution.
|
||
- Incident operator: reboot detection, recovery orchestration, Telegram alerting, SLA countdown.
|
||
|
||
The machine must be able to operate VMware, SSH to service hosts, run public HTTP readbacks, and produce evidence without depending on old chat context.
|
||
|
||
## Agent99 Operating Modes
|
||
|
||
Observe mode:
|
||
|
||
- Read host, VM, service, and website state.
|
||
- Produce evidence only.
|
||
- Default mode for scheduled checks.
|
||
|
||
Controlled apply mode:
|
||
|
||
- Execute approved low-blast-radius actions.
|
||
- Must write evidence before and after each action.
|
||
- Requires explicit operator approval or scheduled startup recovery policy.
|
||
|
||
Break-glass mode:
|
||
|
||
- Reserved for destructive, irreversible, secret-bearing, cost-bearing, or firewall/provider cutover actions.
|
||
- Must not be hidden inside automatic tasks.
|
||
|
||
## Required Local Capabilities On 99
|
||
|
||
Base runtime:
|
||
|
||
- Windows PowerShell 5+ or PowerShell 7.
|
||
- Windows OpenSSH client.
|
||
- `curl.exe`.
|
||
- VMware Workstation.
|
||
- `vmrun.exe`.
|
||
- Windows Task Scheduler.
|
||
|
||
Agent filesystem:
|
||
|
||
- `C:\Wooo\Agent99\bin`
|
||
- `C:\Wooo\Agent99\config`
|
||
- `C:\Wooo\Agent99\evidence`
|
||
- `C:\Wooo\Agent99\logs`
|
||
- `C:\Wooo\Agent99\queue`
|
||
- `C:\Wooo\Agent99\requests`
|
||
- `C:\Wooo\Agent99\alerts\incoming`
|
||
- `C:\Wooo\Agent99\playbooks`
|
||
|
||
Agent scripts:
|
||
|
||
- `agent99-bootstrap.ps1`
|
||
- `agent99-control-plane.ps1`
|
||
- `agent99-submit-request.ps1`
|
||
- `agent99-telegram-inbox.ps1`
|
||
- `agent99-sre-alert-inbox.ps1`
|
||
- `host-reboot-scorecard.ps1`
|
||
- `agent99.config.json`
|
||
|
||
## Command Ingress Without Codex Quota
|
||
|
||
Codex is not the runtime command plane. Agent99 must accept operator work through local 99 channels even when Codex is unavailable.
|
||
|
||
Supported command channels:
|
||
|
||
- Local desktop: `Agent99 Submit Request.lnk`.
|
||
- Local queue: write allowlisted command JSON to `C:\Wooo\Agent99\queue`.
|
||
- AwoooI SRE Telegram group: send `/agent99 ...`, `!agent99 ...`, `#agent99 ...`, `agent99 ...`, or `@agent99 ...`.
|
||
- SRE war room alerts: write alert JSON to `C:\Wooo\Agent99\alerts\incoming`.
|
||
- AwoooI SRE Telegram monitoring alerts: non-command messages that look like Alertmanager/FIRING/critical/warning/502/CPU/disk/Harbor/K3s/backup/host-down/provider-freshness alerts are auto-ingested.
|
||
|
||
Command processing requirements:
|
||
|
||
- Every command resolves to an allowlisted mode only.
|
||
- Website plus 502 plus recover/repair intent resolves to `Recover`, not passive smoke-only checking.
|
||
- Alert-driven remediation uses controlled apply only for allowlisted modes such as `Recover`, `HarborRepair`, `AwoooRepair`, `Perf`, and `LoadShed`.
|
||
- Every accepted command writes a request record, queue record, execution result, and evidence pointer.
|
||
- Telegram messages must be human-readable: source, instruction, mode, result, and evidence path. Raw Python, raw stack trace, token, chat id, cookies, sessions, or `.env` must never be sent.
|
||
- Telegram SRE group commands must be recorded with `source=telegram-sre-war-room`.
|
||
- Telegram SRE monitoring alerts must be recorded with `source=telegram-sre-war-room-monitoring`.
|
||
- Telegram SRE group matching uses `allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE`, so the command path is not limited to the default private chat id.
|
||
- When a command originates from a matched Telegram group, completion replies are routed back to the same Telegram origin while evidence stores only `telegram-origin`, not the raw chat id.
|
||
- After automated recovery completes, Agent99 must report the result back to Telegram so the project AI Agents and human operators know whether the abnormality is resolved.
|
||
- Every queue execution must update problem counts, write a Playbook/KM record, and append a RAG-import JSONL line.
|
||
- Operator command and remediation result Telegram reports are not deduped; each completed repair emits a completion report. Noisy warning/critical signals still use dedupe.
|
||
|
||
SRE Telegram command examples:
|
||
|
||
- `/agent99 檢查所有網站是否正常，若502就自動修復`
|
||
- `/agent99 檢查110和188 CPU負載，必要時執行允許的降載`
|
||
- `/agent99 檢查AWOOOI K3s與Harbor，若ImagePullBackOff就修復`
|
||
- `/agent99 檢查備份健康狀態`
|
||
- `/agent99 建立 provider freshness candidate，要求 provider window、last seen、direct ref 與 verifier readback`
|
||
|
||
Monitoring alert auto-ingest examples:
|
||
|
||
- `Alertmanager FIRING critical public_route_502 service=awoooi host=192.168.0.120 HTTP 502`
|
||
- `HostHighCpuLoad warning host=192.168.0.110`
|
||
- `HarborRegistryDown critical host=192.168.0.110`
|
||
- `KubePodCrashLooping critical namespace=awoooi-prod`
|
||
- `provider_freshness_signal target=provider_freshness_signal freshness last_seen missing direct_ref required`
|
||
|
||
Problem memory outputs:
|
||
|
||
- Problem counters: `C:\Wooo\Agent99\state\problem-counts.json`.
|
||
- Full event ledger: `C:\Wooo\Agent99\playbooks\agent99-incident-records.jsonl`.
|
||
- Playbook/KM summary: `C:\Wooo\Agent99\playbooks\agent99-incident-km.md`.
|
||
- RAG import feed: `C:\Wooo\Agent99\playbooks\rag-import\agent99-incidents.jsonl`.
|
||
- Dashboard: `C:\Wooo\Agent99\dashboard\index.html`, section `Problem Counts`.
|
||
|
||
## SRE War Room Alert Contract
|
||
|
||
AWOOOI SRE monitors and other AI Agent work windows can hand alerts to Agent99 by writing JSON files to:
|
||
|
||
`C:\Wooo\Agent99\alerts\incoming`
|
||
|
||
Minimum fields:
|
||
|
||
```json
|
||
{
|
||
"id": "alert-unique-id",
|
||
"source": "awoooi-sre-war-room",
|
||
"severity": "critical",
|
||
"kind": "public_route_502",
|
||
"service": "awoooi",
|
||
"host": "192.168.0.120",
|
||
"title": "AWOOOI public route returned 502",
|
||
"message": "The public health route is returning 502.",
|
||
"createdAt": "2026-07-09T22:50:00+08:00"
|
||
}
|
||
```
|
||
|
||
Routing rules:
|
||
|
||
- `502`, `site_502`, `public_route_502`, `route_down`, `host_down`, VM down, or reboot detected -> `Recover`.
|
||
- CPU, memory, disk, performance pressure -> `Perf` with controlled apply.
|
||
- Harbor or registry -> `HarborRepair`.
|
||
- AWOOOI, K3s, image pull, pod, deployment -> `AwoooRepair`.
|
||
- Backup, snapshot, restore drill, or cron -> `BackupCheck`.
|
||
- Unknown actionable alert -> `Status` evidence first.
|
||
|
||
Alert processing evidence:
|
||
|
||
- Inbox evidence: `agent99-SreAlertInbox-*.json`.
|
||
- Queued request: `C:\Wooo\Agent99\requests`.
|
||
- Execution evidence: `agent99-Recover-*.json`, `agent99-Perf-*.json`, or the resolved mode evidence file.
|
||
|
||
## VM Control Requirements
|
||
|
||
Agent99 must be able to operate these VM hosts:
|
||
|
||
- 110
|
||
- 112
|
||
- 120
|
||
- 121
|
||
- 188
|
||
|
||
Required actions:
|
||
|
||
- Detect VM powered-off state.
|
||
- Start missing VMs through `vmrun.exe`.
|
||
- Detect VMware config lock symptoms.
|
||
- Avoid deleting VM lock files unless VMware process state proves the VM is not running.
|
||
- Prefer guest shutdown/restart first; use reset only when the guest is stuck and SLA requires recovery.
|
||
|
||
## Engineering Recovery Requirements
|
||
|
||
Agent99 must support these engineering actions:
|
||
|
||
- SSH readback to 110 and 120.
|
||
- Harbor core / Redis / registry checks on 110.
|
||
- K3s AWOOOI deployment and pod checks on 120.
|
||
- Public route readback for user-visible websites.
|
||
- Evidence capture after every action.
|
||
- Telegram alert hooks.
|
||
|
||
## Main Scheduled Automations
|
||
|
||
Startup recovery:
|
||
|
||
- Trigger: 99 boot or operator logon.
|
||
- Frequency: once at startup, then repeat during first recovery window.
|
||
- Mode: controlled apply allowed for approved actions.
|
||
- Purpose: start VMs, restore Harbor, restore K3s pods, verify public routes.
|
||
|
||
Heartbeat:
|
||
|
||
- Trigger: every 5 minutes.
|
||
- Mode: observe by default.
|
||
- Purpose: detect host down, 502, service degraded, backup job missing, or stale freshness.
|
||
|
||
Daily evidence summary:
|
||
|
||
- Trigger: daily.
|
||
- Mode: observe.
|
||
- Purpose: produce a short host/service/site/backup status packet.
|
||
|
||
## Telegram Alert Classes
|
||
|
||
The agent must send alerts for:
|
||
|
||
- `host_down`
|
||
- `host_reboot_detected`
|
||
- `vm_not_running`
|
||
- `vm_start_attempted`
|
||
- `service_recovering`
|
||
- `harbor_unhealthy`
|
||
- `k3s_image_pull_failure`
|
||
- `site_502`
|
||
- `site_recovered`
|
||
- `service_degraded`
|
||
- `backup_missing`
|
||
- `backup_failed`
|
||
- `recovery_sla_risk`
|
||
- `recovery_complete`
|
||
|
||
Telegram credentials must be stored outside the repo, preferably in Windows Credential Manager or machine environment variables.
|
||
|
||
## 2026-07-09 Recovery Logic To Embed
|
||
|
||
When AWOOOI or other sites return 502:
|
||
|
||
1. Do not only inspect nginx.
|
||
2. Check 120 K3s workloads first.
|
||
3. If AWOOOI pods are `ImagePullBackOff`, check 110 Harbor before deleting pods.
|
||
4. If Harbor token endpoint returns 500 or pulls fail, inspect:
|
||
- Docker engine.
|
||
- Harbor Redis.
|
||
- `harbor-core`.
|
||
- `registry`.
|
||
- `registryctl`.
|
||
5. If `harbor-core` is not healthy:
|
||
- Restart Redis and `harbor-core`.
|
||
- If Docker/systemd hangs, execute controlled 110 guest restart via VMware.
|
||
6. After `harbor-core` is healthy:
|
||
- Delete only failed AWOOOI pods.
|
||
- Wait for deployment readiness.
|
||
- Verify public HTTP.
|
||
|
||
## First Implementation Milestone
|
||
|
||
Milestone M1 is complete when:
|
||
|
||
- Agent99 directory exists on 99.
|
||
- Config file exists.
|
||
- Startup and heartbeat scheduled tasks exist.
|
||
- Scorecard can run in observe mode.
|
||
- Recovery can run in controlled apply mode.
|
||
- Evidence JSON is written.
|
||
- No secrets are stored in repo files.
|
||
|
||
## 2026-07-09 M1 Receipt
|
||
|
||
M1 is operational on 99 as of 2026-07-09 16:49 Asia/Taipei.
|
||
|
||
- `C:\Wooo\Agent99` exists on `WOOO-SUPER`.
|
||
- `agent99-control-plane.ps1`, `host-reboot-scorecard.ps1`, and task registration scripts are deployed.
|
||
- `agent99.config.json` includes VMX selectors for 110, 188, 112, 120, and 121.
|
||
- `Wooo-Agent99-Heartbeat` is registered and produced `LastTaskResult=0`.
|
||
- `Wooo-Agent99-Startup-Recovery` is registered and produced `LastTaskResult=0`.
|
||
- Latest heartbeat evidence showed Harbor healthy, AWOOOI deploy ready, and public routes non-502.
|
||
- Latest recovery evidence showed all five configured VM hosts already reachable and no recovery action required.
|
||
|
||
Remaining M2 work:
|
||
|
||
- Install Telegram credentials with environment variables or Windows Credential Manager, then verify real delivery.
|
||
- Add backup-job freshness readback and alert classes.
|
||
- Add Windows Update no-surprise-reboot policy verification.
|
||
- Promote this into a long-running Windows service after the scheduled-task lane stays clean.
|
||
|
||
## 2026-07-09 M2 Performance Guard Receipt
|
||
|
||
Agent99 performance guard is operational on 99 as of 2026-07-09 17:15 Asia/Taipei.
|
||
|
||
- `Wooo-Agent99-Performance-Guard` is registered in Windows Task Scheduler.
|
||
- Trigger interval: every 1 minute.
|
||
- Last manual verification: `LastTaskResult=0`.
|
||
- Latest evidence: `C:\Wooo\Agent99\evidence\agent99-Perf-20260709-171502.json`.
|
||
- Readback coverage:
|
||
- 110: OK, load-per-core 0.35, memory available 62.93%, disk used 82%.
|
||
- 112: OK, load-per-core 0.07, memory available 61.06%, disk used 47%.
|
||
- 120: OK, load-per-core 0.05, memory available 91.76%, disk used 44%.
|
||
- 121: OK, load-per-core 0.00, memory available 92.79%, disk used 27%.
|
||
- 188: OK, load-per-core 0.08, memory available 90.19%, disk used 89%.
|
||
- 99 SSH access was completed for:
|
||
- 112 with user `kali`.
|
||
- 188 with user `ollama`.
|
||
- Load-shedding is controlled by allowlist only.
|
||
- Current allowlisted action: pause `vibework-dedicated-runner` on 110 only when `load_per_core_critical` is present.
|
||
- Disk or memory pressure only alerts; it does not trigger the CPU runner pause action.
|
||
- Telegram audit is enabled in config and called for Agent99 start, warnings, critical events, load-shed attempts/results, and completion.
|
||
- 99 local env aliases are supported for `AGENT99_*`, `AWOOOI_TG_*`, `OPENCLAW_TG_*`, `TELEGRAM_*`, and `TG_*`.
|
||
- If 99 does not have local Telegram env values, Agent99 relays through the existing 110 `stockplatform-v2-api-1` container and uses that container's existing `TELEGRAM_BOT_TOKEN` / `TELEGRAM_CHAT_ID`.
|
||
- 2026-07-09 17:27 verification: `Wooo-Agent99-Performance-Guard` produced `LastTaskResult=0`, `agent99-Perf-20260709-172712.json`, `tgSent=2`, and `relayOk=True`.
|
||
- No token or chat id is stored in repo files or Agent99 evidence.
|
||
|
||
## 2026-07-09 OpenClaw Receipt
|
||
|
||
OpenClaw is configured and now included in Agent99 evidence.
|
||
|
||
- Source truth:
|
||
- AWOOOI prod config: `OPENCLAW_URL=http://192.168.0.188:8088`.
|
||
- 188 Ansible service target: OpenClaw on port `8088`.
|
||
- Runtime truth:
|
||
- `192.168.0.188:8088/health` returns `status=healthy`, `version=6.0`, `service=ClawBot`, `environment=production`, and `telegram_bot=connected`.
|
||
- `clawbot.service` on 188 is `active`.
|
||
- `192.168.0.188:8089` is not the current production endpoint.
|
||
- Agent99 truth:
|
||
- Latest status evidence: `C:\Wooo\Agent99\evidence\agent99-Status-20260709-173813.json`.
|
||
- `aiServices.openclaw`: `ok=True`, `status=healthy`, `url=http://192.168.0.188:8088/health`.
|
||
- Telegram relay evidence: `tgSent=2`, `relayOk=2`.
|
||
|
||
## 2026-07-09 M3 Command And Alert Ingress Receipt
|
||
|
||
Agent99 no longer depends on Codex as the only command channel.
|
||
|
||
- Telegram command inbox deployed: `C:\Wooo\Agent99\bin\agent99-telegram-inbox.ps1`.
|
||
- SRE alert inbox deployed: `C:\Wooo\Agent99\bin\agent99-sre-alert-inbox.ps1`.
|
||
- Scheduled tasks registered:
|
||
- `Wooo-Agent99-Telegram-Inbox`
|
||
- `Wooo-Agent99-SRE-Alert-Inbox`
|
||
- PowerShell parse readback: all five Agent99 scripts parsed successfully on 99.
|
||
- Live config includes both scheduled tasks and freshness contracts.
|
||
- Telegram inbox evidence: `C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260709-230031.json`.
|
||
- `ok=True`
|
||
- `updatesSeen=0`
|
||
- `processedCount=0`
|
||
- no Python code or token text in evidence.
|
||
- SRE synthetic alert evidence: `C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260709-230107.json`.
|
||
- input kind: `public_route_502`
|
||
- source: `awoooi-sre-war-room`
|
||
- resolved mode: `Recover`
|
||
- queued count: `1`
|
||
- failed count: `0`
|
||
- `controlExitCode=0`
|
||
- Recovery execution evidence: `C:\Wooo\Agent99\evidence\agent99-Recover-20260709-230108.json`.
|
||
- public route failures: `0`
|
||
- host reachability failures: `0`
|
||
- Harbor core healthy: `True`
|
||
- AWOOOI deploy ready: `True`
|
||
- Queue processor evidence: `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-230108.json`.
|
||
- processed count: `1`
|
||
- source: `agent99-sre-alert-inbox`
|
||
- reason: `sre_alert`
|
||
- mode: `Recover`
|
||
- exit code: `0`
|
||
- Telegram relay: `sent=True`, `relayOk=True`
|
||
- evidence contains no Python code.
|
||
- Self-health evidence: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-230223.json`.
|
||
- severity: `ok`
|
||
- task failures: `0`
|
||
- evidence critical: `0`
|
||
- evidence warning: `0`
|
||
- Telegram inbox task: OK
|
||
- SRE alert inbox task: OK
|
||
- Dashboard JSON: `C:\Wooo\Agent99\dashboard\agent99-control-center.latest.json`.
|
||
- queue pending: `0`
|
||
- alert pending: `0`
|
||
|
||
## 2026-07-09 M4 SRE Telegram Command And Problem Memory Receipt
|
||
|
||
Agent99 now accepts AwoooI SRE Telegram-group commands as first-class work orders.
|
||
|
||
- Telegram SRE command source mapping:
|
||
- source: `telegram-sre-war-room`
|
||
- request path: `C:\Wooo\Agent99\requests`
|
||
- queue path: `C:\Wooo\Agent99\queue`
|
||
- command result event: `operator_command_result`
|
||
- Synthetic SRE Telegram command verification:
|
||
- instruction: `檢查所有網站是否恢復正常，若502就自動修復`
|
||
- request id: `request-20260709-231157-d53f53d8`
|
||
- resolved mode: `Recover`
|
||
- control exit code: `0`
|
||
- evidence: `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-231158.json`
|
||
- source: `telegram-sre-war-room`
|
||
- processed mode: `Recover`
|
||
- processed ok: `True`
|
||
- processed exit code: `0`
|
||
- Telegram relay: sent `1`
|
||
- Telegram evidence contains no Python code.
|
||
- Problem memory readback:
|
||
- problem key: `telegram-sre-war-room_agent99_operator_instruction`
|
||
- occurrences: `1`
|
||
- resolved: `1`
|
||
- failed: `0`
|
||
- stats path: `C:\Wooo\Agent99\state\problem-counts.json`
|
||
- KM exists: `True`
|
||
- RAG feed exists: `True`
|
||
- dashboard problem events: `1`
|
||
- dashboard resolved: `1`
|
||
- dashboard failed: `0`
|
||
- Post-check:
|
||
- Telegram inbox evidence: `ok=True`, `updatesSeen=0`, `processedCount=0`
|
||
- Self-health evidence: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-231255.json`
|
||
- self-health severity: `ok`
|
||
- task failures: `0`
|
||
- queue pending: `0`
|
||
- alert pending: `0`
|
||
- SRE group routing hardening:
|
||
- Existing relay default chat readback was a private chat, so group commands must not depend only on `TELEGRAM_CHAT_ID`.
|
||
- Live config was updated with `allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE`.
|
||
- Telegram inbox accepts matched group commands and carries a redacted `telegram-origin` reply route for completion reports.
|
||
|
||
## 2026-07-09 M5 SRE Telegram Monitoring Alert Auto-Ingest Receipt
|
||
|
||
Agent99 now ingests monitoring alerts from the AwoooI SRE Telegram group, not only `/agent99` commands.
|
||
|
||
- Runtime switch:
|
||
- `autoIngestMonitoringAlerts=True`
|
||
- `allowedCommandChatTitlePattern=AwoooI SRE|AWOOOI SRE|SRE`
|
||
- Encoding hardening:
|
||
- Runtime PowerShell scripts are ASCII-only.
|
||
- Chinese alert keyword support uses Unicode codepoint checks instead of raw Chinese strings, avoiding Windows PowerShell 5 parser failures.
|
||
- Synthetic SRE Telegram monitoring alert:
|
||
- text: `Alertmanager FIRING critical public_route_502 service=awoooi host=192.168.0.120 HTTP 502`
|
||
- no `/agent99` prefix.
|
||
- Telegram inbox evidence: `C:\Wooo\Agent99\evidence\agent99-TelegramInbox-20260709-233230.json`
|
||
- `inboxOk=True`
|
||
- `alertedCount=1`
|
||
- SRE alert bridge evidence:
|
||
- `C:\Wooo\Agent99\evidence\agent99-SreAlertInbox-20260709-233231.json`
|
||
- queued count: `1`
|
||
- failed count: `0`
|
||
- Control/recovery evidence:
|
||
- `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-233232.json`
|
||
- source: `telegram-sre-war-room-monitoring`
|
||
- kind: `public_route_502`
|
||
- mode: `Recover`
|
||
- processed ok: `True`
|
||
- exit code: `0`
|
||
- Telegram attempts: `1`
|
||
- Telegram sent: `1`
|
||
- evidence contains no Python code.
|
||
- Problem memory:
|
||
- total events before: `2`
|
||
- total events after: `3`
|
||
- total resolved after: `3`
|
||
- total failed after: `0`
|
||
|
||
## 2026-07-09 M6 Provider Freshness Signal Receipt
|
||
|
||
Provider freshness alerts must not be treated as passive route health.
|
||
|
||
- Gap found:
|
||
- Telegram Inbox and SRE Alert Inbox were running, but `provider_freshness_signal` was not present in Agent99 alerts, queue, or execution evidence after 2026-07-09 23:30 Asia/Taipei.
|
||
- Only SelfCheck `evidenceFreshness` fields matched the word `freshness`, so the original alert was not processed as an actionable provider freshness signal.
|
||
- Runtime behavior added:
|
||
- Telegram auto-ingest recognizes `provider_freshness_signal`, `source_provider_freshness_triage`, `raw_signal_normalized`, `controlled_runtime_candidate`, `freshness`, `last seen`, `last_seen`, and `ingestion`.
|
||
- SRE Alert Inbox routes freshness signals to `ProviderFreshness`.
|
||
- `ProviderFreshness` creates a read-only candidate under `C:\Wooo\Agent99\state\provider-freshness-candidates`.
|
||
- Candidate evidence must require provider window, last seen, direct reference, and verifier readback.
|
||
- Guardrail remains: no provider switch, no paid model call, no env change, no reload.
|
||
|
||
## 2026-07-09 M7 AWOOI API To Agent99 Alert Bridge
|
||
|
||
Agent99 must not depend only on Telegram text polling for AwoooI SRE alerts.
|
||
|
||
- Producer-side bridge:
|
||
- AWOOI API `POST /api/v1/webhooks/alertmanager` now schedules a fail-open Agent99 SRE bridge after alert normalization and fingerprinting.
|
||
- The bridge POSTs the same Agent99 alert contract to `AGENT99_SRE_ALERT_RELAY_URL` when configured.
|
||
- If relay URL is empty, the bridge writes the contract expected by `C:\Wooo\Agent99\alerts\incoming` when `AGENT99_SRE_ALERT_INBOX_PATH` is configured.
|
||
- Empty relay/path means bridge disabled; Alertmanager response must not fail or retry-storm because Agent99 inbox is unavailable.
|
||
- Safety:
|
||
- Payloads are sanitized before writing.
|
||
- Labels with token, secret, password, authorization, cookie, session, or private-key names are redacted.
|
||
- Provider freshness signals map to `ProviderFreshness`.
|
||
- Route 502 maps to `Recover`; CPU/memory/disk/load maps to `Perf`; backup maps to `BackupCheck`; Harbor maps to `HarborRepair`; K3s/pod/image-pull maps to `AwoooRepair`.
|
||
- Guardrail:
|
||
- This bridge does not switch provider, call paid models, edit env, reload services, or read raw secrets.
|
||
- problem key: `agent99-sre-alert-inbox_awoooi_192.168.0.120_public_route_502`
|
||
- problem occurrences: `2`
|
||
|
||
## 2026-07-10 M8 Agent99 Direct SRE Relay Contract
|
||
|
||
Telegram text polling is not enough for SLA-grade SRE automation.
|
||
|
||
- 99-side relay:
|
||
- Script: `C:\Wooo\Agent99\bin\agent99-sre-alert-relay.ps1`.
|
||
- Scheduled task: `Wooo-Agent99-SRE-Alert-Relay`.
|
||
- Default prefix: `http://+:8787/agent99/sre-alert/`.
|
||
- Accepted sources: local and `192.168.0.*` by default.
|
||
- Optional token env: `AGENT99_SRE_RELAY_TOKEN`.
|
||
- Production producer:
|
||
- `k8s/awoooi-prod/04-configmap.yaml` sets `AGENT99_SRE_ALERT_RELAY_URL=http://192.168.0.99:8787/agent99/sre-alert/`.
|
||
- `AGENT99_SRE_ALERT_RELAY_TOKEN` is optional and must come from Secret/env only.
|
||
- API bridge uses relay first, then mounted file inbox fallback.
|
||
- Runtime behavior:
|
||
- Relay validates POST + JSON, writes into `C:\Wooo\Agent99\alerts\incoming`, and triggers `agent99-sre-alert-inbox.ps1 -RunNow`.
|
||
- SRE Alert Inbox resolves mode and queues only allowlisted work.
|
||
- Control Loop writes evidence, Telegram completion, problem counts, KM, and RAG import records.
|
||
- Relay must not execute arbitrary shell, read secrets, switch providers, reload services, or perform destructive actions.
|
||
|
||
## 2026-07-09 Windows Update Policy Receipt
|
||
|
||
99 is configured to prevent surprise Windows Update reboots.
|
||
|
||
- Evidence: `C:\Wooo\Agent99\evidence\agent99-windows-update-policy-20260709-173933.json`.
|
||
- `NoAutoRebootWithLoggedOnUsers=1`.
|
||
- `AlwaysAutoRebootAtScheduledTime=0`.
|
||
- `AUOptions=3`.
|
||
- `ActiveHoursStart=0`, `ActiveHoursEnd=23`.
|
||
- `gpupdateOk=True`.
|
||
|
||
## 2026-07-09 M3 Self-Health Receipt
|
||
|
||
Agent99 now monitors its own automation loop.
|
||
|
||
- Scheduled task: `Wooo-Agent99-Self-Health`.
|
||
- Trigger interval: every 5 minutes.
|
||
- Task Scheduler verification:
|
||
- `LastTaskResult=0`.
|
||
- Latest triggered evidence: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-175218.json`.
|
||
- `evidenceAfterTrigger=True`.
|
||
- Self-check coverage:
|
||
- Startup recovery task, heartbeat task, performance guard task, and self-health task.
|
||
- Evidence freshness for `agent99-Perf-*.json`, `agent99-Status-*.json`, `agent99-SelfCheck-*.json`, and `agent99-Recover-*.json`.
|
||
- Latest performance evidence must include all five hosts: 110, 112, 120, 121, and 188.
|
||
- Recent attempted Telegram delivery must be successful through direct env credentials or the 110 relay; a quiet interval with no alert attempts is OK and must not create a false critical self-check.
|
||
- Latest result:
|
||
- `selfHealth.severity=ok`.
|
||
- `taskFailures=0`.
|
||
- `performanceSeverity=ok`.
|
||
- `telegramSeverity=ok`.
|
||
- `tgSent=3`, `relayOk=3`.
|
||
|
||
## 2026-07-09 M4 110 Load Auto-Reduction Receipt
|
||
|
||
Agent99 now handles the 110 CPU/load pressure pattern found after reboot.
|
||
|
||
- Incident findings:
|
||
- The high-load source was not the runner.
|
||
- Normal high consumers included Gitea and StockPlatform Postgres/API.
|
||
- Backup load came from `/backup/scripts/backup-awoooi.sh`, `gzip`, and remote `pg_dump`.
|
||
- A random `alpine "tar czf - /data"` container named `mystifying_napier` was consuming CPU and logging compressed binary output.
|
||
- Manual controlled action applied once:
|
||
- Backup PIDs were set to nice 15 / ionice best-effort 7.
|
||
- `mystifying_napier` was stopped and retained as a dead container for evidence.
|
||
- Automation added to `performance.loadShedding.actions`:
|
||
- `stop-110-runaway-alpine-tar-data`: warning-or-critical action for allowlisted runaway `alpine tar czf - /data` containers.
|
||
- `renice-110-awoooi-backup-pressure`: warning-or-critical action for backup/gzip/remote-pg-dump pressure.
|
||
- `pause-110-ci-runner-pressure`: critical-only action for runner pressure, unchanged.
|
||
- Verification:
|
||
- Evidence: `C:\Wooo\Agent99\evidence\agent99-Perf-20260709-180704.json`.
|
||
- 110: severity OK, load-per-core 0.67, memory available 62.02%, disk used 80%.
|
||
- 112, 120, 121, and 188: severity OK.
|
||
- `perfCritical=0`, `perfWarning=0`.
|
||
- Telegram relay evidence: `tgSent=2`, `relayOk=2`.
|
||
|
||
## 2026-07-09 M5 Backup Health Receipt
|
||
|
||
Agent99 now monitors backup freshness and backup-job contracts.
|
||
|
||
- Scheduled task: `Wooo-Agent99-Backup-Health`.
|
||
- Trigger interval: every 15 minutes.
|
||
- Task Scheduler verification:
|
||
- `LastTaskResult=0`.
|
||
- Latest triggered evidence: `C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260709-190617.json`.
|
||
- `evidenceAfterTrigger=True`.
|
||
- BackupCheck coverage:
|
||
- 13 snapshot targets: `awoooi`, `gitea`, `harbor`, `momo`, `langfuse`, `monitoring`, `signoz`, `clawbot`, `open-webui`, `sentry`, `ai-artifacts`, `configs`, and `public-routes`.
|
||
- 7 file/status checks: `backup.log`, `backup-status.log`, legacy `status.log`, legacy `cron.log`, integrity check status, restore drill status, and configs last-status JSON.
|
||
- 6 cron contract patterns: `backup-all.sh`, `backup-awoooi-frequent.sh`, `backup-health-textfile-exporter.py`, `check-backup-integrity.sh`, `sync-offsite-backups.sh --mode status`, and `backup-status.sh`.
|
||
- Latest result:
|
||
- `backupHealth.severity=ok`.
|
||
- `targetCritical=0`, `targetWarning=0`.
|
||
- `fileCritical=0`, `fileWarning=0`.
|
||
- `cronCritical=0`, `cronWarning=0`.
|
||
- `tgSent=2`, `relayOk=2`.
|
||
- Agent self-health now includes backup monitoring:
|
||
- Evidence: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-190714.json`.
|
||
- `Wooo-Agent99-Backup-Health`: OK, `LastTaskResult=0`.
|
||
- Backup evidence freshness: OK, latest file `agent99-BackupCheck-20260709-190617.json`.
|
||
|
||
## 2026-07-09 M6 Alert-to-Remediation Receipt
|
||
|
||
Agent99 now treats alerts as action triggers, not as the final output.
|
||
|
||
- Telegram relay bug fixed:
|
||
- The relay sender now decodes the message argument from `sys.argv[2]`, so Telegram receives the alert text instead of the Python relay source.
|
||
- Info/start/complete events are suppressed by default: `alertAllOperations=false`, `alertInfoEvents=false`.
|
||
- Duplicate warning/critical events are deduped for 30 minutes using stable event keys.
|
||
- Controlled remediation behavior:
|
||
- `Perf -ControlledApply` suppresses immediate performance spam, runs allowlisted load-shedding/remediation first, then alerts the controlled remediation result.
|
||
- If no remediation applies, Agent99 still alerts the original warning/critical performance issue.
|
||
- 188 disk pressure now has an allowlisted action: `prune-188-docker-build-cache-disk-pressure`.
|
||
- The 188 prune action uses `timeoutSeconds=240` and a post-apply verifier: `postVerifyMaxDiskUsedPercent=89`.
|
||
- A command timeout is not reported as final critical when the post-verifier proves the host is back under the disk threshold.
|
||
- False critical prevention:
|
||
- Self-health no longer treats "no recent alerts were needed" as Telegram failure.
|
||
- Telegram health is critical only when recent alert attempts exist and delivery failed.
|
||
- Telegram delivery health uses a 60-minute lookback across recent evidence files, not only the newest file per evidence family.
|
||
- Transient performance warnings are no longer promoted to self-health critical; only missing or critical host performance evidence is critical.
|
||
- Backup monitor false critical prevention:
|
||
- Backup snapshot readback now supports `snapshotReadTimeoutSeconds`.
|
||
- 99 live config sets `snapshotReadTimeoutSeconds=90`.
|
||
- Harbor backup timeout at 22:10 was verified as a readback timeout, not a missing backup: `/backup/harbor/snapshots` had latest metadata from 2026-07-09 17:48 Asia/Taipei and remained within the 1800-minute SLA.
|
||
- Evidence hygiene:
|
||
- Harbor status readback no longer stores raw `docker inspect` JSON.
|
||
- Agent99 now records only selected Harbor health fields and sets `inspectOutput=[redacted: raw docker inspect omitted]`.
|
||
- Existing Status evidence on 99 was sanitized: 77 files checked; `inspectOutputNotRedactedCount=0`; `unsafeHarborFieldCount=0`.
|
||
- Runtime readback after deployment:
|
||
- Performance evidence: `C:\Wooo\Agent99\evidence\agent99-Perf-20260709-220718.json`.
|
||
- Self-health evidence: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-220829.json`, `selfHealth.severity=ok`.
|
||
- Status evidence: `C:\Wooo\Agent99\evidence\agent99-Status-20260709-221147.json`.
|
||
- Host reachability: 110, 112, 120, 121, and 188 all ping OK.
|
||
- Public routes: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502.
|
||
- Harbor: core, Redis, and registry healthy with redacted evidence.
|
||
- AWOOOI: `deployReady=true`, no image pull failure, no crash failure.
|
||
- Backup evidence after timeout fix: `C:\Wooo\Agent99\evidence\agent99-BackupCheck-20260709-221759.json`, `backupHealth.severity=ok`, all target/file/cron critical and warning counts are 0.
|
||
- Final self-health evidence: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-222236.json`, `selfHealth.severity=ok`, `taskFailures=0`, `performanceSeverity=ok`, `telegramSeverity=ok`.
|
||
|
||
## 2026-07-09 M7 Visible Control Center Receipt
|
||
|
||
Agent99 is now visible and operable on the 99 Windows desktop instead of being only background scheduled tasks.
|
||
|
||
- New mode:
|
||
- `ControlTick`.
|
||
- Runs the Agent99 visible control loop.
|
||
- Updates dashboard HTML/JSON.
|
||
- Processes allowlisted operator command queue files.
|
||
- Writes evidence and alerts operator command results to Telegram.
|
||
- Scheduled task:
|
||
- `Wooo-Agent99-Control-Loop`.
|
||
- Trigger interval: every 1 minute.
|
||
- Self-health now checks this task.
|
||
- Desktop shortcuts created:
|
||
- `C:\Users\Public\Desktop\Agent99 Control Center.lnk`.
|
||
- `C:\Users\Public\Desktop\Agent99 Queue.lnk`.
|
||
- `C:\Users\Public\Desktop\Agent99 Evidence.lnk`.
|
||
- `D:\Desktop\Agent99 Control Center.lnk`.
|
||
- `D:\Desktop\Agent99 Queue.lnk`.
|
||
- `D:\Desktop\Agent99 Evidence.lnk`.
|
||
- Dashboard paths:
|
||
- HTML: `C:\Wooo\Agent99\dashboard\index.html`.
|
||
- JSON: `C:\Wooo\Agent99\dashboard\agent99-control-center.latest.json`.
|
||
- Queue processor:
|
||
- Input: `C:\Wooo\Agent99\queue\*.json`.
|
||
- Processed output: `C:\Wooo\Agent99\queue\processed\processed-<id>.json`.
|
||
- Failed output: `C:\Wooo\Agent99\queue\failed\failed-<id>.json`.
|
||
- Allowed modes: `Status`, `Recover`, `StartVMs`, `PublicSmoke`, `HarborRepair`, `AwoooRepair`, `Perf`, `LoadShed`, `SelfCheck`, `BackupCheck`.
|
||
- Operator command results are Telegram-alerted even when info/start/complete noise is suppressed.
|
||
- Verification:
|
||
- Control evidence: `C:\Wooo\Agent99\evidence\agent99-ControlTick-20260709-223309.json`.
|
||
- Processed command: `codex-visible-publicsmoke-20260709-223308`.
|
||
- Processed result: `ok=true`, `exitCode=0`, `durationSeconds=2.2`.
|
||
- Command evidence: `C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-223309.json`.
|
||
- Public smoke: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502.
|
||
- Telegram result: `operator_command_result`, `severity=info`, `sent=true`, `relayOk=true`.
|
||
- Self-health evidence after control-loop install: `C:\Wooo\Agent99\evidence\agent99-SelfCheck-20260709-223340.json`, `selfHealth.severity=ok`, `Wooo-Agent99-Control-Loop=ok`.
|
||
- Latest performance evidence: `C:\Wooo\Agent99\evidence\agent99-Perf-20260709-223419.json`, `perfCritical=0`, `perfWarning=0`.
|
||
|
||
## 2026-07-09 M8 Operator Instruction Intake Receipt
|
||
|
||
Agent99 now accepts operator text instructions and turns them into allowlisted queue commands.
|
||
|
||
- New submit tool:
|
||
- `C:\Wooo\Agent99\bin\agent99-submit-request.ps1`.
|
||
- Interactive launcher: `C:\Wooo\Agent99\agent99-submit-request.cmd`.
|
||
- Desktop shortcut: `Agent99 Submit Request.lnk`.
|
||
- Request flow:
|
||
- Operator enters one text instruction.
|
||
- Agent99 writes the original request to `C:\Wooo\Agent99\requests`.
|
||
- Agent99 resolves the request into an allowlisted mode.
|
||
- Agent99 writes a queue command to `C:\Wooo\Agent99\queue`.
|
||
- `ControlTick` processes the command, writes evidence, and sends Telegram `operator_command_result`.
|
||
- Current allowlisted instruction targets:
|
||
- website / route / 502 checks -> `PublicSmoke`.
|
||
- reboot / restore / repair -> `Recover`.
|
||
- VM / VMware start -> `StartVMs`.
|
||
- CPU / load / memory / disk / performance -> `Perf`.
|
||
- backup / snapshot / cron -> `BackupCheck`.
|
||
- self / agent / health -> `SelfCheck`.
|
||
- Harbor / registry -> `HarborRepair`.
|
||
- AWOOOI / K3s / pod -> `AwoooRepair`.
|
||
- Verification:
|
||
- Test instruction: `check all websites are normal` equivalent in Traditional Chinese.
|
||
- Request ID: `request-20260709-224133-0dc045fe`.
|
||
- Resolved mode: `PublicSmoke`.
|
||
- Processed result: `ok=true`, `exitCode=0`, `durationSeconds=1.9`.
|
||
- Evidence: `C:\Wooo\Agent99\evidence\agent99-PublicSmoke-20260709-224134.json`.
|
||
- Public smoke: AWOOOI, StockPlatform, FIFA, Gitea, and Harbor all HTTP 200 / non-502.
|
||
- Telegram result: `operator_command_result`, `severity=info`, `sent=true`, `relayOk=true`.
|
||
|
||
## 2026-07-10 M9 Source-Trusted Transport And Verified Remediation Receipt
|
||
|
||
Agent99 now uses a source-trusted deployment path, a dedicated bounded SSH transport, and verifier-authoritative remediation outcomes.
|
||
|
||
- Source authority and deployment:
|
||
- Gitea commit SHA is the source revision; a local scratch checkout or a Windows file timestamp is not deployment truth.
|
||
- `agent99-deploy.ps1` validates PowerShell parse, Agent99 contracts, synthetic checks, staging content, backup, atomic promotion, task registration, and rollback.
|
||
- `C:\Wooo\Agent99\state\runtime-manifest.json.sourceRevision` must match the deployed Gitea SHA. A successful source test without this runtime readback is not a completed deployment.
|
||
- Dedicated transport:
|
||
- Private identity: `C:\Wooo\Agent99\keys\agent99_ed25519`; it remains on 99, is ACL-restricted, and is never copied into evidence, Telegram, logs, or the repository.
|
||
- Public-key authorization is limited with `from="192.168.0.99",restrict` on 110 user `wooo` and 112 user `kali`.
|
||
- Route map: 110 and 112 direct; 120, 121, and 188 prefer 110 as the jump host.
|
||
- Live public-key fingerprint receipt: `SHA256:4ccZZSe0buoWyivi2mfTMXE9LZfCmFBPr/evCzZ0nzA`.
|
||
- Automatic recovery:
|
||
- Agent99 persists boot identity in `C:\Wooo\Agent99\state\boot-state.json`.
|
||
- A changed boot observation queues one Recover request only; duplicate requests are suppressed by single-flight state and a 10-minute cooldown.
|
||
- `recovery_slo_result` records the recovery window. It may claim success only after a real reboot drill proves all required hosts, services, routes, and freshness checks inside 10 minutes.
|
||
- Verifier-authoritative remediation:
|
||
- When an action configures `postVerify`, command success alone is insufficient. The verifier result is the effective action result.
|
||
- A failed verifier creates only the 10-minute failure backoff; it cannot create a success cooldown.
|
||
- `policyVersion` migration updates stale live policies and prevents an old command or marker from surviving a new source deployment.
|
||
- Bounded SSH process lifecycle:
|
||
- Every SSH call has a timeout; timeout cleanup terminates the SSH process tree and verifies that the client exited.
|
||
- Every Agent99 process acquires the shared `C:\Wooo\Agent99\state\ssh-transport.lock` with an exclusive file handle before starting SSH. This serializes scheduled tasks and operator runs across Windows sessions; the OS releases the handle if a process exits.
|
||
- Lock acquisition waits at most 90 seconds. Timeout evidence is `ssh_transport_lock_timeout`; it must not start another SSH client or be reported as a service outage.
|
||
- Each performance row records `transportSerialized` and `transportLockWaitMs`, so lock behavior is verifier evidence rather than an implementation-only claim.
|
||
- Production scheduled tasks run as the local `WOOO` S4U principal. An interactive `Administrator` run that cannot read `keys\agent99_ed25519` is an identity mismatch, not host downtime; do not broaden the private-key ACL just to make an operator shell pass.
|
||
- A long-running SRE alert relay remains healthy while `Wooo-Agent99-SRE-Alert-Relay` is `Running/ok`; stale process-start evidence is annotated as `running_task_supersedes_stale_start_evidence` and does not page.
|
||
- Before controlled `Perf` or `LoadShed`, the stale-process guard only considers Agent99 allowlisted targets using `BatchMode=yes` and explicitly excludes `-N`, `-L`, `-R`, and `-D` tunnels.
|
||
- A process is eligible after 15 minutes with a missing parent, or after the 60-minute hard limit. Evidence records selected process metadata but never the command line.
|
||
- Live receipts:
|
||
- Key provisioning: `C:\Wooo\Agent99\evidence\agent99-ssh-key-provision-final.json`.
|
||
- False-success marker correction: `C:\Wooo\Agent99\evidence\agent99-load-shed-marker-correction.json`.
|
||
- Controlled stale-client cleanup: `C:\Wooo\Agent99\evidence\agent99-stale-ssh-cleanup-v2.json`; 7 orphan clients stopped.
|
||
- Performance readback: `C:\Wooo\Agent99\evidence\agent99-Perf-20260710-224454.json`; all five hosts OK, 188 disk reduced from 90% to 88%.
|
||
- The deployment receipt and runtime manifest must contain the final deployed Gitea SHA; paths or source-test success alone are not accepted as runtime proof.
|
||
- Remaining acceptance truth:
|
||
- This receipt closes the transport stall, orphan SSH, 188 disk pressure, and false-green remediation defects.
|
||
- It does not close the whole-host 10-minute cold-start objective until the next controlled reboot produces a fresh scorecard with all required gates green.
|