wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity

feat(aiops): add ssh_docker_prune to auto-repair flywheel for disk-full alerts

Browse Source 
Adds Group B SSH MCP tool ssh_docker_prune (image+volume+builder prune
with ≥75% disk usage gate) and routes "docker prune" actions through it.
Flips HostDiskUsageHigh from auto_repair=false to true with mcp_provider
routing labels so the flywheel can self-heal next disk-full event without
hitting the emergency_channel Telegram path.

Trigger: 2026-05-01 → 05-02 Telegram alert storm (peak 53/hr) caused by
empty ssh-mcp-key/known_hosts secret rejecting all SSH and forcing every
disk-full alert through "Host key is not trusted → escalate" loop.
known_hosts patched live; this commit closes the playbook gap so the
next occurrence resolves without manual intervention.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Your Name
2026-05-02 11:42:48 +08:00
parent 8cf559215c
commit 3156ff1c69
6 changed files with 1472 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
apps/api/src/plugins/mcp/providers/ssh_provider.py
View File

@@ -126,8 +126,13 @@ GROUP_B_TOOLS = {
"ssh_clear_docker_logs",
"ssh_renew_ssl",
"ssh_reload_nginx",
"ssh_docker_prune",
}
# Disk usage gate for ssh_docker_prune (only run when usage >= this %)
# 2026-05-02 ogt + Claude Sonnet 4.6: prevent accidental prune on healthy hosts
DOCKER_PRUNE_DISK_GATE_PCT = 75
ALL_TOOLS = GROUP_A_TOOLS | GROUP_B_TOOLS
MIN_TRUST_SCORE_FOR_GROUP_B = 0.8
@@ -335,6 +340,19 @@ class SSHProvider(MCPToolProvider):
}, "required": ["host", "trust_score"]},
server_name=self.name,
),
MCPTool(
name="ssh_docker_prune",
description=(
"Reclaim disk space via docker image+volume+builder prune. "
f"Gated: only runs when root filesystem usage >= {DOCKER_PRUNE_DISK_GATE_PCT}%. "
"Requires trust_score >= 0.8."
),
input_schema={"type": "object", "properties": {
"host": {"type": "string"},
"trust_score": {"type": "number"},
}, "required": ["host", "trust_score"]},
server_name=self.name,
),
]
# =========================================================================
@@ -551,6 +569,21 @@ class SSHProvider(MCPToolProvider):
if tool_name == "ssh_reload_nginx":
return "nginx -t 2>&1 && systemctl reload nginx && echo 'Nginx reloaded'"
if tool_name == "ssh_docker_prune":
# Disk-gated docker prune: only acts when the alerting condition still holds.
# 2026-05-02 ogt + Claude Sonnet 4.6: ADR-068 飛輪 — disk full SOP
gate = DOCKER_PRUNE_DISK_GATE_PCT
return (
"USAGE=$(df --output=pcent / | tail -1 | tr -dc '0-9'); "
f"if [ -z \"$USAGE\" ] || [ \"$USAGE\" -lt {gate} ]; then "
f"echo \"skip: disk usage ${{USAGE}}% < {gate}% gate\"; exit 0; fi; "
"echo '=== DISK BEFORE ==='; df -h /; "
"echo '=== IMAGE PRUNE ==='; docker image prune -a -f 2>&1 | tail -3; "
"echo '=== VOLUME PRUNE ==='; docker volume prune -f 2>&1 | tail -3; "
"echo '=== BUILDER PRUNE ==='; docker builder prune -a -f 2>&1 | tail -3; "
"echo '=== DISK AFTER ==='; df -h /"
)
raise ValueError(f"No command builder for tool: {tool_name}")
async def _ssh_exec(