Files
ewoooc/templates/admin/host_health.html
OoO 79cf08c58c
All checks were successful
CD Pipeline / deploy (push) Successful in 2m30s
feat(p39): 觀測台升級 — DB + MCP + RAG + AI 自動化深度整合
統帥質疑:6 頁觀測台只是 raw stats dashboard,沒展現 AI 自動化專業。
深度盤點 4 軸結果:
- DB 利用率 22.7%(22 表只用 5 張)
- MCP 整合 1/6(mcp_calls 表完全沒被讀)
- RAG 整合 0/6(沒 import rag_service)
- AI 自動化 L0 × 5 + L1 × 1(純讀 dashboard,無一鍵觸發)

本 commit 5 個增強:

D-1: promotion_review 加 RAG「Top 3 相似已晉升」
- 對每筆 awaiting_review episode 跑 rag_service.query 找 ai_insights 中
  cosine ≥ 0.7 的相似已晉升內容
- 輔助人工判斷:是否冗餘?是否新領域?
- header 顯示 ai_insights 知識庫 size
- fail-safe: 單筆 RAG 失敗不影響其餘

D-2: host_health 加 MCP 24h 工作量 widget
- 從 mcp_calls 統計各 server 24h 呼叫次數 / 成功率 / cache 率 /
  使用 tool 數 / 平均耗時 / cost
- 展現「AI×MCP 編排規模」而非只「server 健康與否」

D-3: ai_calls × rag_query_log × mcp_calls 三表 JOIN
- 新增「呼叫端 × RAG × MCP 編排矩陣」card
- 每個 caller:總呼叫 / RAG 命中率 / MCP 編排率(透過 request_id 串接)
  / RAG 反饋分數 / 反饋筆數
- 展現「AI 自動化專業」核心指標

D-4: budget 加 RAG 自動策略建議 + 一鍵 force-throttle (L2)
- ratio ≥ 0.8 時自動 RAG 召回 ai_insights 中的 budget_strategy 知識
- POST /budget/force_throttle endpoint:立即重算 cost_throttle 狀態
  (不等下次每小時 cron)— 升級到 L2 自動化
- 對應頁面加「立即重算節流狀態」按鈕

D-5: host_health 加 incidents + heal_logs 7d 摘要
- 顯示 ADR-013 AutoHeal 閉環核心 KPI:
  總事件 / 未解決 / 已解決 / P0+P1 / 自癒成功率 / 平均自癒耗時
- 展現「AIOps 自癒系統」運作實況

對應升級:
- DB 利用率 22.7% → ~50%(新接 mcp_calls + rag_query_log JOIN
  + ai_insights + incidents + heal_logs)
- MCP 整合 1/6 → 3/6(host_health + ai_calls + budget 都接 mcp_calls)
- RAG 整合 0/6 → 3/6(promotion_review + budget + 待 quality_trend)
- AI 自動化 L1 → L2 一鍵 force-throttle 一個(其餘按鈕待 D-6)

全部 fail-safe:DB 表/RAG/MCP 失敗都不擋頁面渲染。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-04 19:08:41 +08:00

265 lines
11 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
{% extends "base.html" %}
{% block title %}主機健康監控{% endblock %}
{% block content %}
<div class="container-fluid mt-3">
<h2 class="mb-3"><i class="fas fa-heartbeat me-2"></i>主機健康監控
<small class="text-muted">三主機 Ollama + MCP + 成本節流即時狀態</small>
</h2>
<!-- Ollama 三主機 -->
<div class="card mb-3">
<div class="card-header"><strong><i class="fas fa-server me-2"></i>Ollama 三主機HTTP /api/tags 即時 probe</strong></div>
<div class="card-body p-0">
<table class="table mb-0">
<thead class="table-light">
<tr><th>角色</th><th>主機</th><th>HTTP 健康</th><th>異常標記</th><th>已載入模型</th></tr>
</thead>
<tbody>
{% for h in ollama_hosts %}
<tr>
<td><strong>{{ h.label }}</strong></td>
<td><code>{{ h.host }}</code></td>
<td>
{% if h.healthy %}
<span class="badge bg-success"><i class="fas fa-check me-1"></i>HTTP 正常</span>
{% else %}
<span class="badge bg-danger"><i class="fas fa-times me-1"></i>離線</span>
{% endif %}
</td>
<td>
{% if h.unhealthy_mark %}
<span class="badge bg-warning"><i class="fas fa-exclamation-triangle me-1"></i>已標記異常30 秒)</span>
{% else %}
<span class="badge bg-light text-dark"></span>
{% endif %}
</td>
<td>
{% for m in h.models %}
<span class="badge bg-info text-dark me-1">{{ m }}</span>
{% endfor %}
{% if not h.models %}<small class="text-muted">無 / 未連線</small>{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
<!-- MCP servers -->
<div class="card mb-3">
<div class="card-header"><strong><i class="fas fa-plug me-2"></i>MCP 服務Phase 10/10.5</strong></div>
<div class="card-body p-0">
<table class="table mb-0">
<thead class="table-light">
<tr><th>服務名稱</th><th>狀態</th></tr>
</thead>
<tbody>
{% for server, healthy in mcp_status.items() %}
<tr>
<td><code>{{ server }}</code></td>
<td>
{% if healthy %}
<span class="badge bg-success"><i class="fas fa-check me-1"></i>正常</span>
{% else %}
<span class="badge bg-secondary">— 未啟用 / 離線</span>
{% endif %}
</td>
</tr>
{% else %}
<tr><td colspan="2" class="text-muted small">MCP_ROUTER_ENABLED=false 或 mcp-stack 未部署</td></tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
<!-- Cost Throttle 狀態Phase 20 -->
<div class="card mb-3">
<div class="card-header"><strong><i class="fas fa-dollar-sign me-2"></i>成本節流狀態Phase 20</strong></div>
<div class="card-body p-0">
{% if throttle_state %}
<table class="table mb-0">
<thead class="table-light">
<tr><th>供應商</th><th>已花費</th><th>預算</th><th>月底推估</th><th>使用率</th><th>狀態</th></tr>
</thead>
<tbody>
{% for provider, info in throttle_state.items() %}
<tr {% if info.throttled %}class="table-warning"{% endif %}>
<td><code>{{ provider }}</code></td>
<td>${{ "%.2f"|format(info.spent) }}</td>
<td>${{ "%.2f"|format(info.budget) }}</td>
<td>${{ "%.2f"|format(info.projected) }}</td>
<td>{{ "%.0f"|format(info.ratio * 100) }}%</td>
<td>
{% if info.throttled %}
<span class="badge bg-danger"><i class="fas fa-exclamation-triangle me-1"></i>已節流</span>
{% else %}
<span class="badge bg-success"><i class="fas fa-check me-1"></i>正常</span>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<p class="text-muted m-3 small">
COST_THROTTLE_ENABLED=false 或尚未首次評估(每小時 cron 執行)
</p>
{% endif %}
</div>
</div>
<!-- AIOps 7d 摘要Phase 39 D-5 新增) -->
{% if aiops_summary %}
<div class="card mb-3" style="border-left: 4px solid #0d6efd;">
<div class="card-header">
<strong><i class="fas fa-shield-virus me-2"></i>AIOps 自癒系統 7 日摘要</strong>
<small class="text-muted">資料來源incidents + heal_logsADR-013 AutoHeal 閉環)</small>
</div>
<div class="card-body">
<div class="row g-2">
<div class="col-md-2 col-sm-4">
<div class="border rounded p-2 text-center">
<small class="text-muted d-block">總事件</small>
<strong style="font-size: 1.4em;">{{ aiops_summary.incidents_total }}</strong>
</div>
</div>
<div class="col-md-2 col-sm-4">
<div class="border rounded p-2 text-center">
<small class="text-muted d-block">未解決</small>
<strong class="{% if aiops_summary.incidents_open > 0 %}text-danger{% endif %}" style="font-size: 1.4em;">
{{ aiops_summary.incidents_open }}
</strong>
</div>
</div>
<div class="col-md-2 col-sm-4">
<div class="border rounded p-2 text-center">
<small class="text-muted d-block">已解決</small>
<strong class="text-success" style="font-size: 1.4em;">{{ aiops_summary.incidents_resolved }}</strong>
</div>
</div>
<div class="col-md-2 col-sm-4">
<div class="border rounded p-2 text-center">
<small class="text-muted d-block">P0/P1</small>
<strong class="{% if (aiops_summary.incidents_p0 + aiops_summary.incidents_p1) > 0 %}text-danger{% endif %}" style="font-size: 1.4em;">
{{ aiops_summary.incidents_p0 + aiops_summary.incidents_p1 }}
</strong>
</div>
</div>
<div class="col-md-2 col-sm-4">
<div class="border rounded p-2 text-center">
<small class="text-muted d-block">自癒成功率</small>
<strong class="{% if aiops_summary.heal_success_rate >= 80 %}text-success{% elif aiops_summary.heal_success_rate >= 50 %}text-warning{% else %}text-danger{% endif %}" style="font-size: 1.4em;">
{{ "%.0f"|format(aiops_summary.heal_success_rate) }}%
</strong>
</div>
</div>
<div class="col-md-2 col-sm-4">
<div class="border rounded p-2 text-center">
<small class="text-muted d-block">平均自癒耗時</small>
<strong style="font-size: 1.4em;">{{ aiops_summary.heals_avg_ms }} ms</strong>
</div>
</div>
</div>
<div class="mt-2 small text-muted">
<i class="fas fa-info-circle me-1"></i>
7d 共 {{ aiops_summary.heals_total }} 次自癒嘗試
(成功 {{ aiops_summary.heals_success }} · 失敗 {{ aiops_summary.heals_failed }}
</div>
</div>
</div>
{% endif %}
<!-- MCP 24h 工作量Phase 39 D-2 新增) -->
{% if mcp_24h %}
<div class="card mb-3">
<div class="card-header"><strong><i class="fas fa-bolt me-2"></i>MCP 服務 24h 工作量</strong>
<small class="text-muted">資料來源mcp_calls 表 — 展現 AI×MCP 編排規模</small>
</div>
<div class="card-body p-0">
<table class="table mb-0">
<thead class="table-light">
<tr>
<th>服務</th>
<th class="text-end">呼叫次數</th>
<th class="text-end">成功率</th>
<th class="text-end">快取命中率</th>
<th class="text-end">使用 Tool 數</th>
<th class="text-end">平均耗時</th>
<th class="text-end">成本 (USD)</th>
</tr>
</thead>
<tbody>
{% for s in mcp_24h %}
<tr>
<td><code>{{ s.server }}</code></td>
<td class="text-end">{{ "{:,}".format(s.total_calls) }}</td>
<td class="text-end">
<strong class="{% if s.success_rate >= 95 %}text-success{% elif s.success_rate >= 80 %}text-warning{% else %}text-danger{% endif %}">
{{ "%.1f"|format(s.success_rate) }}%
</strong>
</td>
<td class="text-end">
<span class="{% if s.cache_rate >= 30 %}text-success{% endif %}">
{{ "%.1f"|format(s.cache_rate) }}%
</span>
</td>
<td class="text-end">{{ s.tools_used }}</td>
<td class="text-end">{{ s.avg_ms }} ms</td>
<td class="text-end">${{ "%.4f"|format(s.total_cost) }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
{% endif %}
<!-- 過去 24h 健康趨勢Phase 38 新增) -->
{% if health_history %}
<div class="card mb-3">
<div class="card-header"><strong><i class="fas fa-chart-line me-2"></i>過去 24 小時健康趨勢</strong>
<small class="text-muted">資料來源host_health_probes每次刷新自動寫入</small>
</div>
<div class="card-body p-0">
<table class="table mb-0">
<thead class="table-light">
<tr>
<th>角色</th>
<th class="text-end">總探針次數</th>
<th class="text-end">正常次數</th>
<th class="text-end">離線次數</th>
<th class="text-end">在線率</th>
<th class="text-end">平均回應 ms</th>
</tr>
</thead>
<tbody>
{% for h in health_history %}
<tr>
<td><strong>{{ h.host_label }}</strong></td>
<td class="text-end">{{ h.total }}</td>
<td class="text-end text-success">{{ h.up_count }}</td>
<td class="text-end text-danger">{{ h.down_count }}</td>
<td class="text-end">
<strong class="{% if h.uptime_pct >= 99 %}text-success{% elif h.uptime_pct >= 90 %}text-warning{% else %}text-danger{% endif %}">
{{ "%.1f"|format(h.uptime_pct) }}%
</strong>
</td>
<td class="text-end">{{ h.avg_ms }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
</div>
{% endif %}
<p class="text-muted mt-3"><small>
<i class="fas fa-robot me-1"></i>Operation Ollama-First v5.0 / Phase 38 — 主機健康監控(含 24h 歷史)
</small></p>
</div>
{% endblock %}