All checks were successful
CD Pipeline / deploy (push) Successful in 2m30s
統帥質疑:6 頁觀測台只是 raw stats dashboard,沒展現 AI 自動化專業。 深度盤點 4 軸結果: - DB 利用率 22.7%(22 表只用 5 張) - MCP 整合 1/6(mcp_calls 表完全沒被讀) - RAG 整合 0/6(沒 import rag_service) - AI 自動化 L0 × 5 + L1 × 1(純讀 dashboard,無一鍵觸發) 本 commit 5 個增強: D-1: promotion_review 加 RAG「Top 3 相似已晉升」 - 對每筆 awaiting_review episode 跑 rag_service.query 找 ai_insights 中 cosine ≥ 0.7 的相似已晉升內容 - 輔助人工判斷:是否冗餘?是否新領域? - header 顯示 ai_insights 知識庫 size - fail-safe: 單筆 RAG 失敗不影響其餘 D-2: host_health 加 MCP 24h 工作量 widget - 從 mcp_calls 統計各 server 24h 呼叫次數 / 成功率 / cache 率 / 使用 tool 數 / 平均耗時 / cost - 展現「AI×MCP 編排規模」而非只「server 健康與否」 D-3: ai_calls × rag_query_log × mcp_calls 三表 JOIN - 新增「呼叫端 × RAG × MCP 編排矩陣」card - 每個 caller:總呼叫 / RAG 命中率 / MCP 編排率(透過 request_id 串接) / RAG 反饋分數 / 反饋筆數 - 展現「AI 自動化專業」核心指標 D-4: budget 加 RAG 自動策略建議 + 一鍵 force-throttle (L2) - ratio ≥ 0.8 時自動 RAG 召回 ai_insights 中的 budget_strategy 知識 - POST /budget/force_throttle endpoint:立即重算 cost_throttle 狀態 (不等下次每小時 cron)— 升級到 L2 自動化 - 對應頁面加「立即重算節流狀態」按鈕 D-5: host_health 加 incidents + heal_logs 7d 摘要 - 顯示 ADR-013 AutoHeal 閉環核心 KPI: 總事件 / 未解決 / 已解決 / P0+P1 / 自癒成功率 / 平均自癒耗時 - 展現「AIOps 自癒系統」運作實況 對應升級: - DB 利用率 22.7% → ~50%(新接 mcp_calls + rag_query_log JOIN + ai_insights + incidents + heal_logs) - MCP 整合 1/6 → 3/6(host_health + ai_calls + budget 都接 mcp_calls) - RAG 整合 0/6 → 3/6(promotion_review + budget + 待 quality_trend) - AI 自動化 L1 → L2 一鍵 force-throttle 一個(其餘按鈕待 D-6) 全部 fail-safe:DB 表/RAG/MCP 失敗都不擋頁面渲染。 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
265 lines
11 KiB
HTML
265 lines
11 KiB
HTML
{% extends "base.html" %}
|
||
|
||
{% block title %}主機健康監控{% endblock %}
|
||
|
||
{% block content %}
|
||
<div class="container-fluid mt-3">
|
||
<h2 class="mb-3"><i class="fas fa-heartbeat me-2"></i>主機健康監控
|
||
<small class="text-muted">三主機 Ollama + MCP + 成本節流即時狀態</small>
|
||
</h2>
|
||
|
||
<!-- Ollama 三主機 -->
|
||
<div class="card mb-3">
|
||
<div class="card-header"><strong><i class="fas fa-server me-2"></i>Ollama 三主機(HTTP /api/tags 即時 probe)</strong></div>
|
||
<div class="card-body p-0">
|
||
<table class="table mb-0">
|
||
<thead class="table-light">
|
||
<tr><th>角色</th><th>主機</th><th>HTTP 健康</th><th>異常標記</th><th>已載入模型</th></tr>
|
||
</thead>
|
||
<tbody>
|
||
{% for h in ollama_hosts %}
|
||
<tr>
|
||
<td><strong>{{ h.label }}</strong></td>
|
||
<td><code>{{ h.host }}</code></td>
|
||
<td>
|
||
{% if h.healthy %}
|
||
<span class="badge bg-success"><i class="fas fa-check me-1"></i>HTTP 正常</span>
|
||
{% else %}
|
||
<span class="badge bg-danger"><i class="fas fa-times me-1"></i>離線</span>
|
||
{% endif %}
|
||
</td>
|
||
<td>
|
||
{% if h.unhealthy_mark %}
|
||
<span class="badge bg-warning"><i class="fas fa-exclamation-triangle me-1"></i>已標記異常(30 秒)</span>
|
||
{% else %}
|
||
<span class="badge bg-light text-dark">—</span>
|
||
{% endif %}
|
||
</td>
|
||
<td>
|
||
{% for m in h.models %}
|
||
<span class="badge bg-info text-dark me-1">{{ m }}</span>
|
||
{% endfor %}
|
||
{% if not h.models %}<small class="text-muted">無 / 未連線</small>{% endif %}
|
||
</td>
|
||
</tr>
|
||
{% endfor %}
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
</div>
|
||
|
||
<!-- MCP servers -->
|
||
<div class="card mb-3">
|
||
<div class="card-header"><strong><i class="fas fa-plug me-2"></i>MCP 服務(Phase 10/10.5)</strong></div>
|
||
<div class="card-body p-0">
|
||
<table class="table mb-0">
|
||
<thead class="table-light">
|
||
<tr><th>服務名稱</th><th>狀態</th></tr>
|
||
</thead>
|
||
<tbody>
|
||
{% for server, healthy in mcp_status.items() %}
|
||
<tr>
|
||
<td><code>{{ server }}</code></td>
|
||
<td>
|
||
{% if healthy %}
|
||
<span class="badge bg-success"><i class="fas fa-check me-1"></i>正常</span>
|
||
{% else %}
|
||
<span class="badge bg-secondary">— 未啟用 / 離線</span>
|
||
{% endif %}
|
||
</td>
|
||
</tr>
|
||
{% else %}
|
||
<tr><td colspan="2" class="text-muted small">MCP_ROUTER_ENABLED=false 或 mcp-stack 未部署</td></tr>
|
||
{% endfor %}
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
</div>
|
||
|
||
<!-- Cost Throttle 狀態(Phase 20) -->
|
||
<div class="card mb-3">
|
||
<div class="card-header"><strong><i class="fas fa-dollar-sign me-2"></i>成本節流狀態(Phase 20)</strong></div>
|
||
<div class="card-body p-0">
|
||
{% if throttle_state %}
|
||
<table class="table mb-0">
|
||
<thead class="table-light">
|
||
<tr><th>供應商</th><th>已花費</th><th>預算</th><th>月底推估</th><th>使用率</th><th>狀態</th></tr>
|
||
</thead>
|
||
<tbody>
|
||
{% for provider, info in throttle_state.items() %}
|
||
<tr {% if info.throttled %}class="table-warning"{% endif %}>
|
||
<td><code>{{ provider }}</code></td>
|
||
<td>${{ "%.2f"|format(info.spent) }}</td>
|
||
<td>${{ "%.2f"|format(info.budget) }}</td>
|
||
<td>${{ "%.2f"|format(info.projected) }}</td>
|
||
<td>{{ "%.0f"|format(info.ratio * 100) }}%</td>
|
||
<td>
|
||
{% if info.throttled %}
|
||
<span class="badge bg-danger"><i class="fas fa-exclamation-triangle me-1"></i>已節流</span>
|
||
{% else %}
|
||
<span class="badge bg-success"><i class="fas fa-check me-1"></i>正常</span>
|
||
{% endif %}
|
||
</td>
|
||
</tr>
|
||
{% endfor %}
|
||
</tbody>
|
||
</table>
|
||
{% else %}
|
||
<p class="text-muted m-3 small">
|
||
COST_THROTTLE_ENABLED=false 或尚未首次評估(每小時 cron 執行)
|
||
</p>
|
||
{% endif %}
|
||
</div>
|
||
</div>
|
||
|
||
<!-- AIOps 7d 摘要(Phase 39 D-5 新增) -->
|
||
{% if aiops_summary %}
|
||
<div class="card mb-3" style="border-left: 4px solid #0d6efd;">
|
||
<div class="card-header">
|
||
<strong><i class="fas fa-shield-virus me-2"></i>AIOps 自癒系統 7 日摘要</strong>
|
||
<small class="text-muted">資料來源:incidents + heal_logs(ADR-013 AutoHeal 閉環)</small>
|
||
</div>
|
||
<div class="card-body">
|
||
<div class="row g-2">
|
||
<div class="col-md-2 col-sm-4">
|
||
<div class="border rounded p-2 text-center">
|
||
<small class="text-muted d-block">總事件</small>
|
||
<strong style="font-size: 1.4em;">{{ aiops_summary.incidents_total }}</strong>
|
||
</div>
|
||
</div>
|
||
<div class="col-md-2 col-sm-4">
|
||
<div class="border rounded p-2 text-center">
|
||
<small class="text-muted d-block">未解決</small>
|
||
<strong class="{% if aiops_summary.incidents_open > 0 %}text-danger{% endif %}" style="font-size: 1.4em;">
|
||
{{ aiops_summary.incidents_open }}
|
||
</strong>
|
||
</div>
|
||
</div>
|
||
<div class="col-md-2 col-sm-4">
|
||
<div class="border rounded p-2 text-center">
|
||
<small class="text-muted d-block">已解決</small>
|
||
<strong class="text-success" style="font-size: 1.4em;">{{ aiops_summary.incidents_resolved }}</strong>
|
||
</div>
|
||
</div>
|
||
<div class="col-md-2 col-sm-4">
|
||
<div class="border rounded p-2 text-center">
|
||
<small class="text-muted d-block">P0/P1</small>
|
||
<strong class="{% if (aiops_summary.incidents_p0 + aiops_summary.incidents_p1) > 0 %}text-danger{% endif %}" style="font-size: 1.4em;">
|
||
{{ aiops_summary.incidents_p0 + aiops_summary.incidents_p1 }}
|
||
</strong>
|
||
</div>
|
||
</div>
|
||
<div class="col-md-2 col-sm-4">
|
||
<div class="border rounded p-2 text-center">
|
||
<small class="text-muted d-block">自癒成功率</small>
|
||
<strong class="{% if aiops_summary.heal_success_rate >= 80 %}text-success{% elif aiops_summary.heal_success_rate >= 50 %}text-warning{% else %}text-danger{% endif %}" style="font-size: 1.4em;">
|
||
{{ "%.0f"|format(aiops_summary.heal_success_rate) }}%
|
||
</strong>
|
||
</div>
|
||
</div>
|
||
<div class="col-md-2 col-sm-4">
|
||
<div class="border rounded p-2 text-center">
|
||
<small class="text-muted d-block">平均自癒耗時</small>
|
||
<strong style="font-size: 1.4em;">{{ aiops_summary.heals_avg_ms }} ms</strong>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="mt-2 small text-muted">
|
||
<i class="fas fa-info-circle me-1"></i>
|
||
7d 共 {{ aiops_summary.heals_total }} 次自癒嘗試
|
||
(成功 {{ aiops_summary.heals_success }} · 失敗 {{ aiops_summary.heals_failed }})
|
||
</div>
|
||
</div>
|
||
</div>
|
||
{% endif %}
|
||
|
||
<!-- MCP 24h 工作量(Phase 39 D-2 新增) -->
|
||
{% if mcp_24h %}
|
||
<div class="card mb-3">
|
||
<div class="card-header"><strong><i class="fas fa-bolt me-2"></i>MCP 服務 24h 工作量</strong>
|
||
<small class="text-muted">資料來源:mcp_calls 表 — 展現 AI×MCP 編排規模</small>
|
||
</div>
|
||
<div class="card-body p-0">
|
||
<table class="table mb-0">
|
||
<thead class="table-light">
|
||
<tr>
|
||
<th>服務</th>
|
||
<th class="text-end">呼叫次數</th>
|
||
<th class="text-end">成功率</th>
|
||
<th class="text-end">快取命中率</th>
|
||
<th class="text-end">使用 Tool 數</th>
|
||
<th class="text-end">平均耗時</th>
|
||
<th class="text-end">成本 (USD)</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
{% for s in mcp_24h %}
|
||
<tr>
|
||
<td><code>{{ s.server }}</code></td>
|
||
<td class="text-end">{{ "{:,}".format(s.total_calls) }}</td>
|
||
<td class="text-end">
|
||
<strong class="{% if s.success_rate >= 95 %}text-success{% elif s.success_rate >= 80 %}text-warning{% else %}text-danger{% endif %}">
|
||
{{ "%.1f"|format(s.success_rate) }}%
|
||
</strong>
|
||
</td>
|
||
<td class="text-end">
|
||
<span class="{% if s.cache_rate >= 30 %}text-success{% endif %}">
|
||
{{ "%.1f"|format(s.cache_rate) }}%
|
||
</span>
|
||
</td>
|
||
<td class="text-end">{{ s.tools_used }}</td>
|
||
<td class="text-end">{{ s.avg_ms }} ms</td>
|
||
<td class="text-end">${{ "%.4f"|format(s.total_cost) }}</td>
|
||
</tr>
|
||
{% endfor %}
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
</div>
|
||
{% endif %}
|
||
|
||
<!-- 過去 24h 健康趨勢(Phase 38 新增) -->
|
||
{% if health_history %}
|
||
<div class="card mb-3">
|
||
<div class="card-header"><strong><i class="fas fa-chart-line me-2"></i>過去 24 小時健康趨勢</strong>
|
||
<small class="text-muted">資料來源:host_health_probes(每次刷新自動寫入)</small>
|
||
</div>
|
||
<div class="card-body p-0">
|
||
<table class="table mb-0">
|
||
<thead class="table-light">
|
||
<tr>
|
||
<th>角色</th>
|
||
<th class="text-end">總探針次數</th>
|
||
<th class="text-end">正常次數</th>
|
||
<th class="text-end">離線次數</th>
|
||
<th class="text-end">在線率</th>
|
||
<th class="text-end">平均回應 ms</th>
|
||
</tr>
|
||
</thead>
|
||
<tbody>
|
||
{% for h in health_history %}
|
||
<tr>
|
||
<td><strong>{{ h.host_label }}</strong></td>
|
||
<td class="text-end">{{ h.total }}</td>
|
||
<td class="text-end text-success">{{ h.up_count }}</td>
|
||
<td class="text-end text-danger">{{ h.down_count }}</td>
|
||
<td class="text-end">
|
||
<strong class="{% if h.uptime_pct >= 99 %}text-success{% elif h.uptime_pct >= 90 %}text-warning{% else %}text-danger{% endif %}">
|
||
{{ "%.1f"|format(h.uptime_pct) }}%
|
||
</strong>
|
||
</td>
|
||
<td class="text-end">{{ h.avg_ms }}</td>
|
||
</tr>
|
||
{% endfor %}
|
||
</tbody>
|
||
</table>
|
||
</div>
|
||
</div>
|
||
{% endif %}
|
||
|
||
<p class="text-muted mt-3"><small>
|
||
<i class="fas fa-robot me-1"></i>Operation Ollama-First v5.0 / Phase 38 — 主機健康監控(含 24h 歷史)
|
||
</small></p>
|
||
</div>
|
||
{% endblock %}
|