feat(p40): 觀測台收官 — 4 頁升 L2 + RAG 根因 + 蒸餾池監控
All checks were successful
CD Pipeline / deploy (push) Successful in 2m36s

接續 Phase 39 (commit 79cf08c),本 commit 完成 Phase D 最後 4 項:

D-6: quality_trend 蒸餾池 + RAG 根因
- 新「蒸餾池狀態」card:learning_episodes 各 promotion_status 分布
  (pending / awaiting_review / approved / rejected_quality /
   rejected_hallucination / rejected_duplicate / rejected_human / expired)
- 對最差 3 名 caller (avg_score < 3 且反饋 ≥ 3) 自動 RAG 根因建議
- RAG 從 ai_insights 召回相似低品質案例

D-7: ai_calls 一鍵 Code Review (L2)
- 新 POST /observability/ai_calls/trigger_code_review
  讀 git rev-parse HEAD + diff-tree 取最新變更檔案
  在 daemon thread 跑 CodeReviewPipeline.run() (5 step Hermes→
  OpenClaw→EA→NemoTron)
- 頁面新增「觸發 Code Review Pipeline」按鈕

D-8: ppt_audit 失敗 row 一鍵 AiderHeal (L2)
- 新 POST /observability/ppt_audit/trigger_aider_heal
  接收 pptx_filename + error_msg,呼叫 services/aider_heal_executor::
  execute_code_fix 自動修 services/ppt_generator.py
  AiderHeal 修完會 git push 觸發 CD
- audit_records 表中 status='failed'/'error' 的 row 自動顯示按鈕

D-9: host_health 一鍵 AutoHeal (L2)
- 新 POST /observability/host_health/trigger_autoheal
  接收 host_label,白名單對應 OLLAMA_HOST_PRIMARY/SECONDARY/FALLBACK
  防 SSRF。已標記 unhealthy 的 host 才允許觸發
  呼叫 auto_heal_service.handle_exception(error_type='ollama_unhealthy')
  跑 ADR-013 playbook(DOCKER_RESTART / SSH_CMD / ALERT_ONLY)
- 三主機 row 中 unhealthy / down 的 host 自動顯示按鈕

升級對應:
- AI 自動化:L2 從 1 個 → 4 個(budget force_throttle / Code Review /
  AiderHeal / AutoHeal)
- DB 利用率 ~60%:新增 learning_episodes 分布查詢
- RAG 整合 4/6(promotion_review + budget + quality_trend + 待 ppt_audit)

Phase 38+39+40 累計:6 commits 完成觀測台從 raw dashboard
升級到 AI 自動化專業舞台。

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
OoO
2026-05-04 19:13:39 +08:00
parent 79cf08c58c
commit 65f236da2d
5 changed files with 350 additions and 6 deletions

View File

@@ -12,6 +12,17 @@
<div class="alert alert-warning"><strong><i class="fas fa-exclamation-triangle me-1"></i></strong> {{ error }}</div>
{% endif %}
<!-- Phase 40 D-7: 一鍵觸發 Code Review (L2 自動化) -->
<div class="mb-3">
<button class="btn btn-warning btn-sm" onclick="triggerCodeReview()">
<i class="fas fa-microscope me-1"></i>觸發 Code Review Pipeline (5 step)
</button>
<small class="text-muted ms-2">
看到錯誤率飆高?一鍵觸發 Hermes→OpenClaw→EA→NemoTron 5 步審查最新 commit
在背景執行,完成後 Telegram 通知。
</small>
</div>
<!-- 篩選 bar -->
<form method="get" class="row g-2 mb-3">
<div class="col-auto">
@@ -172,7 +183,24 @@
</div>
<p class="text-muted mt-2"><small>
<i class="fas fa-robot me-1"></i>Operation Ollama-First v5.0 / Phase 29 — AI 呼叫總覽
<i class="fas fa-robot me-1"></i>Operation Ollama-First v5.0 / Phase 40 — AI 呼叫總覽(含 RAG/MCP 編排矩陣)
</small></p>
</div>
<script>
async function triggerCodeReview() {
if (!confirm('觸發 Code Review Pipeline\n\n會對最新 commit 跑 5 step 審查Hermes 掃描 → OpenClaw 摘要 → EA 決策 → NemoTron 行動),背景執行。')) return;
try {
const r = await fetch('/observability/ai_calls/trigger_code_review', {method: 'POST'});
const d = await r.json();
if (d.ok) {
alert(`${d.message}\n\nPipeline ID: ${d.pipeline_id}\nCommit: ${d.commit_sha}\n變更檔案: ${d.changed_files_count}`);
} else {
alert('❌ ' + (d.error || '觸發失敗'));
}
} catch (e) {
alert('Error: ' + e);
}
}
</script>
{% endblock %}

View File

@@ -14,7 +14,7 @@
<div class="card-body p-0">
<table class="table mb-0">
<thead class="table-light">
<tr><th>角色</th><th>主機</th><th>HTTP 健康</th><th>異常標記</th><th>已載入模型</th></tr>
<tr><th>角色</th><th>主機</th><th>HTTP 健康</th><th>異常標記</th><th>已載入模型</th><th>動作</th></tr>
</thead>
<tbody>
{% for h in ollama_hosts %}
@@ -41,6 +41,16 @@
{% endfor %}
{% if not h.models %}<small class="text-muted">無 / 未連線</small>{% endif %}
</td>
<td>
{% if h.unhealthy_mark or not h.healthy %}
<button class="btn btn-sm btn-outline-danger"
onclick="triggerAutoHeal({{ h.label|tojson }})">
<i class="fas fa-band-aid me-1"></i>AutoHeal
</button>
{% else %}
<small class="text-muted"></small>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
@@ -258,7 +268,29 @@
{% endif %}
<p class="text-muted mt-3"><small>
<i class="fas fa-robot me-1"></i>Operation Ollama-First v5.0 / Phase 38 — 主機健康監控(含 24h 歷史)
<i class="fas fa-robot me-1"></i>Operation Ollama-First v5.0 / Phase 40 — 主機健康監控(含 24h 歷史 / MCP / AIOps / AutoHeal L2
</small></p>
</div>
<script>
async function triggerAutoHeal(hostLabel) {
if (!confirm(`觸發 AutoHeal\n\n主機:${hostLabel}\n\n會跑對應 ADR-013 playbookDOCKER_RESTART / SSH_CMD / ALERT_ONLY並寫入 incidents 表。`)) return;
try {
const r = await fetch('/observability/host_health/trigger_autoheal', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({host_label: hostLabel}),
});
const d = await r.json();
if (d.ok) {
alert(`✅ AutoHeal 已派出\n動作:${d.action || '—'}\n訊息:${d.message || ''}`);
window.location.reload();
} else {
alert('❌ ' + (d.error || d.message || '觸發失敗'));
}
} catch (e) {
alert('Error: ' + e);
}
}
</script>
{% endblock %}

View File

@@ -55,7 +55,7 @@
<tr>
<th>審核時間</th><th>檔名</th><th>結果</th>
<th class="text-end">問題數</th><th class="text-end">信心度</th>
<th class="text-end">耗時 ms</th><th>錯誤訊息</th>
<th class="text-end">耗時 ms</th><th>錯誤訊息</th><th>動作</th>
</tr>
</thead>
<tbody>
@@ -80,9 +80,17 @@
<td class="text-end">{{ "%.2f"|format(r.confidence) }}</td>
<td class="text-end">{{ r.duration_ms }}</td>
<td><small class="text-muted">{{ (r.error_msg or '')[:80] }}</small></td>
<td>
{% if r.audit_status in ('failed', 'error') %}
<button class="btn btn-sm btn-outline-warning"
onclick="triggerAiderHeal({{ r.pptx_filename|tojson }}, {{ (r.error_msg or '')|tojson }})">
<i class="fas fa-wrench me-1"></i>AiderHeal
</button>
{% endif %}
</td>
</tr>
{% else %}
<tr><td colspan="7" class="text-center text-muted">尚無審核紀錄migration 030 跑過後即會累積)</td></tr>
<tr><td colspan="8" class="text-center text-muted">尚無審核紀錄migration 030 跑過後即會累積)</td></tr>
{% endfor %}
</tbody>
</table>
@@ -96,7 +104,28 @@
</p>
<p class="text-muted mt-3"><small>
<i class="fas fa-robot me-1"></i>Operation Ollama-First v5.0 / Phase 29 — PPT 視覺審核歷史
<i class="fas fa-robot me-1"></i>Operation Ollama-First v5.0 / Phase 40 — PPT 視覺審核歷史(含 AiderHeal L2
</small></p>
</div>
<script>
async function triggerAiderHeal(pptxFilename, errorMsg) {
if (!confirm(`觸發 AiderHeal 自動修復?\n\n檔案:${pptxFilename}\n錯誤:${(errorMsg || '').substring(0, 200)}\n\nAiderHeal 會嘗試修 services/ppt_generator.py 並 git push 到 main 觸發 CD。`)) return;
try {
const r = await fetch('/observability/ppt_audit/trigger_aider_heal', {
method: 'POST',
headers: {'Content-Type': 'application/json'},
body: JSON.stringify({pptx_filename: pptxFilename, error_msg: errorMsg || ''}),
});
const d = await r.json();
if (d.ok) {
alert(`✅ AiderHeal 已派出\n動作:${d.action || '—'}\n訊息:${d.message || ''}`);
} else {
alert('❌ ' + (d.error || d.message || '觸發失敗'));
}
} catch (e) {
alert('Error: ' + e);
}
}
</script>
{% endblock %}

View File

@@ -23,6 +23,63 @@
<div class="col-auto"><button class="btn btn-primary btn-sm">查詢</button></div>
</form>
{% if episode_distribution %}
<div class="card mb-3">
<div class="card-header"><strong><i class="fas fa-flask me-2"></i>蒸餾池狀態learning_episodes 過去 {{ days }} 日)</strong>
<small class="text-muted">資料來源learning_episodes — 展現 RAG 學習鏈路飽和度</small>
</div>
<div class="card-body">
<div class="row g-2">
{% for status, cnt in episode_distribution.items() %}
<div class="col-md-2 col-sm-4">
<div class="border rounded p-2 text-center">
<small class="text-muted d-block">
{% if status == 'pending' %}<i class="fas fa-hourglass-start"></i> 待處理
{% elif status == 'awaiting_review' %}<i class="fas fa-user-clock"></i> 待審核
{% elif status == 'approved' %}<i class="fas fa-check-circle text-success"></i> 已晉升
{% elif status == 'rejected_quality' %}<i class="fas fa-times text-danger"></i> 品質拒
{% elif status == 'rejected_hallucination' %}<i class="fas fa-times text-danger"></i> 幻覺拒
{% elif status == 'rejected_duplicate' %}<i class="fas fa-clone text-warning"></i> 重複拒
{% elif status == 'rejected_human' %}<i class="fas fa-user-times text-danger"></i> 人工拒
{% elif status == 'expired' %}<i class="fas fa-clock text-muted"></i> 已過期
{% else %}{{ status }}{% endif %}
</small>
<strong style="font-size: 1.4em;">{{ cnt }}</strong>
</div>
</div>
{% endfor %}
</div>
</div>
</div>
{% endif %}
{% if rag_root_causes %}
<div class="card mb-3" style="border-left: 4px solid #6f42c1;">
<div class="card-header bg-light">
<strong><i class="fas fa-stethoscope me-2"></i>RAG 自動根因建議</strong>
<small class="text-muted">— 對最差 3 名 caller 自動從 ai_insights 召回相似案例</small>
</div>
<div class="card-body p-2">
{% for rc in rag_root_causes %}
<div class="mb-3 p-2" style="background: #fafafa; border-radius: 6px;">
<strong><code>{{ rc.caller }}</code></strong>
<span class="badge bg-danger ms-1">{{ "%.2f"|format(rc.avg_score) }}/5</span>
<span class="badge bg-secondary ms-1">{{ rc.feedback_n }} 筆反饋</span>
<ul class="list-unstyled mt-2 mb-0 small">
{% for h in rc.hits %}
<li class="mb-1">
<span class="badge bg-info text-dark me-1">{{ h.insight_type }}</span>
<span class="badge bg-light text-dark me-1">相似度 {{ "%.2f"|format(h.similarity) }}</span>
{{ h.content }}{% if h.content|length >= 200 %}…{% endif %}
</li>
{% endfor %}
</ul>
</div>
{% endfor %}
</div>
</div>
{% endif %}
{% if recommendations %}
<div class="card mb-3">
<div class="card-header bg-warning"><strong><i class="fas fa-lightbulb me-2"></i>智能建議</strong></div>