feat(aiops): capacity_forecaster — Phase 4 Holt-Winters MVP (predict_linear)
Some checks failed
CD Pipeline / build-and-deploy (push) Has been cancelled

統帥批准 4 項下階段候選之一完成: AI 容量預測.

新增 capacity_forecaster_job.py (~220 行):
  每日 05:00 Taipei 跑預測 (02:00 scanner → 03:00 compliance →
  04:00 Hermes → 05:00 forecaster 形成完整日鏈).

預測方法論 (MVP):
  Prometheus predict_linear(metric[7d], 86400*7) — 基於過去 7d 做線性外推
  3 個預測 query:
    1. disk_saturation_7d: predict_linear(node_filesystem_avail_bytes[7d], 7d) < 0
    2. mem_saturation_7d: predict_linear(MemAvailable[7d], 7d) / MemTotal < 10%
    3. cpu_high_7d_trend: avg_over_time(cpu_used_pct[7d]) > 70%

發現高風險 host → 寫 aol(capacity_recommendation) + 推 Telegram
  - input: host + horizon + findings count
  - output: findings list + proposed_actions + requires_human_decision=true

proposed_actions 依 findings 推導:
  - disk: 清理 log/docker/PG WAL 或擴容
  - mem: top consumer / JVM 調整
  - cpu: scale out / vCPU 擴充

統帥鐵律對齊:
   只推建議不自動 scale up
   7d window 有足夠樣本
   AI 預測 + 人工決策

未來 TODO:
  - 真 Holt-Winters (含季節性) — 需 Python statsmodels
  - 業務週期調整 (週一高峰/週末低谷)

Wire main.py lifespan asyncio.create_task()

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
OG T
2026-04-19 20:00:36 +08:00
parent 53618b25c9
commit a391dfc389
2 changed files with 278 additions and 0 deletions

View File

@@ -450,6 +450,16 @@ async def lifespan(_app: FastAPI) -> AsyncGenerator[None, None]:
except Exception as e:
logger.warning("hermes_rule_quality_loop_schedule_failed", error=str(e))
# ADR-090 § Phase 4 Capacity Forecaster (2026-04-19 ogt + Claude Opus 4.7 Asia/Taipei)
# 每日 05:00 Taipei 用 Prometheus predict_linear 預測未來 7d disk/mem/cpu 飽和
# 高風險 host 寫 aol(capacity_recommendation) + Telegram 建議
try:
from src.jobs.capacity_forecaster_job import run_capacity_forecaster_loop
asyncio.create_task(run_capacity_forecaster_loop())
logger.info("capacity_forecaster_loop_scheduled", daily_trigger_hour_taipei=5)
except Exception as e:
logger.warning("capacity_forecaster_loop_schedule_failed", error=str(e))
# ADR-076 Task 4: 每日 08:00 台北時間自動日度巡檢報告
# 2026-04-14 Claude Haiku 4.5 Asia/Taipei
try: