14c81f728f
新增: - ADR-025: 告警鏈路 E2E 驗證架構 (2026-03-26 事故教訓) 更新: - ADR-011: 新增 DNS 規則最佳實踐 (附錄 B) - Skill 04: 新增 NetworkPolicy DNS 規則 + CoreDNS 設定 - Skill 05: 新增告警鏈路 Smoke Test 要求 - CLAUDE.md: 新增告警鏈路驗證到任務前必讀 事故根因: 1. URL 路徑錯誤 (webhook vs webhooks) 2. NetworkPolicy DNS 規則標籤不匹配 3. CoreDNS 上游 DNS 依賴 systemd-resolved Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
340 lines
12 KiB
Markdown
340 lines
12 KiB
Markdown
# ADR-011: NetworkPolicy 變更治理架構
|
||
|
||
**狀態**: 提案
|
||
**日期**: 2026-03-23
|
||
**決策者**: 統帥
|
||
**觸發**: 多次 NetworkPolicy 變更導致生產事故
|
||
|
||
## 問題陳述
|
||
|
||
```
|
||
事故時間線：
|
||
├── 2026-03-20: Worker 無法連 Redis → 發現 Egress 被阻擋
|
||
├── 2026-03-22: OTEL 上報失敗 → 發現 Port 24317/24318 未開
|
||
├── 2026-03-23: Y 按鈕執行超時 → K8s API 192.168.0.120:6443 未開
|
||
└── 每次都是「事後診斷」才發現 NetworkPolicy 問題
|
||
```
|
||
|
||
**根本原因**:
|
||
1. 任何有 kubectl 權限的人都可以直接修改 NetworkPolicy
|
||
2. 修改後沒有任何告警或審計
|
||
3. Git 版本與叢集版本經常不同步
|
||
4. 沒有 Dry-Run / Diff 機制防止錯誤
|
||
|
||
---
|
||
|
||
## 決策：三層防護架構
|
||
|
||
```
|
||
┌──────────────────────────────────────────────────────────────────────┐
|
||
│ NetworkPolicy 變更治理架構 │
|
||
├──────────────────────────────────────────────────────────────────────┤
|
||
│ │
|
||
│ Layer 1: GitOps (Single Source of Truth) │
|
||
│ ══════════════════════════════════════════ │
|
||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||
│ │ 開發者 │ ──▶ │ PR 審核 │ ──▶ │ ArgoCD │ │
|
||
│ │ 修改 YAML │ │ 至少 2 人 │ │ 自動同步 │ │
|
||
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||
│ │ │ │ │
|
||
│ ▼ ▼ ▼ │
|
||
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
|
||
│ │ Git │ │ GitHub │ │ K8s │ │
|
||
│ │ Commit │ │ Actions │ │ Cluster │ │
|
||
│ └─────────────┘ └─────────────┘ └─────────────┘ │
|
||
│ │
|
||
│ Layer 2: Policy Validation (防錯) │
|
||
│ ═══════════════════════════════════ │
|
||
│ ┌─────────────────────────────────────────────────────────────┐ │
|
||
│ │ Kyverno / OPA Gatekeeper │ │
|
||
│ │ ──────────────────────────────── │ │
|
||
│ │ Rule 1: NetworkPolicy 必須有 system 標籤 │ │
|
||
│ │ Rule 2: 禁止 podSelector: {} (空選擇器) 覆蓋既有規則 │ │
|
||
│ │ Rule 3: Egress 必須明確指定 Port (禁止開放全部) │ │
|
||
│ │ Rule 4: 生產環境 NetworkPolicy 必須有註解說明 │ │
|
||
│ └─────────────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
│ Layer 3: 變更告警 (監控) │
|
||
│ ════════════════════════════ │
|
||
│ ┌─────────────────────────────────────────────────────────────┐ │
|
||
│ │ Falco / kube-state-metrics + Alertmanager │ │
|
||
│ │ ──────────────────────────────────────────── │ │
|
||
│ │ Alert: NetworkPolicy 被創建/修改/刪除 │ │
|
||
│ │ Alert: 直接 kubectl apply (繞過 GitOps) │ │
|
||
│ │ Alert: 非白名單用戶修改關鍵資源 │ │
|
||
│ └─────────────────────────────────────────────────────────────┘ │
|
||
│ │
|
||
└──────────────────────────────────────────────────────────────────────┘
|
||
```
|
||
|
||
---
|
||
|
||
## 實施計畫
|
||
|
||
### Phase 1: 立即措施 (Day 1) ✅
|
||
|
||
```yaml
|
||
# 1. 為所有 NetworkPolicy 加上變更元數據
|
||
metadata:
|
||
annotations:
|
||
awoooi.io/last-modified: "2026-03-23T15:30:00Z"
|
||
awoooi.io/modified-by: "ogt"
|
||
awoooi.io/change-reason: "修復 K8s API 連線 - Y 按鈕執行超時"
|
||
awoooi.io/ticket: "AWOOOI-123"
|
||
```
|
||
|
||
```bash
|
||
# 2. 設定 kubectl 審計 (立即啟用)
|
||
# 在 K3s server 加入:
|
||
--audit-policy-file=/etc/rancher/k3s/audit-policy.yaml
|
||
--audit-log-path=/var/log/k3s/audit.log
|
||
```
|
||
|
||
### Phase 2: GitOps (Week 1)
|
||
|
||
```yaml
|
||
# ArgoCD Application (輕量版)
|
||
apiVersion: argoproj.io/v1alpha1
|
||
kind: Application
|
||
metadata:
|
||
name: awoooi-networkpolicy
|
||
namespace: argocd
|
||
spec:
|
||
project: default
|
||
source:
|
||
repoURL: https://github.com/your-org/awoooi.git
|
||
path: k8s/awoooi-prod
|
||
targetRevision: main
|
||
destination:
|
||
server: https://kubernetes.default.svc
|
||
namespace: awoooi-prod
|
||
syncPolicy:
|
||
automated:
|
||
prune: false # 禁止自動刪除
|
||
selfHeal: true # 有人手動改會自動復原
|
||
syncOptions:
|
||
- CreateNamespace=false
|
||
```
|
||
|
||
### Phase 3: Policy Validation (Week 2)
|
||
|
||
```yaml
|
||
# Kyverno Policy: 強制 NetworkPolicy 必須有註解
|
||
apiVersion: kyverno.io/v1
|
||
kind: ClusterPolicy
|
||
metadata:
|
||
name: require-networkpolicy-annotations
|
||
spec:
|
||
validationFailureAction: Enforce
|
||
rules:
|
||
- name: require-change-reason
|
||
match:
|
||
resources:
|
||
kinds:
|
||
- NetworkPolicy
|
||
namespaces:
|
||
- awoooi-prod
|
||
validate:
|
||
message: "NetworkPolicy 必須有 awoooi.io/change-reason 註解"
|
||
pattern:
|
||
metadata:
|
||
annotations:
|
||
awoooi.io/change-reason: "?*"
|
||
awoooi.io/modified-by: "?*"
|
||
```
|
||
|
||
### Phase 4: 變更告警 (Week 2)
|
||
|
||
```yaml
|
||
# Falco Rule: NetworkPolicy 變更告警
|
||
- rule: NetworkPolicy Modified
|
||
desc: 偵測 NetworkPolicy 被創建、修改或刪除
|
||
condition: >
|
||
k8s_audit and
|
||
ka.target.resource = "networkpolicies" and
|
||
ka.verb in (create, update, patch, delete)
|
||
output: >
|
||
🚨 NetworkPolicy 變更告警
|
||
[%ka.verb] %ka.target.namespace/%ka.target.name
|
||
by %ka.user.name from %ka.sourceips
|
||
priority: WARNING
|
||
tags: [network, security, k8s]
|
||
```
|
||
|
||
```yaml
|
||
# Prometheus Alert: 直接 kubectl 修改 (繞過 GitOps)
|
||
- alert: DirectKubectlNetworkPolicyChange
|
||
expr: |
|
||
increase(apiserver_audit_event_total{
|
||
verb=~"create|update|patch|delete",
|
||
resource="networkpolicies",
|
||
user_agent!~"argocd.*"
|
||
}[5m]) > 0
|
||
for: 0m
|
||
labels:
|
||
severity: critical
|
||
annotations:
|
||
summary: "有人直接 kubectl 修改 NetworkPolicy (繞過 GitOps)"
|
||
description: "{{ $labels.user }} 在 {{ $labels.namespace }} 修改了 NetworkPolicy"
|
||
```
|
||
|
||
---
|
||
|
||
## PR 審核流程 (CODEOWNERS)
|
||
|
||
```
|
||
# .github/CODEOWNERS
|
||
# NetworkPolicy 變更需要 CIO + SRE 雙重審核
|
||
k8s/*/02-network-policy.yaml @awoooi/cio @awoooi/sre-team
|
||
k8s/*/networkpolicy*.yaml @awoooi/cio @awoooi/sre-team
|
||
```
|
||
|
||
---
|
||
|
||
## 回滾機制 (統帥要求補充)
|
||
|
||
### 基線管理
|
||
|
||
```bash
|
||
k8s/awoooi-prod/
|
||
├── 02-network-policy.yaml # 當前版本
|
||
├── .baselines/
|
||
│ ├── LATEST_KNOWN_GOOD.yaml # 最後驗證通過版本
|
||
│ └── 02-network-policy.{date}.yaml # 歷史版本
|
||
```
|
||
|
||
### 一鍵回滾
|
||
|
||
```bash
|
||
# 緊急回滾 (< 30 秒)
|
||
./scripts/rollback-networkpolicy.sh awoooi-prod
|
||
|
||
# 回滾到特定日期
|
||
./scripts/rollback-networkpolicy.sh awoooi-prod 2026-03-22
|
||
```
|
||
|
||
### Drift 偵測 + 人工決策 (禁止盲目自動回滾)
|
||
|
||
- 每 5 分鐘比對叢集狀態 vs Git 基線
|
||
- 偵測到差異 → 立即告警 + 顯示差異
|
||
- **人工判斷**: 合法變更 → 同步到 Git；錯誤變更 → 執行回滾
|
||
- 唯一自動回滾條件: 變更導致健康檢查失敗 + 變更在 10 分鐘內 + 基線曾健康
|
||
|
||
### 回滾 vs 修復 決策樹
|
||
|
||
```
|
||
偵測到問題
|
||
│
|
||
├─ 影響生產？
|
||
│ ├─ 是 → 立即回滾 (< 30 秒)
|
||
│ └─ 否 → 評估後決定
|
||
│
|
||
└─ 回滾後
|
||
├─ 記錄 RCA
|
||
├─ 修復根因
|
||
└─ 更新基線
|
||
```
|
||
|
||
---
|
||
|
||
## 驗收標準
|
||
|
||
| 項目 | 狀態 |
|
||
|------|------|
|
||
| NetworkPolicy 只能透過 PR 修改 | ⬜ |
|
||
| PR 需要 2 人審核 | ⬜ |
|
||
| 直接 kubectl apply 會觸發告警 | ⬜ |
|
||
| 告警發送到 Telegram + OpenClaw | ⬜ |
|
||
| 審計日誌保留 90 天 | ⬜ |
|
||
| Kyverno 強制註解規則 | ⬜ |
|
||
| **基線快照每日保存** | ⬜ |
|
||
| **一鍵回滾腳本可用** | ⬜ |
|
||
| **Drift 偵測每 5 分鐘執行** | ⬜ |
|
||
| **回滾耗時 < 30 秒** | ⬜ |
|
||
|
||
---
|
||
|
||
## 成本
|
||
|
||
| 元件 | 方案 | 成本 |
|
||
|------|------|------|
|
||
| GitOps | ArgoCD (已有) | $0 |
|
||
| Policy | Kyverno (開源) | $0 |
|
||
| 告警 | Falco + Alertmanager (開源) | $0 |
|
||
| **總計** | | **$0** |
|
||
|
||
---
|
||
|
||
## 附錄 A: 2026-03-23 事故根因分析
|
||
|
||
```
|
||
2026-03-23 Y 按鈕執行超時
|
||
|
||
根因:
|
||
NetworkPolicy allow-required-egress 遺漏 K8s API 實際端點
|
||
|
||
問題鏈:
|
||
1. ClusterIP 10.43.0.1:443 已允許 ✓
|
||
2. 但實際流量路由到 192.168.0.120:6443 ✗
|
||
3. 192.168.0.120 在 192.168.0.0/16 排除範圍內 → 被阻擋
|
||
|
||
修復:
|
||
新增 192.168.0.120:6443 到 allow-required-egress
|
||
|
||
教訓:
|
||
1. K8s Service ClusterIP ≠ 實際 Endpoint
|
||
2. NetworkPolicy 需要允許完整路由路徑
|
||
3. 變更前應該用 dry-run 驗證
|
||
```
|
||
|
||
---
|
||
|
||
## 附錄 B: 2026-03-26 DNS 規則事故根因分析
|
||
|
||
```
|
||
2026-03-26 兩天無 Telegram 告警
|
||
|
||
根因 1: Alertmanager URL 路徑錯誤
|
||
設定: /api/v1/webhook/alertmanager (單數)
|
||
實際: /api/v1/webhooks/alertmanager (複數)
|
||
結果: 404 Not Found
|
||
|
||
根因 2: NetworkPolicy DNS 規則標籤錯誤
|
||
設定: podSelector 要求 environment=prod, system=awoooi
|
||
實際: CoreDNS 只有 k8s-app=kube-dns
|
||
結果: Pod 無法連接 CoreDNS
|
||
|
||
根因 3: CoreDNS 上游 DNS 設定錯誤
|
||
設定: forward . /etc/resolv.conf → 127.0.0.53 (systemd-resolved)
|
||
實際: 容器內無法使用 127.0.0.53
|
||
結果: 外部 DNS 解析失敗
|
||
|
||
修復:
|
||
1. Alertmanager ConfigMap 修正 URL 路徑
|
||
2. NetworkPolicy 使用正確的 namespaceSelector
|
||
3. CoreDNS 改用 8.8.8.8 1.1.1.1
|
||
|
||
教訓:
|
||
1. URL 路徑必須經過 E2E 測試驗證 (ADR-025)
|
||
2. NetworkPolicy DNS 規則必須使用 namespace selector
|
||
3. CoreDNS 不能依賴宿主機的 systemd-resolved
|
||
```
|
||
|
||
### DNS 規則最佳實踐
|
||
|
||
```yaml
|
||
# ✅ 正確的 DNS 規則寫法
|
||
- ports:
|
||
- port: 53
|
||
protocol: UDP
|
||
- port: 53
|
||
protocol: TCP
|
||
to:
|
||
- namespaceSelector:
|
||
matchLabels:
|
||
kubernetes.io/metadata.name: kube-system
|
||
podSelector:
|
||
matchLabels:
|
||
k8s-app: kube-dns
|
||
```
|