wooo/awoooi
1
1
Fork 0
Code Issues 2 Pull Requests 1 Actions 2 Packages Projects Releases Wiki Activity
6,496 Commits 219 Branches 2 Tags
c9bb4bbfc6af05c41df7c3af546ba44cbcf45708
Commit Graph

42 Commits

Author SHA1 Message Date
Your Name
d84be37e26 fix(reboot): repair 110 ssh control path on startup
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
Details
CD Pipeline / cancel-stale-cd (push) Has been skipped
Details
CD Pipeline / tests (push) Successful in 48s
Details
CD Pipeline / post-deploy-checks (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
2026-07-02 10:53:16 +08:00
Your Name
acf1d0e127 fix(recovery): auto-start controlled 110 drain lane
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
Details
CD Pipeline / cancel-stale-cd (push) Has been skipped
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
CD Pipeline / post-deploy-checks (push) Has been cancelled
Details
CD Pipeline / tests (push) Has been cancelled
Details
2026-06-30 23:33:19 +08:00
Your Name
f1758cdf0c fix(runner): verify 110 controlled cd lane readiness
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
Details
CD Pipeline / cancel-stale-cd (push) Has been skipped
Details
CD Pipeline / tests (push) Failing after 3m10s
Details
CD Pipeline / build-and-deploy (push) Has been skipped
Details
AWOOOI Harbor 110 Local Repair / workflow-shape (push) Successful in 1s
Details
CD Pipeline / post-deploy-checks (push) Has been skipped
Details
AWOOOI Harbor 110 Local Repair / harbor-110-local-repair (push) Has been cancelled
Details
2026-06-30 21:37:24 +08:00
Your Name
3397f5a9aa fix(recovery): open controlled drain lane after guardrails
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
Details
CD Pipeline / cancel-stale-cd (push) Has been skipped
Details
CD Pipeline / tests (push) Failing after 3m13s
Details
CD Pipeline / build-and-deploy (push) Has been skipped
Details
CD Pipeline / post-deploy-checks (push) Has been skipped
Details
AWOOOI Harbor 110 Local Repair / workflow-shape (push) Successful in 1s
Details
AWOOOI Harbor 110 Local Repair / harbor-110-local-repair (push) Has been cancelled
Details
2026-06-30 21:26:47 +08:00
Your Name
66ec3c92e5 fix(recovery): quarantine corrupt docker metadata on boot
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
Details
CD Pipeline / cancel-stale-cd (push) Has been skipped
Details
CD Pipeline / tests (push) Failing after 10m8s
Details
CD Pipeline / build-and-deploy (push) Has been skipped
Details
CD Pipeline / post-deploy-checks (push) Has been skipped
Details
2026-06-30 18:02:43 +08:00
Your Name
d5c9d467c1 fix(recovery): reopen controlled drain lane source 2026-06-28 19:28:37 +08:00
Your Name
6840c3578b Revert "fix(recovery): reopen controlled cd lane source" 
This reverts commit f4a7c01eef.
 2026-06-28 19:02:06 +08:00
Your Name
f4a7c01eef fix(recovery): reopen controlled cd lane source 2026-06-28 18:54:40 +08:00
Your Name
f4d1b99da4 Revert "fix(recovery): disable runner failclosed authority source" 
This reverts commit 1fdbc96a9b.
 2026-06-28 18:39:12 +08:00
Your Name
1fdbc96a9b fix(recovery): disable runner failclosed authority source
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
Details
CD Pipeline / cancel-stale-cd (push) Has been skipped
Details
CD Pipeline / tests (push) Successful in 1m41s
Details
Code Review / ai-code-review (push) Successful in 15s
Details
CD Pipeline / build-and-deploy (push) Successful in 5m37s
Details
CD Pipeline / post-deploy-checks (push) Successful in 1m38s
Details
2026-06-28 16:40:14 +08:00
Your Name
f52ec0db26 fix(recovery): add runner failclosed cron authority [skip ci] 2026-06-28 16:32:27 +08:00
Your Name
d7f56351f2 fix(recovery): reopen controlled automation after failclosed regression
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
Details
CD Pipeline / cancel-stale-cd (push) Has been skipped
Details
Code Review / ai-code-review (push) Has been cancelled
Details
CD Pipeline / tests (push) Failing after 14m8s
Details
Type Sync Check / check-type-sync (push) Successful in 42s
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
CD Pipeline / post-deploy-checks (push) Has been cancelled
Details
2026-06-28 16:01:40 +08:00
Your Name
ba054e698d fix(recovery): seal runner failclosed disablers [skip ci] 2026-06-28 15:58:06 +08:00
Your Name
9a6c724bdc fix(ci): open controlled cd lane guards
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
Details
CD Pipeline / cancel-stale-cd (push) Has been skipped
Details
CD Pipeline / tests (push) Successful in 1m41s
Details
Code Review / ai-code-review (push) Successful in 13s
Details
CD Pipeline / build-and-deploy (push) Failing after 2m35s
Details
CD Pipeline / post-deploy-checks (push) Has been cancelled
Details
2026-06-28 15:04:48 +08:00
Your Name
723207315c fix(ci): keep 110 cd lane fail closed [skip ci] 2026-06-28 14:59:32 +08:00
Your Name
4414ec991f fix(ci): reopen hard-limited controlled cd lane
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 0s
Details
CD Pipeline / cancel-stale-cd (push) Has been skipped
Details
CD Pipeline / tests (push) Successful in 1m42s
Details
Code Review / ai-code-review (push) Successful in 15s
Details
CD Pipeline / build-and-deploy (push) Successful in 6m33s
Details
CD Pipeline / post-deploy-checks (push) Successful in 3m10s
Details
2026-06-28 11:53:42 +08:00
Your Name
f109b11478 fix(recovery): seal 110 cd lane restore sources [skip ci] 2026-06-28 11:37:01 +08:00
Your Name
a7f6bf72aa fix(recovery): preserve controlled drain lane assets
Some checks failed
Code Review / ai-code-review (push) Has been cancelled
Details
2026-06-28 11:32:15 +08:00
Your Name
e97b252475 fix(cd): reopen controlled runtime deploy lane
Some checks failed
CD Pipeline / tests (push) Failing after 7s
Details
CD Pipeline / build-and-deploy (push) Has been skipped
Details
CD Pipeline / post-deploy-checks (push) Has been skipped
Details
Code Review / ai-code-review (push) Successful in 17s
Details
2026-06-28 11:09:42 +08:00
Your Name
241cbe067e fix(recovery): freeze 110 cd lane and source-aware 188 gates [skip ci] 2026-06-28 10:58:41 +08:00
Your Name
b7b475983f fix(runner): accept controlled cd lane drain readback 2026-06-28 10:42:35 +08:00
Your Name
913779a703 fix(runner): preserve verified controlled cd lane
Some checks failed
Ansible / Reboot Recovery Contract / validate (push) Has been cancelled
Details
CD Pipeline / tests (push) Has been cancelled
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
CD Pipeline / post-deploy-checks (push) Has been cancelled
Details
Code Review / ai-code-review (push) Has been cancelled
Details
2026-06-28 10:21:10 +08:00
Your Name
3bebf324c1 fix(runner): keep controlled cd lane alive
Some checks failed
Ansible / Reboot Recovery Contract / validate (push) Successful in 1m13s
Details
CD Pipeline / tests (push) Failing after 40s
Details
CD Pipeline / build-and-deploy (push) Has been skipped
Details
CD Pipeline / post-deploy-checks (push) Has been skipped
Details
Code Review / ai-code-review (push) Has been cancelled
Details
2026-06-28 10:16:47 +08:00
Your Name
0531050934 fix(runner): split controlled cd lane guard [skip ci] 2026-06-28 09:56:31 +08:00
Your Name
00db624e5f fix(reboot): fail closed direct cd lane pressure path [skip ci] 2026-06-28 09:46:46 +08:00
Your Name
b204840841 fix(runner): keep controlled cd lane open
Some checks failed
Code Review / ai-code-review (push) Successful in 25s
Details
Ansible / Reboot Recovery Contract / validate (push) Failing after 12m24s
Details
2026-06-28 09:35:20 +08:00
Your Name
8f402983ee fix(reboot): enforce direct runner fail-closed guard [skip ci] 2026-06-28 09:16:43 +08:00
Your Name
54eea881e3 fix(ci): fail closed host runner pressure guards [skip ci] 2026-06-28 08:38:15 +08:00
Your Name
b6c600e24d fix(reboot): open host runner controlled automation
Some checks failed
CD Pipeline / tests (push) Waiting to run
Details
Ansible / Reboot Recovery Contract / validate (push) Successful in 1m16s
Details
Code Review / ai-code-review (push) Successful in 22s
Details
CD Pipeline / build-and-deploy (push) Has been cancelled
Details
CD Pipeline / post-deploy-checks (push) Has been cancelled
Details
2026-06-28 02:09:23 +08:00
Your Name
7c3b1c0ab9 fix(reboot): require double key for host runner sentinel
Some checks failed
Ansible / Reboot Recovery Contract / validate (push) Successful in 1m37s
Details
Code Review / ai-code-review (push) Has been cancelled
Details
2026-06-28 02:06:24 +08:00
Your Name
77ba5ed517 fix(reboot): honor runner sentinel during deploy window 2026-06-28 02:02:56 +08:00
Your Name
2a1cd3cc8b fix(reboot): fail closed host runner startup
Some checks failed
Ansible / Reboot Recovery Contract / validate (push) Successful in 1m16s
Details
Code Review / ai-code-review (push) Successful in 17s
Details
AI 技術雷達監控 / ai-technology-watch (push) Failing after 11m45s
Details
2026-06-28 01:59:15 +08:00
Your Name
4c951b2996 fix(ci): keep 110 runner inactive until pressure clears 2026-06-27 20:15:01 +08:00
ogt
5f37de539c fix(ci): isolate 110 runner labels and gate host pressure [skip ci] 2026-06-27 19:38:15 +08:00
ogt
7f706feded fix(ops): recognize repo-scoped CI containers in load guard [skip ci] 2026-06-27 12:45:45 +08:00
Your Name
9b465ee140 ci(runner): drain legacy docker act runner safely
All checks were successful
Code Review / ai-code-review (push) Successful in 11s
Details
2026-05-21 18:53:45 +08:00
Your Name
cb5ab900c4 fix(ci): preserve gitea runner jobs on shutdown
All checks were successful
Code Review / ai-code-review (push) Successful in 46s
Details
2026-05-01 16:16:27 +08:00
OG T
3f7a742683 fix(infra): 首席架構師 Review 修正 — C1/I1/I2/I3/I4/S1 
C1: 移除 deploy-to-110.sh 密碼明文，改用 SSH key + sudoers NOPASSWD
I1: 加入 /var/lock/harbor-repair.lock 防止 watchdog 與 startup 並行修復
I2: docker compose 的 stderr 不再靜默（改用 tee -a log | while read 輸出）
I3: watchdog while loop 包在子 shell + || true，子 shell 異常不終止 watchdog
I4: repair_harbor 關鍵指令（harbor-log 啟動）加入退出碼捕捉
S1: 修復後驗證等待從 5s/10s 改為 30s（harbor-core 初始化需要足夠時間）
S2: docker ps 改用 --filter status=exited 取代 grep/awk

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 12:18:41 +08:00
OG T
66b12bf9eb fix(infra): 根治 Harbor Exited(128) Race Condition + harbor-watchdog 常駐自愈 
問題根因：
  awoooi-startup-110.sh 在 Harbor 啟動時，第一次 compose up -d 會同時
  啟動所有容器。harbor-core/db/portal 嘗試連 syslog:1514（harbor-log 未就緒），
  失敗後 exit(128)，restart:always 重試直到 backoff 放棄。
  即使後來 harbor-log healthy，其他容器已不再重試。

修復 1 — startup-110.sh Harbor 時序（4 Phase 策略）：
  Phase 1: 清除所有 Exited Harbor 容器（打破 backoff 死鎖）
  Phase 2: 只啟動 harbor-log
  Phase 3: 等 harbor-log healthy（最多 90s）
  Phase 4: 啟動全組件

修復 2 — harbor-watchdog.service（常駐自愈）：
  Type=simple 常駐進程，每 60s 輪詢 http://127.0.0.1:5000/v2/
  不健康 → 等 5s 再確認 → 執行 Phase 1-4 完整修復
  修復重開機時序問題無法覆蓋的「運行中崩潰」場景

Bug Fix：curl -f 會把 HTTP 401 視為失敗（exit 22），
  Harbor /v2/ 正常回傳 401（需認證），改用 curl -s 不加 -f

REBOOT-RECOVERY-SOP.md → v5.0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 12:13:21 +08:00
OG T
4ba62132e2 ops(startup): startup-110.sh 加入 Step 7 Sentry 自動啟動 
Sentry 已安裝於 /opt/sentry (2026-03-24)，但重開機後未自動啟動
加入非阻塞啟動 + 重開機損壞修復邏輯:
  - sentry-postgres WAL 損壞 → pg_resetwal -f 自動修復
  - sentry-redis dump.rdb 損壞 → 自動刪除重建
  - 啟動後 20s 非阻塞健康驗證

根因: 2026-04-05 重開機後 PostgreSQL WAL + Redis RDB + ClickHouse parts 全部損壞

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 03:09:20 +08:00
OG T
ad4abefcd9 fix(k8s+ops): 修復告警鏈路 + Gitea runner 自動啟動
Some checks failed
CD Pipeline / build-and-deploy (push) Failing after 21s
Details 
## 修復項目

1. NetworkPolicy allow-nginx-ingress 加入 192.168.0.110
   - Alertmanager (在 110) 需要從 110 直接 POST webhook 到 API pod
   - 修復前: 110 被 NetworkPolicy default-deny 阻擋，webhook timeout
   - 修復後: 110 加入 ingress 白名單，告警鏈路恢復

2. awoooi-startup-110.sh 加入 Gitea Act Runner
   - Step 6: 啟動 /home/wooo/act-runner (gitea-runner container)
   - 修復前: 重開機後 runner 離線，CD pipeline 全面失效
   - 修復後: runner 自動重啟，若配置過期自動清除重新註冊

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 01:42:52 +08:00
OG T
f4f454fd98 feat(api): 重開機後自動 warm-up Redis Working Memory from PostgreSQL 
- main.py lifespan: 啟動時從 DB restore INVESTIGATING/MITIGATING incidents
- scripts/reboot-recovery: 188 + 110 自動化腳本 + systemd services
- scripts/reboot-recovery: aiops-network 自動建立 (ClawBot 依賴)
- docs/runbooks/REBOOT-RECOVERY-SOP.md: 完整改寫，含自動化腳本說明

Why: 重開機後 Redis 清空導致前端 incidents 顯示 0 筆（DB 完整保存）
統帥批准: 「所有數據必須被長久記錄下來」

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-04-05 00:39:20 +08:00