feat(sre): enforce typed controlled automation
Some checks failed
CD Pipeline / workflow-shape (push) Successful in 0s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 2m38s
CD Pipeline / build-and-deploy (push) Failing after 24m19s
CD Pipeline / post-deploy-checks (push) Has been skipped

This commit is contained in:
ogt
2026-07-16 17:32:01 +08:00
parent dfec0c9f7f
commit 541a32a9cb
130 changed files with 14849 additions and 1738 deletions

View File

@@ -0,0 +1,20 @@
# Portfolio Infrastructure Reconciliation Handoff
1. Current P0 remains `AIA-SRE-013`; the safety queue remains `017 -> 002 -> 004 -> 013 -> 014 -> 015` and this inventory creates no competing P0.
2. Active inventory is 284 lines, SHA `c21416023775cc7fa96952ea3f06be389246269d11130902ae039aa0bd463df4`; all 131 non-summary rows map once.
3. Canonical assets are 144, imported omissions remain 18, conflicts are 11 across 36 assets; runtime closure remains `0`.
4. Host110 Ollama is removed and retained only as a canonical tombstone; runtime/monitoring absence is not yet verified.
5. Host111 is the third provider hop and runs as a macOS LaunchAgent, with local plus host120/host121 independent verification still pending.
6. Host111 also lacks a fresh canonical production sensor/rule series; LaunchAgent source readiness is not runtime closure.
7. GCP-A/B are public-HTTP sanitized candidates only; unsanitized payloads and every cloud tool loop must fail closed before network access.
8. Exact GCP-A/B Prometheus targets are missing in production; source/config targets do not count as runtime coverage.
9. Claude/Gemini credential material was user-supplied, but raw values are not recorded/read back; protected-secret metadata verification is pending.
10. Claude/Gemini paid canary needs durable Gate5 run authorization, daily/monthly/per-incident caps and one paired enable/rollback receipt.
11. Any partial five-lane result rolls back both paid providers; the sanitized five-lane production canary has not run.
12. AlertChainBroken source now uses host99 Agent99 exact-host read-only polling plus a reduced HTTPS relay, so it is independent of the primary webhook and sends no first-hop credential; atomic deployment, freshness/dedupe, current alert resolution and same-run repair/learning receipts remain P0 runtime work.
13. The gitea-native source candidate has no exact production target while the legacy GitHub exporter remains visible runtime drift.
14. GitHub stays frozen; legacy exporter/ghcr/runner claims cannot be normalized into healthy Gitea coverage.
15. Product alerts remain blocked until exact product-only Telegram routes exist; every card names AI/provider/action/executor/verifier/receipt.
16. Backup/restore remains read-only critical break-glass; every stateful asset needs freshness/offsite/escrow/drill evidence.
17. Source/test, Gitea CD/deploy marker, production runtime, visible UI/Telegram and learning writeback are separate evidence layers.
18. No lower evidence layer may mark a higher layer complete; no work item is runtime-closed for this program SHA.
19. Next: complete focused contracts, deploy one SHA, then verify tombstones/targets/second ingress/paid gates with same-run receipts.

View File

@@ -0,0 +1,614 @@
{
"schema_version": "portfolio_infrastructure_asset_reconciliation_v1",
"governance_version": "global_product_governance_v2",
"program_id": "AIA-SRE-P0-20260715",
"generated_at": "2026-07-16T15:18:00+08:00",
"status": "all_imported_rows_mapped_source_conflicts_and_omissions_recorded_runtime_unverified",
"scope_complete": false,
"source_inventory": {
"active_version_id": "inventory-c214160-20260716",
"label": "WOOO 基礎設施服務工具盤點表(完整版)",
"source_path": "/Users/ogt/.gemini/antigravity-ide/brain/4e1bfd7f-cc1d-4f4f-94b0-df62ecb40386/infrastructure_inventory.md",
"sha256": "c21416023775cc7fa96952ea3f06be389246269d11130902ae039aa0bd463df4",
"declared_updated_date": "2026-07-16",
"source_line_count": 284,
"inventory_item_row_count": 131,
"source_category_counts": {
"hosts": 5,
"databases": 15,
"cache_queue": 8,
"monitoring_observability": 18,
"error_tracking": 3,
"ai_llm": 5,
"workflow_automation": 1,
"container_registry_cicd": 9,
"storage_backup": 2,
"gateway_proxy": 7,
"host112_security": 23,
"applications": 24,
"k3s_components": 11
},
"live_truth_accepted": false,
"runtime_probe_performed": false,
"interpretation": "The imported docker-ps and health labels are historical evidence only. Every asset remains runtime-unverified until an exact production receipt is correlated to this source revision.",
"version_history": [
{
"version_id": "inventory-2e8580c-20260716",
"sha256": "2e8580c6465fe072990bc3736625c1a81614201d6cb7acadcc27338f96acaf22",
"source_line_count": 241,
"inventory_item_row_count": 112,
"disposition": "superseded_by_inventory-c214160-20260716_preserved_as_reconciliation_history"
}
]
},
"governance": {
"pipeline": [
"Alert",
"Canonical Asset Normalize",
"Typed Domain Router",
"HolmesGPT Investigator",
"Ollama RCA / Gemini Critic",
"Deterministic Policy",
"Single Controlled Executor",
"Independent Verifier",
"Incident Closure + KM/RAG/MCP/PlayBook"
],
"provider_order": ["ollama_gcp_a", "ollama_gcp_b", "ollama_local_host111", "claude", "gemini"],
"github_frozen": true,
"cross_domain_fallback_allowed": false,
"unknown_asset_terminal": "asset_identity_unresolved",
"snapshot_health_is_live_truth": false,
"critical_backup_restore_default": "read_only_investigation_break_glass_for_write",
"alert_attribution_required": [
"decision_engine",
"ai_models_used",
"agent_actions",
"executor",
"independent_verifier",
"same_run_receipt"
]
},
"asset_groups": [
{
"group_id": "imported-hosts",
"category": "host",
"common": {
"product_id": "shared-infrastructure",
"project_id": "wooo-platform",
"site_id": "on-prem-lan",
"owner_lane": "infrastructure_ops",
"source_truth_state": "source_registry_partial",
"live_truth_state": "not_probed_this_reconciliation",
"domain_router": "host_systemd",
"executor": "host_ansible_executor",
"verifier": "host_runtime_independent_verifier",
"monitoring": {"signals": ["node_reachability", "cpu", "memory", "disk", "service_freshness"], "coverage": "partial"},
"alerting": {"lifecycle": "canonical_incident_only", "cross_domain_fallback_allowed": false, "required_ai_attribution": true},
"telegram_destination": "awoooi_sre_war_room_for_shared_P0_P1_lifecycle_only",
"backup_restore": {"mode": "readback_first", "target": "host_config_and_service_specific_contracts", "freshness": "unverified"},
"learning_targets": ["KM", "RAG", "MCP", "PlayBook"],
"priority": "P0",
"dependencies": ["PORT-001"],
"next_action": "Run exact-host read-only identity and service probes, then bind each result to a durable asset receipt."
},
"members": [
{"canonical_id": "host:110", "inventory_labels": ["DevOps", "192.168.0.110"], "source_rows": ["L012"], "runtime_identity": "192.168.0.110", "reconciliation_state": "matched_with_source_drift", "findings": ["Inventory claims healthy containers but no same-run receipt exists", "Host110 Ollama has been removed; every former endpoint/proxy identity must resolve only to a retired tombstone", "Source tombstone and monitoring cleanup are not production proof of endpoint absence", "Legacy GitHub exporter runtime remains visible while the exact gitea-native target is missing"], "source_refs": ["infra/ansible/inventory/hosts.yml", "infra/ansible/inventory/group_vars/host_110.yml", "ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml"], "next_action": "Verify the Host110 Ollama tombstone against runtime and monitoring, verify gitea-native target freshness, and retire only the exact legacy GitHub exporter under bounded apply."},
{"canonical_id": "host:188", "inventory_labels": ["AI/Data", "192.168.0.188"], "source_rows": ["L013"], "runtime_identity": "192.168.0.188", "reconciliation_state": "matched_with_role_conflict", "findings": ["Inventory says AI/Data without GPU", "Imported local Ollama identity conflicts with the approved host111 provider route", "Disk 64 percent is an unverified point-in-time claim"], "source_refs": ["infra/ansible/inventory/hosts.yml", "infra/ansible/inventory/group_vars/host_188.yml", "ops/config/service-registry.yaml"]},
{"canonical_id": "host:120", "inventory_labels": ["K3s Master", "K3s Master (192.168.0.120)"], "source_rows": ["L014", "L249"], "runtime_identity": "192.168.0.120", "reconciliation_state": "matched_runtime_unverified", "findings": ["Control-plane role is source-visible; health is not live-verified"], "source_refs": ["infra/ansible/inventory/hosts.yml", "ops/monitoring/service-registry.yaml"], "domain_router": "control_plane_recovery", "executor": "Agent99", "verifier": "cold_start_independent_scorecard_verifier"},
{"canonical_id": "host:121", "inventory_labels": ["K3s Backup", "K3s Backup (192.168.0.121)"], "source_rows": ["L015", "L250"], "runtime_identity": "192.168.0.121", "reconciliation_state": "role_conflict", "findings": ["Imported inventory calls this a backup/control-plane node while monitoring registry calls it k3s-worker"], "source_refs": ["infra/ansible/inventory/hosts.yml", "ops/monitoring/service-registry.yaml"], "domain_router": "control_plane_recovery", "executor": "Agent99", "verifier": "cold_start_independent_scorecard_verifier", "next_action": "Resolve exact K3s node roles from live kubectl node identity before any recovery action."},
{"canonical_id": "host:112", "inventory_labels": ["Security", "192.168.0.112"], "source_rows": ["L016"], "runtime_identity": "192.168.0.112", "reconciliation_state": "matched_runtime_unverified", "findings": ["Wazuh/Kali source identity exists; imported healthy state is not a current receipt"], "source_refs": ["infra/ansible/inventory/hosts.yml", "ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml"], "owner_lane": "security_ops"}
]
},
{
"group_id": "imported-databases",
"category": "database",
"common": {
"product_id": "shared-infrastructure",
"project_id": "database-estate",
"site_id": "on-prem-lan",
"owner_lane": "database_ops",
"source_truth_state": "inventory_claim_and_partial_registry",
"live_truth_state": "not_probed_this_reconciliation",
"domain_router": "database",
"executor": "db_bounded_executor",
"verifier": "db_independent_verifier",
"monitoring": {"signals": ["readiness", "connections", "replication", "capacity", "backup_freshness"], "coverage": "partial"},
"alerting": {"lifecycle": "database_typed_incident", "cross_domain_fallback_allowed": false, "required_ai_attribution": true},
"telegram_destination": "awoooi_sre_war_room_for_shared_P0_P1_lifecycle_only",
"backup_restore": {"mode": "critical_read_only_default", "target": "gap:no_exact_product_restore_contract", "freshness": "unverified"},
"learning_targets": ["KM", "RAG", "MCP", "PlayBook"],
"priority": "P0",
"dependencies": ["PORT-001", "PORT-007"],
"next_action": "Resolve exact database/container/port identity, collect read-only health and backup freshness, then create bounded DB verifier work."
},
"members": [
{"canonical_id": "database:awoooi:prod", "inventory_labels": ["awoooi_prod"], "source_rows": ["L025"], "runtime_identity": "host188/postgresql/awoooi_prod", "reconciliation_state": "identity_scope_conflict", "findings": ["Inventory combines AWOOOI business database and K3s datastore into one purpose", "Canonical service registry names the container postgres, not awoooi_prod"], "source_refs": ["ops/config/service-registry.yaml", "scripts/backup/backup-awoooi.sh"], "product_id": "awoooi", "project_id": "awoooi", "backup_restore": {"mode": "critical_read_only_default", "target": "scripts/backup/backup-awoooi.sh", "freshness": "unverified"}},
{"canonical_id": "database:momo-pro:postgres", "inventory_labels": ["momo-db"], "source_rows": ["L026"], "runtime_identity": "host188/docker/momo-db", "reconciliation_state": "matched_source_runtime_unverified", "findings": ["Exact exposed port and database name are not proven by the inventory row"], "source_refs": ["ops/config/service-registry.yaml", "scripts/backup/backup-momo.sh"], "product_id": "momo-pro", "project_id": "momo-pro-system", "owner_lane": "momo_product_ops", "telegram_destination": "blocked:momo_configured_chat_unresolved", "backup_restore": {"mode": "critical_read_only_default", "target": "scripts/backup/backup-momo.sh", "freshness": "unverified"}},
{"canonical_id": "database:vibework:postgres", "inventory_labels": ["vibework-production-postgres-1"], "source_rows": ["L027"], "runtime_identity": "host188/docker/vibework-production-postgres-1", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["No exact AWOOOI service registry entry or committed restore contract found"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "vibework", "project_id": "vibework", "owner_lane": "vibework_product_ops", "telegram_destination": "blocked:vibework_route_not_implemented"},
{"canonical_id": "database:2026fifa:timescaledb", "inventory_labels": ["current-fifa2026-postgres-1", "TimescaleDB current-fifa2026-postgres-1"], "source_rows": ["L028", "L043"], "runtime_identity": "host188/docker/current-fifa2026-postgres-1", "reconciliation_state": "duplicate_inventory_row_and_registry_gap", "findings": ["Same instance is counted once as PostgreSQL and again as TimescaleDB", "No exact restore drill is registered"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "2026fifa", "project_id": "2026FIFAWorldCup", "owner_lane": "2026fifa_product_ops", "telegram_destination": "blocked:2026fifa_destination_unassigned"},
{"canonical_id": "database:k3s:postgres-recovery", "inventory_labels": ["k3s-postgres-recovery"], "source_rows": ["L029"], "runtime_identity": "host188/docker/k3s-postgres-recovery", "reconciliation_state": "recovery_asset_without_restore_receipt", "findings": ["Running recovery container is not proof of current K3s datastore recovery readiness"], "source_refs": ["docs/security/backup-restore-escrow-inventory.snapshot.json"], "product_id": "shared-infrastructure", "project_id": "k3s-control-plane"},
{"canonical_id": "database:test:pgvector-b5-8801", "inventory_labels": ["pg-test-b5-8801-1"], "source_rows": ["L030"], "runtime_identity": "host188/docker/pg-test-b5-8801-1", "reconciliation_state": "stale_exited_candidate", "findings": ["Imported state says Exited", "No deletion is authorized; ownership and retention must be resolved first"], "source_refs": ["ops/config/service-registry.yaml"], "product_id": "unknown", "project_id": "unknown", "owner_lane": "asset_identity", "domain_router": "unknown", "executor": null, "verifier": "asset_identity_drift_verifier", "next_action": "Create deterministic drift work item; do not prune or restart until owner and data classification are proven."},
{"canonical_id": "database:harbor:postgres", "inventory_labels": ["harbor-db"], "source_rows": ["L031"], "runtime_identity": "host110/docker/harbor-db", "reconciliation_state": "matched_runtime_unverified", "findings": ["Stateful BLOCK policy exists; health and backup freshness remain unverified"], "source_refs": ["ops/config/service-registry.yaml", "scripts/backup/backup-harbor.sh"], "project_id": "harbor", "owner_lane": "devops_team", "backup_restore": {"mode": "critical_read_only_default", "target": "scripts/backup/backup-harbor.sh", "freshness": "unverified"}},
{"canonical_id": "database:langfuse:postgres", "inventory_labels": ["langfuse-db"], "source_rows": ["L032"], "runtime_identity": "host110/docker/langfuse-db", "reconciliation_state": "matched_runtime_unverified", "findings": ["Stateful BLOCK policy exists; LLM trace privacy and restore evidence remain unverified"], "source_refs": ["ops/config/service-registry.yaml", "scripts/backup/backup-langfuse.sh"], "project_id": "langfuse", "owner_lane": "ai_team", "backup_restore": {"mode": "critical_read_only_default", "target": "scripts/backup/backup-langfuse.sh", "freshness": "unverified"}},
{"canonical_id": "database:agent-bounty:postgres", "inventory_labels": ["agent_bounty_db"], "source_rows": ["L033"], "runtime_identity": "host110/docker/agent_bounty_db", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["No exact backup/restore verifier is registered"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "agent-bounty-protocol", "project_id": "agent-bounty-protocol", "owner_lane": "agent_bounty_product_ops", "telegram_destination": "blocked:vibe_ai_agent_receipt_gap"},
{"canonical_id": "database:stockplatform-v2:postgres", "inventory_labels": ["stockplatform-v2-postgres-1"], "source_rows": ["L034"], "runtime_identity": "host110/docker/stockplatform-v2-postgres-1", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Product destination ownership and restore contract are unresolved"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "stockplatform-v2", "project_id": "stockPlatform", "owner_lane": "stockplatform_product_ops", "telegram_destination": "blocked:stockplatform_tsenyang_ownership_drift"},
{"canonical_id": "database:awooogo:postgres", "inventory_labels": ["awooo-go-db-1"], "source_rows": ["L035"], "runtime_identity": "host110/docker/awooo-go-db-1", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Product Telegram egress is disabled and restore evidence is absent"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "awooogo", "project_id": "AwoooGo", "owner_lane": "awooogo_product_ops", "telegram_destination": "disabled:awooogo"},
{"canonical_id": "database:bitan-pharmacy:restore", "inventory_labels": ["bitan-pg-restore"], "source_rows": ["L036"], "runtime_identity": "host110/docker/bitan-pg-restore", "reconciliation_state": "restore_marker_without_drill_receipt", "findings": ["Container name implies restore but no isolated drill receipt is linked"], "source_refs": ["docs/security/backup-restore-escrow-inventory.snapshot.json"], "product_id": "bitan-pharmacy", "project_id": "bitan-pharmacy", "owner_lane": "bitan_product_ops", "telegram_destination": "blocked:bitan_destination_unassigned"},
{"canonical_id": "database:sentry:postgres", "inventory_labels": ["Sentry PG (內建)"], "source_rows": ["L037"], "runtime_identity": "host110/sentry-self-hosted/postgres", "reconciliation_state": "matched_runtime_unverified", "findings": ["Exact container identity is not present in the imported row"], "source_refs": ["ops/config/service-registry.yaml", "scripts/backup/backup-sentry.sh"], "project_id": "sentry", "owner_lane": "devops_team", "backup_restore": {"mode": "critical_read_only_default", "target": "scripts/backup/backup-sentry.sh", "freshness": "unverified"}},
{"canonical_id": "database:signoz:clickhouse", "inventory_labels": ["signoz-clickhouse"], "source_rows": ["L042"], "runtime_identity": "unresolved:host110_or_host188/signoz-clickhouse", "reconciliation_state": "host_identity_conflict", "findings": ["Imported inventory places ClickHouse on host110 while monitoring registry places clickhouse on host188"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "scripts/backup/clickhouse-native-backup.sh"], "project_id": "signoz", "owner_lane": "observability_ops", "backup_restore": {"mode": "critical_read_only_default", "target": "scripts/backup/clickhouse-native-restore-drill.sh", "freshness": "unverified"}, "next_action": "Resolve exact host/container identity before any DB executor or restore drill."}
]
},
{
"group_id": "imported-cache-queue",
"category": "cache_queue",
"common": {
"product_id": "shared-infrastructure",
"project_id": "cache-queue-estate",
"site_id": "on-prem-lan",
"owner_lane": "database_ops",
"source_truth_state": "inventory_claim_and_partial_registry",
"live_truth_state": "not_probed_this_reconciliation",
"domain_router": "database",
"executor": "db_bounded_executor",
"verifier": "db_independent_verifier",
"monitoring": {"signals": ["readiness", "memory", "connections", "queue_depth", "persistence_freshness"], "coverage": "partial"},
"alerting": {"lifecycle": "stateful_cache_queue_incident", "cross_domain_fallback_allowed": false, "required_ai_attribution": true},
"telegram_destination": "awoooi_sre_war_room_for_shared_P0_P1_lifecycle_only",
"backup_restore": {"mode": "critical_read_only_default", "target": "gap:no_exact_persistence_contract", "freshness": "unverified"},
"learning_targets": ["KM", "RAG", "MCP", "PlayBook"],
"priority": "P1",
"dependencies": ["PORT-001", "PORT-007"],
"next_action": "Verify exact instance identity and persistence mode before generating any bounded action."
},
"members": [
{"canonical_id": "cache:awoooi:redis", "inventory_labels": ["redis-server (systemd)"], "source_rows": ["L052"], "runtime_identity": "host188/systemd/redis-server", "reconciliation_state": "service_type_conflict", "findings": ["Imported row says systemd while monitoring registry models redis as Docker on port 6380"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml"], "product_id": "awoooi", "project_id": "awoooi", "priority": "P0"},
{"canonical_id": "cache:clawbot:redis", "inventory_labels": ["clawbot-redis"], "source_rows": ["L053"], "runtime_identity": "host188/docker/clawbot-redis", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Persistence and backup contract are not registered"], "source_refs": ["scripts/backup/backup-clawbot.sh"], "product_id": "clawbot-openclaw", "project_id": "clawbot", "owner_lane": "ai_agent_runtime", "telegram_destination": "blocked:clawbot_cross_product_route"},
{"canonical_id": "cache:2026fifa:redis", "inventory_labels": ["current-fifa2026-redis-1"], "source_rows": ["L054"], "runtime_identity": "host188/docker/current-fifa2026-redis-1", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Product cache has no canonical persistence/restore contract"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "2026fifa", "project_id": "2026FIFAWorldCup", "owner_lane": "2026fifa_product_ops", "telegram_destination": "blocked:2026fifa_destination_unassigned"},
{"canonical_id": "cache:harbor:redis", "inventory_labels": ["harbor-redis"], "source_rows": ["L055"], "runtime_identity": "host110/docker/harbor-redis", "reconciliation_state": "matched_runtime_unverified", "findings": ["Critical HITL stateful policy exists; runtime health remains unverified"], "source_refs": ["ops/config/service-registry.yaml", "scripts/backup/backup-harbor.sh"], "project_id": "harbor", "owner_lane": "devops_team"},
{"canonical_id": "cache:agent-bounty:redis", "inventory_labels": ["agent_bounty_redis"], "source_rows": ["L056"], "runtime_identity": "host110/docker/agent_bounty_redis", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Product route receipt and persistence contract are missing"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "agent-bounty-protocol", "project_id": "agent-bounty-protocol", "owner_lane": "agent_bounty_product_ops", "telegram_destination": "blocked:vibe_ai_agent_receipt_gap"},
{"canonical_id": "cache:stockplatform-v2:redis", "inventory_labels": ["stockplatform-v2-redis-1"], "source_rows": ["L057"], "runtime_identity": "host110/docker/stockplatform-v2-redis-1", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Product ownership and persistence contract are unresolved"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "stockplatform-v2", "project_id": "stockPlatform", "owner_lane": "stockplatform_product_ops", "telegram_destination": "blocked:stockplatform_tsenyang_ownership_drift"},
{"canonical_id": "cache:sentry:redis", "inventory_labels": ["Sentry Redis (內建)"], "source_rows": ["L058"], "runtime_identity": "host110/sentry-self-hosted/redis", "reconciliation_state": "matched_runtime_unverified", "findings": ["Exact container and queue health receipt are absent"], "source_refs": ["ops/config/service-registry.yaml", "scripts/backup/backup-sentry.sh"], "project_id": "sentry", "owner_lane": "devops_team"},
{"canonical_id": "queue:signoz:zookeeper", "inventory_labels": ["signoz-zookeeper-1"], "source_rows": ["L063"], "runtime_identity": "host110/docker/signoz-zookeeper-1", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Host placement and dependent ClickHouse/Kafka topology are not reconciled"], "source_refs": ["ops/monitoring/service-registry.yaml", "scripts/backup/backup-signoz.sh"], "project_id": "signoz", "owner_lane": "observability_ops"}
]
},
{
"group_id": "imported-observability",
"category": "monitoring_observability",
"common": {
"product_id": "shared-infrastructure",
"project_id": "observability-platform",
"site_id": "on-prem-and-k3s",
"owner_lane": "observability_ops",
"source_truth_state": "monitoring_registry_and_inventory_conflict",
"live_truth_state": "not_probed_this_reconciliation",
"domain_router": "docker_container",
"executor": "host_ansible_executor_with_exact_container_playbook",
"verifier": "container_runtime_independent_verifier",
"monitoring": {"signals": ["self_health", "pipeline_freshness", "target_coverage", "delivery_receipts"], "coverage": "partial"},
"alerting": {"lifecycle": "observability_platform_incident", "cross_domain_fallback_allowed": false, "required_ai_attribution": true},
"telegram_destination": "awoooi_sre_war_room_for_shared_P0_P1_lifecycle_only",
"backup_restore": {"mode": "critical_read_only_default", "target": "scripts/backup/backup-monitoring.sh", "freshness": "unverified"},
"learning_targets": ["KM", "RAG", "MCP", "PlayBook"],
"priority": "P1",
"dependencies": ["PORT-001", "PORT-002", "PORT-010"],
"next_action": "Resolve exact deployment identity, verify generated target parity, and attach lifecycle receipts before claiming observability coverage."
},
"members": [
{"canonical_id": "service:prometheus", "inventory_labels": ["Prometheus"], "source_rows": ["L071"], "runtime_identity": "host110/docker/prometheus", "source_truth_state": "source_reconciled_runtime_pending", "reconciliation_state": "source_identity_reconciled_runtime_pending", "findings": ["Monitoring registry now declares the exact host110 Docker identity", "Production container, generated target parity and scrape freshness remain unverified"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "k8s/monitoring/prometheus.yml", "scripts/ops/deploy-alertmanager-runtime-scrape.sh"], "priority": "P0", "verifier": "prometheus_runtime_target_and_rule_verifier"},
{"canonical_id": "service:grafana", "inventory_labels": ["Grafana"], "source_rows": ["L072"], "runtime_identity": "host110/docker/grafana", "reconciliation_state": "source_partial_runtime_unverified", "findings": ["Source policy exists but live dashboard/runtime parity is not proven"], "source_refs": ["ops/config/service-registry.yaml", "docs/security/monitoring-alerting-observability-inventory.snapshot.json"]},
{"canonical_id": "service:alertmanager", "inventory_labels": ["Alertmanager"], "source_rows": ["L073"], "runtime_identity": "host110/docker/alertmanager", "source_truth_state": "source_reconciled_runtime_pending", "reconciliation_state": "source_control_path_reconciled_runtime_pending", "findings": ["Monitoring registry and typed router now declare the exact host110 Docker identity", "Alert rule now reads Alertmanager monotonic delivery counters instead of process-local API counters", "Exact bounded Ansible recovery and independent verifier are source-ready; production same-run receipt is pending"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "k8s/monitoring/alert-chain-monitor.yaml", "ops/alertmanager/alertmanager.yml", "infra/ansible/playbooks/110-alertmanager-delivery-recovery.yml", "scripts/ops/deploy-alertmanager-runtime-scrape.sh"], "priority": "P0", "verifier": "alertmanager_delivery_chain_independent_verifier", "next_action": "Deploy the canonical scrape/rule and exact playbook, then correlate receiver, API ingest, Telegram receipt, retry/rollback and KM writeback under one run."},
{"canonical_id": "service:signoz", "inventory_labels": ["SignOz"], "source_rows": ["L074"], "runtime_identity": "unresolved:host110-ui-host188-collector", "reconciliation_state": "split_plane_identity_conflict", "findings": ["Inventory collapses UI, collector, and storage into one host110 service while source registries split them across hosts"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "scripts/backup/backup-signoz.sh"]},
{"canonical_id": "service:otel-collector:signoz", "inventory_labels": ["OTel Collector"], "source_rows": ["L075"], "runtime_identity": "unresolved:host110-or-host188/otel-collector", "reconciliation_state": "host_identity_conflict", "findings": ["Inventory says host110 while monitoring registry says host188 port 24317"], "source_refs": ["ops/monitoring/service-registry.yaml", "docs/security/monitoring-alerting-observability-inventory.snapshot.json"], "domain_router": "unknown", "executor": null, "verifier": "asset_identity_drift_verifier"},
{"canonical_id": "service:blackbox-exporter", "inventory_labels": ["Blackbox Exporter"], "source_rows": ["L076"], "runtime_identity": "host110/docker/blackbox-exporter", "reconciliation_state": "matched_runtime_unverified", "findings": ["Generated target freshness and live probe results are not current truth"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/generated/blackbox-targets-generated.yaml"]},
{"canonical_id": "service:node-exporter:host110", "inventory_labels": ["Node Exporter host110"], "source_rows": ["L077"], "runtime_identity": "host110/docker/node-exporter", "reconciliation_state": "inventory_only_runtime_unverified", "findings": ["No exact canonical service entry links this exporter to host110"], "source_refs": ["ops/monitoring/service-registry.yaml"]},
{"canonical_id": "service:node-exporter:host188", "inventory_labels": ["Node Exporter host188"], "source_rows": ["L078"], "runtime_identity": "host188/docker/node-exporter:9100", "reconciliation_state": "inventory_only_runtime_unverified", "findings": ["No exact canonical service entry links this exporter to host188"], "source_refs": ["ops/monitoring/service-registry.yaml"]},
{"canonical_id": "service:cadvisor:host110", "inventory_labels": ["cAdvisor"], "source_rows": ["L079"], "runtime_identity": "host110/docker/cadvisor", "reconciliation_state": "inventory_only_runtime_unverified", "findings": ["Image/version claim and scrape freshness need source and runtime receipts"], "source_refs": ["docs/security/monitoring-alerting-observability-inventory.snapshot.json"]},
{"canonical_id": "service:redis-exporter:host188", "inventory_labels": ["Redis Exporter"], "source_rows": ["L080"], "runtime_identity": "host188/docker/redis-exporter:9121", "reconciliation_state": "inventory_only_runtime_unverified", "findings": ["Exporter target must be reconciled with the conflicting Redis systemd/Docker identity"], "source_refs": ["ops/monitoring/generated/prometheus-scrape-generated.yaml"]},
{"canonical_id": "service:postgres-exporter:host188", "inventory_labels": ["Postgres Exporter"], "source_rows": ["L081"], "runtime_identity": "host188/docker/postgres-exporter:9187", "reconciliation_state": "inventory_only_runtime_unverified", "findings": ["One exporter does not prove coverage for every product database"], "source_refs": ["docs/security/monitoring-alerting-observability-inventory.snapshot.json"]},
{"canonical_id": "service:nginx-exporter:host188", "inventory_labels": ["Nginx Exporter"], "source_rows": ["L082"], "runtime_identity": "host188/docker/nginx-exporter:9113", "reconciliation_state": "inventory_only_runtime_unverified", "findings": ["Target freshness and exact Nginx instance coverage are unverified"], "source_refs": ["ops/monitoring/generated/prometheus-scrape-generated.yaml"]},
{"canonical_id": "service:gitea-exporter:host110", "inventory_labels": ["GitHub/Gitea Exporter"], "source_rows": ["L083"], "runtime_identity": "host110/docker/promhippie-github-exporter", "source_truth_state": "source_reconciled_runtime_pending", "reconciliation_state": "source_retirement_ready_runtime_pending", "findings": ["GitHub exporter and token references are removed from the source compose contract", "Prometheus contains a Gitea-native source candidate, but the exact production target is missing", "The legacy GitHub exporter remains visible in production and requires bounded removal plus independent readback", "Source retirement cannot be promoted to runtime closure until gitea-native freshness and legacy absence are both verified"], "source_refs": ["k8s/monitoring/docker-compose-110.yml", "k8s/monitoring/prometheus.yml", "scripts/generate_monitoring.py"], "priority": "P0", "monitoring": {"signals": ["gitea_native_target_up", "gitea_native_scrape_freshness", "legacy_github_exporter_absence"], "coverage": "source_candidate_runtime_target_missing"}, "next_action": "First make the exact gitea-native target observable; then use a bounded host110 compose apply to retire only the legacy exporter and independently verify target freshness plus absence of GitHub runtime traffic."},
{"canonical_id": "k8s:kube-state-metrics", "inventory_labels": ["kube-state-metrics"], "source_rows": ["L084"], "runtime_identity": "k3s/kube-state-metrics/deployment", "reconciliation_state": "source_visible_runtime_unverified", "findings": ["K3s deployment is registered; live rollout and scrape freshness are unverified"], "source_refs": ["ops/monitoring/service-registry.yaml"], "domain_router": "kubernetes_workload", "executor": "kubernetes_controlled_executor", "verifier": "kubernetes_rollout_verifier"},
{"canonical_id": "k8s:node-problem-detector", "inventory_labels": ["Node Problem Detector"], "source_rows": ["L085"], "runtime_identity": "k3s/node-problem-detector", "reconciliation_state": "missing_from_monitoring_service_registry", "findings": ["Inventory claims running but no exact registry member is present"], "source_refs": ["k8s/monitoring"], "domain_router": "kubernetes_workload", "executor": "kubernetes_controlled_executor", "verifier": "kubernetes_rollout_verifier"},
{"canonical_id": "k8s:otel-daemonset", "inventory_labels": ["OTel DaemonSet"], "source_rows": ["L086"], "runtime_identity": "k3s/observability/otel-daemonset", "reconciliation_state": "source_partial_runtime_unverified", "findings": ["Node coverage and telemetry delivery freshness need independent verification"], "source_refs": ["k8s/monitoring"], "domain_router": "kubernetes_workload", "executor": "kubernetes_controlled_executor", "verifier": "kubernetes_rollout_verifier"},
{"canonical_id": "k8s:event-exporter", "inventory_labels": ["Event Exporter"], "source_rows": ["L087"], "runtime_identity": "k3s/observability/event-exporter", "reconciliation_state": "source_visible_runtime_unverified", "findings": ["Registry entry exists; event-to-metric freshness is unverified"], "source_refs": ["ops/monitoring/service-registry.yaml"], "domain_router": "kubernetes_workload", "executor": "kubernetes_controlled_executor", "verifier": "kubernetes_rollout_verifier"},
{"canonical_id": "service:langfuse", "inventory_labels": ["Langfuse"], "source_rows": ["L088"], "runtime_identity": "host110/docker/langfuse", "reconciliation_state": "matched_runtime_unverified", "findings": ["Trace ingest, privacy boundary, and backup freshness are unverified"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "scripts/backup/backup-langfuse.sh"], "owner_lane": "ai_team", "backup_restore": {"mode": "critical_read_only_default", "target": "scripts/backup/backup-langfuse.sh", "freshness": "unverified"}}
]
},
{
"group_id": "imported-error-tracking",
"category": "error_tracking",
"common": {
"product_id": "shared-infrastructure",
"project_id": "sentry-self-hosted",
"site_id": "host110",
"owner_lane": "observability_ops",
"source_truth_state": "partial_exact_container_registry",
"live_truth_state": "not_probed_this_reconciliation",
"domain_router": "docker_container",
"executor": "host_ansible_executor_with_exact_container_playbook",
"verifier": "container_runtime_independent_verifier",
"monitoring": {"signals": ["container_health", "ingestion", "queue_lag", "error_event_freshness"], "coverage": "partial"},
"alerting": {"lifecycle": "sentry_exact_container_incident", "cross_domain_fallback_allowed": false, "required_ai_attribution": true},
"telegram_destination": "awoooi_sre_war_room_for_shared_P0_P1_lifecycle_only",
"backup_restore": {"mode": "critical_read_only_default", "target": "scripts/backup/backup-sentry.sh", "freshness": "unverified"},
"learning_targets": ["KM", "RAG", "MCP", "PlayBook"],
"priority": "P1",
"dependencies": ["PORT-001", "PORT-004"],
"next_action": "Bind exact host110 container aliases to bounded PlayBooks and require independent queue/ingest verification."
},
"members": [
{"canonical_id": "service:sentry:web", "inventory_labels": ["Sentry Self-Hosted"], "source_rows": ["L096"], "runtime_identity": "host110/sentry-self-hosted/web", "reconciliation_state": "matched_runtime_unverified", "findings": ["Exact-container recovery catalog exists but production runtime receipt is pending"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml"]},
{"canonical_id": "service:sentry:relay", "inventory_labels": ["Sentry Relay"], "source_rows": ["L097"], "runtime_identity": "host110/sentry-self-hosted/relay", "reconciliation_state": "image_supply_chain_conflict", "findings": ["Imported image points to ghcr.io, which is forbidden by the GitHub freeze unless replaced by an internal immutable mirror"], "source_refs": ["docs/evaluations/package_supply_chain_inventory_2026-06-04.json", "ops/config/service-registry.yaml"], "priority": "P0"},
{"canonical_id": "service:sentry:snuba", "inventory_labels": ["Snuba"], "source_rows": ["L098"], "runtime_identity": "host110/sentry-self-hosted/snuba", "reconciliation_state": "image_supply_chain_and_exact_worker_gap", "findings": ["Imported image points to ghcr.io", "Only the profiling consumer has an exact recovery catalog; other workers need identity-only monitoring until registered"], "source_refs": ["ops/config/service-registry.yaml"], "priority": "P0"}
]
},
{
"group_id": "imported-ai-runtime",
"category": "ai_llm",
"common": {
"product_id": "awoooi",
"project_id": "ai-control-plane",
"site_id": "on-prem-lan",
"owner_lane": "ai_team",
"source_truth_state": "conflicts_with_approved_provider_route",
"live_truth_state": "not_probed_this_reconciliation",
"domain_router": "docker_container",
"executor": "host_ansible_executor_with_exact_container_playbook",
"verifier": "container_runtime_independent_verifier",
"monitoring": {"signals": ["health", "latency", "tokens", "cost", "route_state", "receipt_freshness"], "coverage": "partial"},
"alerting": {"lifecycle": "ai_provider_or_agent_incident", "cross_domain_fallback_allowed": false, "required_ai_attribution": true},
"telegram_destination": "awoooi_sre_war_room_for_shared_P0_P1_lifecycle_only",
"backup_restore": {"mode": "readback_first", "target": "scripts/backup/backup-ai-artifacts.sh", "freshness": "unverified"},
"learning_targets": ["KM", "RAG", "MCP", "PlayBook"],
"priority": "P0",
"dependencies": ["PORT-001", "PORT-003", "PORT-011"],
"next_action": "Reconcile provider and agent identity before route evaluation; never treat a proxy host as the AI provider."
},
"members": [
{"canonical_id": "service:ollama:host188", "inventory_labels": ["Ollama systemd host188"], "source_rows": ["L106"], "runtime_identity": "host188/systemd/ollama:11434", "reconciliation_state": "stale_provider_identity_conflict", "findings": ["Imported inventory identifies host188 Ollama, while the approved route uses host111 as the local hop", "No fallback is allowed to select another host/domain by generic target"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "docs/operations/sre-k3s-controlled-automation-work-items.snapshot.json"], "domain_router": "host_systemd", "executor": "host_ansible_executor", "verifier": "host111_ollama_route_identity_verifier", "next_action": "Create identity drift work; determine whether host188 Ollama is retired, non-routing, or a separately owned asset without changing the five-hop order."},
{"canonical_id": "model:host188:nemotron-mini", "inventory_labels": ["nemotron-mini:latest"], "source_rows": ["L107"], "runtime_identity": "host188/ollama/model/nemotron-mini:latest", "reconciliation_state": "model_catalog_conflict", "findings": ["Imported model conflicts with monitoring registry local model list and approved provider identity", "Latest tag is mutable and not an immutable deployment identity"], "source_refs": ["ops/monitoring/service-registry.yaml"], "domain_router": "control_plane_service", "executor": "single_control_plane_executor", "verifier": "model_catalog_and_digest_verifier"},
{"canonical_id": "service:open-webui", "inventory_labels": ["Open WebUI"], "source_rows": ["L108"], "runtime_identity": "host188/docker/open-webui:3010", "reconciliation_state": "source_partial_runtime_unverified", "findings": ["UI availability does not prove Ollama route correctness"], "source_refs": ["infra/ansible/inventory/group_vars/host_188.yml", "scripts/backup/backup-open-webui.sh"]},
{"canonical_id": "service:openclaw", "inventory_labels": ["OpenClaw"], "source_rows": ["L109"], "runtime_identity": "host188/docker/openclaw:8089", "reconciliation_state": "port_and_service_identity_conflict", "findings": ["Ansible source maps openclaw to port 8088 while imported inventory says 8089", "No current durable agent decision receipt is implied by container health"], "source_refs": ["infra/ansible/inventory/group_vars/host_188.yml", "ops/monitoring/service-registry.yaml"]},
{"canonical_id": "service:clawbot:v5", "inventory_labels": ["ClawBot v5", "clawbot"], "source_rows": ["L110", "L232"], "runtime_identity": "host188/docker/clawbot:8088", "reconciliation_state": "duplicate_inventory_row_and_route_drift", "findings": ["Same service appears in AI and application tables", "Telegram route is blocked by cross-product bot ownership drift"], "source_refs": ["scripts/backup/backup-clawbot.sh", "docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "clawbot-openclaw", "project_id": "clawbot", "owner_lane": "ai_agent_runtime", "telegram_destination": "blocked:clawbot_cross_product_route"}
]
},
{
"group_id": "imported-workflow-automation",
"category": "workflow_automation",
"common": {
"product_id": "shared-infrastructure",
"project_id": "workflow-automation",
"site_id": "host188",
"owner_lane": "automation_ops",
"source_truth_state": "ansible_source_partial",
"live_truth_state": "not_probed_this_reconciliation",
"domain_router": "docker_container",
"executor": "host_ansible_executor_with_exact_container_playbook",
"verifier": "container_runtime_independent_verifier",
"monitoring": {"signals": ["health", "workflow_failures", "queue_depth", "delivery_receipts"], "coverage": "gap"},
"alerting": {"lifecycle": "workflow_incident", "cross_domain_fallback_allowed": false, "required_ai_attribution": true},
"telegram_destination": "awoooi_sre_war_room_for_shared_P0_P1_lifecycle_only",
"backup_restore": {"mode": "critical_read_only_default", "target": "gap:no_n8n_restore_contract", "freshness": "unverified"},
"learning_targets": ["KM", "RAG", "MCP", "PlayBook"],
"priority": "P1",
"dependencies": ["PORT-001", "PORT-011"],
"next_action": "Register n8n workflow inventory, owners, failure receipts, and exact-host restore verifier."
},
"members": [
{"canonical_id": "service:n8n:host188", "inventory_labels": ["n8n"], "source_rows": ["L118"], "runtime_identity": "host188/docker/n8n:5678", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Ansible directory exists but workflow inventory, backup, and failure routing are absent"], "source_refs": ["infra/ansible/inventory/group_vars/host_188.yml"]}
]
},
{
"group_id": "imported-cicd-registry",
"category": "container_registry_cicd",
"common": {
"product_id": "shared-infrastructure",
"project_id": "gitea-harbor-cd",
"site_id": "on-prem-and-k3s",
"owner_lane": "devops_team",
"source_truth_state": "source_partial_runtime_unverified",
"live_truth_state": "not_probed_this_reconciliation",
"domain_router": "docker_container",
"executor": "host_ansible_executor_with_exact_container_playbook",
"verifier": "container_runtime_independent_verifier",
"monitoring": {"signals": ["health", "queue", "build", "deploy_marker", "artifact_digest", "runner_freshness"], "coverage": "partial"},
"alerting": {"lifecycle": "gitea_cd_or_registry_incident", "cross_domain_fallback_allowed": false, "required_ai_attribution": true},
"telegram_destination": "awoooi_sre_war_room_for_shared_P0_P1_lifecycle_only",
"backup_restore": {"mode": "critical_read_only_default", "target": "service_specific_backup_contract", "freshness": "unverified"},
"learning_targets": ["KM", "RAG", "MCP", "PlayBook"],
"priority": "P0",
"dependencies": ["PORT-001", "PORT-004", "PORT-007"],
"next_action": "Verify Gitea-only source control, internal artifact provenance, exact runner ownership, and backup/restore freshness."
},
"members": [
{"canonical_id": "service:harbor", "inventory_labels": ["Harbor"], "source_rows": ["L126"], "runtime_identity": "host110/docker/harbor-core:5000", "reconciliation_state": "matched_runtime_unverified", "findings": ["Critical HITL policy exists; current health, immutable image, and backup receipts are unverified"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "scripts/backup/backup-harbor.sh"], "backup_restore": {"mode": "critical_read_only_default", "target": "scripts/backup/backup-harbor.sh", "freshness": "unverified"}},
{"canonical_id": "service:harbor:trivy", "inventory_labels": ["Trivy via Harbor"], "source_rows": ["L127"], "runtime_identity": "host110/docker/trivy-adapter", "reconciliation_state": "inventory_only_runtime_unverified", "findings": ["Scanner freshness and vulnerability database provenance are not linked"], "source_refs": ["docs/evaluations/package_supply_chain_inventory_2026-06-04.json"]},
{"canonical_id": "service:docker-registry:backup", "inventory_labels": ["Docker Registry 備用"], "source_rows": ["L128"], "runtime_identity": "unresolved:host110-or-host188/docker-registry", "reconciliation_state": "host_identity_conflict", "findings": ["Imported row says host110 while host188 Ansible variables define docker_registry", "Backup registry promotion policy is not registered"], "source_refs": ["infra/ansible/inventory/group_vars/host_188.yml"], "domain_router": "unknown", "executor": null, "verifier": "asset_identity_drift_verifier"},
{"canonical_id": "service:gitea", "inventory_labels": ["Gitea"], "source_rows": ["L129"], "runtime_identity": "host110/docker/gitea:3001", "reconciliation_state": "matched_runtime_unverified", "findings": ["Gitea is the sole source-control authority; runtime and backup freshness need exact receipts", "The exact gitea-native Prometheus target is missing in production, so source configuration cannot prove metrics coverage", "The legacy GitHub exporter must remain a separate drift until bounded removal and absence verification"], "source_refs": ["ops/config/service-registry.yaml", "product.awoooi.yaml", "scripts/backup/backup-gitea.sh", "k8s/monitoring/prometheus.yml"], "monitoring": {"signals": ["gitea_native_target_up", "gitea_native_scrape_freshness", "legacy_exporter_absence"], "coverage": "runtime_target_missing"}, "backup_restore": {"mode": "critical_read_only_default", "target": "scripts/backup/gitea-full-backup-restore-drill.sh", "freshness": "unverified"}, "next_action": "Restore and verify the exact gitea-native metrics target before retiring the legacy exporter; keep both checks separate from Gitea application health."},
{"canonical_id": "runner:gitea:vibework-host110", "inventory_labels": ["Act Runner VibeWork 110"], "source_rows": ["L130"], "runtime_identity": "host110/gitea-act-runner/vibework", "reconciliation_state": "paused_capacity_and_legacy_registry_gap", "findings": ["Imported state says paused", "Monitoring registry still describes a GitHub Actions runner instead of this Gitea runner"], "source_refs": ["ops/monitoring/service-registry.yaml"]},
{"canonical_id": "runner:gitea:vibework-deployer-host188", "inventory_labels": ["Act Runner VibeWork deployer"], "source_rows": ["L131"], "runtime_identity": "host188/gitea-act-runner/vibework-deployer", "reconciliation_state": "missing_from_monitoring_registry", "findings": ["Current health and single-writer/CD ownership are unverified"], "source_refs": ["ops/monitoring/service-registry.yaml"]},
{"canonical_id": "runner:gitea:vibework-dedicated-host188", "inventory_labels": ["Act Runner VibeWork dedicated"], "source_rows": ["L132"], "runtime_identity": "host188/gitea-act-runner/vibework-dedicated", "reconciliation_state": "missing_from_monitoring_registry", "findings": ["Exact labels, capacity, and production deploy scope are unverified"], "source_refs": ["ops/monitoring/service-registry.yaml"]},
{"canonical_id": "runner:gitea:stockplatform-host188", "inventory_labels": ["Act Runner StockPlatform"], "source_rows": ["L133"], "runtime_identity": "host188/gitea-act-runner/stockplatform", "reconciliation_state": "missing_from_monitoring_registry", "findings": ["Exact labels, capacity, and product ownership are unverified"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "stockplatform-v2", "project_id": "stockPlatform", "owner_lane": "stockplatform_product_ops", "telegram_destination": "blocked:stockplatform_tsenyang_ownership_drift"},
{"canonical_id": "k8s:argocd", "inventory_labels": ["ArgoCD", "K3s ArgoCD"], "source_rows": ["L134", "L251"], "runtime_identity": "k3s/argocd", "reconciliation_state": "source_visible_runtime_unverified", "findings": ["Multiple ArgoCD components are registered; live sync and rollout truth remain unverified"], "source_refs": ["ops/monitoring/service-registry.yaml", "docs/security/k8s-argocd-manifest-inventory.snapshot.json"], "domain_router": "kubernetes_workload", "executor": "kubernetes_controlled_executor", "verifier": "kubernetes_rollout_verifier"}
]
},
{
"group_id": "imported-storage-backup",
"category": "storage_backup",
"common": {
"product_id": "shared-infrastructure",
"project_id": "backup-dr",
"site_id": "on-prem-and-k3s",
"owner_lane": "dr_ops",
"source_truth_state": "source_policy_visible_runtime_unverified",
"live_truth_state": "not_probed_this_reconciliation",
"domain_router": "backup_restore",
"executor": "backup_restore_break_glass",
"verifier": "backup_restore_readback_verifier",
"monitoring": {"signals": ["freshness", "integrity", "offsite", "escrow", "restore_drill"], "coverage": "partial"},
"alerting": {"lifecycle": "dr_readback_incident", "cross_domain_fallback_allowed": false, "required_ai_attribution": true},
"telegram_destination": "awoooi_sre_war_room_for_shared_P0_P1_lifecycle_only",
"backup_restore": {"mode": "critical_read_only_default", "target": "backup_status_and_isolated_restore_drill", "freshness": "unverified"},
"learning_targets": ["KM", "RAG", "MCP", "PlayBook"],
"priority": "P0",
"dependencies": ["PORT-001", "PORT-007"],
"next_action": "Read backup status, offsite, escrow, and isolated restore receipts; do not run restore or write markers without break-glass."
},
"members": [
{"canonical_id": "storage:minio:host188", "inventory_labels": ["MinIO"], "source_rows": ["L142"], "runtime_identity": "host188/docker/minio:9000", "reconciliation_state": "matched_runtime_unverified", "findings": ["MinIO service identity exists; backup object freshness, offsite, and escrow remain unverified"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "docs/security/backup-restore-escrow-inventory.snapshot.json"]},
{"canonical_id": "k8s:velero", "inventory_labels": ["Velero", "K3s Velero"], "source_rows": ["L143", "L252"], "runtime_identity": "k3s/velero", "reconciliation_state": "source_visible_runtime_unverified", "findings": ["Deployment presence is not proof of backup freshness or restore success"], "source_refs": ["ops/monitoring/service-registry.yaml", "docs/security/backup-restore-escrow-inventory.snapshot.json"], "domain_router": "kubernetes_workload", "executor": "kubernetes_controlled_executor", "verifier": "velero_readback_and_rollout_verifier"}
]
},
{
"group_id": "imported-gateways",
"category": "gateway_proxy",
"common": {
"product_id": "shared-infrastructure",
"project_id": "network-gateway",
"site_id": "on-prem-lan",
"owner_lane": "network_ops",
"source_truth_state": "inventory_and_ansible_partial",
"live_truth_state": "not_probed_this_reconciliation",
"domain_router": "docker_container",
"executor": "host_ansible_executor_with_exact_container_playbook",
"verifier": "container_runtime_independent_verifier",
"monitoring": {"signals": ["health", "tls", "route", "upstream", "latency"], "coverage": "partial"},
"alerting": {"lifecycle": "gateway_incident", "cross_domain_fallback_allowed": false, "required_ai_attribution": true},
"telegram_destination": "awoooi_sre_war_room_for_shared_P0_P1_lifecycle_only",
"backup_restore": {"mode": "readback_first", "target": "scripts/backup/backup-configs.sh", "freshness": "unverified"},
"learning_targets": ["KM", "RAG", "MCP", "PlayBook"],
"priority": "P1",
"dependencies": ["PORT-001", "PORT-009"],
"next_action": "Resolve exact gateway identity and run route/TLS/upstream readback before bounded reload or restart."
},
"members": [
{"canonical_id": "service:nginx:host188", "inventory_labels": ["Nginx systemd"], "source_rows": ["L151"], "runtime_identity": "host188/systemd/nginx:80,443", "reconciliation_state": "source_visible_runtime_unverified", "findings": ["Imported purpose mentions multiple products but exact route ownership is not reconciled"], "source_refs": ["infra/ansible/inventory/group_vars/host_188.yml", "scripts/backup/backup-public-routes.sh"], "domain_router": "host_systemd", "executor": "host_ansible_executor", "verifier": "host_route_tls_independent_verifier"},
{"canonical_id": "service:harbor:nginx", "inventory_labels": ["Nginx Harbor"], "source_rows": ["L152"], "runtime_identity": "host110/docker/harbor-nginx:5000", "reconciliation_state": "source_partial_runtime_unverified", "findings": ["Exact container health and upstream route receipt are absent"], "source_refs": ["ops/config/service-registry.yaml", "scripts/backup/backup-harbor.sh"], "project_id": "harbor"},
{"canonical_id": "service:sentry:nginx", "inventory_labels": ["Nginx Sentry"], "source_rows": ["L153"], "runtime_identity": "host110/sentry-self-hosted/nginx:9000", "reconciliation_state": "source_partial_runtime_unverified", "findings": ["Exact container health and Sentry upstream receipt are absent"], "source_refs": ["ops/config/service-registry.yaml", "scripts/backup/backup-sentry.sh"], "project_id": "sentry"},
{"canonical_id": "service:stockplatform-v2:nginx", "inventory_labels": ["Nginx stockplatform-v2"], "source_rows": ["L154"], "runtime_identity": "host110/docker/stockplatform-v2-nginx:31235", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Product route ownership and alert destination are unresolved"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "stockplatform-v2", "project_id": "stockPlatform", "owner_lane": "stockplatform_product_ops", "telegram_destination": "blocked:stockplatform_tsenyang_ownership_drift"},
{"canonical_id": "service:bitan:nginx", "inventory_labels": ["Nginx Bitan"], "source_rows": ["L155"], "runtime_identity": "host110/docker/bitan-nginx", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Product gateway has no canonical Telegram route or backup verifier"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "bitan-pharmacy", "project_id": "bitan-pharmacy", "owner_lane": "bitan_product_ops", "telegram_destination": "blocked:bitan_destination_unassigned"},
{"canonical_id": "service:bitan:postgrest", "inventory_labels": ["PostgREST"], "source_rows": ["L156"], "runtime_identity": "host110/docker/bitan-postgrest:3000", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Database API must use Bitan-specific policy and DB verifier; generic container restart is insufficient"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "bitan-pharmacy", "project_id": "bitan-pharmacy", "owner_lane": "bitan_product_ops", "telegram_destination": "blocked:bitan_destination_unassigned"},
{"canonical_id": "network:keepalived:vip-unresolved", "inventory_labels": ["Keepalived"], "source_rows": ["L157"], "runtime_identity": "unresolved:vip-192.168.0.125-or-192.168.0.200", "reconciliation_state": "critical_vip_identity_conflict", "findings": ["Imported inventory claims VIP 192.168.0.125 on host188 plus K3s", "Ansible host110/host188 vars claim VIP 192.168.0.200", "No network cutover is authorized"], "source_refs": ["infra/ansible/inventory/group_vars/host_110.yml", "infra/ansible/inventory/group_vars/host_188.yml"], "priority": "P0", "domain_router": "unknown", "executor": null, "verifier": "vip_identity_drift_verifier", "next_action": "Create fail-closed drift item and reconcile live VIP ownership read-only before any keepalived action."}
]
},
{
"group_id": "imported-security",
"category": "security",
"common": {
"product_id": "shared-infrastructure",
"project_id": "security-platform",
"site_id": "on-prem-lan",
"owner_lane": "security_ops",
"source_truth_state": "source_registry_partial",
"live_truth_state": "not_probed_this_reconciliation",
"domain_router": "host_systemd",
"executor": "host_ansible_executor",
"verifier": "host_runtime_independent_verifier",
"monitoring": {"signals": ["service_health", "agent_freshness", "security_events", "scan_receipts"], "coverage": "partial"},
"alerting": {"lifecycle": "security_incident", "cross_domain_fallback_allowed": false, "required_ai_attribution": true},
"telegram_destination": "awoooi_sre_war_room_for_security_incident_lifecycle",
"backup_restore": {"mode": "critical_read_only_default", "target": "security_config_and_evidence_contract", "freshness": "unverified"},
"learning_targets": ["KM", "RAG", "MCP", "PlayBook"],
"priority": "P0",
"dependencies": ["PORT-001", "PORT-005"],
"next_action": "Run bounded no-secret posture readback and attach exact-host verifier receipts; active scans remain break-glass."
},
"members": [
{"canonical_id": "service:wazuh-manager", "inventory_labels": ["Wazuh Manager"], "source_rows": ["L168"], "runtime_identity": "host112/systemd/wazuh-manager", "reconciliation_state": "bounded_source_ready_runtime_unverified", "findings": ["Imported Active 22h and memory labels remain unverified point-in-time claims", "Exact no-write posture catalog exists; active response and arbitrary mutation remain unavailable"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml"], "verifier": "iwooos_wazuh_controlled_executor_runtime"},
{"canonical_id": "service:wazuh-indexer:host112", "inventory_labels": ["Wazuh Indexer"], "source_rows": ["L169"], "runtime_identity": "host112/systemd/wazuh-indexer:9200,9300", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Indexer state, disk, shard health, and backup/restore verifier are not registered", "OpenSearch-compatible data mutation remains database-bounded"], "source_refs": ["ops/monitoring/alerts-unified.yml"], "domain_router": "database", "executor": "db_bounded_executor", "verifier": "wazuh_indexer_independent_verifier"},
{"canonical_id": "service:wazuh-dashboard:host112", "inventory_labels": ["Wazuh Dashboard"], "source_rows": ["L170"], "runtime_identity": "host112/systemd/wazuh-dashboard:443", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["TLS, auth-safe UI health, and upstream Indexer/Manager receipts are not registered"], "source_refs": ["ops/monitoring/alerts-unified.yml"], "verifier": "wazuh_dashboard_independent_verifier"},
{"canonical_id": "service:wazuh-agent:host188", "inventory_labels": ["Wazuh Agent host188"], "source_rows": ["L171"], "runtime_identity": "host188/systemd/wazuh-agent", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Agent-manager connectivity and event freshness are not registered"], "source_refs": ["infra/ansible/inventory/hosts.yml"]},
{"canonical_id": "service:kali-scanner:host112", "inventory_labels": ["kali-scanner.service"], "source_rows": ["L176"], "runtime_identity": "host112/systemd/kali-scanner:8080", "reconciliation_state": "source_partial_runtime_unverified", "findings": ["Only TCP blackbox is registered", "Active or credentialed scans remain critical break-glass"], "source_refs": ["ops/monitoring/service-registry.yaml"]},
{"canonical_id": "service:awoooi-guardian:host112", "inventory_labels": ["awoooi-guardian.service"], "source_rows": ["L177"], "runtime_identity": "host112/systemd/awoooi-guardian", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Multi-host monitoring scope, action boundary, and independent verifier are not registered"], "source_refs": ["ops/monitoring/alerts-unified.yml"]},
{"canonical_id": "service:filebeat:host112", "inventory_labels": ["filebeat.service"], "source_rows": ["L178"], "runtime_identity": "host112/systemd/filebeat", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Log delivery freshness and Wazuh Indexer backpressure receipts are missing"], "source_refs": ["ops/monitoring/alerts-unified.yml"], "verifier": "filebeat_delivery_independent_verifier"},
{"canonical_id": "service:wg-easy:host112", "inventory_labels": ["wg-easy"], "source_rows": ["L183"], "runtime_identity": "host112/docker/wg-easy:51820,51821", "reconciliation_state": "github_freeze_and_network_boundary_conflict", "findings": ["Imported image points to ghcr.io and conflicts with the GitHub freeze", "VPN/firewall cutover is critical break-glass", "Web UI exposure and route ownership are unverified"], "source_refs": ["docs/evaluations/package_supply_chain_inventory_2026-06-04.json"], "domain_router": "docker_container", "executor": "host_ansible_executor_with_exact_container_playbook", "verifier": "wireguard_route_and_container_independent_verifier"},
{"canonical_id": "service:node-exporter:host112", "inventory_labels": ["node-exporter host112"], "source_rows": ["L184"], "runtime_identity": "host112/docker/node-exporter:9100", "reconciliation_state": "missing_from_monitoring_service_registry", "findings": ["Current scrape freshness and exact image provenance are not registered"], "source_refs": ["ops/monitoring/generated/prometheus-scrape-generated.yaml"], "domain_router": "docker_container", "executor": "host_ansible_executor_with_exact_container_playbook", "verifier": "node_exporter_scrape_independent_verifier"},
{"canonical_id": "package:wazuh-suite:host112:4.9.2", "inventory_labels": ["Wazuh Manager/Indexer/Dashboard 4.9.2"], "source_rows": ["L189"], "runtime_identity": "host112/packages/wazuh-suite:4.9.2", "reconciliation_state": "package_version_claim_runtime_unverified", "findings": ["Composite package row duplicates three runtime services", "Upgrade and rollback require component-specific replay and canary"], "source_refs": ["ops/config/service-registry.yaml"], "verifier": "host_package_version_independent_verifier"},
{"canonical_id": "package:nmap-zenmap:host112", "inventory_labels": ["nmap / zenmap"], "source_rows": ["L190"], "runtime_identity": "host112/packages/nmap-zenmap", "reconciliation_state": "security_tool_inventory_only", "findings": ["Installed tool is not authorization for active scanning", "Version pair requires immutable package provenance"], "source_refs": ["docs/evaluations/package_supply_chain_inventory_2026-06-04.json"], "verifier": "security_tool_inventory_verifier"},
{"canonical_id": "package:metasploit:host112", "inventory_labels": ["Metasploit Framework"], "source_rows": ["L191"], "runtime_identity": "host112/packages/metasploit", "reconciliation_state": "critical_security_tool_inventory_only", "findings": ["Credentialed exploit activity is critical break-glass", "Installed version and package provenance are unverified"], "source_refs": ["docs/evaluations/package_supply_chain_inventory_2026-06-04.json"], "verifier": "security_tool_inventory_verifier"},
{"canonical_id": "package:burp-suite:host112", "inventory_labels": ["Burp Suite"], "source_rows": ["L192"], "runtime_identity": "host112/packages/burp-suite", "reconciliation_state": "critical_security_tool_inventory_only", "findings": ["Active web scanning is critical break-glass", "Edition, license, and package provenance are unverified"], "source_refs": ["docs/evaluations/package_supply_chain_inventory_2026-06-04.json"], "verifier": "security_tool_inventory_verifier"},
{"canonical_id": "package:sqlmap:host112", "inventory_labels": ["sqlmap"], "source_rows": ["L193"], "runtime_identity": "host112/packages/sqlmap", "reconciliation_state": "critical_security_tool_inventory_only", "findings": ["SQL injection scanning is critical break-glass", "Installed version and package provenance are unverified"], "source_refs": ["docs/evaluations/package_supply_chain_inventory_2026-06-04.json"], "verifier": "security_tool_inventory_verifier"},
{"canonical_id": "package:nikto:host112", "inventory_labels": ["nikto"], "source_rows": ["L194"], "runtime_identity": "host112/packages/nikto", "reconciliation_state": "critical_security_tool_inventory_only", "findings": ["Active web scanning is critical break-glass", "Installed version and package provenance are unverified"], "source_refs": ["docs/evaluations/package_supply_chain_inventory_2026-06-04.json"], "verifier": "security_tool_inventory_verifier"},
{"canonical_id": "package:openvas:host112", "inventory_labels": ["OpenVAS / ospd-openvas"], "source_rows": ["L195"], "runtime_identity": "host112/packages/openvas-ospd", "reconciliation_state": "critical_security_tool_inventory_only", "findings": ["Active vulnerability scanning is critical break-glass", "Feed freshness and daemon state are not registered"], "source_refs": ["docs/evaluations/package_supply_chain_inventory_2026-06-04.json"], "verifier": "security_tool_and_feed_independent_verifier"},
{"canonical_id": "package:python-elasticsearch:host112", "inventory_labels": ["python3-elasticsearch"], "source_rows": ["L196"], "runtime_identity": "host112/packages/python3-elasticsearch", "reconciliation_state": "package_inventory_only", "findings": ["Client compatibility with Wazuh Indexer 4.9.2 is not verified"], "source_refs": ["docs/evaluations/package_supply_chain_inventory_2026-06-04.json"], "verifier": "host_package_version_independent_verifier"},
{"canonical_id": "service:haveged:host112", "inventory_labels": ["haveged"], "source_rows": ["L197"], "runtime_identity": "host112/systemd/haveged", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["Entropy service health and dependency need are unverified"], "source_refs": ["infra/ansible/inventory/hosts.yml"]},
{"canonical_id": "service:lightdm:host112", "inventory_labels": ["LightDM"], "source_rows": ["L198"], "runtime_identity": "host112/systemd/lightdm", "reconciliation_state": "missing_from_canonical_service_registry", "findings": ["GUI display-manager health is not a Wazuh completion receipt", "Recovery must remain exact-host and independently verified"], "source_refs": ["docs/operations/host112-guest-recovery-20260711.snapshot.json"]},
{"canonical_id": "schedule:host112:wazuh-guardian-hourly", "inventory_labels": ["每小時整點 host112 security schedule"], "source_rows": ["L203"], "runtime_identity": "host112/crontab/hourly-security-monitor", "reconciliation_state": "schedule_command_and_receipt_unregistered", "findings": ["Schedule label alone does not identify exact command, owner, idempotency, or output receipt"], "source_refs": ["ops/monitoring/alerts-unified.yml"], "verifier": "host_schedule_freshness_independent_verifier"},
{"canonical_id": "schedule:host112:daily-0800", "inventory_labels": ["每日 08:00 host112 security schedule"], "source_rows": ["L204"], "runtime_identity": "host112/crontab/daily-0800", "reconciliation_state": "schedule_command_and_receipt_unregistered", "findings": ["Exact command, owner, risk, and delivery receipt are absent"], "source_refs": ["ops/monitoring/alerts-unified.yml"], "verifier": "host_schedule_freshness_independent_verifier"},
{"canonical_id": "schedule:host112:weekly-sunday-0900", "inventory_labels": ["每週日 09:00 host112 security schedule"], "source_rows": ["L205"], "runtime_identity": "host112/crontab/weekly-sunday-0900", "reconciliation_state": "schedule_command_and_receipt_unregistered", "findings": ["Exact command, active-scan boundary, and result receipt are absent"], "source_refs": ["ops/monitoring/alerts-unified.yml"], "verifier": "host_schedule_freshness_independent_verifier"},
{"canonical_id": "schedule:host112:daily-0200", "inventory_labels": ["每日 02:00 host112 security schedule"], "source_rows": ["L206"], "runtime_identity": "host112/crontab/daily-0200", "reconciliation_state": "schedule_command_and_receipt_unregistered", "findings": ["Exact command, backup/log retention boundary, and result receipt are absent"], "source_refs": ["ops/monitoring/alerts-unified.yml"], "verifier": "host_schedule_freshness_independent_verifier"},
{"canonical_id": "service:fail2ban:host188", "inventory_labels": ["Fail2Ban historical inventory row"], "source_rows": [], "historical_source_rows": ["inventory-2e8580c-20260716:L167"], "runtime_identity": "host188/systemd/fail2ban", "reconciliation_state": "missing_from_imported_inventory", "findings": ["Asset was present in the 241-line inventory but disappeared from active c214160 without a retirement receipt", "Ban policy, rollback, and independent verifier remain unregistered"], "source_refs": ["infra/ansible/inventory/hosts.yml"], "next_action": "Create removal drift work; verify whether Fail2Ban is retired or omitted without restarting or changing bans."}
]
},
{
"group_id": "imported-product-applications",
"category": "application_service",
"common": {
"product_id": "unknown",
"project_id": "unknown",
"site_id": "on-prem-lan",
"owner_lane": "asset_identity",
"source_truth_state": "imported_runtime_claim_without_product_manifest",
"live_truth_state": "not_probed_this_reconciliation",
"domain_router": "docker_container",
"executor": "host_ansible_executor_with_exact_container_playbook",
"verifier": "container_runtime_independent_verifier",
"monitoring": {"signals": ["health", "latency", "errors", "data_freshness", "worker_lag"], "coverage": "gap"},
"alerting": {"lifecycle": "product_specific_incident", "cross_domain_fallback_allowed": false, "required_ai_attribution": true},
"telegram_destination": "blocked:canonical_product_route_required",
"backup_restore": {"mode": "critical_read_only_default", "target": "gap:product_backup_restore_contract", "freshness": "unverified"},
"learning_targets": ["KM", "RAG", "MCP", "PlayBook"],
"priority": "P1",
"dependencies": ["PORT-001", "PORT-006", "PORT-007", "PORT-008"],
"next_action": "Adopt a product manifest, exact container registry, product-only Telegram route, backup contract, and same-run verifier before controlled apply."
},
"members": [
{"canonical_id": "service:momo-pro:web", "inventory_labels": ["momo-pro-system"], "source_rows": ["L218"], "runtime_identity": "host188/docker/momo-pro-system:5003", "reconciliation_state": "product_manifest_and_registry_gap", "findings": ["Imported health is unverified; product destination returns chat_not_found"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json", "scripts/backup/backup-momo.sh"], "product_id": "momo-pro", "project_id": "momo-pro-system", "owner_lane": "momo_product_ops", "telegram_destination": "blocked:momo_configured_chat_unresolved", "backup_restore": {"mode": "critical_read_only_default", "target": "scripts/backup/backup-momo.sh", "freshness": "unverified"}},
{"canonical_id": "service:momo-pro:scheduler", "inventory_labels": ["momo-scheduler"], "source_rows": ["L219"], "runtime_identity": "host188/docker/momo-scheduler", "reconciliation_state": "product_manifest_and_registry_gap", "findings": ["Scheduler freshness and idempotent replay contract are missing"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "momo-pro", "project_id": "momo-pro-system", "owner_lane": "momo_product_ops", "telegram_destination": "blocked:momo_configured_chat_unresolved"},
{"canonical_id": "service:momo-pro:telegram-bot", "inventory_labels": ["momo-telegram-bot"], "source_rows": ["L220"], "runtime_identity": "host188/docker/momo-telegram-bot", "reconciliation_state": "destination_unreachable", "findings": ["Configured destination returns chat_not_found; sending remains blocked"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "momo-pro", "project_id": "momo-pro-system", "owner_lane": "momo_product_ops", "telegram_destination": "blocked:momo_configured_chat_unresolved"},
{"canonical_id": "service:vibework:web", "inventory_labels": ["vibework-production-web-1"], "source_rows": ["L221"], "runtime_identity": "host188/docker/vibework-production-web-1:32336", "reconciliation_state": "product_manifest_and_route_gap", "findings": ["Standalone VibeWork Telegram route is not implemented"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "vibework", "project_id": "vibework", "owner_lane": "vibework_product_ops", "telegram_destination": "blocked:vibework_route_not_implemented"},
{"canonical_id": "service:vibework:notification-worker", "inventory_labels": ["vibework-production-notification-worker-1"], "source_rows": ["L222"], "runtime_identity": "host188/docker/vibework-production-notification-worker-1", "reconciliation_state": "product_manifest_and_receipt_gap", "findings": ["Notification durability and delivery receipts are unverified"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "vibework", "project_id": "vibework", "owner_lane": "vibework_product_ops", "telegram_destination": "blocked:vibework_route_not_implemented"},
{"canonical_id": "site:tsenyang-website", "inventory_labels": ["tsenyang-website"], "source_rows": ["L223"], "runtime_identity": "host188/docker/tsenyang-website:3000", "reconciliation_state": "route_ownership_conflict", "findings": ["Telegram private route is shared with StockPlatform and ownership is unresolved"], "source_refs": ["infra/ansible/inventory/group_vars/host_188.yml", "docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "tsenyang-website", "project_id": "tsenyang-website", "owner_lane": "tsenyang_site_ops", "telegram_destination": "blocked:stockplatform_tsenyang_ownership_drift"},
{"canonical_id": "service:2026fifa:web", "inventory_labels": ["current-fifa2026-web-1"], "source_rows": ["L224"], "runtime_identity": "host188/docker/current-fifa2026-web-1:3108", "reconciliation_state": "product_manifest_and_route_gap", "findings": ["Canonical product destination and durable receipt backend are missing"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "2026fifa", "project_id": "2026FIFAWorldCup", "owner_lane": "2026fifa_product_ops", "telegram_destination": "blocked:2026fifa_destination_unassigned"},
{"canonical_id": "service:2026fifa:backend", "inventory_labels": ["current-fifa2026-backend-1"], "source_rows": ["L225"], "runtime_identity": "host188/docker/current-fifa2026-backend-1:8000", "reconciliation_state": "product_manifest_and_route_gap", "findings": ["API SLO and backup contract are not registered in this repo"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "2026fifa", "project_id": "2026FIFAWorldCup", "owner_lane": "2026fifa_product_ops", "telegram_destination": "blocked:2026fifa_destination_unassigned"},
{"canonical_id": "service:2026fifa:odds-worker", "inventory_labels": ["current-fifa2026-odds-worker-1"], "source_rows": ["L226"], "runtime_identity": "host188/docker/current-fifa2026-odds-worker-1", "reconciliation_state": "freshness_and_route_gap", "findings": ["Odds source freshness, provider ownership, and replay verifier are missing"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "2026fifa", "project_id": "2026FIFAWorldCup", "owner_lane": "2026fifa_product_ops", "telegram_destination": "blocked:2026fifa_destination_unassigned"},
{"canonical_id": "service:2026fifa:fixtures-worker", "inventory_labels": ["current-fifa2026-fixtures-worker-1"], "source_rows": ["L227"], "runtime_identity": "host188/docker/current-fifa2026-fixtures-worker-1", "reconciliation_state": "freshness_and_route_gap", "findings": ["Fixture source freshness and replay verifier are missing"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "2026fifa", "project_id": "2026FIFAWorldCup", "owner_lane": "2026fifa_product_ops", "telegram_destination": "blocked:2026fifa_destination_unassigned"},
{"canonical_id": "service:2026fifa:calendar-cache-worker", "inventory_labels": ["current-fifa2026-calendar-cache-worker-1"], "source_rows": ["L228"], "runtime_identity": "host188/docker/current-fifa2026-calendar-cache-worker-1", "reconciliation_state": "freshness_and_route_gap", "findings": ["Cache freshness and rebuild verifier are missing"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "2026fifa", "project_id": "2026FIFAWorldCup", "owner_lane": "2026fifa_product_ops", "telegram_destination": "blocked:2026fifa_destination_unassigned"},
{"canonical_id": "service:2026fifa:agent-review-worker", "inventory_labels": ["current-fifa2026-agent-review-worker-1"], "source_rows": ["L229"], "runtime_identity": "host188/docker/current-fifa2026-agent-review-worker-1", "reconciliation_state": "ai_review_receipt_gap", "findings": ["AI reviewer model, decision receipt, and cost attribution are not registered"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "2026fifa", "project_id": "2026FIFAWorldCup", "owner_lane": "2026fifa_product_ops", "telegram_destination": "blocked:2026fifa_destination_unassigned"},
{"canonical_id": "service:2026fifa:news-worker", "inventory_labels": ["current-fifa2026-news-worker-1"], "source_rows": ["L230"], "runtime_identity": "host188/docker/current-fifa2026-news-worker-1", "reconciliation_state": "source_freshness_and_route_gap", "findings": ["News source provenance/freshness and replay verifier are missing"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "2026fifa", "project_id": "2026FIFAWorldCup", "owner_lane": "2026fifa_product_ops", "telegram_destination": "blocked:2026fifa_destination_unassigned"},
{"canonical_id": "service:2026fifa:alerts", "inventory_labels": ["current-fifa2026-alerts-1"], "source_rows": ["L231"], "runtime_identity": "host188/docker/current-fifa2026-alerts-1", "reconciliation_state": "alert_destination_and_receipt_gap", "findings": ["Product alert service has no canonical destination or durable delivery receipt"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "2026fifa", "project_id": "2026FIFAWorldCup", "owner_lane": "2026fifa_product_ops", "telegram_destination": "blocked:2026fifa_destination_unassigned"},
{"canonical_id": "service:agent-bounty:web", "inventory_labels": ["agent_bounty_web"], "source_rows": ["L233"], "runtime_identity": "host110/docker/agent_bounty_web:3005", "reconciliation_state": "durable_route_receipt_gap", "findings": ["Bot-to-group route is source-proven but durable delivery receipt is missing"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "agent-bounty-protocol", "project_id": "agent-bounty-protocol", "owner_lane": "agent_bounty_product_ops", "telegram_destination": "blocked:vibe_ai_agent_receipt_gap"},
{"canonical_id": "service:awooogo:web", "inventory_labels": ["awooo-go-web-1"], "source_rows": ["L234"], "runtime_identity": "host110/docker/awooo-go-web-1:32190", "reconciliation_state": "telegram_disabled_and_manifest_gap", "findings": ["Product Telegram egress is intentionally disabled"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "awooogo", "project_id": "AwoooGo", "owner_lane": "awooogo_product_ops", "telegram_destination": "disabled:awooogo"},
{"canonical_id": "service:awooogo:api", "inventory_labels": ["awooo-go-api-1"], "source_rows": ["L235"], "runtime_identity": "host110/docker/awooo-go-api-1", "reconciliation_state": "telegram_disabled_and_manifest_gap", "findings": ["Product API monitoring and backup contract are not registered here"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "awooogo", "project_id": "AwoooGo", "owner_lane": "awooogo_product_ops", "telegram_destination": "disabled:awooogo"},
{"canonical_id": "service:awooogo:security-agent", "inventory_labels": ["awooo-go-security-agent-1"], "source_rows": ["L236"], "runtime_identity": "host110/docker/awooo-go-security-agent-1", "reconciliation_state": "security_agent_contract_gap", "findings": ["Exact authority, active-response boundary, and verifier are not registered"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "awooogo", "project_id": "AwoooGo", "owner_lane": "awooogo_product_ops", "telegram_destination": "disabled:awooogo", "priority": "P0"},
{"canonical_id": "service:bitan-pharmacy:postgrest", "inventory_labels": ["bitan-pharmacy-postgrest"], "source_rows": ["L237"], "runtime_identity": "host110/docker/bitan-pharmacy-postgrest", "reconciliation_state": "product_route_and_db_verifier_gap", "findings": ["Canonical product destination and bounded DB/API verifier are missing"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "bitan-pharmacy", "project_id": "bitan-pharmacy", "owner_lane": "bitan_product_ops", "telegram_destination": "blocked:bitan_destination_unassigned"},
{"canonical_id": "service:open-design", "inventory_labels": ["open-design"], "source_rows": ["L238"], "runtime_identity": "host110/docker/open-design", "reconciliation_state": "asset_identity_unresolved", "findings": ["Product owner, source repository, monitoring, alert route, and backup contract are absent"], "source_refs": ["ops/config/service-registry.yaml"], "domain_router": "unknown", "executor": null, "verifier": "asset_identity_drift_verifier", "next_action": "Create deterministic drift work item; prohibit fallback AUTO until product ownership and source truth are registered."},
{"canonical_id": "service:stockplatform-v2:web", "inventory_labels": ["stockplatform-v2-web-1"], "source_rows": ["L239"], "runtime_identity": "host110/docker/stockplatform-v2-web-1:3000", "reconciliation_state": "route_ownership_and_manifest_gap", "findings": ["Telegram private route ownership conflicts with TSENYANG Website"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "stockplatform-v2", "project_id": "stockPlatform", "owner_lane": "stockplatform_product_ops", "telegram_destination": "blocked:stockplatform_tsenyang_ownership_drift"},
{"canonical_id": "service:stockplatform-v2:admin", "inventory_labels": ["stockplatform-v2-admin-1"], "source_rows": ["L240"], "runtime_identity": "host110/docker/stockplatform-v2-admin-1:3001", "reconciliation_state": "admin_surface_and_route_gap", "findings": ["Admin auth, smoke, and alert ownership are not linked to a product manifest"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "stockplatform-v2", "project_id": "stockPlatform", "owner_lane": "stockplatform_product_ops", "telegram_destination": "blocked:stockplatform_tsenyang_ownership_drift"},
{"canonical_id": "service:stockplatform-v2:api", "inventory_labels": ["stockplatform-v2-api-1"], "source_rows": ["L241"], "runtime_identity": "host110/docker/stockplatform-v2-api-1:8000", "reconciliation_state": "api_data_and_route_gap", "findings": ["Market-data freshness, API SLO, and product alert ownership are not registered here"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "product_id": "stockplatform-v2", "project_id": "stockPlatform", "owner_lane": "stockplatform_product_ops", "telegram_destination": "blocked:stockplatform_tsenyang_ownership_drift"}
]
},
{
"group_id": "imported-k3s-controls",
"category": "k3s_component",
"common": {
"product_id": "shared-infrastructure",
"project_id": "k3s-control-plane",
"site_id": "k3s",
"owner_lane": "k3s_ops",
"source_truth_state": "inventory_claim_manifest_lookup_required",
"live_truth_state": "not_probed_this_reconciliation",
"domain_router": "kubernetes_workload",
"executor": "kubernetes_controlled_executor",
"verifier": "kubernetes_rollout_verifier",
"monitoring": {"signals": ["rollout", "readiness", "events", "policy_drift"], "coverage": "partial"},
"alerting": {"lifecycle": "k3s_component_incident", "cross_domain_fallback_allowed": false, "required_ai_attribution": true},
"telegram_destination": "awoooi_sre_war_room_for_shared_P0_P1_lifecycle_only",
"backup_restore": {"mode": "readback_first", "target": "k3s_manifest_and_velero_contract", "freshness": "unverified"},
"learning_targets": ["KM", "RAG", "MCP", "PlayBook"],
"priority": "P1",
"dependencies": ["PORT-001", "PORT-005", "PORT-010"],
"next_action": "Resolve exact namespace/kind/name and verify rollout/policy state before Kubernetes controlled apply."
},
"members": [
{"canonical_id": "k8s:kured", "inventory_labels": ["Kured"], "source_rows": ["L253"], "runtime_identity": "k3s/kured", "reconciliation_state": "critical_reboot_boundary_gap", "findings": ["Host reboot is critical break-glass; inventory presence cannot authorize Kured actions"], "source_refs": ["k8s"], "priority": "P0"},
{"canonical_id": "k8s:descheduler", "inventory_labels": ["Descheduler"], "source_rows": ["L254"], "runtime_identity": "k3s/descheduler", "reconciliation_state": "manifest_and_runtime_verifier_gap", "findings": ["Rescheduling blast radius and post-verifier are not linked"], "source_refs": ["k8s"]},
{"canonical_id": "k8s:vpa", "inventory_labels": ["VPA"], "source_rows": ["L255"], "runtime_identity": "k3s/vpa", "reconciliation_state": "recommendation_mode_unverified", "findings": ["Imported row says recommendation mode but live mode and recommendation freshness are unverified"], "source_refs": ["k8s"]},
{"canonical_id": "k8s:hpa", "inventory_labels": ["HPA"], "source_rows": ["L256"], "runtime_identity": "k3s/hpa", "reconciliation_state": "policy_coverage_unverified", "findings": ["Target workload coverage and scaling verifier are not enumerated"], "source_refs": ["k8s"]},
{"canonical_id": "k8s:pdb", "inventory_labels": ["PDB"], "source_rows": ["L257"], "runtime_identity": "k3s/pdb", "reconciliation_state": "policy_coverage_unverified", "findings": ["Protected workload coverage and disruption allowance are not enumerated"], "source_refs": ["k8s"]},
{"canonical_id": "k8s:network-policy", "inventory_labels": ["NetworkPolicy"], "source_rows": ["L258"], "runtime_identity": "k3s/networkpolicy", "reconciliation_state": "policy_coverage_unverified", "findings": ["Presence does not prove every workload/provider path is allowed or isolated"], "source_refs": ["k8s/awoooi-prod/02-network-policy.yaml"]},
{"canonical_id": "k8s:rbac", "inventory_labels": ["RBAC"], "source_rows": ["L259"], "runtime_identity": "k3s/rbac", "reconciliation_state": "policy_coverage_unverified", "findings": ["Presence does not prove least privilege or executor identity"], "source_refs": ["k8s"]}
]
},
{
"group_id": "missing-control-plane-and-runtime-assets",
"category": "inventory_omission",
"common": {
"product_id": "awoooi",
"project_id": "ai-control-plane",
"site_id": "production",
"owner_lane": "AIControlPlane",
"source_truth_state": "source_visible_missing_from_imported_inventory",
"live_truth_state": "not_probed_this_reconciliation",
"reconciliation_state": "missing_from_imported_inventory",
"domain_router": "control_plane_service",
"executor": "single_control_plane_executor",
"verifier": "control_plane_independent_verifier",
"monitoring": {"signals": ["health", "freshness", "same_run_receipt", "coverage"], "coverage": "inventory_gap"},
"alerting": {"lifecycle": "canonical_incident_only", "cross_domain_fallback_allowed": false, "required_ai_attribution": true},
"telegram_destination": "awoooi_sre_war_room_for_shared_P0_P1_lifecycle_only",
"backup_restore": {"mode": "readback_first", "target": "component_specific_contract_required", "freshness": "unverified"},
"learning_targets": ["KM", "RAG", "MCP", "PlayBook"],
"priority": "P0",
"dependencies": ["PORT-001", "PORT-005", "PORT-011"],
"next_action": "Add the canonical asset to the reconciled source registry and require same-run production readback."
},
"members": [
{"canonical_id": "windows-vmware:host_99", "inventory_labels": ["Agent99 Windows/VMware control plane"], "source_rows": [], "runtime_identity": "192.168.0.99/Agent99", "findings": ["Agent99 is required for Windows/VMware and control-plane recovery but absent from the imported inventory", "Agent99 also owns the read-only host110 Alertmanager poll/reduced relay coordination role; Linux remediation still routes only to the host Ansible executor", "The atomic 15-file runtime bundle includes the poller but production deployment/readback is pending"], "source_refs": ["docs/operations/agent99-enterprise-ai-automation-work-items.snapshot.json", "k8s/awoooi-prod/04-configmap.yaml", "agent99-alertmanager-alertchain-poll.ps1", "scripts/reboot-recovery/deploy-agent99-via-windows99-ssh.sh"], "owner_lane": "Agent99", "domain_router": "windows_vmware", "executor": "Agent99", "verifier": "agent99_independent_runtime_verifier"},
{"canonical_id": "service:ollama:host110", "inventory_labels": ["retired Host110 Ollama tombstone"], "source_rows": [], "runtime_identity": "retired-tombstone:host110/ollama/no-endpoint", "source_truth_state": "retired_tombstone_source_ready_runtime_absence_pending", "reconciliation_state": "retired_tombstone_source_ready_runtime_absence_pending", "findings": ["Host110 Ollama was removed and is forbidden as a provider, proxy, primary, secondary or failover target", "A tombstone is retained so stale sensors and routes create drift instead of resurrecting the endpoint", "Production runtime and Prometheus absence receipts are still missing"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "docs/operations/sre-k3s-controlled-automation-work-items.snapshot.json"], "owner_lane": "ai_team", "domain_router": "host_systemd", "executor": "host_ansible_executor", "verifier": "host110_ollama_runtime_and_monitoring_absence_verifier", "monitoring": {"signals": ["runtime_endpoint_absence", "prometheus_target_absence", "route_reference_absence"], "coverage": "source_tombstone_runtime_absence_pending"}, "next_action": "Verify no Host110 Ollama process, proxy, target, rule or provider route exists; preserve the tombstone and create drift on any recurrence."},
{"canonical_id": "host:111", "inventory_labels": ["host111 local Ollama"], "source_rows": [], "runtime_identity": "192.168.0.111/macos-launchd/ollama", "findings": ["Approved third provider hop and Ansible host are absent from the imported inventory", "The runtime manager is a macOS LaunchAgent, not systemd", "The exact LaunchAgent playbook requires local verification plus independent origin probes from host120 and host121", "The canonical host111 Ollama sensor/rule has no fresh production series"], "source_refs": ["infra/ansible/inventory/hosts.yml", "infra/ansible/playbooks/111-ollama-fallback.yml", "ops/monitoring/service-registry.yaml", "ops/monitoring/alerts-unified.yml", "apps/api/src/services/awooop_ansible_post_verifier.py"], "owner_lane": "ai_team", "domain_router": "host_systemd", "executor": "host_ansible_executor", "verifier": "host111_launchagent_local_and_120_121_origin_verifier", "monitoring": {"signals": ["launchagent_state", "local_generation", "host120_origin_generation", "host121_origin_generation", "canonical_alert_series"], "coverage": "source_candidate_runtime_sensor_missing"}, "next_action": "Check/apply the exact LaunchAgent catalog, verify local plus host120/host121 origins, and require a fresh canonical sensor series before provider readiness."},
{"canonical_id": "ai-provider:ollama_gcp_a", "inventory_labels": ["GCP-A Ollama"], "source_rows": [], "runtime_identity": "34.143.170.20:11434", "source_truth_state": "source_reconciled_runtime_pending", "findings": ["Approved first provider hop is absent from the imported inventory", "Observed transport is public HTTP and is candidate-only until secure mesh/TLS promotion", "Only sanitized prompts may cross this boundary; unsanitized input and every tool loop fail closed before network access", "The exact Prometheus provider target is missing in production"], "source_refs": ["ops/monitoring/service-registry.yaml", "k8s/monitoring/prometheus.yml", "docs/operations/sre-k3s-controlled-automation-work-items.snapshot.json", "apps/api/src/services/ai_provider_policy.py"], "owner_lane": "ai_team", "domain_router": "ai_provider", "executor": "deterministic_provider_router", "verifier": "provider_health_generation_and_transport_receipt_verifier", "monitoring": {"signals": ["exact_target_up", "scrape_freshness", "sanitization_receipt", "tool_loop_denial", "transport_boundary"], "coverage": "source_candidate_runtime_target_missing"}, "transport_policy": {"observed": "public_http", "execution_scope": "sanitized_candidate_only", "tool_loop": "fail_closed", "promotion_requires": "secure_mesh_or_tls_plus_runtime_readback"}, "next_action": "Deploy the exact blackbox target, verify fresh series and sanitization/tool-loop denials, then retain candidate-only status until secure transport promotion."},
{"canonical_id": "ai-provider:ollama_gcp_b", "inventory_labels": ["GCP-B Ollama"], "source_rows": [], "runtime_identity": "34.21.145.224:11434", "source_truth_state": "source_reconciled_runtime_pending", "findings": ["Approved second provider hop is absent from the imported inventory", "Observed transport is public HTTP and is candidate-only until secure mesh/TLS promotion", "Only sanitized prompts may cross this boundary; unsanitized input and every tool loop fail closed before network access", "The exact Prometheus provider target is missing in production"], "source_refs": ["ops/monitoring/service-registry.yaml", "k8s/monitoring/prometheus.yml", "docs/operations/sre-k3s-controlled-automation-work-items.snapshot.json", "apps/api/src/services/ai_provider_policy.py"], "owner_lane": "ai_team", "domain_router": "ai_provider", "executor": "deterministic_provider_router", "verifier": "provider_health_generation_and_transport_receipt_verifier", "monitoring": {"signals": ["exact_target_up", "scrape_freshness", "sanitization_receipt", "tool_loop_denial", "transport_boundary"], "coverage": "source_candidate_runtime_target_missing"}, "transport_policy": {"observed": "public_http", "execution_scope": "sanitized_candidate_only", "tool_loop": "fail_closed", "promotion_requires": "secure_mesh_or_tls_plus_runtime_readback"}, "next_action": "Deploy the exact blackbox target, verify fresh series and sanitization/tool-loop denials, then retain candidate-only status until secure transport promotion."},
{"canonical_id": "ai-provider:claude", "inventory_labels": ["Anthropic Claude API"], "source_rows": [], "runtime_identity": "protected-secret-reference/anthropic", "findings": ["Approved fourth provider hop and paid canary are absent from the imported inventory", "User-supplied credential material was noted but no raw value is recorded or readable from this snapshot", "A protected-secret metadata verifier and durable Gate5 authorization bound to the run are pending", "Paid enablement requires paired Claude/Gemini rollback, daily/monthly/per-incident cost caps and production cost readback", "The sanitized five-lane paid canary has not run"], "source_refs": ["docs/operations/sre-k3s-controlled-automation-work-items.snapshot.json", "apps/api/src/services/paid_provider_canary_validation.py"], "owner_lane": "ai_team", "domain_router": "ai_provider", "executor": "deterministic_provider_router", "verifier": "paid_provider_generation_and_cost_receipt_verifier", "credential_metadata": {"user_supplied_credential_noted": true, "raw_value_recorded": false, "raw_value_readback_allowed": false, "protected_secret_reference_status": "pending"}, "authorization": {"gate": "Gate5", "durable_run_binding_required": true, "status": "pending"}, "canary": {"percent": 5, "status": "pending", "paired_rollback_required": true, "cost_caps_verified": false}},
{"canonical_id": "ai-provider:gemini", "inventory_labels": ["Gemini API"], "source_rows": [], "runtime_identity": "protected-secret-reference/gemini", "findings": ["Approved final provider hop and paid canary are absent from the imported inventory", "User-supplied credential material was noted but no raw value is recorded or readable from this snapshot", "A protected-secret metadata verifier and durable Gate5 authorization bound to the run are pending", "Paid enablement requires paired Claude/Gemini rollback, daily/monthly/per-incident cost caps and production cost readback", "The sanitized five-lane paid canary has not run"], "source_refs": ["docs/operations/sre-k3s-controlled-automation-work-items.snapshot.json", "apps/api/src/services/paid_provider_canary_validation.py"], "owner_lane": "ai_team", "domain_router": "ai_provider", "executor": "deterministic_provider_router", "verifier": "paid_provider_generation_and_cost_receipt_verifier", "credential_metadata": {"user_supplied_credential_noted": true, "raw_value_recorded": false, "raw_value_readback_allowed": false, "protected_secret_reference_status": "pending"}, "authorization": {"gate": "Gate5", "durable_run_binding_required": true, "status": "pending"}, "canary": {"percent": 5, "status": "pending", "paired_rollback_required": true, "cost_caps_verified": false}},
{"canonical_id": "control-plane:holmesgpt-investigator", "inventory_labels": ["HolmesGPT Investigator"], "source_rows": [], "runtime_identity": "source_candidate/runtime_not_promoted", "findings": ["Required pipeline investigator is absent from the infrastructure inventory and runtime promotion remains pending"], "source_refs": ["docs/operations/sre-k3s-controlled-automation-work-items.snapshot.json"], "owner_lane": "InvestigationPlane"},
{"canonical_id": "k8s:awoooi-prod:api", "inventory_labels": ["AWOOOI production API"], "source_rows": [], "runtime_identity": "k3s/awoooi-prod/deployment/awoooi-api", "findings": ["Production API is in the monitoring registry but absent from the imported infrastructure inventory"], "source_refs": ["ops/monitoring/service-registry.yaml", "k8s/awoooi-prod/06-deployment-api.yaml"], "owner_lane": "backend_team", "domain_router": "kubernetes_workload", "executor": "kubernetes_controlled_executor", "verifier": "kubernetes_rollout_verifier"},
{"canonical_id": "k8s:awoooi-prod:web", "inventory_labels": ["AWOOOI production Web"], "source_rows": [], "runtime_identity": "k3s/awoooi-prod/deployment/awoooi-web", "findings": ["Production Web is in the monitoring registry but absent from the imported infrastructure inventory"], "source_refs": ["ops/monitoring/service-registry.yaml", "k8s/awoooi-prod"], "owner_lane": "frontend_team", "domain_router": "kubernetes_workload", "executor": "kubernetes_controlled_executor", "verifier": "kubernetes_rollout_verifier"},
{"canonical_id": "k8s:awoooi-prod:worker", "inventory_labels": ["AWOOOI production Worker"], "source_rows": [], "runtime_identity": "k3s/awoooi-prod/deployment/awoooi-worker", "findings": ["Worker is in the monitoring registry but absent from the imported infrastructure inventory"], "source_refs": ["ops/monitoring/service-registry.yaml", "k8s/awoooi-prod"], "owner_lane": "backend_team", "domain_router": "kubernetes_workload", "executor": "kubernetes_controlled_executor", "verifier": "kubernetes_rollout_verifier"},
{"canonical_id": "k8s:awoooi-prod:broker", "inventory_labels": ["AWOOOI production Broker"], "source_rows": [], "runtime_identity": "k3s/awoooi-prod/deployment/awoooi-broker", "findings": ["Broker is part of production deployment/readback but absent from the imported inventory and monitoring service registry"], "source_refs": ["k8s/awoooi-prod", "docs/operations/sre-k3s-controlled-automation-work-items.snapshot.json"], "owner_lane": "backend_team", "domain_router": "kubernetes_workload", "executor": "kubernetes_controlled_executor", "verifier": "kubernetes_rollout_verifier"},
{"canonical_id": "signal:alertmanager-webhook-chain", "inventory_labels": ["AlertChainBroken_Alertmanager"], "source_rows": [], "runtime_identity": "alertmanager->awoooi-api-webhook->incident->telegram plus host99 Agent99 exact-host pull fallback", "source_truth_state": "source_reconciled_runtime_pending", "findings": ["Canonical signal is locked to container:host_110:alertmanager", "Rule, concise attribution card, exact bounded recovery and independent verifier are source-ready", "Host99 Agent99 independently polls only the exact active/critical AlertChain signal from host110 without a first-hop credential or raw-payload persistence", "The reduced event is relayed over HTTPS with stable dedupe while Linux execution remains in the host Ansible lane", "Production deployment, poll freshness/dedupe, current alert resolution and same-run learning acknowledgements remain pending"], "source_refs": ["k8s/monitoring/alert-chain-monitor.yaml", "ops/alertmanager/alertmanager.yml", "agent99-alertmanager-alertchain-poll.ps1", "agent99-sre-alert-relay.ps1", "apps/api/src/api/v1/webhooks.py", "apps/api/src/services/controlled_alert_target_router.py", "infra/ansible/playbooks/110-alertmanager-delivery-recovery.yml", "scripts/reboot-recovery/deploy-agent99-via-windows99-ssh.sh"], "owner_lane": "observability_ops", "verifier": "alert_delivery_chain_plus_agent99_pull_independent_verifier", "monitoring": {"signals": ["alertmanager_delivery_counters", "primary_webhook_health", "agent99_pull_freshness", "dedupe_receipt", "telegram_delivery_receipt"], "coverage": "source_primary_and_independent_pull_ready_runtime_pending"}, "next_action": "Deploy the receiver-scoped scrape/rule/playbook and the atomic Agent99 15-file bundle; run check/apply/verify under one trace/run/work item; then require Alertmanager resolution, Telegram and KM/RAG/MCP/PlayBook acknowledgements before closure."},
{"canonical_id": "control-plane:telegram-routing-registry", "inventory_labels": ["Canonical Telegram routing registry"], "source_rows": [], "runtime_identity": "docs/security/telegram-canonical-routing-registry.snapshot.json", "findings": ["Nine product routes are blocked/disabled/not implemented; scattering alerts is forbidden"], "source_refs": ["docs/security/telegram-canonical-routing-registry.snapshot.json"], "owner_lane": "NotificationGovernance", "verifier": "telegram_route_and_delivery_receipt_verifier"},
{"canonical_id": "control-plane:km-rag", "inventory_labels": ["Knowledge Base and RAG"], "source_rows": [], "runtime_identity": "awoooi-api/knowledge-rag", "findings": ["Required learning target is implemented in source but absent from the imported inventory and runtime closure is unverified"], "source_refs": ["apps/api/src/api/v1/knowledge.py", "apps/api/src/api/v1/rag.py"], "owner_lane": "LearningPlane", "verifier": "km_rag_durable_write_ack_verifier"},
{"canonical_id": "control-plane:mcp-gateway", "inventory_labels": ["Internal MCP Gateway"], "source_rows": [], "runtime_identity": "awoooi-api/mcp-control-plane", "findings": ["MCP catalog has broad product coverage but is absent from the imported inventory; write adapters must remain policy controlled"], "source_refs": ["docs/operations/mcp-control-plane-catalog.snapshot.json"], "owner_lane": "MCPControlPlane", "verifier": "mcp_gateway_audit_and_runtime_verifier"},
{"canonical_id": "control-plane:playbook-registry", "inventory_labels": ["PlayBook registry"], "source_rows": [], "runtime_identity": "awoooi-api/playbook-service", "findings": ["Required candidate/rollback/trust asset is absent from the imported inventory"], "source_refs": ["apps/api/src/services/playbook_service.py"], "owner_lane": "AutomationControlPlane", "verifier": "playbook_trust_and_execution_receipt_verifier"},
{"canonical_id": "control-plane:independent-verifier-registry", "inventory_labels": ["Independent verifier registry"], "source_rows": [], "runtime_identity": "awoooi-api/verifier-registry", "findings": ["All assets require an independent verifier, but the inventory does not enumerate verifier ownership or coverage"], "source_refs": ["docs/operations/sre-k3s-controlled-automation-work-items.snapshot.json"], "owner_lane": "VerifierPlane", "verifier": "verifier_registry_self_check"}
]
}
],
"conflict_register": [
{"id": "PORT-CONFLICT-001", "priority": "P0", "assets": ["service:alertmanager", "signal:alertmanager-webhook-chain"], "conflict": "Alertmanager source identity, receiver-scoped metrics, exact repair path and host99 Agent99 independent pull/reduced relay are reconciled in source; production deployment, poll freshness/dedupe, alert resolution and same-run closure receipts remain pending.", "linked_work_items": ["AIA-SRE-002", "AIA-SRE-003", "AIA-SRE-004", "AIA-SRE-014", "AIA-SRE-015", "AIA-SRE-017"]},
{"id": "PORT-CONFLICT-002", "priority": "P0", "assets": ["service:ollama:host110", "service:ollama:host188", "host:111", "ai-provider:ollama_gcp_a", "ai-provider:ollama_gcp_b", "ai-provider:claude", "ai-provider:gemini"], "conflict": "Host110 is source-tombstoned but runtime absence is unverified; host111 is a LaunchAgent without local plus host120/121 same-run verification or a fresh sensor; GCP-A/GCP-B remain public-HTTP sanitized candidates with exact Prometheus targets missing; Claude/Gemini still lack durable Gate5 authorization, protected-secret metadata readback, verified cost caps, paired rollback and a completed five-lane canary.", "linked_work_items": ["AIA-SRE-002", "AIA-SRE-004", "AIA-SRE-008", "AIA-SRE-013", "AIA-SRE-014"]},
{"id": "PORT-CONFLICT-003", "priority": "P0", "assets": ["service:gitea-exporter:host110", "service:sentry:relay", "service:sentry:snuba", "runner:gitea:vibework-host110"], "conflict": "GitHub exporter source retirement is ready, but the exact gitea-native production target is missing and the legacy exporter remains visible; remaining ghcr/runner claims also conflict with the global GitHub freeze.", "linked_work_items": ["AIA-SRE-002", "AIA-SRE-004", "AIA-SRE-014", "AIA-SRE-018"]},
{"id": "PORT-CONFLICT-004", "priority": "P0", "assets": ["network:keepalived:vip-unresolved"], "conflict": "Inventory VIP 192.168.0.125 conflicts with Ansible VIP 192.168.0.200 and host membership differs.", "linked_work_items": ["AIA-SRE-002", "AIA-SRE-003"]},
{"id": "PORT-CONFLICT-005", "priority": "P0", "assets": ["service:signoz", "service:otel-collector:signoz", "database:signoz:clickhouse"], "conflict": "SignOz UI, collector, and ClickHouse host placement is inconsistent across imported inventory and registries.", "linked_work_items": ["AIA-SRE-002", "AIA-SRE-014"]},
{"id": "PORT-CONFLICT-006", "priority": "P0", "assets": ["database:awoooi:prod", "database:k3s:postgres-recovery"], "conflict": "AWOOOI business database and K3s datastore/recovery identities are conflated; multiple same-host port 5432 claims are not externally disambiguated.", "linked_work_items": ["AIA-SRE-002", "AIA-SRE-010", "AIA-SRE-011"]},
{"id": "PORT-CONFLICT-007", "priority": "P0", "assets": ["control-plane:telegram-routing-registry"], "conflict": "Most product Telegram routes are blocked, disabled, or not implemented; shared-war-room fallback is forbidden for raw product monitoring.", "linked_work_items": ["AIA-SRE-017"]},
{"id": "PORT-CONFLICT-008", "priority": "P0", "assets": ["control-plane:km-rag", "control-plane:mcp-gateway", "control-plane:playbook-registry", "control-plane:independent-verifier-registry"], "conflict": "Imported inventory omits the learning and verifier control planes required to make AI automation complete.", "linked_work_items": ["AIA-SRE-014", "AIA-SRE-015"]},
{"id": "PORT-CONFLICT-009", "priority": "P0", "assets": ["windows-vmware:host_99", "host:111", "k8s:awoooi-prod:api", "k8s:awoooi-prod:web", "k8s:awoooi-prod:worker", "k8s:awoooi-prod:broker"], "conflict": "Core execution and production workload assets are absent from the imported inventory.", "linked_work_items": ["AIA-SRE-006", "AIA-SRE-007", "AIA-SRE-009", "AIA-SRE-013"]},
{"id": "PORT-CONFLICT-010", "priority": "P1", "assets": ["all-imported-assets"], "conflict": "All imported healthy/up labels are uncorrelated point-in-time claims and cannot close runtime work items.", "linked_work_items": ["AIA-SRE-014", "AIA-SRE-016", "AIA-SRE-018"]},
{"id": "PORT-CONFLICT-011", "priority": "P0", "assets": ["service:wg-easy:host112", "package:metasploit:host112", "package:burp-suite:host112", "package:sqlmap:host112", "package:nikto:host112", "package:openvas:host112", "service:fail2ban:host188"], "conflict": "The expanded host112 inventory adds a forbidden ghcr image claim and multiple active-scan tools without authorization/receipt contracts, while Fail2Ban disappeared without a retirement receipt.", "linked_work_items": ["AIA-SRE-002", "AIA-SRE-003", "AIA-SRE-008", "AIA-SRE-014", "AIA-SRE-016"]}
],
"priority_policy": {
"current_p0_unchanged": "AIA-SRE-013",
"P0": "identity, execution safety, active critical alert chain, provider route, backup integrity, monitoring and learning closure",
"P1": "portfolio manifest adoption and 24-hour effectiveness scorecard after P0 control paths are safe",
"ordering_rule": "all P0 reconciliation work precedes P1; declared dependencies are mandatory and an external blocker may not stop unrelated ready P0 work"
},
"reconciliation_work_items": [
{"id": "PORT-001", "order": 1, "priority": "P0", "title": "Canonical portfolio asset normalization and exact identity graph", "dependencies": [], "linked_program_items": ["AIA-SRE-001", "AIA-SRE-002", "AIA-SRE-003"], "terminal_condition": "Every imported and omitted asset has exact owner, runtime identity, source truth, and fail-closed drift receipt."},
{"id": "PORT-002", "order": 2, "priority": "P0", "title": "Alertmanager webhook chain same-run diagnosis, bounded repair, verifier, and closure", "dependencies": ["PORT-001"], "linked_program_items": ["AIA-SRE-004", "AIA-SRE-014", "AIA-SRE-015", "AIA-SRE-017"], "terminal_condition": "Alertmanager receiver through API ingest and Telegram durable receipt closes under one run; host99 Agent99 exact-host read-only polling creates the reduced repair event when the primary webhook is fully broken, without a first-hop credential or raw-payload persistence; repeated unchanged alerts are suppressed without hiding recurrence."},
{"id": "PORT-003", "order": 3, "priority": "P0", "title": "Ollama provider identity and retired transport cleanup", "dependencies": ["PORT-001"], "linked_program_items": ["AIA-SRE-002", "AIA-SRE-004", "AIA-SRE-008", "AIA-SRE-013", "AIA-SRE-014"], "terminal_condition": "Provider readback shows GCP-A, GCP-B, host111, Claude, Gemini in order; Host110 is only a verified retired tombstone; host111 LaunchAgent closes local and host120/121 verification with a fresh sensor; GCP public-HTTP hops remain sanitized candidate-only and tool loops fail closed."},
{"id": "PORT-004", "order": 4, "priority": "P0", "title": "Sunset generic auto-repair and GitHub legacy supply/runtime paths", "dependencies": ["PORT-001"], "linked_program_items": ["AIA-SRE-002", "AIA-SRE-004", "AIA-SRE-014", "AIA-SRE-018"], "terminal_condition": "No generic SSH/docker/delete/cross-provider fallback or GitHub source/image/API path can execute; gitea-native exact metrics are fresh; the legacy GitHub exporter is absent; exact typed executors and internal immutable artifacts are verified."},
{"id": "PORT-005", "order": 5, "priority": "P0", "title": "Register host99, host111, K3s nodes, host112 security tools/schedules and control-plane assets", "dependencies": ["PORT-001", "PORT-004"], "linked_program_items": ["AIA-SRE-006", "AIA-SRE-007", "AIA-SRE-008", "AIA-SRE-009", "AIA-SRE-014", "AIA-SRE-016"], "terminal_condition": "Host and control-plane inventories reconcile with Agent99/Ansible/Kubernetes single-executor routes; host112 services, containers, packages and schedules have no-secret inventory, active-scan break-glass boundaries and independent readbacks."},
{"id": "PORT-006", "order": 6, "priority": "P0", "title": "Canonical per-product Telegram destinations and concise AI-work attribution", "dependencies": ["PORT-001", "PORT-002"], "linked_program_items": ["AIA-SRE-017"], "terminal_condition": "Every emitted alert has exact product route, decision engine/models/actions/executor/verifier/receipt and no raw product alert leaks into the shared war room."},
{"id": "PORT-007", "order": 7, "priority": "P0", "title": "Portfolio backup/restore, escrow, freshness and isolated drill coverage", "dependencies": ["PORT-001", "PORT-004"], "linked_program_items": ["AIA-SRE-010", "AIA-SRE-011"], "terminal_condition": "Every stateful asset has readback freshness, escrow/offsite state and an isolated restore verifier; writes remain critical break-glass."},
{"id": "PORT-009", "order": 8, "priority": "P0", "title": "Resolve VIP, SignOz, database, port and service placement conflicts", "dependencies": ["PORT-001", "PORT-005"], "linked_program_items": ["AIA-SRE-002", "AIA-SRE-003", "AIA-SRE-010", "AIA-SRE-014"], "terminal_condition": "All identity conflicts have read-only live evidence, canonical resolution, source update, and drift verifier without blind mutation."},
{"id": "PORT-010", "order": 9, "priority": "P0", "title": "Generated monitoring/alert coverage and source-to-runtime parity", "dependencies": ["PORT-001", "PORT-004", "PORT-009"], "linked_program_items": ["AIA-SRE-002", "AIA-SRE-013", "AIA-SRE-014", "AIA-SRE-016", "AIA-SRE-017", "AIA-SRE-018"], "terminal_condition": "Every canonical asset has fresh metrics/logs/traces/alerts, exact owner, generated target parity, and a tested lifecycle route; source targets alone never count as runtime coverage; GCP-A/GCP-B, host111 and gitea-native exact series are production-visible."},
{"id": "PORT-011", "order": 10, "priority": "P0", "title": "KM/RAG/MCP/PlayBook and verifier target attachment for every asset", "dependencies": ["PORT-001", "PORT-004", "PORT-007", "PORT-010"], "linked_program_items": ["AIA-SRE-012", "AIA-SRE-014", "AIA-SRE-015"], "terminal_condition": "Every verified run writes durable acknowledgements to all learning targets and never marks learning complete from metadata-only refs."},
{"id": "PORT-008", "order": 11, "priority": "P1", "title": "Adopt product manifests for every imported project/site", "dependencies": ["PORT-001", "PORT-004"], "linked_program_items": ["AIA-SRE-002", "AIA-SRE-015"], "terminal_condition": "Every product/site has Gitea authority, runtime, contracts, operation boundaries and owner without creating repos or using GitHub."},
{"id": "PORT-012", "order": 12, "priority": "P1", "title": "24-hour replay, cost, recurrence and production closure scorecard", "dependencies": ["PORT-002", "PORT-003", "PORT-005", "PORT-006", "PORT-007", "PORT-008", "PORT-009", "PORT-010", "PORT-011"], "linked_program_items": ["AIA-SRE-013", "AIA-SRE-016", "AIA-SRE-018"], "terminal_condition": "Production same-run closure publishes MTTA, MTTR, false positive, recurrence, human intervention, verifier, rollback, freshness, coverage and paid-provider cost without false green; paid evidence includes durable Gate5 authorization, protected-secret metadata only, daily/monthly/per-incident caps and paired Claude/Gemini rollback on any partial result."}
],
"rollups": {
"imported_item_rows": 131,
"canonical_assets": 144,
"inventory_omissions": 18,
"conflicts": 11,
"conflict_assets": 36,
"reconciliation_work_items": 12,
"source_reconciled_assets": 6,
"runtime_closed_assets": 0,
"program_completion_percent": 0,
"asset_coverage_status": "all_imported_rows_mapped_but_identity_and_runtime_coverage_partial",
"runtime_closure_status": "not_started_for_this_reconciliation"
}
}

View File

@@ -2,14 +2,14 @@
"schema_version": "sre_k3s_controlled_automation_work_items_v1",
"governance_version": "global_product_governance_v2",
"program_id": "AIA-SRE-P0-20260715",
"generated_at": "2026-07-15T16:15:01+08:00",
"status": "phase_1_canonical_identity_and_typed_routing_in_progress",
"generated_at": "2026-07-16T15:18:00+08:00",
"status": "phase_3_paid_provider_bounded_canary_in_progress",
"scope_complete": false,
"current_p0": {
"id": "AIA-SRE-004",
"title": "Canonical Asset Normalize 與 Typed Domain Router production closure",
"phase": "phase_1",
"terminal_condition": "同一 source SHA 通過 focused/full testsGitea CD、production registry parity、typed route replay 與 Sentry exact-container verifier readback"
"id": "AIA-SRE-013",
"title": "Claude/Gemini protected-secret 受控上線與五路 SRE 效果驗證",
"phase": "phase_3",
"terminal_condition": "同一 source SHA 通過 focused/full testsGitea CDdurable Gate5 authorization receipt 綁定同一 runprotected-secret metadata、GCP-A/GCP-B/host111/Claude/Gemini 同 fixture、Claude/Gemini finalized receipt、latency/token/cost、paired atomic rollback 與 production status readback 全部可驗證;不代表尚未完成的 HolmesGPT 主 pipeline promotion"
},
"architecture": {
"pipeline": [
@@ -17,7 +17,7 @@
"Canonical Asset Normalize",
"Typed Domain Router",
"HolmesGPT Investigator",
"Ollama RCA / Claude Fallback / Gemini Critic",
"Ollama RCA / Gemini Critic",
"Deterministic Policy",
"Single Controlled Executor",
"Independent Verifier",
@@ -30,7 +30,8 @@
"trace_id",
"run_id",
"work_item_id"
]
],
"provider_failover_is_not_a_pipeline_stage": true
},
"provider_policy": {
"ordered_route": [
@@ -71,12 +72,70 @@
"ollama_role": "primary_rca_and_action_planning",
"claude_role": "paid_architecture_rca_and_debug_fallback",
"gemini_role": "final_fallback_and_optional_critic_only",
"claude_paid_call_allowed": false,
"gemini_paid_call_allowed": false,
"claude_cost_cap_status": "source_configured_shadow_runtime_readback_pending",
"gemini_cost_cap_status": "source_configured_runtime_readback_pending",
"claude_paid_call_allowed": true,
"gemini_paid_call_allowed": true,
"paid_execution_mode": "canary",
"paid_canary_percent": 5,
"paid_canary_authorization_ref": "AIA-SRE-013-user-approved-20260716",
"paid_canary_authorization_status": "user_direction_received_durable_gate5_receipt_pending",
"durable_gate5_authorization": {
"required": true,
"binding": [
"trace_id",
"run_id",
"work_item_id",
"provider_pair",
"cost_caps",
"expiry"
],
"accepted_source": "durable_operation_log_or_authorization_receipt",
"free_form_authorization_ref_is_sufficient": false,
"status": "pending"
},
"paid_canary_scope": "one committed sanitized fixture per provider; no infrastructure remediation",
"claude_cost_cap_status": "source_configured_canary_runtime_readback_pending",
"gemini_cost_cap_status": "source_configured_canary_runtime_readback_pending",
"claude_credential_status": "runtime_secret_reference_verifier_pending_no_secret_value_read",
"gemini_credential_status": "runtime_secret_reference_verifier_pending_no_secret_value_read",
"credential_metadata": {
"claude": {
"user_supplied_credential_noted": true,
"raw_value_recorded_in_repository": false,
"raw_value_readback_allowed": false,
"protected_secret_reference_status": "pending_secure_ingress_and_metadata_verifier"
},
"gemini": {
"user_supplied_credential_noted": true,
"raw_value_recorded_in_repository": false,
"raw_value_readback_allowed": false,
"protected_secret_reference_status": "pending_secure_ingress_and_metadata_verifier"
}
},
"cloud_transport_contract": {
"ollama_gcp_a": {
"observed_transport": "public_http",
"execution_scope": "sanitized_candidate_only",
"tool_loop_terminal": "fail_closed",
"pre_generation_receipt": "same_run_api_tags_probe_plus_redis_exact_readback_source_ready_runtime_pending",
"exact_prometheus_target_runtime_status": "missing"
},
"ollama_gcp_b": {
"observed_transport": "public_http",
"execution_scope": "sanitized_candidate_only",
"tool_loop_terminal": "fail_closed",
"pre_generation_receipt": "same_run_api_tags_probe_plus_redis_exact_readback_source_ready_runtime_pending",
"exact_prometheus_target_runtime_status": "missing"
},
"production_sensitive_payload_allowed": false,
"secure_mesh_or_tls_required_for_promotion": true
},
"paid_pair_contract": {
"enable_and_rollback_as_pair": true,
"partial_five_lane_terminal": "rollback_both_paid_providers_and_fail_canary",
"daily_monthly_and_per_incident_cost_caps_required": true,
"source_status": "same_run_gate5_atomic_single_use_claim_and_aggregate_cap_ready",
"runtime_status": "paid_canary_pending"
},
"production_provider_route_switch_allowed": false,
"required_before_paid_provider_enablement": [
"offline_replay_scorecard",
@@ -153,6 +212,57 @@
"verifier": null,
"terminal": "asset_identity_unresolved",
"fallback": "forbidden"
},
{
"domain": "control_plane_recovery",
"executor": "Agent99",
"verifier": "cold_start_independent_scorecard_verifier",
"fallback": "forbidden",
"exact_target_required": true
}
],
"immediate_execution_queue": [
{
"position": 1,
"work_item_id": "AIA-SRE-017",
"scope": "alert_chain_delivery_truth_and_emergency_card_attribution",
"blocking_reason": "firing AlertChainBroken invalidates downstream automation and verifier trust",
"terminal_condition": "Alertmanager monotonic delivery metrics are scraped; Agent99 on host99 independently polls the exact host110 active/critical AlertChain signal without a first-hop credential or raw-payload persistence and relays only the reduced envelope over HTTPS; the false NodePort-counter alert resolves; emergency cards state AI/Agent/executor/verifier truth"
},
{
"position": 2,
"work_item_id": "AIA-SRE-002",
"scope": "retire_host110_ollama_identity_and_reconcile_portfolio_inventory",
"blocking_reason": "stale provider identity produces false failover alerts and corrupts provider comparison",
"terminal_condition": "Host110 Ollama is a canonical retired tombstone; GCP-A, GCP-B and host111 are direct identities; host111 LaunchAgent is independently verified from host120/121; exact provider sensors are fresh; no host110 Ollama endpoint remains in runtime or monitoring"
},
{
"position": 3,
"work_item_id": "AIA-SRE-004",
"scope": "critical_fail_closed_and_exact_alert_chain_domain_route",
"blocking_reason": "critical or cross-domain candidates must not reach an executor through legacy fallback",
"terminal_condition": "critical override stays no-write and AlertChainBroken resolves only to host110 Alertmanager container lane"
},
{
"position": 4,
"work_item_id": "AIA-SRE-013",
"scope": "five_lane_paid_provider_canary",
"blocking_reason": "paid calls are allowed only after positions 1-3 have independent readback",
"terminal_condition": "all five providers return comparable safe receipts; public-cloud prompts remain sanitized and tool loops fail closed; the Gate5 authorization UUID is the exact execution run/trace, has an atomic one-use claim and canary bucket proof; Claude/Gemini use a run-owned lease, paired rollback and verified aggregate cost caps"
},
{
"position": 5,
"work_item_id": "AIA-SRE-014",
"scope": "independent_verifier_coverage",
"blocking_reason": "no apply may close without a domain-specific verifier",
"terminal_condition": "every mutating catalog and provider lane has an independent terminal readback"
},
{
"position": 6,
"work_item_id": "AIA-SRE-015",
"scope": "incident_and_learning_writeback",
"blocking_reason": "source, CD or Telegram visibility does not prove learning closure",
"terminal_condition": "same-run incident, KM, RAG, MCP and PlayBook acknowledgements are durable"
}
],
"work_items": [
@@ -190,14 +300,34 @@
],
"source_refs": [
"ops/config/service-registry.yaml",
"ops/monitoring/service-registry.yaml",
"ops/monitoring/generated/prometheus-scrape-generated.yaml",
"ops/monitoring/generated/blackbox-targets-generated.yaml",
"docs/operations/portfolio-infrastructure-asset-reconciliation.snapshot.json",
"docs/operations/portfolio-infrastructure-asset-reconciliation-handoff.md",
"apps/api/src/services/portfolio_infrastructure_asset_reconciliation.py",
"apps/api/src/api/v1/agents.py",
"apps/api/tests/test_portfolio_infrastructure_asset_reconciliation_api.py",
"scripts/ops/render-service-registry-configmap.py",
"k8s/awoooi-prod/15-service-registry-configmap.yaml"
],
"executor": "service registry renderer plus Gitea CD",
"verifier": "source hash and exact embedded YAML parity",
"rollback": "reapply previous ConfigMap from prior Gitea SHA",
"exit_condition": "production ConfigMap source hash and API identity lookup match Gitea main",
"next_action": "deploy and read back ConfigMap annotation"
"exit_condition": "production ConfigMap, monitoring inventory and portfolio asset graph match live canonical identities; Host110 Ollama is present only as a retired tombstone; GCP-A/GCP-B exact targets and the host111 LaunchAgent sensor are fresh; inventory snapshots are never treated as runtime health",
"confirmed_truth": [
"Host110 Ollama has been removed and must remain a retired tombstone, never a fallback endpoint",
"Host111 is the third provider hop and runs as a macOS LaunchAgent, not systemd",
"GCP-A and GCP-B source targets are public HTTP sanitized candidates only",
"gitea-native source target is not production runtime evidence while the legacy GitHub exporter remains visible"
],
"runtime_gaps": [
"Host110 Ollama absence has no same-run production tombstone verifier receipt",
"Host111 LaunchAgent lacks completed local plus host120/host121 origin readback and a fresh exact sensor series",
"GCP-A/GCP-B exact Prometheus target series are missing in production",
"gitea-native exact target is missing and legacy GitHub exporter runtime drift is unresolved"
],
"next_action": "deploy the reconciled tombstone/targets, then verify Host110 absence, Host111 LaunchAgent from local and host120/121, GCP-A/GCP-B exact Prometheus series, gitea-native target freshness and legacy exporter absence under one source SHA"
},
{
"id": "AIA-SRE-003",
@@ -237,13 +367,18 @@
"apps/api/src/services/controlled_alert_target_router.py",
"apps/api/src/services/awooop_ansible_audit_service.py",
"apps/api/src/services/awooop_ansible_check_mode_service.py",
"apps/api/src/api/v1/agents.py"
"apps/api/src/api/v1/agents.py",
"ops/monitoring/alerts-unified.yml"
],
"executor": "typed policy router",
"verifier": "domain/catalog/host/canonical ID scope verifier",
"rollback": "revert router source while keeping unknown fail-closed",
"exit_condition": "every candidate is exact-domain scoped and circuit-open never selects another host/domain",
"next_action": "run full Ansible replay regression"
"exit_condition": "every candidate is exact-domain scoped; circuit-open never selects another host/domain; GCP-A/GCP-B unsanitized prompts and every cloud tool loop fail closed before a network call",
"runtime_gaps": [
"source tests do not prove production router enforcement",
"Agent99 exact-host Alertmanager pull and reduced HTTPS relay are source-implemented but have no host99 production receipt or primary-path outage verifier"
],
"next_action": "run critical override, AlertChainBroken exact-host, cloud sanitized-input/tool-loop fail-closed and full Ansible replay regressions; then verify the same policy in production"
},
{
"id": "AIA-SRE-005",
@@ -332,8 +467,12 @@
"executor": "host_ansible_executor",
"verifier": "host_runtime_independent_verifier",
"rollback": "catalog-specific compensating action",
"exit_condition": "every host action has one exact inventory host, check mode, bounded apply and postcondition",
"next_action": "close catalogs still backed by broad convergence playbooks"
"exit_condition": "every host action has one exact inventory host, runtime manager identity, check mode, bounded apply and postcondition; host111 uses its LaunchAgent catalog and is verified locally plus from host120/host121",
"runtime_gaps": [
"host111 LaunchAgent catalog and sensor are source candidates without production apply receipt",
"host120 and host121 independent origin probes have no same-run production verifier receipt"
],
"next_action": "close broad catalogs, then check/apply the exact host111 LaunchAgent playbook and verify local, host120 and host121 readbacks plus the canonical sensor rule"
},
{
"id": "AIA-SRE-009",
@@ -429,7 +568,7 @@
"order": 13,
"priority": "P0",
"phase": "phase_3",
"title": "Ollama RCA、Claude fallback 與 Gemini Critic/cost gate",
"title": "GCP-A/GCP-B/host111 Ollama、Claude、Gemini RCA/critic 與付費 canary/cost gate",
"owner_lane": "ModelRouter",
"risk": "critical",
"status": "source_implemented_runtime_pending",
@@ -440,13 +579,35 @@
"apps/api/src/services/ai_router.py",
"apps/api/src/services/ollama_failover_manager.py",
"apps/api/src/services/ai_providers/claude.py",
"apps/api/src/services/ai_provider_policy.py"
"apps/api/src/services/ai_providers/gemini.py",
"apps/api/src/services/ai_provider_policy.py",
"apps/api/src/services/failover_alerter.py",
"apps/api/src/services/cloud_transport_receipt.py",
"apps/api/src/services/paid_provider_canary_authorization.py",
"apps/api/src/services/paid_provider_canary_gate5_run.py",
"apps/api/src/services/paid_provider_canary_validation.py",
"apps/api/src/services/ai_rate_limiter.py",
"scripts/ops/run-paid-provider-canary.py",
"k8s/awoooi-prod/04-configmap.yaml",
"k8s/awoooi-prod/06-deployment-api.yaml"
],
"executor": "ordered model router",
"verifier": "provider-attributed replay accuracy, latency and cost readback",
"rollback": "Claude/Gemini disabled; remain on ordered Ollama chain",
"exit_condition": "provider order is enforced and paid providers cannot run without explicit cost cap",
"next_action": "deploy the disabled-by-default source route, verify secret metadata without reading values, then run replay, shadow comparison and bounded canary"
"rollback": "Claude/Gemini are disabled together on any partial five-lane result; remain on the ordered Ollama chain",
"exit_condition": "provider order is enforced; GCP public-HTTP hops remain sanitized candidate-only with cloud tool loops fail-closed until secure transport promotion; paid providers remain 5 percent canary only; the durable Gate5 UUID is the same execution run/trace and is atomically single-use; protected-secret metadata, paired rollback, aggregate cost cap, finalized receipts, latency/token/cost and production readback are verified without secret-value exposure",
"confirmed_truth": [
"The user supplied Claude and Gemini credential material, but raw values are not repository data and must not be copied or read back",
"GCP-A/GCP-B are currently public HTTP boundaries and cannot receive unsanitized or tool-loop payloads",
"Claude and Gemini enablement, canary outcome and rollback form one atomic pair"
],
"runtime_gaps": [
"durable Gate5 authorization receipt bound to the canary run is missing",
"Claude/Gemini protected-secret reference metadata verification is pending",
"daily/monthly/per-incident cost-cap and production cost readback are pending",
"GCP-A/GCP-B exact Prometheus targets are absent from production",
"sanitized five-lane paid canary has not run and no paired rollback receipt exists"
],
"next_action": "first close the ordered alert-chain, Host110 tombstone/Host111 verifier and critical fail-closed safety queue; then create the durable Gate5 receipt, verify protected-secret metadata without reading values, deploy cost caps, execute one sanitized five-lane fixture, and roll back both paid providers unless all five lanes are comparable"
},
{
"id": "AIA-SRE-014",
@@ -471,8 +632,12 @@
"executor": "none",
"verifier": "domain-specific external readback",
"rollback": "failed verifier triggers same-domain retry or compensating action",
"exit_condition": "100 percent of mutating catalogs have independent postconditions",
"next_action": "reconcile catalog IDs against postcondition registry"
"exit_condition": "100 percent of mutating catalogs have independent production postconditions; host111 includes local plus host120/host121 origin verification and provider monitoring requires exact fresh series",
"runtime_gaps": [
"source-level verifier definitions are not production verifier receipts",
"Host111 LaunchAgent, GCP provider targets, gitea-native metrics and the host99 Alertmanager pull/relay remain without independent runtime closure"
],
"next_action": "reconcile catalog IDs against postcondition registry, then attach same-run runtime receipts for Host111 origins, GCP exact targets, gitea-native metrics and the host99 Alertmanager pull/relay"
},
{
"id": "AIA-SRE-015",
@@ -528,13 +693,26 @@
"AIA-SRE-015"
],
"source_refs": [
"ops/config/telegram-routing-registry.yaml"
"ops/config/telegram-routing-registry.yaml",
"ops/alertmanager/alertmanager.yml",
"k8s/monitoring/prometheus.yml",
"ops/monitoring/alerts-unified.yml",
"scripts/ops/deploy-alertmanager-runtime-scrape.sh",
"agent99-alertmanager-alertchain-poll.ps1",
"agent99-sre-alert-relay.ps1",
"scripts/reboot-recovery/deploy-agent99-via-windows99-ssh.sh",
"apps/api/src/services/failover_alerter.py",
"apps/api/src/services/telegram_gateway.py"
],
"executor": "canonical Telegram gateway",
"verifier": "delivery receipt plus desktop/mobile visible smoke",
"rollback": "suppress duplicate projection; retain incident ledger",
"exit_condition": "no alert is sprayed across groups/bots and every card links one canonical run",
"next_action": "reconcile project/product/alert class routing against the typed domain"
"exit_condition": "no alert is sprayed across groups/bots; every card states AI/provider/Agent action, executor, verifier and receipt truth; AlertChainBroken uses receiver-scoped delivery counters plus host99 Agent99 exact-host read-only polling independent of the broken webhook; UI status polling never emits failover alerts",
"runtime_gaps": [
"the host99 Agent99 independent pull/reduced relay has no production deployment, freshness, dedupe or controlled-repair receipt",
"the proposed rule/scrape/card path has no production alert-resolution or same-run learning receipt"
],
"next_action": "deploy Alertmanager runtime scrape/canonical rules and the host99 Agent99 exact-host poller through the atomic 15-file bundle; verify receiver-scoped freshness, dedupe and the current firing alert resolution; then reconcile every project/product/alert class against its typed domain and canonical Telegram destination"
},
{
"id": "AIA-SRE-018",
@@ -556,13 +734,19 @@
"AIA-SRE-015"
],
"source_refs": [
".gitea/workflows/cd.yaml"
".gitea/workflows/cd.yaml",
"docs/operations/portfolio-infrastructure-asset-reconciliation.snapshot.json"
],
"executor": "Gitea controlled CD",
"verifier": "deploy marker, image SHA, API readback and desktop/mobile smoke",
"rollback": "previous immutable image revision and ConfigMap source SHA",
"exit_condition": "source/test/CD/runtime/UI evidence all match one deployed SHA",
"next_action": "commit phase 1, normal push after shared writer release, then terminal CD/readback"
"exit_condition": "source/test/CD/runtime/UI evidence all match one deployed SHA; gitea-native exact metrics are fresh and the retired legacy GitHub exporter is absent from production",
"runtime_gaps": [
"no program item is runtime closed for this source SHA",
"gitea-native exact target is missing while legacy GitHub exporter runtime drift remains",
"source/test readiness, CD terminal, production runtime and visible UI evidence are separate incomplete layers"
],
"next_action": "commit phase 1, normal push after shared writer release, then terminal CD plus production API/runtime/UI readback and exact gitea-native/legacy-exporter verification"
}
],
"rollups": {
@@ -598,6 +782,14 @@
"km_rag_mcp_playbook_writeback_ack"
],
"terminal_without_all_stages": "partial_or_degraded_with_safe_next_action",
"source_test_cd_green_is_runtime_closure": false
"source_test_cd_green_is_runtime_closure": false,
"evidence_layers": {
"source_test": "partial_source_evidence_only",
"gitea_cd_deploy_marker": "pending_for_this_program_sha",
"production_runtime": "zero_work_items_closed",
"visible_ui_telegram": "pending_same_run_receipts",
"learning_writeback": "pending_same_run_durable_acknowledgements"
},
"promotion_rule": "a lower evidence layer cannot promote an incomplete higher evidence layer"
}
}

View File

@@ -32,7 +32,7 @@ Git repo
|---|---|---|
| 主機 inventory | `infra/ansible/inventory/hosts.yml` | 記錄 110 / 120 / 121 / 188 / 112 |
| 188 public nginx routes | `infra/ansible/roles/nginx/templates/*` + `playbooks/nginx-sync.yml` | `/etc/nginx/sites-enabled/*` |
| 110 Ollama proxy | `110-ollama-proxy.conf.j2` | `/etc/nginx/sites-enabled/110-ollama-proxy.conf` |
| 111 Ollama local boundary | `playbooks/111-ollama-fallback.yml` | K3s 120/121 allowlist + direct `/api/tags` readbackhost110 不在路徑 |
| 110 cold-start monitor | `roles/cold-start-monitor` | `/home/wooo/scripts`、cron、node-exporter textfile |
| 110 runner guardrails | `roles/runner-guardrails` | `actions.runner.*` systemd drop-ins |
| 110/188 Docker/systemd/storage/backup/runaway-process textfile exporters | `roles/host-textfile-exporters` | `/home/*/node_exporter_textfiles/docker_stats.prom``storage_health.prom``backup_health.prom`、110 `systemd_units.prom`、110 `host_runaway_process.prom` |

View File

@@ -144,8 +144,8 @@ Every request must emit:
Set provider env back to raw endpoints:
```yaml
OLLAMA_URL: "http://192.168.0.110:11435"
OLLAMA_SECONDARY_URL: "http://192.168.0.110:11436"
OLLAMA_URL: "http://34.143.170.20:11434"
OLLAMA_SECONDARY_URL: "http://34.21.145.224:11434"
OLLAMA_FALLBACK_URL: "http://192.168.0.111:11434"
```

View File

@@ -1,225 +1,13 @@
# GCP Ollama Nginx Proxy 部署指南
# Host110 GCP Ollama Proxy已退役
> ADR-110 三層容災 — 啟用 GCP Ollama 的關鍵步驟
>
> 2026-05-05 修正:此 runbook 只保留為過渡 / rollback bridge。正式方案改用
> ADR-125 的 WireGuard private mesh 與 AwoooP Inference Gateway。新部署不得把
> GCP `11434/tcp` 對 `0.0.0.0/0` 長期開放。
`superseded_by=global_product_governance_v2`
---
- ownerAssetIdentity
- expiry2026-07-16
- replacementGCP-A → GCP-B → host111 → Claude API → Gemini API
- verifier`apps/api/tests/test_ollama_prod_manifest_order.py`
## 背景
GCP Ollama (34.143.170.20 / 34.21.145.224) 已部署完成。
最初透過 192.168.0.110 (DevOps 金庫) 架設 nginx 反向代理,讓 K3s Pod 走內網連線 GCP Ollama。
2026-05-06 現況K3s Pod 已可直連 GCP-A/GCP-B `11434/tcp`,且 production
暫時改用 direct endpoint避開 110 上舊 `conf.d/ollama-gcp-proxy.conf`
的 120s `proxy_read_timeout`。正式長期方案仍是 ADR-125 WireGuard mesh。
---
## 部署檔案
| 檔案 | 用途 |
|-----|------|
| `infra/ansible/roles/nginx/templates/110-ollama-proxy.conf.j2` | nginx 配置模板 |
| `infra/ansible/playbooks/nginx-sync.yml` | Ansible Playbook |
---
## 執行部署
```bash
# 1. 進入 Ansible 目錄
cd /Users/ogt/awoooi/infra/ansible
# 2. 部署到 110 (Dry-run 先驗證)
ansible-playbook -i inventory/hosts.yml playbooks/nginx-sync.yml --tags 110 --check
# 3. 正式部署
ansible-playbook -i inventory/hosts.yml playbooks/nginx-sync.yml --tags 110
```
---
## 驗證部署
### 從 110 本機驗證
```bash
# 測試 GCP-A proxy
curl http://127.0.0.1:11435/api/tags
# 測試 GCP-B proxy
curl http://127.0.0.1:11436/api/tags
```
### 從 K3s Node 驗證
```bash
# 進入 K3s node (120 或 121)
ssh wooo@192.168.0.120
# 測試連線 110 proxy
curl http://192.168.0.110:11435/api/tags
curl http://192.168.0.110:11436/api/tags
```
### 從 K8s Pod 驗證
```bash
# 進入 API Pod
kubectl exec -it -n awoooi-prod deployment/awoooi-api -- bash
# 測試連線
apt-get update && apt-get install -y curl
curl http://192.168.0.110:11435/api/tags
```
---
## 啟用 GCP Ollama
代理部署完成後,修改 ConfigMap 啟用 GCP 端點:
```bash
# 編輯 ConfigMap
kubectl edit configmap -n awoooi-prod awoooi-config
```
若使用 nginx bridge修改以下欄位
```yaml
# 修改前
OLLAMA_URL: "http://192.168.0.111:11434"
OLLAMA_SECONDARY_URL: "http://192.168.0.110:11435"
OLLAMA_FALLBACK_URL: "http://192.168.0.110:11436"
# 修改後 (啟用 GCP-A 作為 Primary)
OLLAMA_URL: "http://192.168.0.110:11435" # GCP-A via proxy
OLLAMA_SECONDARY_URL: "http://192.168.0.110:11436" # GCP-B via proxy
OLLAMA_FALLBACK_URL: "http://192.168.0.111:11434" # Local GPU 最後防線
```
重啟 Deployment
```bash
kubectl rollout restart deployment/awoooi-api -n awoooi-prod
```
若需要繞過 110 bridge timeout使用 direct GCP endpoint
```yaml
OLLAMA_URL: "http://34.143.170.20:11434"
OLLAMA_SECONDARY_URL: "http://34.21.145.224:11434"
OLLAMA_FALLBACK_URL: "http://192.168.0.111:11434"
OLLAMA_DIAGNOSE_TIMEOUT_SECONDS: "300"
INCIDENT_LLM_TIMEOUT_SECONDS: "360"
```
---
## 確認模型已載入
GCP Ollama 必須已載入以下模型:
```bash
# GCP-A 檢查
curl http://34.143.170.20:11434/api/tags | jq '.models[].name'
# 必須包含:
# - bge-m3:latest (embedding)
# - qwen2.5:7b-instruct (health check)
# - qwen3:14b (RCA analysis)
# - hermes3:latest (tool calling)
# - deepseek-r1:14b (reasoning)
```
若模型未載入SSH 到 GCP 主機執行:
```bash
ollama pull bge-m3:latest
ollama pull qwen2.5:7b-instruct
ollama pull qwen3:14b
ollama pull hermes3:latest
ollama pull deepseek-r1:14b
```
---
## 部署檢查清單
- [ ] Ansible playbook 執行成功 (110)
- [ ] 110:11435 監聽確認 (`ss -tlnp | grep 11435`)
- [ ] 110:11436 監聽確認 (`ss -tlnp | grep 11436`)
- [ ] K3s node 可連線 110:11435/11436
- [ ] K8s Pod 可連線 110:11435/11436
- [ ] GCP-A/B 模型已載入
- [ ] ConfigMap 已修改
- [ ] Deployment 已重啟
- [ ] API Pod 啟動無錯誤
- [ ] 推理測試成功 (檢查 latency < 10s)
---
## 常見問題
### 1. K3s Pod 連線被拒絕
檢查 NetworkPolicy
```bash
kubectl describe networkpolicy -n awoooi-prod allow-required-egress
```
確認包含:
```yaml
- to:
- ipBlock:
cidr: 192.168.0.110/32
ports:
- protocol: TCP
port: 11435
- protocol: TCP
port: 11436
```
### 2. nginx 無法連線 GCP
檢查 110 外網連線:
```bash
curl -v http://34.143.170.20:11434/api/tags
```
若失敗,只允許短時間確認 GCP 防火牆是否對 110 的固定出口 IP 開放
`11434/tcp`。不得把 `0.0.0.0/0:11434` 當成正式設定。
正式切換請改走 [GCP-OLLAMA-WIREGUARD-MESH.md](GCP-OLLAMA-WIREGUARD-MESH.md)。
### 3. 模型載入但推理失敗
檢查 GCP VM 記憶體/CPU 使用率:
```bash
# GCP Console → Compute Engine → VM 執行個體 → 監控
```
若記憶體不足,升級機型或減少同時載入模型數量。
---
## 相關文件
- ADR-110: GCP 三層容災架構
- ADR-125: GCP Ollama Private Mesh and AwoooP Inference Gateway
- `k8s/awoooi-prod/04-configmap.yaml`
- `k8s/awoooi-prod/02-network-policy.yaml`
- `docs/runbooks/RUNBOOK-OLLAMA-FAILOVER.md`
- `docs/runbooks/GCP-OLLAMA-WIREGUARD-MESH.md`
- `docs/runbooks/AWOOOP-INFERENCE-GATEWAY.md`
---
## 負責人
- 建立: Claude Sonnet 4.6 — 2026-05-04
- 審查: 首席架構師 ogt
本文件保留原路徑只為阻止舊連結失效沒有可執行部署手順。Host110 不再是
Ollama provider、proxy 或 rollback target任何工具都不得重建舊 Nginx 配置或
listener。現行操作請使用 `docs/runbooks/RUNBOOK-OLLAMA-FAILOVER.md`transport
目標請使用 `docs/runbooks/GCP-OLLAMA-WIREGUARD-MESH.md`

View File

@@ -19,8 +19,8 @@ Current production endpoints:
| Variable | Endpoint | Meaning |
|----------|----------|---------|
| `OLLAMA_URL` | `http://192.168.0.110:11435` | GCP-A through 110 nginx |
| `OLLAMA_SECONDARY_URL` | `http://192.168.0.110:11436` | GCP-B through 110 nginx |
| `OLLAMA_URL` | GCP-A direct public HTTP | Temporary degraded stopgap |
| `OLLAMA_SECONDARY_URL` | GCP-B direct public HTTP | Temporary degraded stopgap |
| `OLLAMA_FALLBACK_URL` | `http://192.168.0.111:11434` | Local 111 |
This is a bridge. Do not treat the public proxy as the final architecture.
@@ -29,7 +29,6 @@ This is a bridge. Do not treat the public proxy as the final architecture.
| Host | WireGuard IP | Notes |
|------|--------------|-------|
| 110 | `10.77.114.10` | DevOps host and rollback bridge |
| 120 | `10.77.114.120` | K3s node |
| 121 | `10.77.114.121` | K3s node |
| 111 | `10.77.114.111` | Local Ollama fallback |
@@ -213,8 +212,8 @@ After mesh cutover, allow only mesh endpoints for Ollama:
port: 11434
```
Do not remove the `192.168.0.110:11435/11436` rules until rollback is no longer
needed.
Host110 proxy rules are retired and must never be restored. Keep the exact
direct GCP public egress only until mesh canary and rollback verification pass.
## Shadow Validation
@@ -261,13 +260,13 @@ bash scripts/ops/ollama-topology-check.sh
```bash
kubectl -n awoooi-prod set env deploy/awoooi-api \
OLLAMA_URL=http://192.168.0.110:11435 \
OLLAMA_SECONDARY_URL=http://192.168.0.110:11436 \
OLLAMA_URL=http://34.143.170.20:11434 \
OLLAMA_SECONDARY_URL=http://34.21.145.224:11434 \
OLLAMA_FALLBACK_URL=http://192.168.0.111:11434
kubectl -n awoooi-prod set env deploy/awoooi-worker \
OLLAMA_URL=http://192.168.0.110:11435 \
OLLAMA_SECONDARY_URL=http://192.168.0.110:11436 \
OLLAMA_URL=http://34.143.170.20:11434 \
OLLAMA_SECONDARY_URL=http://34.21.145.224:11434 \
OLLAMA_FALLBACK_URL=http://192.168.0.111:11434
```
@@ -277,4 +276,3 @@ kubectl -n awoooi-prod set env deploy/awoooi-worker \
- Alert lane Gemini usage is zero except documented all-Ollama outages.
- Public GCP `11434/tcp` is closed.
- Operator runbook records peer public keys and rollback owner.

View File

@@ -4,10 +4,14 @@
`192.168.0.188:11434` 已不再是 AWOOOI / AwoooP 的 Ollama provider。正式順序是
1. GCP-A`192.168.0.110:11435`
2. GCP-B`192.168.0.110:11436`
3. Local 111`192.168.0.110:11437`
4. Gemini / Claude雲端備援不是主路徑
1. GCP-Acanonical direct GCP Ollamacloud boundary
2. GCP-Bcanonical direct GCP Ollamacloud boundary
3. Local 111`192.168.0.111:11434`local boundary
4. Claude API付費 canaryfallback
5. Gemini API最後付費備援
Host110 provider/proxy 已由 `global_product_governance_v2` 退役;不得作為
rollback target。GCP direct public HTTP 仍是 mesh 上線前的 degraded stopgap。
188 主機仍承載 PostgreSQL、Redis、SigNoz、OpenClaw 等服務,所以不能把「解除 188 Ollama」誤解成「動 188 主機」。退場只針對 `ollama.service``:11434`
@@ -105,7 +109,7 @@ curl -sS --max-time 3 http://192.168.0.188:11434/api/tags || echo LAN_CLOSED
## 今日發現
- `awoooi-prod` provider registry 已沒有 `ollama_188`
- `awoooi-dev` 原本仍使用 `OLLAMA_URL=http://192.168.0.188:11434`已 live patch 到 GCP-A/GCP-B/111 proxy
- `awoooi-dev` 原本仍使用 `OLLAMA_URL=http://192.168.0.188:11434`source 已改為 GCP-A/GCP-B/111 direct routeproduction readback 另行驗證
- Prometheus 舊 blackbox target 已移除並 reload避免退役後誤報。
- 188 journal 中近期少量推理 POST 來源為 `192.168.0.88`;後續確認 `.88` 是 188 的 default gateway不是下游主機。
- 2026-05-06 14:53 短窗口 Gate 綠燈repo、K8s env、Prometheus target、dev health check 均已避開 188。

View File

@@ -1,263 +1,74 @@
# RUNBOOK-OLLAMA-FAILOVER.md
# Ollama 容災監控 Runbook
# 2026-04-26 P2.3 by Claude Sonnet 4.6 (tool-expert)
# 2026-06-04 Codex 更新ADR-110 現行順序為 GCP-A → GCP-B → 111 local → Gemini111 經 110:11437 Nginx proxy 進入,不直接由 K8s Pod 打 111。
# 對應告警規則: ops/monitoring/ollama_health_rules.yaml
# 對應 Dashboard: ops/monitoring/grafana/dashboards/ollama_failover.json
# Ollama付費 Critic 容災 Runbook
---
狀態:`global_product_governance_v2`2026-07-16 Asia/Taipei。
## 現行拓樸與一鍵診斷
## 唯一 production 順序
現行 AI 推理順序:
1. `ollama_gcp_a` — GCP-A Ollamacloud data boundary。
2. `ollama_gcp_b` — GCP-B Ollamacloud data boundary。
3. `ollama_local` — host111 Ollama唯一 local-only hop。
4. `claude` — Anthropic Claude API付費 canaryfallback。
5. `gemini` — Gemini API最後付費備援。
| 順位 | Provider | 入口 |
|------|----------|------|
| 1 | GCP-A | `http://192.168.0.110:11435` |
| 2 | GCP-B | `http://192.168.0.110:11436` |
| 3 | 111 local fallback | `http://192.168.0.110:11437``192.168.0.111:11434` |
| 4 | Gemini | 只在三層 Ollama 都不可用後作最後備援 |
Host110 不再是 Ollama provider、proxy 或 fallback。任何告警、runtime
manifest、Ansible PlayBook、verifier 或 UI 若將 host110 顯示為 Ollama
identity必須標成 `asset_identity_drift`,不得切換或自動重建舊 transport。
111 是 Mac / LaunchAgent 管理的 local fallback不要用舊的 `systemctl` / `nvidia-smi` 手順判斷它。若 `11437``502`,優先分辨是 110 proxy 本身壞掉,還是 110 到 111 的 LAN/ARP 不通。
GCP-A/B 目前的 direct public HTTP 是暫時 transport不能算完整 production
closure。P0 replacement 是 WireGuard `10.77.114.x` mesh完成 replay、canary、
runtime readback 後才能關閉 public listener。
只讀診斷
## 只讀診斷
```bash
bash scripts/ops/ollama111-fallback-proxy-diagnose.sh
```
常見判讀:
輸出只包含 canonical provider、data boundary、HTTP 健康狀態與 production
route readback 狀態;不輸出 key、prompt、response 或 host110 proxy 資訊。
| 證據 | 判讀 | 行動 |
|------|------|------|
| `ollama_gcp_a=up``ollama_gcp_b=up`,但 `ollama_local=down` | 第三層 fallback 單點故障,核心 AI lane 仍可用 | 不要改 provider order先修 111 主機或 LAN |
| 110 `ip neigh 192.168.0.111 INCOMPLETE` | 110 找不到 111 MAC通常是 111 關機、睡眠、Wi-Fi/LAN 斷線或 IP 漂移 | 恢復 111 電源/網路,再重跑診斷 |
| 110 direct `192.168.0.111:11434` OK`127.0.0.1:11437` 502 | 110 Nginx proxy 配置或程序異常 | 檢查 `110-ollama-proxy.conf` 與 Nginx reload |
| SSH 到 `ollama-111-gpu` OK`127.0.0.1:11434` 失敗 | 111 本機 Ollama 或 allowlist LaunchAgent 異常 | 檢查 `com.momo.ollama111-allow-proxy` 與 Ollama |
判讀:
---
| 證據 | 判定 | 安全下一步 |
|---|---|---|
| GCP-A healthy | 保持第一順位 | 不切換 |
| GCP-A unavailable、GCP-B healthy | 第二順位可用 | 建立 GCP-A transport work item不改固定順序 |
| GCP-A/B unavailable、host111 healthy | local hop 可用 | 保持 local-only privacy boundary |
| host111 unavailable | local-only 任務 fail closed | 由 host111 Ansible executor check/apply/verify |
| Claude/Gemini 被使用 | 前三跳皆有 bounded unavailable receipts | 驗證 sanitized receipt、token/cost cap、rollback |
| 任一 circuit open | 該 candidate 不得滑到別台主機或別 domain | 保留 exact failure receipt等待 cooldown同 domain repair |
## Grafana Dashboard 使用說明
## Alert 與自動化契約
Dashboard 路徑:`Ollama 容災監控`uid: `ollama-failover-p23`
匯入方式Grafana UI → Dashboards → Import → Upload JSON file → 選 `ops/monitoring/grafana/dashboards/ollama_failover.json`
每個 failoverrecovery lifecycle 必須顯示:
### Panel 1 — Ollama 可用性 (Stat)
- canonical source 與 provider identity
- Ollama RCAClaude 或 Gemini Critic 是否實際呼叫;
- deterministic policy decision
- executor若沒有必須明示 `none`
- independent verifier 與 receipt ID
- token、cost、latency及 paid budget state
- rollbackno-write terminal
- KM、RAG、MCP、PlayBook durable acknowledgement。
**看什麼**`up{job=~"ollama_gcp_a|ollama_gcp_b|ollama_local|ollama_111"}` × 100顯示每個 Ollama provider endpoint 的 scrape 存活狀態
只讀 route/status 查詢不得觸發 failover Telegram、切換 provider 或修改 Redis
同一 canonical transition 依 TTL 去重selected primary 與 observed fallback 必須
分開呈現,不能把 fallback 說成 primary 已切換。
| 顏色 | 意義 |
|------|------|
| 綠色 100% | Prometheus 探測正常,主機在線 |
| 黃色 | 部分 endpoint 離線,系統應進入容災 |
| 紅色 0% | Ollama provider pool 全離線,高風險 |
## Privacy 與成本
**注意**:此面板反映 Prometheus scrape 狀態,需要 scrape job 命名對齊 `ollama_gcp_a` / `ollama_gcp_b` / `ollama_local`
設定檔位於 `ops/monitoring/generated/prometheus-scrape-generated.yaml`
- `require_local=true` 只能使用 host111GCP-A/B 不得因自架 Ollama 被誤認為 local。
- Claude/Gemini 只接收已提交 synthetic fixture 或具可信 sanitization receipt 的資料
- provider key 只由 protected runtime secret reference 注入;禁止讀回或記錄明文。
- 付費 lane 必須先有 run-owned TTL lease、request/token/daily/monthly cost cap、
generation accounting receipt 和 paired rollback。
- 單一 canary scenario 只能比較該 fixture 的正確率、延遲與成本,不能宣稱模型全面勝出。
---
## Runtime closure
### Panel 2 — 推理延遲 P50 / P99 (Time Series)
不能用「health 200」、「CD success」或 Telegram 卡片單獨結案。完整 terminal
**看什麼**:推理延遲分位數。
`Detect → Normalize → Correlate → Decide → Check → Controlled Apply → Verify → Retry/Rollback → Learn/Writeback`
| 門檻 | 含義 |
|------|------|
| < 10s (P50) | HEALTHY — 正常使用 111 |
| 1030s (P50) | SLOW — 系統已切至 Gemini |
| > 30s (P99) | DEGRADED — 應觸發 failover |
**⚠️ BACKLOG 警告**`ollama_inference_duration_seconds_bucket` 尚未在 API 暴露(需在 `_check_inference()` 加 Histogram.observe())。
面板顯示 "No Data" 是正常的,等 backlog 補完後啟用。
---
### Panel 3 — AI Provider 路由分布 (Pie Chart)
**看什麼**:過去 5 分鐘各 provider 被選中的請求比例。
| 分布 | 意義 |
|------|------|
| ollama / ollama_gcp_a 佔 >90% | 正常GCP-A 健康 |
| ollama_gcp_b 佔多數 | GCP-A SLOW/DEGRADED/OFFLINE容災到 GCP-B |
| ollama_local 出現 | GCP-A/B 均不可用,容災到 111 local |
| gemini 佔多數 | Ollama provider pool 全部不可用,使用付費備援 |
| 全部 nemotron/claude | 極端情況,所有主力 provider 失敗 |
---
### Panel 4 — Failover / Recovery 觸發次數 (Bar Chart)
**看什麼**:每小時 failover和 recovery的觸發次數。
| 模式 | 意義 |
|------|------|
| 兩條都接近 0 | 正常111 穩定運行中 |
| 橘色上升後綠色跟上 | Auto recovery 正常:切出後又切回 |
| 橘色上升,綠色不動 | **`OllamaRecoveryStuck` alert**,見下方 runbook |
| 橘色持續高頻(>5/h | **`OllamaFailoverFrequent` alert**111 不穩定 |
---
## Alert Runbook
### `OllamaInstanceDown` — Ollama 主機離線
**觸發條件**`up{job=~"ollama_gcp_a|ollama_gcp_b|ollama_local|ollama_111"} == 0` 持續 2 分鐘。
**影響評估**
- 系統應依序切至 GCP-B / 111 local / Gemini查 Panel 3 與 `/api/v1/health` 確認)
- 查 Panel 4 是否有 Failover 計數上升
**排查步驟**
```bash
# 步驟 1跑完整只讀診斷先分辨是 110 proxy、111 LAN、或 111 本機服務
bash scripts/ops/ollama111-fallback-proxy-diagnose.sh
# 步驟 2若 110 到 111 是 No route to host / ip neigh INCOMPLETE
# 代表 111 主機或網路不可達;先恢復 111 電源/網路,不要重啟 API。
# 步驟 3若 SSH 到 111 可行,再查本機 Ollama 與 LaunchAgent
ssh ollama-111-gpu 'curl -sS -m 5 http://127.0.0.1:11434/api/tags'
ssh ollama-111-gpu 'launchctl print gui/501/com.momo.ollama111-allow-proxy | head -80'
```
**恢復確認**
`/api/v1/health``ollama_local.status=up`,且 `scripts/ops/ollama111-fallback-proxy-diagnose.sh` 中 110 `proxy_11437` 回 HTTP 200。
---
### `OllamaFailoverFrequent` — Failover 頻率過高
**觸發條件**`rate(ollama_failover_triggered_total[1h]) > 5` 持續 10 分鐘(每小時超過 5 次切換)。
**影響評估**
- 服務本身仍可用Gemini 在接手)
- 但 Gemini 配額消耗加速,有觸發 `GeminiQuotaApproaching` 的風險
**排查步驟**
```bash
# 步驟 1確認 111 近況(反覆 OFFLINE/HEALTHY 之間跳動?)
bash scripts/ops/ollama111-fallback-proxy-diagnose.sh
# 步驟 2查 API log 找 failover 原因
kubectl logs -n awoooi-prod deploy/api --since=30m | grep "ollama_failover_triggered"
# 步驟 3查推理延遲是否長期在 SLOW 邊界?)
kubectl logs -n awoooi-prod deploy/api --since=30m | grep "ollama_health_checked"
# 步驟 4如果是 111 本機 Ollama 問題,需依 LaunchAgent runbook 執行維護;不可直接用 Linux systemctl 手順。
```
---
### `OllamaRecoveryStuck` — Auto Recovery 停滯
**觸發條件**`ollama_health_status{host="111"} == 1 AND ollama_current_primary_is_ollama == 0` 持續 5 分鐘。
111 已 HEALTHY 但路由仍走 Gemini
**影響評估**
- API 功能正常Gemini 在服務)
- 但 Gemini 配額持續消耗111 GPU 資源浪費
**排查步驟**
```bash
# 步驟 1確認 OllamaAutoRecoveryService 是否在運行
kubectl logs -n awoooi-prod deploy/api --since=10m | grep "ollama_auto_recovery"
# 步驟 2查 recovery service 狀態
kubectl logs -n awoooi-prod deploy/api --since=10m | grep -E "ollama_auto_recovery_started|ollama_auto_recovery_stopped|ollama_auto_recovery_loop_error"
# 步驟 3查 current_primary Redis key
kubectl exec -n awoooi-prod deploy/api -- python -c "
import asyncio
from src.core.redis_client import get_redis
async def check():
r = get_redis()
val = await r.get('ollama:current_primary')
print('current_primary:', val)
asyncio.run(check())
"
# 步驟 4如果 recovery service 掛了,重啟 API pod會重新啟動 lifespan
kubectl rollout restart deployment/api -n awoooi-prod
kubectl rollout status deployment/api -n awoooi-prod
```
---
### `GeminiQuotaApproaching` — Gemini 配額 >80%
**觸發條件**`gemini_daily_call_count / gemini_daily_quota > 0.8` 持續 5 分鐘。
**注意**`gemini_daily_quota` 來自 `settings.GEMINI_DAILY_QUOTA`(預設 1000
`gemini_daily_call_count` 從 Redis key `ollama:gemini_daily_count:{YYYY-MM-DD}` 讀取並刷新 Gauge。
**影響評估**
- 當日 Gemini 配額即將耗盡
- 耗盡後系統會自動切至 188 CPU-only 備援qwen2.5:7b-instruct速度較慢
**行動步驟**
```bash
# 步驟 1確認當日 Gemini 使用量
kubectl exec -n awoooi-prod deploy/api -- python -c "
import asyncio, datetime
from src.core.redis_client import get_redis
async def check():
r = get_redis()
today = datetime.date.today().isoformat()
val = await r.get(f'ollama:gemini_daily_count:{today}')
print(f'gemini_daily_count[{today}]:', val)
asyncio.run(check())
"
# 步驟 2確認 111 是否能快速恢復(讓流量切回 Ollama
bash scripts/ops/ollama111-fallback-proxy-diagnose.sh
# 步驟 3如需增加配額修改 settings
# k8s/awoooi-prod/04-configmap.yaml.patch-* 找 GEMINI_DAILY_QUOTA
# 改完後 kubectl apply + rollout restart
# 步驟 4緊急手動重置計數謹慎使用只在確認誤計時才用
# kubectl exec -n awoooi-prod deploy/api -- redis-cli DEL "ollama:gemini_daily_count:$(date +%Y-%m-%d)"
```
---
## Metric 清單
| Metric | 類型 | 狀態 | 說明 |
|--------|------|------|------|
| `up{job="ollama_gcp_a"}` | Gauge | ✅ 現有 | Prometheus scrape 存活 |
| `up{job="ollama_gcp_b"}` | Gauge | ✅ 現有 | Prometheus scrape 存活 |
| `up{job="ollama_local"}` | Gauge | ✅ 現有 | Prometheus scrape 存活 |
| `ollama_failover_triggered_total` | Counter | ✅ P2.3 補入 | failover 切換次數labels: from_provider, to_provider |
| `ollama_recovery_triggered_total` | Counter | ✅ P2.3 補入 | recovery 切回次數labels: from_provider |
| `ollama_health_status{host}` | Gauge | ✅ P2.3 補入 | 健康狀態 1=healthy, 0=not_healthy |
| `ollama_current_primary_is_ollama` | Gauge | ✅ P2.3 補入 | 1=primary 是 ollama, 0=failover 中 |
| `ai_router_selected_provider_total` | Counter | ✅ P2.3 補入 | AI router 選擇次數labels: provider |
| `gemini_daily_call_count` | Gauge | ✅ P2.3 補入 | 今日 Gemini 呼叫次數 |
| `gemini_daily_quota` | Gauge | ✅ P2.3 補入 | Gemini 每日配額 |
| `ollama_inference_duration_seconds` | Histogram | ⏳ BACKLOG | 推理延遲分布,需在 `_check_inference()` 加 observe |
| `post_execution_verification_total` | Counter | ⏳ BACKLOG | Verifier 執行次數,需 auto_repair_service.py 補入 |
| `post_execution_verification_failed_total` | Counter | ⏳ BACKLOG | Verifier 失敗次數,需 auto_repair_service.py 補入 |
## Backlog 補完指引
### `ollama_inference_duration_seconds`
`apps/api/src/services/ollama_health_monitor.py``_check_inference()` 方法結尾,加:
```python
from src.core.metrics import OLLAMA_INFERENCE_DURATION # 需先在 metrics.py 加 Histogram
OLLAMA_INFERENCE_DURATION.labels(host=host_label).observe(latency_ms / 1000)
```
### `post_execution_verification_*`
`apps/api/src/services/auto_repair_service.py` 的 verifier 路徑,加 Counter inc()。
需先確認 verifier 執行點grep `post_execution``verif` 找入口)。
若 GCP mesh、host111 reachability、paid accounting、independent verifier 或 learning
ack 任一缺失,狀態維持 `partial/degraded/blocked_with_safe_next_action`