fix(alerting): align Alertmanager metric schema
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 2s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 2m31s
CD Pipeline / build-and-deploy (push) Successful in 15m18s
AWOOOI Harbor 110 Local Repair / workflow-shape (push) Successful in 1s
AWOOOI Harbor 110 Local Repair / harbor-110-local-repair (push) Successful in 22s
CD Pipeline / post-deploy-checks (push) Successful in 5m35s
All checks were successful
CD Pipeline / workflow-shape (push) Successful in 2s
CD Pipeline / cancel-stale-cd (push) Has been skipped
CD Pipeline / tests (push) Successful in 2m31s
CD Pipeline / build-and-deploy (push) Successful in 15m18s
AWOOOI Harbor 110 Local Repair / workflow-shape (push) Successful in 1s
AWOOOI Harbor 110 Local Repair / harbor-110-local-repair (push) Successful in 22s
CD Pipeline / post-deploy-checks (push) Successful in 5m35s
This commit is contained in:
ogt
@@ -196,7 +196,7 @@
|
||||
"members": [
|
||||
{"canonical_id": "service:prometheus", "inventory_labels": ["Prometheus"], "source_rows": ["L071"], "runtime_identity": "host110/docker/prometheus", "source_truth_state": "source_reconciled_runtime_pending", "reconciliation_state": "source_identity_reconciled_runtime_pending", "findings": ["Monitoring registry now declares the exact host110 Docker identity", "Production container, generated target parity and scrape freshness remain unverified"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "k8s/monitoring/prometheus.yml", "scripts/ops/deploy-alertmanager-runtime-scrape.sh"], "priority": "P0", "verifier": "prometheus_runtime_target_and_rule_verifier"},
|
||||
{"canonical_id": "service:grafana", "inventory_labels": ["Grafana"], "source_rows": ["L072"], "runtime_identity": "host110/docker/grafana", "reconciliation_state": "source_partial_runtime_unverified", "findings": ["Source policy exists but live dashboard/runtime parity is not proven"], "source_refs": ["ops/config/service-registry.yaml", "docs/security/monitoring-alerting-observability-inventory.snapshot.json"]},
|
||||
{"canonical_id": "service:alertmanager", "inventory_labels": ["Alertmanager"], "source_rows": ["L073"], "runtime_identity": "host110/docker/alertmanager", "source_truth_state": "source_reconciled_runtime_pending", "reconciliation_state": "source_control_path_reconciled_runtime_pending", "findings": ["Monitoring registry and typed router now declare the exact host110 Docker identity", "Alert rule now reads Alertmanager monotonic delivery counters instead of process-local API counters", "Exact bounded Ansible recovery and independent verifier are source-ready; production same-run receipt is pending"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "k8s/monitoring/alert-chain-monitor.yaml", "ops/alertmanager/alertmanager.yml", "infra/ansible/playbooks/110-alertmanager-delivery-recovery.yml", "scripts/ops/deploy-alertmanager-runtime-scrape.sh"], "priority": "P0", "verifier": "alertmanager_delivery_chain_independent_verifier", "next_action": "Deploy the canonical scrape/rule and exact playbook, then correlate receiver, API ingest, Telegram receipt, retry/rollback and KM writeback under one run."},
|
||||
{"canonical_id": "service:alertmanager", "inventory_labels": ["Alertmanager"], "source_rows": ["L073"], "runtime_identity": "host110/docker/alertmanager", "source_truth_state": "source_reconciled_runtime_pending", "reconciliation_state": "source_control_path_reconciled_runtime_pending", "findings": ["Monitoring registry and typed router now declare the exact host110 Docker identity", "Production readback proves this Alertmanager exports notification counters with integration labels but no native receiver label", "The source contract has exactly one webhook receiver and Prometheus stamps receiver_contract=awoooi-webhook only on the two webhook counter families", "The corrected bounded dry-run passed promtool with no persistent write; apply and production same-run receipt remain pending"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "k8s/monitoring/alert-chain-monitor.yaml", "ops/alertmanager/alertmanager.yml", "infra/ansible/playbooks/110-alertmanager-delivery-recovery.yml", "scripts/ops/deploy-alertmanager-runtime-scrape.sh"], "priority": "P0", "verifier": "alertmanager_delivery_chain_independent_verifier", "next_action": "Deploy the production-schema-compatible scrape/rule and exact playbook, then correlate receiver contract, API ingest, Telegram receipt, retry/rollback and KM writeback under one run."},
|
||||
{"canonical_id": "service:signoz", "inventory_labels": ["SignOz"], "source_rows": ["L074"], "runtime_identity": "unresolved:host110-ui-host188-collector", "reconciliation_state": "split_plane_identity_conflict", "findings": ["Inventory collapses UI, collector, and storage into one host110 service while source registries split them across hosts"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "scripts/backup/backup-signoz.sh"]},
|
||||
{"canonical_id": "service:otel-collector:signoz", "inventory_labels": ["OTel Collector"], "source_rows": ["L075"], "runtime_identity": "unresolved:host110-or-host188/otel-collector", "reconciliation_state": "host_identity_conflict", "findings": ["Inventory says host110 while monitoring registry says host188 port 24317"], "source_refs": ["ops/monitoring/service-registry.yaml", "docs/security/monitoring-alerting-observability-inventory.snapshot.json"], "domain_router": "unknown", "executor": null, "verifier": "asset_identity_drift_verifier"},
|
||||
{"canonical_id": "service:blackbox-exporter", "inventory_labels": ["Blackbox Exporter"], "source_rows": ["L076"], "runtime_identity": "host110/docker/blackbox-exporter", "reconciliation_state": "matched_runtime_unverified", "findings": ["Generated target freshness and live probe results are not current truth"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/generated/blackbox-targets-generated.yaml"]},
|
||||
@@ -545,7 +545,7 @@
|
||||
},
|
||||
"members": [
|
||||
{"canonical_id": "windows-vmware:host_99", "inventory_labels": ["Agent99 Windows/VMware control plane"], "source_rows": [], "runtime_identity": "192.168.0.99/Agent99", "findings": ["Agent99 is required for Windows/VMware and control-plane recovery but absent from the imported inventory", "Agent99 also owns the read-only host110 Alertmanager poll/reduced relay coordination role; Linux remediation still routes only to the host Ansible executor", "The atomic 15-file runtime bundle includes the poller but production deployment/readback is pending"], "source_refs": ["docs/operations/agent99-enterprise-ai-automation-work-items.snapshot.json", "k8s/awoooi-prod/04-configmap.yaml", "agent99-alertmanager-alertchain-poll.ps1", "scripts/reboot-recovery/deploy-agent99-via-windows99-ssh.sh"], "owner_lane": "Agent99", "domain_router": "windows_vmware", "executor": "Agent99", "verifier": "agent99_independent_runtime_verifier"},
|
||||
{"canonical_id": "service:ollama:host110", "inventory_labels": ["retired Host110 Ollama tombstone"], "source_rows": [], "runtime_identity": "retired-tombstone:host110/ollama/no-endpoint", "source_truth_state": "retired_tombstone_source_ready_runtime_absence_pending", "reconciliation_state": "retired_tombstone_source_ready_runtime_absence_pending", "findings": ["Host110 Ollama was removed and is forbidden as a provider, proxy, primary, secondary or failover target", "A tombstone is retained so stale sensors and routes create drift instead of resurrecting the endpoint", "Production runtime and Prometheus absence receipts are still missing"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "docs/operations/sre-k3s-controlled-automation-work-items.snapshot.json"], "owner_lane": "ai_team", "domain_router": "host_systemd", "executor": "host_ansible_executor", "verifier": "host110_ollama_runtime_and_monitoring_absence_verifier", "monitoring": {"signals": ["runtime_endpoint_absence", "prometheus_target_absence", "route_reference_absence"], "coverage": "source_tombstone_runtime_absence_pending"}, "next_action": "Verify no Host110 Ollama process, proxy, target, rule or provider route exists; preserve the tombstone and create drift on any recurrence."},
|
||||
{"canonical_id": "service:ollama:host110", "inventory_labels": ["retired Host110 Ollama tombstone"], "source_rows": [], "runtime_identity": "retired-tombstone:host110/ollama/no-endpoint", "source_truth_state": "retired_tombstone_runtime_absence_verified", "reconciliation_state": "retired_tombstone_runtime_absence_verified", "findings": ["Host110 Ollama is forbidden as a provider, proxy, primary, secondary or failover target", "A tombstone is retained so stale sensors and routes create drift instead of resurrecting the endpoint", "AIA-SRE-002 run-aia-sre-002-20260716-1752 removed the exact stale Nginx 11435 proxy with a root-only rollback backup", "Independent post-verifier proved listeners 11435/11434=0, Ollama containers and systemd units=0, stale config paths=0, Nginx active/config valid and Prometheus host110 Ollama targets=0"], "source_refs": ["ops/config/service-registry.yaml", "ops/monitoring/service-registry.yaml", "docs/operations/sre-k3s-controlled-automation-work-items.snapshot.json", "scripts/ops/retire-host110-ollama-proxy.sh"], "owner_lane": "ai_team", "domain_router": "host_systemd", "executor": "host_ansible_executor", "bounded_executor_receipt": "run-aia-sre-002-20260716-1752", "runtime_closure": {"status": "verified_absent", "trace_id": "trace-aia-sre-002-20260716-1752", "run_id": "run-aia-sre-002-20260716-1752", "work_item_id": "AIA-SRE-002", "post_verifier": "host110_ollama_runtime_and_monitoring_absence_verifier"}, "verifier": "host110_ollama_runtime_and_monitoring_absence_verifier", "monitoring": {"signals": ["runtime_endpoint_absence", "prometheus_target_absence", "route_reference_absence"], "coverage": "runtime_absence_verified_20260716"}, "next_action": "Preserve the tombstone and create a drift work item on any Host110 Ollama process, proxy, listener, target, rule or provider-route recurrence."},
|
||||
{"canonical_id": "host:111", "inventory_labels": ["host111 local Ollama"], "source_rows": [], "runtime_identity": "192.168.0.111/macos-launchd/ollama", "findings": ["Approved third provider hop and Ansible host are absent from the imported inventory", "The runtime manager is a macOS LaunchAgent, not systemd", "The exact LaunchAgent playbook requires local verification plus independent origin probes from host120 and host121", "The canonical host111 Ollama sensor/rule has no fresh production series"], "source_refs": ["infra/ansible/inventory/hosts.yml", "infra/ansible/playbooks/111-ollama-fallback.yml", "ops/monitoring/service-registry.yaml", "ops/monitoring/alerts-unified.yml", "apps/api/src/services/awooop_ansible_post_verifier.py"], "owner_lane": "ai_team", "domain_router": "host_systemd", "executor": "host_ansible_executor", "verifier": "host111_launchagent_local_and_120_121_origin_verifier", "monitoring": {"signals": ["launchagent_state", "local_generation", "host120_origin_generation", "host121_origin_generation", "canonical_alert_series"], "coverage": "source_candidate_runtime_sensor_missing"}, "next_action": "Check/apply the exact LaunchAgent catalog, verify local plus host120/host121 origins, and require a fresh canonical sensor series before provider readiness."},
|
||||
{"canonical_id": "ai-provider:ollama_gcp_a", "inventory_labels": ["GCP-A Ollama"], "source_rows": [], "runtime_identity": "34.143.170.20:11434", "source_truth_state": "source_reconciled_runtime_pending", "findings": ["Approved first provider hop is absent from the imported inventory", "Observed transport is public HTTP and is candidate-only until secure mesh/TLS promotion", "Only sanitized prompts may cross this boundary; unsanitized input and every tool loop fail closed before network access", "The exact Prometheus provider target is missing in production"], "source_refs": ["ops/monitoring/service-registry.yaml", "k8s/monitoring/prometheus.yml", "docs/operations/sre-k3s-controlled-automation-work-items.snapshot.json", "apps/api/src/services/ai_provider_policy.py"], "owner_lane": "ai_team", "domain_router": "ai_provider", "executor": "deterministic_provider_router", "verifier": "provider_health_generation_and_transport_receipt_verifier", "monitoring": {"signals": ["exact_target_up", "scrape_freshness", "sanitization_receipt", "tool_loop_denial", "transport_boundary"], "coverage": "source_candidate_runtime_target_missing"}, "transport_policy": {"observed": "public_http", "execution_scope": "sanitized_candidate_only", "tool_loop": "fail_closed", "promotion_requires": "secure_mesh_or_tls_plus_runtime_readback"}, "next_action": "Deploy the exact blackbox target, verify fresh series and sanitization/tool-loop denials, then retain candidate-only status until secure transport promotion."},
|
||||
{"canonical_id": "ai-provider:ollama_gcp_b", "inventory_labels": ["GCP-B Ollama"], "source_rows": [], "runtime_identity": "34.21.145.224:11434", "source_truth_state": "source_reconciled_runtime_pending", "findings": ["Approved second provider hop is absent from the imported inventory", "Observed transport is public HTTP and is candidate-only until secure mesh/TLS promotion", "Only sanitized prompts may cross this boundary; unsanitized input and every tool loop fail closed before network access", "The exact Prometheus provider target is missing in production"], "source_refs": ["ops/monitoring/service-registry.yaml", "k8s/monitoring/prometheus.yml", "docs/operations/sre-k3s-controlled-automation-work-items.snapshot.json", "apps/api/src/services/ai_provider_policy.py"], "owner_lane": "ai_team", "domain_router": "ai_provider", "executor": "deterministic_provider_router", "verifier": "provider_health_generation_and_transport_receipt_verifier", "monitoring": {"signals": ["exact_target_up", "scrape_freshness", "sanitization_receipt", "tool_loop_denial", "transport_boundary"], "coverage": "source_candidate_runtime_target_missing"}, "transport_policy": {"observed": "public_http", "execution_scope": "sanitized_candidate_only", "tool_loop": "fail_closed", "promotion_requires": "secure_mesh_or_tls_plus_runtime_readback"}, "next_action": "Deploy the exact blackbox target, verify fresh series and sanitization/tool-loop denials, then retain candidate-only status until secure transport promotion."},
|
||||
@@ -566,8 +566,8 @@
|
||||
}
|
||||
],
|
||||
"conflict_register": [
|
||||
{"id": "PORT-CONFLICT-001", "priority": "P0", "assets": ["service:alertmanager", "signal:alertmanager-webhook-chain"], "conflict": "Alertmanager source identity, receiver-scoped metrics, exact repair path and host99 Agent99 independent pull/reduced relay are reconciled in source; production deployment, poll freshness/dedupe, alert resolution and same-run closure receipts remain pending.", "linked_work_items": ["AIA-SRE-002", "AIA-SRE-003", "AIA-SRE-004", "AIA-SRE-014", "AIA-SRE-015", "AIA-SRE-017"]},
|
||||
{"id": "PORT-CONFLICT-002", "priority": "P0", "assets": ["service:ollama:host110", "service:ollama:host188", "host:111", "ai-provider:ollama_gcp_a", "ai-provider:ollama_gcp_b", "ai-provider:claude", "ai-provider:gemini"], "conflict": "Host110 is source-tombstoned but runtime absence is unverified; host111 is a LaunchAgent without local plus host120/121 same-run verification or a fresh sensor; GCP-A/GCP-B remain public-HTTP sanitized candidates with exact Prometheus targets missing; Claude/Gemini still lack durable Gate5 authorization, protected-secret metadata readback, verified cost caps, paired rollback and a completed five-lane canary.", "linked_work_items": ["AIA-SRE-002", "AIA-SRE-004", "AIA-SRE-008", "AIA-SRE-013", "AIA-SRE-014"]},
|
||||
{"id": "PORT-CONFLICT-001", "priority": "P0", "assets": ["service:alertmanager", "signal:alertmanager-webhook-chain"], "conflict": "Alertmanager source identity, production-compatible integration counters plus sole-webhook receiver contract, exact repair path and host99 Agent99 independent pull/reduced relay are reconciled in source; production apply, poll freshness/dedupe, alert resolution and same-run closure receipts remain pending.", "linked_work_items": ["AIA-SRE-002", "AIA-SRE-003", "AIA-SRE-004", "AIA-SRE-014", "AIA-SRE-015", "AIA-SRE-017"]},
|
||||
{"id": "PORT-CONFLICT-002", "priority": "P0", "assets": ["service:ollama:host110", "service:ollama:host188", "host:111", "ai-provider:ollama_gcp_a", "ai-provider:ollama_gcp_b", "ai-provider:claude", "ai-provider:gemini"], "conflict": "Host110 tombstone runtime absence is verified under AIA-SRE-002 and must remain absent; host111 is a LaunchAgent without local plus host120/121 same-run verification or a fresh sensor; GCP-A/GCP-B remain public-HTTP sanitized candidates with exact Prometheus targets missing; Claude/Gemini still lack a completed five-lane canary and durable cost receipt.", "linked_work_items": ["AIA-SRE-002", "AIA-SRE-004", "AIA-SRE-008", "AIA-SRE-013", "AIA-SRE-014"]},
|
||||
{"id": "PORT-CONFLICT-003", "priority": "P0", "assets": ["service:gitea-exporter:host110", "service:sentry:relay", "service:sentry:snuba", "runner:gitea:vibework-host110"], "conflict": "GitHub exporter source retirement is ready, but the exact gitea-native production target is missing and the legacy exporter remains visible; remaining ghcr/runner claims also conflict with the global GitHub freeze.", "linked_work_items": ["AIA-SRE-002", "AIA-SRE-004", "AIA-SRE-014", "AIA-SRE-018"]},
|
||||
{"id": "PORT-CONFLICT-004", "priority": "P0", "assets": ["network:keepalived:vip-unresolved"], "conflict": "Inventory VIP 192.168.0.125 conflicts with Ansible VIP 192.168.0.200 and host membership differs.", "linked_work_items": ["AIA-SRE-002", "AIA-SRE-003"]},
|
||||
{"id": "PORT-CONFLICT-005", "priority": "P0", "assets": ["service:signoz", "service:otel-collector:signoz", "database:signoz:clickhouse"], "conflict": "SignOz UI, collector, and ClickHouse host placement is inconsistent across imported inventory and registries.", "linked_work_items": ["AIA-SRE-002", "AIA-SRE-014"]},
|
||||
@@ -606,9 +606,9 @@
|
||||
"conflict_assets": 36,
|
||||
"reconciliation_work_items": 12,
|
||||
"source_reconciled_assets": 6,
|
||||
"runtime_closed_assets": 0,
|
||||
"program_completion_percent": 0,
|
||||
"runtime_closed_assets": 1,
|
||||
"program_completion_percent": 1,
|
||||
"asset_coverage_status": "all_imported_rows_mapped_but_identity_and_runtime_coverage_partial",
|
||||
"runtime_closure_status": "not_started_for_this_reconciliation"
|
||||
"runtime_closure_status": "started_partial_1_of_144"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -308,6 +308,7 @@
|
||||
"apps/api/src/services/portfolio_infrastructure_asset_reconciliation.py",
|
||||
"apps/api/src/api/v1/agents.py",
|
||||
"apps/api/tests/test_portfolio_infrastructure_asset_reconciliation_api.py",
|
||||
"scripts/ops/retire-host110-ollama-proxy.sh",
|
||||
"scripts/ops/render-service-registry-configmap.py",
|
||||
"k8s/awoooi-prod/15-service-registry-configmap.yaml"
|
||||
],
|
||||
@@ -317,17 +318,17 @@
|
||||
"exit_condition": "production ConfigMap, monitoring inventory and portfolio asset graph match live canonical identities; Host110 Ollama is present only as a retired tombstone; GCP-A/GCP-B exact targets and the host111 LaunchAgent sensor are fresh; inventory snapshots are never treated as runtime health",
|
||||
"confirmed_truth": [
|
||||
"Host110 Ollama has been removed and must remain a retired tombstone, never a fallback endpoint",
|
||||
"AIA-SRE-002 run-aia-sre-002-20260716-1752 removed the exact stale Nginx 11435 proxy with rollback backup and independently verified listeners 11435/11434=0, Ollama containers=0, stale config paths=0, nginx active/test pass and Prometheus host110 Ollama targets=0",
|
||||
"Host111 is the third provider hop and runs as a macOS LaunchAgent, not systemd",
|
||||
"GCP-A and GCP-B source targets are public HTTP sanitized candidates only",
|
||||
"gitea-native source target is not production runtime evidence while the legacy GitHub exporter remains visible"
|
||||
],
|
||||
"runtime_gaps": [
|
||||
"Host110 Ollama absence has no same-run production tombstone verifier receipt",
|
||||
"Host111 LaunchAgent lacks completed local plus host120/host121 origin readback and a fresh exact sensor series",
|
||||
"GCP-A/GCP-B exact Prometheus target series are missing in production",
|
||||
"gitea-native exact target is missing and legacy GitHub exporter runtime drift is unresolved"
|
||||
],
|
||||
"next_action": "deploy the reconciled tombstone/targets, then verify Host110 absence, Host111 LaunchAgent from local and host120/121, GCP-A/GCP-B exact Prometheus series, gitea-native target freshness and legacy exporter absence under one source SHA"
|
||||
"next_action": "deploy the reconciled tombstone/targets, preserve the verified Host110 absence receipt, then verify Host111 LaunchAgent from local and host120/121, GCP-A/GCP-B exact Prometheus series, gitea-native target freshness and legacy exporter absence under one source SHA"
|
||||
},
|
||||
{
|
||||
"id": "AIA-SRE-003",
|
||||
@@ -707,12 +708,12 @@
|
||||
"executor": "canonical Telegram gateway",
|
||||
"verifier": "delivery receipt plus desktop/mobile visible smoke",
|
||||
"rollback": "suppress duplicate projection; retain incident ledger",
|
||||
"exit_condition": "no alert is sprayed across groups/bots; every card states AI/provider/Agent action, executor, verifier and receipt truth; AlertChainBroken uses receiver-scoped delivery counters plus host99 Agent99 exact-host read-only polling independent of the broken webhook; UI status polling never emits failover alerts",
|
||||
"exit_condition": "no alert is sprayed across groups/bots; every card states AI/provider/Agent action, executor, verifier and receipt truth; AlertChainBroken uses Alertmanager integration-scoped delivery counters stamped with the independently tested sole-webhook receiver contract plus host99 Agent99 exact-host read-only polling independent of the broken webhook; UI status polling never emits failover alerts",
|
||||
"runtime_gaps": [
|
||||
"the host99 Agent99 independent pull/reduced relay has no production deployment, freshness, dedupe or controlled-repair receipt",
|
||||
"the proposed rule/scrape/card path has no production alert-resolution or same-run learning receipt"
|
||||
"production Alertmanager exposes integration-only notification counters; the bounded check now passes after adding a scrape-time receiver_contract label, but apply, alert resolution and same-run learning receipts remain pending"
|
||||
],
|
||||
"next_action": "deploy Alertmanager runtime scrape/canonical rules and the host99 Agent99 exact-host poller through the atomic 15-file bundle; verify receiver-scoped freshness, dedupe and the current firing alert resolution; then reconcile every project/product/alert class against its typed domain and canonical Telegram destination"
|
||||
"next_action": "deploy the production-schema-compatible Alertmanager runtime scrape/canonical rules and the host99 Agent99 exact-host poller through the atomic 15-file bundle; verify receiver-contract freshness, dedupe and the current firing alert resolution; then reconcile every project/product/alert class against its typed domain and canonical Telegram destination"
|
||||
},
|
||||
{
|
||||
"id": "AIA-SRE-018",
|
||||
|
||||
Reference in New Issue
Block a user