feat(ai): add bounded model-aware decision fallback

docs/guides/ai_automation_mainline_work_items.md

@@ -1,6 +1,6 @@
 # AI Automation Mainline Work Items
 
-> Updated: 2026-07-17 10:28 Asia/Taipei
+> Updated: 2026-07-22 Asia/Taipei
 > Governance: `global_product_governance_v2` + ADR-038
 > Current P0: `GROWTH-P0-001 comparison coverage truth + autonomous refresh`
 
@@ -17,16 +17,16 @@
 | Order | ID | Status | Work item | Exit evidence / next machine action |
 |---:|---|---|---|---|
 | 1 | `SEC-P0-001` | Completed | Deny-by-default route access control | `governance/evidence/SEC-P0-001-20260711T122758Z.json` plus the current runtime receipt prove anonymous matrix 8/8 denied, public `/metrics` 404, exact internal target up, EwoooC product markers present, Prometheus identity preserved and `momo-db` unchanged. |
-| 2 | `GROWTH-P0-001` | In progress (`runtime_partial`) | Comparison coverage truth + autonomous refresh | V10.813 is live and retains the growth runtime introduced at exact Gitea object `2647632660673e9f1533e61922c96e4fb4adcb40`. Controlled run `34999f5314054f09909663b46fc42ee2` scanned 20, verified/wrote/read back one exact offer and raised the fixed cohort to `25 ready + 1 candidate validation + 24 unmatched`, `50%` count and `NT$211,667 / NT$354,062 = 59.782%` revenue. Follow-up run `e1508ec3a49241d1965bc99d428b0325` found 19 candidates but safely wrote zero because none passed strict identity/variant/unit verification; its independent terminal is `degraded_no_safe_candidate`. Yahoo remains durably `active + enabled + write_enabled`, and the durable readback proves `formal_source_activated=true` even when the latest run is verified no-write. Formal runtime remains `2/15`. Next: retry the unresolved revenue-weighted batch only with fresh source evidence, then add the next approved structured marketplace adapter without relaxing promotion gates. |
+| 2 | `GROWTH-P0-001` | In progress (`runtime_partial`) | Comparison coverage truth + autonomous refresh | V10.815 is live and retains the growth runtime introduced at exact Gitea object `2647632660673e9f1533e61922c96e4fb4adcb40`. Controlled run `34999f5314054f09909663b46fc42ee2` scanned 20, verified/wrote/read back one exact offer and raised the fixed cohort to `25 ready + 1 candidate validation + 24 unmatched`, `50%` count and `NT$211,667 / NT$354,062 = 59.782%` revenue. Follow-up run `e1508ec3a49241d1965bc99d428b0325` found 19 candidates but safely wrote zero because none passed strict identity/variant/unit verification; its independent terminal is `degraded_no_safe_candidate`. Yahoo remains durably `active + enabled + write_enabled`, and the durable readback proves `formal_source_activated=true` even when the latest run is verified no-write. Formal runtime remains `2/15`. Next: retry the unresolved revenue-weighted batch only with fresh source evidence, then add the next approved structured marketplace adapter without relaxing promotion gates. |
 | 3 | `SEC-P0-002` | In progress (`canary_ready`) | Database identity + least-privilege RBAC | V10.789 is live and `governance/auth_identity_runtime_receipt.json` verifies required tables, two active admins, durable lockout, session revocation, trusted proxy policy and no-secret mutation audit readiness. Runtime is intentionally hybrid because zero database-admin success receipts have been captured; `auto` retires shared authority after two durable successes without a manual review gate. Next: capture database-admin login receipts and verify automatic database-only cutover. |
 | 4 | `SEC-P0-003` | In progress | Webhook trust and replay protection | Telegram secret-token verification code exists; production secret activation remains unproven. Exit: secret provisioned outside source, required mode enabled, invalid-secret 401 and authorized callback canary pass. |
 | 5 | `SUPPLY-P0-001` | In progress | Gitea-only secure software supply chain | Gitea-native checkout, secret-safe `.dockerignore`, commit-bound source receipt and governance gate are active. Exit: exact dependency lock, internal SAST/SCA/secret scan, SBOM, image digest/provenance, vulnerability SLA and production digest readback. |
 | 6 | `GOV-P0-001` | In progress | Canonical full asset graph + runtime reconciliation | `governance/ewoooc_asset_inventory.json` seeds hosts, services, data, AI, routes, supply chain, observability and recovery. Exit: same-run probe receipt for every asset; drift auto-creates work items. |
 | 7 | `GOV-P0-002` | Not started | Unified controlled-apply envelope | Introduce one `trace_id/run_id/work_item_id` across sensor, identity, SOT diff, decision, risk, dry-run, execution, verifier, rollback/retry and learning acknowledgement. Start with EventRouter + AutoHeal. |
-| 8 | `RAG-P0-001` | In progress (`source_fix_ready_runtime_canary_pending`) | Internal RAG candidate canary + NemoTron decision-only proof | V10.813 production still preserves the prior fail-closed receipt. Windows 99 audit run `534c6208-2088-428c-ba6f-f7a63a67f422` now proves the old GCP project is `DELETE_REQUESTED`, GCP-A is unreachable, and GCP-B Ollama `0.22.1` has exact `bge-m3` / `qwen3:14b` digests. Its bounded BGE-M3 probe returned 1024 dimensions with zero DB/app writes but required `116.475s`, proving the 10-second consistency timeout was a false-negative source defect. V10.814 source raises only the canary cold-start cap to 150 seconds, verifies `/api/tags` digest before embedding, permits one digest-verified GCP host as degraded shadow quorum while excluding 111, and adds a daily 05:00 NemoTron model decision with zero tool/DB/price/insight writes. Exit: deploy V10.814, pass both production canaries with Telegram/durable receipts, retain redundancy blocker until a canonical replacement GCP-A asset exists, then controlled `RAG_ENABLED` shadow activation plus query/hit/feedback telemetry. |
+| 8 | `RAG-P0-001` | In progress (`rag_canary_passed_nemotron_fallback_source_ready`) | Internal RAG candidate canary + NemoTron decision-only proof | V10.815 production run `736e6de1-fbc3-4c9b-bcfe-9b6ef488b932` passed PixelRAG receipt -> exact-digest BGE-M3 -> read-only pgvector canary with zero business writes; activation remains correctly blocked by `RAG_ENABLED=false` and degraded embedding redundancy. GCP-A is unreachable; GCP-B exact-digest `qwen3:14b` chat timed out at 300s and `qwen2.5:7b-instruct` at 240s, while 111 exact-digest `qwen3:8b` produced the required tool call in 19.053s with no tool/DB/Telegram execution. V10.816 source now implements exact-digest GCP-A -> GCP-B -> 111 model-aware fallback with one 180-second production deadline, bounded 111 context/output, deterministic one-call-per-SKU tool validation, truthful failure footprints and a process-shared `side_effect_started` boundary. Durable side effects remain quarantined for four hours even when dedupe commit verification fails, including mixed forced-review/model batches, so app/scheduler/bot cannot reopen the same SKU and duplicate the action；mixed-path dispatched/skipped metrics preserve input-count truth. Focused model/dedupe regression is `93 passed`. Next: deploy V10.816, obtain shared-store and fresh production `canary_passed_degraded_fallback` receipts plus independent zero-write readback; then restore GCP capacity/redundancy before controlled `RAG_ENABLED` shadow activation and query/hit/feedback telemetry. |
 | 9 | `MCP-P0-001` | In progress (`federation_source_ready`) | MCP/RAG production runtime closure | V10.796 source adds a strict public aggregate receipt for canonical `ewoooc` and `momo-pro-system` identities without opening authenticated internal APIs or exposing endpoint/tool payload data. Exit still requires V10.796 production `/health`, two fresh AWOOOI durable receipts with fingerprint recompute, live MCP servers/router/RAG, approved caller/tool boundary and production query canary. Current source readiness must not be reported as runtime closure. |
 | 10 | `SEC-P0-004` | Not started | Security operations lifecycle and metrics | Add durable security incident state and publish MTTA, MTTR, recurrence, false positive, human intervention, verifier pass, rollback and freshness. Exit: detect-to-learn production receipt. |
-| 11 | `REL-P0-001` | In progress (`runtime_verified_cd_degraded`) | Formal deploy and visible proof discipline | Production V10.813 runs exact Gitea object `a193acc407ef61b9cc6340e8e11aa9d0fb9a39c5`; dev merge `715f90133101e7fdcac243bf999aef05f80e7401` carries the same source. Host 110 still has no matching EwoooC runner, so formal CD remains unavailable and is not replaced by fallback evidence. Windows 99 controlled run `2ef0a084-7aa8-4748-a0db-127fb623d75d` deployed exact archive SHA-256 `65369d5f78c56376bffe8606adc790c046274dc070dfef9b933a6140cd83cc66`; rollback/evidence is retained at `/home/ollama/momo-deploy-backups/ewoooc-20260716T190911Z-a193acc-2ef0a084`. Internal/external `/health` is healthy at V10.813, all 8 source hashes match, app/scheduler/bot identities and immutable `momo-db` ID `cd092451cb5fd555d0ffff70642e109f3b742882c418beeab631793d1e9dc55d` remained unchanged. Public login visibly reports V10.813; authenticated cockpit visual proof remains unavailable in the current browser session. |
+| 11 | `REL-P0-001` | In progress (`v10815_runtime_verified_v10816_pending`) | Formal deploy and visible proof discipline | Production V10.815 runs exact Gitea object `283c8c80c631f5d97315708885413b62ee5a34ea`; dev merge `df83c646cdb3fde5e33f045bff19c5a392f84b86` has an identical tree. Host 110 still has no matching EwoooC runner, so formal CD remains unavailable and is not replaced by fallback evidence. Windows 99 controlled run `736e6de1-fbc3-4c9b-bcfe-9b6ef488b932` corrected and deployed the complete 36-file source set, verified `36/36` hashes and `12/12` runtime env checks, recreated only app/scheduler/bot, and left `momo-db` identity unchanged; rollback is retained at `/home/ollama/momo-deploy-backups/ewoooc-20260722T115155Z-283c8c8-full-736e6de1`. Internal/external `/health` is healthy at V10.815 and public JS/CSS match the tested hashes. V10.816 remains source-only until Gitea integration plus a new Windows 99 controlled deployment and independent runtime readback complete. |
 
 ### GROWTH-P0-001 Fixed Execution Lanes
 
@@ -36,7 +36,7 @@ These lanes are one ordered current P0, not optional side work. They must advanc
 |---|---|---|---|
 | A. Sales freshness | In progress (`sla_runtime_closed_source_redundancy_partial`) | Latest sales date `2026-07-13`; before the `2026-07-15 20:00` cutoff, raw lag is `2` but SLA lag is `0`, state is `grace` and decisions remain released. Scheduler receipt `0f24219e0bbb4740b7ad6645e7952296` and explicit canary receipt `6e2df008f65544fe8e557c11ff0dbb93` both persisted `completed_no_write`; durable decision is `no_candidate_fresh_no_write`. Live and persisted readiness now agree at Google Drive `1/4`; HTTPS, IMAP and local remain disabled. | Continue automatic report-arrival reconciliation. At/after 20:00 require `2026-07-14` or automatically block decision use, emit the bounded upstream action and verify the next arrival receipt; keep source redundancy partial until another approved source is live. |
 | B. Verified same-item evidence | In progress | TOP50 fixed cohort: `25` verified, `1` candidate/source validation, `24` unmatched. Count coverage is `50%`; revenue-weighted coverage is `NT$211,667 / NT$354,062 = 59.782%`. V10.810 preserves fingerprint `7b74504fcc1e1801c2ca2b42`; run `34999f5314054f09909663b46fc42ee2` added one independently verified offer (`+2.0pp` count, `+1.89pp` revenue), while run `e1508ec3a49241d1965bc99d428b0325` correctly ended no-write. Their durable artifact SHA-256 values are `994beba6dfa70ef8c50031f130916ee155657dd77ac8a2f59b6530b491d45d1e` and `e58be7bb07599871a8f318ad63885bd868c786b0144b81ba0980ae80cf556541`. | Retry unresolved candidates only after fresh evidence arrives, preserve deterministic identity/unit/variant gates, and publish count/revenue deltas against the same fingerprint per run. |
-| C. Platform runtime coverage | In progress (`runtime_canary_activated`) | V10.813 is live. Yahoo is durably active and exact-offer canary readback remains valid; an already-active canary now returns `already_active_verified`, `state_changed=false` and `writes_database_count=0`, while latest no-write receipts no longer erase durable activation truth. Formal runtime is `2/15`; PixelRAG remains evidence-only. | Continue bounded refresh on schedule, monitor expiry/recurrence/rollback signals, then implement the next approved structured source contract for Shopee, Coupang, ETMall, Friday or Rakuten without treating blocked pages as product data. |
+| C. Platform runtime coverage | In progress (`runtime_canary_activated`) | V10.815 is live. Yahoo is durably active and exact-offer canary readback remains valid; an already-active canary returns `already_active_verified`, `state_changed=false` and `writes_database_count=0`, while latest no-write receipts do not erase durable activation truth. Formal runtime is `2/15`; PixelRAG remains evidence-only. | Continue bounded refresh on schedule, monitor expiry/recurrence/rollback signals, then implement the next approved structured source contract for Shopee, Coupang, ETMall, Friday or Rakuten without treating blocked pages as product data. |
 
 ### AI Agent Product Integration Acceptance
 
@@ -45,7 +45,7 @@ This is an acceptance surface inside the current growth P0; it does not reorder
 | Layer | Current status | Exit evidence |
 |---|---|---|
 | Source and scheduler wiring | Source ready (`4/4`) | Hermes, NemoTron, OpenClaw and ElephantAlpha source markers plus scheduler ownership are machine-read and reported separately from runtime. |
-| Agent runtime activity | Production partial (`3/4` role-active; `0/4` healthy); V10.814 source canary ready | V10.813 live readback preserves Hermes/OpenClaw call evidence, zero NemoTron runtime activity and ElephantAlpha durable role-owned insights. V10.814 adds a qwen3 decision-only canary that uses the production payload and exact digest but stops before every tool/action/data write; a fresh pass proves shadow model execution only, not formal product action. Exit requires all four role-active and healthy in the bounded window, without treating class presence, configured fallback or shadow canary as an executed business outcome. |
+| Agent runtime activity | Production partial; V10.816 bounded fallback source ready | V10.815 runtime proved the RAG no-write lane and separately proved 111 `qwen3:8b` can execute the NemoTron tool contract in 19.053s, but production dispatcher does not use that model-aware fallback until V10.816 is deployed. A fresh decision-only pass proves shadow model execution only, not formal product action. Exit requires all four Agents role-active and healthy in the bounded window, without treating class presence, configured fallback or shadow canary as an executed business outcome. |
 | MCP/RAG dependency | Runtime disabled / telemetry empty | Production currently reports `MCP_ROUTER_ENABLED=false`, `RAG_ENABLED=false`, zero `mcp_calls` and zero `rag_query_log` activity. Exit requires enabled approved routes, live health, non-zero agent/product telemetry and the internal RAG candidate canary. |
 | Controlled automation closure | Runtime partial | `/api/ai-automation/agent-product-integration`, CLI and smoke must report Detect -> Normalize -> Correlate -> Decide -> Check -> Controlled Apply -> Verify -> Retry/Rollback -> Learn/Writeback. Completion requires bounded execution, linked outcome/incident verification and durable learning evidence; aggregate source presence is insufficient. |
 
@@ -57,8 +57,8 @@ This bounded interruption is closed and control returns to `GROWTH-P0-001` witho
 |---|---|---|
 | Program | Completed for the four primary analysis tabs | Daily sales, sales analysis, growth analysis and monthly summary now share one canonical day/month/range contract; cross-tab links preserve the selected period. |
 | Asset coverage | `4/4` pages and `2/2` sales async APIs verified | Daily KPI/calendar/charts/Top 10, growth KPI/series, monthly KPI/tables/charts and sales KPI/charts/YoY/detail table all use the active period. Historical current-snapshot mixing and blank monthly charts are replaced by explicit honest states. |
-| Runtime closure | Completed in V10.810 at `911393190ded015e384e438c26b68faf50ec260c` | `2026-04` daily stays inside April and bounds extreme DoD/WoW values with traceable markers; growth single-point series remain visible; sales consumes the real `chart_values` payload and aggregates repeated SKU rows before ranking; monthly requests are abortable/deduplicated and publish explicit loading/error/ready states. External HTTPS serves the exact tested JS/CSS checksums and internal/external `/health` report V10.810. |
-| Verification | Passed; formal CD still degraded | Final regression: `2,123 passed / 9 skipped / 0 failed`; focused analytics contracts: `49 passed`. The automated visual guard passed `12/12` combinations (`4` pages x desktop/tablet/mobile), covering `126` chart-target observations, period links, actual pixels/elements, honest empty states, date-tick density, console errors and overflow. Formal CD remains degraded because no matching runner executed the release. |
+| Runtime closure | Completed in V10.815 at `283c8c80c631f5d97315708885413b62ee5a34ea` | The exact `/sales_analysis` URL with blank date fields now resolves one canonical period and applies the same metric/range/filter contract to KPI, charts, YoY and detail rows. Query, chart and export logic are split into dedicated services; repeated SKU rows are aggregated before ranking, date/range links preserve state, and invalid filters fail safely. Windows 99 independent readback verified all 36 target files, runtime env, public `/health` and exact JS/CSS hashes. |
+| Verification | Passed; authenticated visual proof remains pending | V10.815 broad regression: `2,174 passed / 9 skipped / 0 failed`; focused analytics batches: `173 passed` and `70 passed`, plus Python compile, Node syntax, Jinja render and diff checks. Public target route returns the expected login redirect and public assets match SHA-256 `6bb4a985744917ef2243d6d055c85b6803b2baf5c571580f49ef69c536ccdf04` / `d98654ddaac10a49af2351750f350bbbcfc3491d62b14c8a25d02eb4d7803186`. The current browser session cannot provide authenticated chart screenshots, so that evidence is not claimed. |
 
 ## P1
 
@@ -69,10 +69,10 @@ This bounded interruption is closed and control returns to `GROWTH-P0-001` witho
 | 14 | `APPSEC-P1-001` | In progress | CSP and DOM/XSS hardening | Security headers are present and CSP is report-only. Collect violations, remove high-risk `innerHTML`/inline sinks, then enforce CSP by canary. |
 | 15 | `APPSEC-P1-002` | Not started | Unsafe shared-cache serialization removal | Replace writable pickle caches in dashboard/daily-sales/EDM/sales with constrained JSON or signed schema. |
 | 16 | `ARCH-P1-001` | In progress | Split oversized policy/executor/verifier modules | Current top debts include 44k-line PChome mapping and 14k-line smoke service. Split by bounded family and independent tests. |
-| 17 | `UX-P1-001` | In progress | Professional full-site UI/UX | V10.810 closes chart rendering defects across all four primary analysis tabs: real payload values render, single points stay visible, extreme percentages no longer flatten normal data, dense date labels auto-skip, Top 50 vendor content uses bounded zoom, and loading/error/empty states are explicit. The `12/12` desktop/tablet/mobile visual matrix has zero horizontal overflow. The broader site-wide first-viewport, progressive-disclosure, accessibility and loading/error/degraded-state audit remains in progress. |
+| 17 | `UX-P1-001` | In progress | Professional full-site UI/UX | V10.815 closes the exact sales-analysis period/filter linkage and keeps the prior four-tab rendering guards: real payload values, visible single points, bounded extreme percentages, date-label auto-skip, zoom bounds and explicit loading/error/empty states. Public assets are hash-verified; authenticated V10.815 visual proof and the broader site-wide first-viewport, progressive-disclosure, accessibility and loading/error/degraded-state audit remain in progress. |
 | 18 | `PIXELRAG-P1-001` | Not started | Ollama-first multimodal embedding benchmark | Verify approved visual embedding on GCP-A -> GCP-B -> 111 and design pgvector-compatible visual metadata; FAISS remains disallowed without ADR. |
 | 19 | `MARKET-P1-001` | In progress | Marketplace source contracts | Yahoo Shopping remains active in V10.810 production with public-boundary allowlists, bounded streaming/rate, provenance, current product-detail readback, stock/spec/variant guards, source-specific promotion partition, idempotent exact canary activation and durable activation readback across no-write runs. Four fresh verified Yahoo offers now contribute formal evidence across completed batches; non-exact and unit-price candidates do not. Shopee, Coupang, ETMall, Friday and Rakuten still require equivalent structured contracts, and blocked pages remain non-product data. |
-| 20 | `QA-P1-001` | In progress | Deterministic test and CI governance | V10.810 final broad regression is `2,123 passed / 9 skipped / 0 failed`; analytics contracts are `49 passed`, and the new runtime chart guard passes `12/12` page/viewport cases with actual pixel/element and console assertions. Production host parity is `18/18` files and public HTTPS parity is `9/9` key JS/CSS assets. No matching EwoooC runner executed this release, so CI/CD parity is not claimed and exact-object fallback receipts remain separate evidence. |
+| 20 | `QA-P1-001` | In progress | Deterministic test and CI governance | V10.816 source-candidate broad regression is `2,223 passed / 15 skipped / 0 failed`; focused model/dedupe regression is `93 passed`, and independent ninth-round review found no material issue. V10.815 production parity remains `36/36` source hashes plus `12/12` runtime env checks, and public HTTPS serves its exact tested sales JS/CSS. No matching EwoooC runner has executed V10.816, so formal CI/CD or production parity is not claimed and Windows 99 exact-object deployment receipts remain a separate evidence layer. |
 
 ## P2
 
@@ -97,7 +97,7 @@ These are reusable foundations, not proof that the full program is complete.
 | Completed | PromotionGate replay | No production write. |
 | Completed | Embedding-signature guard replay | Signature readiness only. |
 | Completed | Candidate knowledge replay | Internal RAG preview only; no DB/model call. |
-| Source ready; production pending | Internal RAG + NemoTron decision canaries | V10.814 source separates cold-start timeout from normal embedding latency, enforces exact BGE-M3/qwen3 digest readback, permits one approved GCP host only as degraded shadow quorum, and adds zero-tool/zero-data-write NemoTron runtime proof. Production canary receipts, replacement GCP-A control-plane identity, RAG shadow activation and query/outcome telemetry remain P0. |
+| Source ready; production pending | Model-aware NemoTron dispatcher fallback | V10.816 source uses one modular exact-digest candidate registry for production and canary: GCP-A/GCP-B `qwen3:14b` with 60-second attempts, then 111 `qwen3:8b` with 45 seconds, `think=false`, fixed `num_ctx=4096` / `num_predict=512` and one total deadline. Production and canary share the full one-call-per-SKU tool contract; malformed model identity schema fails closed and continues the approved candidate chain. Two-phase dedupe uses `/app/data` shared `flock` + fsync + atomic JSON ownership-token leases across Gunicorn workers and app/scheduler/bot containers. Before any handler, the owner persists `side_effect_started` with a four-hour quarantine; only an explicit DB write, durable insight, delivered notification or durable EventRouter queue acknowledgement counts as dispatched. A later commit-verification failure is surfaced but never releases the quarantine, preventing duplicate action after a durable effect. Privacy-safe state hashes SKU values; privacy-safe attempt summaries preserve label/tier/model/status/digest/error-class in structured and recipient-visible footprints. Production deployment, shared-store canary, fresh degraded-fallback receipt and independent readback remain P0; GCP-A replacement, GCP-B capacity and RAG shadow activation remain unresolved rather than hidden by the fallback. |
 | Completed | PixelRAG application portfolio | Commerce/RAG/UX/ops/marketing/governance inventory. |
 | Completed | Ollama-first VLM route readiness and replay worker | Evidence-bound artifact output; no direct price write. |
 | Completed | Platform probe worker | Shopee/Coupang barriers become structured fallback/backoff receipts. |