# Project Todos Consolidated todo list from specs, plans, codebase review, and session work. --- ## Execution Order Recommended sequence accounting for priority, dependencies, and groupings. Items within a batch are independent and can be parallelized. ### Batch 4 — P2: Code quality, grouped by file 9. **`data-client.ts`:** Uses `new Date()` instead of server timestamp (`data-client.ts:348,335`) 10. **`offline/coda-store.ts`:** file split (MessageStore/ChatStore/BlobStore) still open (472 lines) 11. **`core-api/src/client.ts` file split** (ApiClient/StreamQueryRunner/RetryFetcher) — still 873 lines ### Batch 6 — Infrastructure & ops 14. **Docker smoke tests** — straightforward, no dependencies 15. **Ephemeral environment Jenkins trigger** (Option A recommended) — PR to jenkins-config; small, isolated 16. **Archive Notion pages** — non-code, no dependencies ### Batch 7 — Model/DB migrations 17. **Model capabilities to DB** — move ALLOWED_MODELS + supportsThinking to models table; supersedes the regex fix from Batch 1 #2 ### Batch 8 — Search improvements 18. **Search bearer token auth** — COD-82; add auth middleware to search ConnectRPC routes (admin auth exists but main search routes are unprotected) 19. **OpenAPI search index** — third index over internal services' OpenAPI specs; spec + plan ready. See Batch 8a below. 20. **Glossary target validation at startup** — small, independent; log warning on missing targets 21. **Tool catalog semantic search (embedding integration)** — Add vector stage to `ToolCatalogSearch` via `RemoteEmbeddingProvider` calling the search service. Three pieces: (1) `Embed` RPC on search service (~20 lines, exposes the existing HuggingFace model), (2) `RemoteEmbeddingProvider` in server (~40 lines, implements `EmbeddingProvider` interface via ConnectRPC), (3) pass provider to `HybridSearch.create()` in `catalog.ts`. Pre-compute system tool vectors at startup via `addFromSnapshot()`. Fallback: if search service unreachable, `HybridSearch` degrades to keyword-only (vector stage returns empty, BM25+glossary continue via RRF — proven at MRR@8=0.9529). **Platform-aware design:** expose `add()`/`remove()` on `ToolCatalogSearch` for runtime tool mutations; pre-compute vectors at tool creation time (not server startup); per-tenant index instances when platform service manages tool grants/revocation. For <500 tools per tenant, brute-force cosine over pre-computed vectors is sub-millisecond (no HNSW needed). Design spec: `docs/superpowers/specs/2026-05-13-hybrid-search-migration-design.md` 22. **AGENT_CONSUMABLE tag discovery** — defer until `_AGENT` schema pattern is deployed in Snowflake 23. **KeywordSearchUnit extraction** — decompose HybridSearch's keyword indexing into a composable unit 24. **Glossary enrichment (LLM Wiki)** — add `relationships`, `gotchas`, `related_concepts` fields to glossary entries 25. **Glossary linting endpoint (LLM Wiki)** — new admin `LintEngine` handler checking stale targets, coverage gaps, orphan entries 26. **Pre-query catalog endpoint (LLM Wiki)** — lightweight `GetCatalog` admin RPC returning schema landscape overview 27. **ReportUsage persistence & analytics (LLM Wiki)** — wire the no-op `ReportUsage` stub; phase 2: selection analytics 28. **Compiled entity summaries (LLM Wiki)** — pre-synthesized knowledge layer; pursue when per-query LLM cost is measured as a bottleneck ### Batch 9 — Sandbox enhancements 29. **Tier 4 rate limits (per-user, Redis-backed)** 30. **Sandbox audit logging** 31. **Generic tool progress callback** — evaluate need; if promoted to first-class, do before #32 32. **Client UI for sandbox progress** 33. **Sandbox streaming data bridge** — defer until host-side memory pressure observed 34. **Sandbox process separation (gRPC/sidecar)** — defer until in-process sandbox causes scaling issues ### Batch 10 — Integrations & external services 35. **Google Sheets integration** — requires 3P OAuth credential management (#36) first 36. **3P OAuth credential management** — shared infra for user-scoped OAuth tokens (Google, etc.) 37. **Broader Google Suite tools** — Drive, Docs, Calendar; after Sheets + OAuth are proven ### Batch 11 — Architectural changes (largest scope, external dependencies) 38. **Query sources field** (AI Response Transparency) — design decision required first (structured block vs. markdown) 39. **Explore gRPC/proto pattern for api/** --- ## Codebase Review (2026-03-20) Source: [codebase review remediation TRD](decisions/trds/codebase-review-remediation.md) ### P2 — Medium - [ ] **Uses new Date() not server timestamp** — `data-client.ts:348,335`. ### File Splits - [ ] `packages/core-api/src/client.ts` (873 lines) → `ApiClient`, `StreamQueryRunner`, `RetryFetcher` - [ ] `apps/client/src/offline/coda-store.ts` (472 lines) → `MessageStore`, `ChatStore`, `BlobStore` --- ## Search & Glossary ### Open - [ ] **Drop SortedArray during `@theorchard/data-structures` migration** — SortedArray is a tagged wrapper that doesn't validate sortedness and optimizes away an O(n log n) sort on arrays of <100 entries (negligible). Change `NamedSignal.entries` to `Iterable`, sort unconditionally in fusion, remove SortedArray. TimSort is O(n) on already-sorted input anyway. - [ ] **Snowflake search pipeline enrichments** — Three improvements ranked by value × ease. Plan: `docs/superpowers/plans/2026-04-27-snowflake-search-enrichments.md`. PR for glossary term updates: theorchard/ows-coda#191. - [ ] **Glossary examples in search response** — Surface matched `GlossaryExample[]` (worked SQL) in `SearchSnowflakeResponse.suggested_queries`. Currently examples are only used for embedding enrichment, never returned to caller. High value, easy-medium. - [ ] **Data freshness (`last_altered`) in response** — Add `changedOn` to `TableResult` proto. Already stored on `SnowflakeTableEntry`, just not in the response. 2 lines of code. Medium value, trivial. - [ ] **Column value hints for low-cardinality columns** — Run `SELECT DISTINCT` for `*_STATUS`, `*_TYPE`, `*_CODE`, `OWNER` columns during schema load. Biggest gap from real conversations (users say "filter by AWAL accounts" but agent doesn't know `owner = 'AWAL-UK'`). Very high value, medium effort. - [ ] **Query success tracking** — Wire the no-op `ReportUsage` stub to persist `(query, selected_ids, timestamp)`. Correlate `search_snowflake` → `query_snowflake` calls. Medium value, medium-large effort. - [ ] **Save engine input dumps alongside snapshots** — After each fetch, serialize the raw schema inputs (GraphQL introspection result, Snowflake catalog rows) to a separate S3 key alongside the snapshot (e.g., `{prefix}inputs/{timestamp}.json.gz`). NOT stored in the snapshot itself — parallel diagnostic artifact. Enables debugging ("what schema did the engine see when it built this index?"), cross-deployment diffing, and offline reproduction without live data sources. Retention follows the same policy as snapshots. - [ ] **Glossary target validation at startup** — Log a warning if a glossary target doesn't exist in the schema index. - [ ] **AGENT_CONSUMABLE tag discovery** — When `_AGENT` schema pattern is deployed, boost those tables higher via glossary priority or scoring modifier. - [ ] **Tool catalog semantic search** — See Batch 8 #21 for full design (Embed RPC + RemoteEmbeddingProvider + platform-aware mutations). - [ ] **Search bearer token auth** — COD-82; add auth middleware to search ConnectRPC routes. - [ ] **Glossary enrichment** — Add `relationships`, `gotchas`, `related_concepts` fields to glossary JSON entries. `relationships` and `gotchas` concatenated into `context` for embedding enrichment via existing `buildGlossaryContextMap()`. `related_concepts` enables glossary-to-glossary cross-linking (match "revenue" → also boost "deductions" targets at reduced weight). Zero code changes — richer data flows through existing pipelines. See [LLM Wiki comparison](reference/comparisons/knowledge/llm-wiki-comparison.md#1-glossary-enrichment--wiki-style-entity-context-low-effort-medium-value). - [ ] **Glossary linting endpoint** — New admin `LintEngine` handler. Checks: stale targets (glossary references missing IDs), stale related, coverage gaps (high-degree nodes with no glossary entry), orphan entries, cross-reference gaps, embedding staleness (circuit breaker was open during poll), example drift (glossary SQL examples reference missing columns). Returns structured `LintReport`. Emits `lint.*` events for admin UI. See [LLM Wiki comparison](reference/comparisons/knowledge/llm-wiki-comparison.md#2-glossary-linting--wiki-style-health-checks-low-effort-medium-value). - [ ] **Pre-query catalog endpoint** — Lightweight `GetCatalog` RPC returning structured schema landscape overview: databases with table counts grouped by naming pattern, GraphQL domains grouped by glossary `domain` field. Built from existing glossary `databases` config + glossary domain groupings at startup, zero additional fetching. Agent reads catalog before searching to scope ambiguous queries. See [LLM Wiki comparison](reference/comparisons/knowledge/llm-wiki-comparison.md#4-pre-query-catalog--indexmd-equivalent-low-effort-lowmedium-value). - [ ] **ReportUsage persistence & selection analytics** — Phase 1: wire the no-op `ReportUsage` stub (`server.ts:158`) to persist `(query, selected_ids, timestamp)` tuples (append-only log or Prisma model). Phase 2: periodic analysis — co-selection frequency → candidate glossary `related` links, zero-selection queries → coverage gaps, entry-selection correlation → glossary usefulness validation. Phase 3: automatic glossary proposals via admin endpoint. See [LLM Wiki comparison](reference/comparisons/knowledge/llm-wiki-comparison.md#3-query-knowledge-capture--filing-discoveries-back-medium-effort-medium-value). - [ ] **Compiled entity summaries** — Pre-synthesized knowledge layer. Post-poll compilation generates entity summaries (relationships, business context, usage patterns) for new/changed items. Summaries indexed into BM25 + HNSW alongside raw schema. Can be LLM-generated (~$0.50–2.00 per full rebuild) or rule-based templates (FK graph + glossary context + column metadata, $0). Pursue when per-query LLM interpretation cost is measured as a bottleneck — glossary enrichment is the lightweight first step. See [LLM Wiki comparison](reference/comparisons/knowledge/llm-wiki-comparison.md#5-compiled-entity-summaries--the-full-wiki-layer-high-effort-high-value). - [ ] **Community detection + LLM summaries (GraphRAG)** — Run Louvain community detection on the FK/type graph after each poll. Generate 2-3 sentence LLM summaries per community via Bedrock (search service calls Bedrock directly — already has AWS credentials for S3). Summaries stored in memory, regenerated on graph changes. Enables "global" queries ("what data do we have about finances?") by searching community summaries instead of individual table names. Components: `packages/common/src/graph/communities.ts` (~100 LOC, Louvain algorithm), `apps/search/src/engine/community-summarizer.ts` (~60 LOC), new `SummaryProvider` interface, storage on `SearchEngine`, admin endpoint to view communities. - [ ] **`SearchAll` unified cross-source RPC** — New RPC that runs all engines in parallel, fuses results via RRF into a heterogeneous ranked list. For ambiguous user intent where the AI doesn't know whether to search GraphQL or Snowflake. Returns a union result type with source-tagged entries. Individual source RPCs (`SearchGraphQL`, `SearchSnowflake`) remain for targeted tool use. Requires a response proto that can carry both `QueryFieldEntry` and `SnowflakeTableEntry` shapes. - [ ] **OpenAPI search index** — Third search index over internal services' OpenAPI 3.x specs. Shared service catalog config (`packages/common/`) with `${ENV_VAR}` interpolation, `OpenApiFetcher` + `OpenApiAdapter` following existing patterns, cross-service graph edges via path parameter matching, `SearchOpenAPI` + `GetOpenAPISchema` RPCs. - [x] **Bound the stem cache in tokenize.ts** — done via CircularMap (bounded circular buffer, no external dependency). `packages/search/src/tokenize.ts` now uses `CircularMap` with a fixed capacity cap instead of an unbounded `Map`. - [ ] **Cache proximitySignal BFS results** — `packages/search/src/signals/proximitySignal.ts` runs an O(V+E) multi-source BFS on the FK graph on every search query via the `querySignals` callback. For a 500-table graph with 10 anchor nodes, this visits thousands of nodes per query. Add an LRU cache keyed by sorted anchor IDs (e.g. `"table_a|table_b|table_c"`), invalidated on graph rebuild (poll, ~1h). Cache hit returns precomputed `NamedSignal` entries. ~20 lines. Low urgency at current scale — becomes noticeable at 2,000+ tables or high search QPS. - [ ] **Branded FQN type** — Add `type Fqn = string & { readonly __fqn: unique symbol }` to prevent accidental use of bare table names as FQN keys. - [ ] **Parallel array elimination in IndexSnapshot** — Merge `documents[]` + `vectors[]` into `Array<{ document: T; vector?: Float32Array }>` for structural safety. - [ ] **Eliminate `SnowflakeSchemaState.tables` array** — Derive from `tableMap.values()` instead of maintaining a redundant copy. - [ ] **Collapse `TRaw`/`TDoc` type params** — Both current SearchEngine implementations set `TRaw = TDoc`. Consider collapsing to `SearchEngine` and deferring the raw/doc distinction until a data source actually needs non-trivial transformation. - [ ] **Simplify snapshot retention tiers** — The 4-tier ISO-week retention policy in `retention.ts` is over-engineered for current snapshot frequency (~few/hour, ~50 keys max). Consider simplifying to "keep last N" until scale justifies the tiered approach. - [ ] **Wire OTel fields on SearchEvent** — `traceId`, `spanId`, `durationMs` on `SearchEvent` are aspirational scaffolding. Bridge to an OTel SDK/exporter when observability infrastructure is ready. - [ ] **Populate TraceQuery span details** — The per-stage proto messages (`KeywordDetail`, `VectorDetail`, etc.) exist but aren't filled by `traceQueryHandler`. Requires stages to emit structured events that the handler maps to those typed fields. - [ ] **Guard `toLogFields` against pino field clobbering** — `LoggingSubscriber.toLogFields` does `Object.assign(fields, event.data)`. A payload field named `level` or `msg` would clobber pino reserved fields. Namespace event data under a `data` key or use an allowlist. ### Extract to `packages/common/` - [ ] **EventBus + InMemoryEventBus + createEmitter** — Generic typed event bus with glob-pattern subscriptions. API is stable. Rename `SearchEvent` to `AppEvent` with search-specific fields as an extension. Second consumer: `apps/server` (orchestrator observability). - [ ] **BlobStore interface** — Extract `BlobStore` interface to `packages/common/src/storage/`. Leave `S3BlobStore` in `apps/search` (or a future `packages/storage/`) to avoid adding `@aws-sdk/client-s3` as a common dependency. Extract when a second consumer emerges. - [ ] **TraceSubscriber → EventCollector** — Generic event collector with filter helpers (`ofType`, `ofSource`, `clear`, `dispose`). Extract after EventBus moves to common. 47 lines, clean dependency. - [ ] **S3 async page generator** — `S3BlobStore.list()` eagerly accumulates all pages. Convert to `async function*` yielding pages lazily. `SnapshotManager.load()` could short-circuit after one page. - [ ] **Snowflake stream → async generator** — `exec-sql.ts` `collectStream()` wraps EventEmitter in Promise. Convert to `async function*` yielding rows. Requires carrying over abort signal wiring. - [ ] **`MultiMap`** — Class with `add(key, value)`, `get(key): V[]` for the "build incrementally" case. Eliminates `Map` boilerplate (get → check undefined → create → push). Main beneficiary: `schema-loader.ts` `attachColumns()`. - [ ] **`concurrentMap` async generator** — `concurrentMap(src: AsyncIterable, concurrency: number, fn: (item: T) => Promise): AsyncGenerator`. Generalizes the embedding batch+yield pattern with bounded concurrency and backpressure. --- ## PRDs & TRDs ### Complete | Feature | PRD | TRD | | -------------------------------- | ----------------------------------------- | -------------------------------------------------------- | | DB package extraction | — | [TRD](decisions/trds/db-package-extraction.md) | | Docker Compose local environment | — | [TRD](decisions/trds/docker-compose-environment.md) | | Functional tests | — | [TRD](decisions/trds/functional-tests.md) | | Message feedback | [PRD](decisions/prds/message-feedback.md) | — | | Sandbox engine | [PRD](decisions/prds/sandbox-engine.md) | — | | Get single chat endpoint | — | [TRD](decisions/trds/get-single-chat-endpoint.md) | | Message status stream persister | — | [TRD](decisions/trds/message-status-stream-persister.md) | | Auth error page | [PRD](decisions/prds/auth-error-page.md) | — | | Stream module | — | [TRD](decisions/trds/stream-module.md) | | Collections consistency | — | [TRD](decisions/trds/collections-consistency.md) | | Semantic schema search | — | [TRD](decisions/trds/semantic-schema-search.md) | | Snowflake schema index | — | [TRD](decisions/trds/snowflake-schema-index.md) | | Ephemeral environments | — | [TRD](decisions/trds/ephemeral-environments.md) | | Logging improvements | — | [TRD](decisions/trds/logging-improvements.md) | | Standalone search service | — | [TRD](decisions/trds/search-service.md) | | Search architecture restructure | — | [Architecture](architecture/search.md) | | Workspace restructure | — | — | ### In Progress | Feature | PRD | TRD | | -------------------------------- | --------------------------------------------------------- | ---------------------------------------------------- | | Dashboards | [PRD](decisions/prds/dashboards.md) | — | | Offline client cursor pagination | [PRD](decisions/prds/offline-client-cursor-pagination.md) | — | | File splits & dedup | — | [TRD](decisions/trds/file-splits-dedup.md) | | Codebase review fixes | — | [TRD](decisions/trds/codebase-review-remediation.md) | --- ## Sandbox - [ ] **Enforce ToolPolicy.readOnly in BridgeEnforcer** — `ToolPolicy.readOnly` is defined in the bridge protocol and passed by every consumer, but `BridgeEnforcer` only checks `allowedTools` — it never reads `readOnly`. User code can forward write operations through `dataProxy.request()` regardless of the flag. Add enforcement in `BridgeEnforcer.guardDataRequest()`: accept a method-classification callback (or a static set of read-only method names) and reject write methods when `readOnly: true`. Add tests for the enforcement path. - [ ] **Sandbox streaming data bridge** — Add streaming support to the sandbox bridge protocol so data from tool calls (e.g., Snowflake query results) can be streamed directly into the isolate without intermediate materialization on the host. Requires new bridge message types (`data_stream_chunk`, `data_stream_end`), a new isolate-side API (`dataProxy.requestStream()` returning async iterator), backpressure handling across the `ivm` boundary, and modified timeout semantics. Security model is unaffected — each chunk still uses `ivm.ExternalCopy`. **Revisit when:** profiling shows host-side memory pressure from intermediate result materialization, or result sets routinely approach the 50MB bridge limit. - [ ] **Tier 4 rate limits (per-user, Redis-backed)** — Per-user global limits: max concurrent executions (5), max executions/min (30), max data requests/min (200). Enforced by control plane via Redis sliding windows. Deferred from sandbox-server integration v1. - [ ] **Sandbox audit logging** — Dedicated audit log for all `data_request` events from sandbox code (requestId, method, params, user identity, timestamp). Handler currently logs tool calls, but a structured audit log system is future work. - [ ] **Client UI for sandbox progress** — Sandbox-specific progress rendering in the React client (e.g., code execution indicator, data request progress, stats display). - [ ] **Generic tool progress callback** — Explore adding an `onProgress` callback to the `ToolHandler` type so any tool can report sub-step progress (not just sandbox). Currently the sandbox handler captures the progress callback via closure to avoid changing the universal handler signature. If more tools need progress reporting, promote this to a first-class handler concern. - [ ] **Sandbox process separation (gRPC / sidecar)** — The current integration runs the sandbox engine in-process with the server (sub-project #3). Explore separating the sandbox into its own process for stronger isolation and independent scaling. Progression: (1) **Sidecar** — sandbox runs as a co-located process alongside the server, communicating via gRPC with protobuf-defined messages (the sandbox message protocol is already JSON-serializable, mapping directly to protobufs). Same host, separate memory/crash domain. (2) **Fully separated service** — sandbox runs as an independent service (e.g., Fargate), discovered via Cloud Map, scaled independently based on isolate demand. Corresponds to sub-project #2 (gRPC bridge) from the sandbox engine design spec. **Revisit when:** in-process sandbox causes memory pressure on the server, or isolation/scaling requirements exceed what a single process provides. --- ## Agent Memory (from greenfield review, 2026-04-28) Source: [greenfield review](superpowers/specs/2026-04-28-greenfield-review-agent-memory-pr193.md) ### Immediate - [ ] **Separate `tool_avoidance` prompt label** — `prompt-formatter.ts:8-9` maps both `tool_affinity` and `tool_avoidance` to `"Tool preferences"`. Positive/negative signals are concatenated with no visual separation. Give `tool_avoidance` a distinct label. ### Next iteration - [ ] **Replace `metadata: Record` with discriminated union** — `types.ts:32`. Each `ObservationType` has well-known required keys but the type is `Record`, forcing unsafe casts in `consolidate.ts` and no compile-time enforcement in emitters. Replace with a discriminated union per observation type. ### Improvement - [ ] **Batch observation writes** — `emitMemoryForToolExecution` calls `emitObservation` per observation. Add `appendBatch` to `ObservationLog` and emit per tool round instead of per observation. - [ ] **Simplify `Fact.subject` to bare key** — `subject` redundantly encodes `category` (e.g. `"tool_affinity:get_account"`). `affinity.ts` must string-slice to recover the tool name. Make `subject` the bare key within the category. - [ ] **Create `MemoryObservationLog`/`MemoryFactRepository`** — skip JSON serialization in the dev path. Current in-memory stores use `RedisObservationLog`/`RedisFactRepository` which stringify/parse every object. - [ ] **Make decay constants configurable** — `BASE_DECAY`, `REINFORCE_AMOUNT`, `EXPIRY_THRESHOLD` are duplicated in `exponential.ts` and `adaptive.ts` as module-level constants. Extract `DecayConfig` with defaults, accept via constructor. - [ ] **Add missing test coverage from greenfield review** — ~10 gaps: `observationToFactKey` null branch, empty `factKeys` path, `identityId` fill-in contract, confidence at expiry threshold, concurrent emit+load race, `get_products_by_isrc` fallback fields, non-array `tables`, `failure_pattern` category, `MemorySortedStore` full-removal, `AdaptiveDecay` timestamp equality assertion. --- ## Agent Behavior & Quality (from user session analysis, 2026-04-25) ### P1 — User-facing quality - [ ] **Pre-validate adjustment eligibility** — Before building adjustment previews, check: (1) `exchange_rates_delivered` on target statement period, (2) contract-account linkage exists, (3) currency compatibility. Currently the agent builds elaborate previews, asks for confirmation, then fails on submission with 422. - [ ] **Deep links to Abacus platform entities** — Users ask "send me a link to this account in Abacus." The agent should construct deep links using a configurable base URL pattern (e.g., `https://abacus.theorchard.com/accounts/{accountId}`). Add URL templates to config or extensions package. - [ ] **Capability manifest** — The agent gives vague answers to "can you generate images?" / "can you write code?" / "can you send links?". Add a clear capability manifest (what it can/can't do) to the system prompt or as a queryable tool. ### P2 — Domain knowledge - [ ] **Paythrough/flowthrough revenue path** — Revenue for paythrough contracts flows through adjustments (`LEDGER_ADJUSTMENT_DETAIL`, `LEDGER_ADJUSTMENT_APPLIED`), not the standard balances pipeline. The agent tried standard revenue queries and got empty results. Add this to glossary context and/or system prompt domain knowledge. - [ ] **Business unit distinction (Orchard vs AWAL vs KNR)** — The `owner` field in vendor tables differentiates AWAL-UK, AWAL-Core, AWAL-US, KNR, etc. The agent doesn't know these represent different Sony business units. Add to glossary context for vendor/account entries. - [ ] **Contract lifecycle automation** — `TO_BE_TERMINATED` → `TERMINATED` happens automatically via the Run Controller during accounting runs, not via user action. Add to glossary context for contract entries. - [ ] **Multi-account artist structures** — Labels often have parent/child account structures (e.g., Rimas has 11 sub-accounts). The agent should understand this pattern and search for sub-accounts when a parent account query returns limited data. - [ ] **Cross-account transfer adjustment format** — A transfer between accounts requires separate debit/credit rows per account, potentially in different currencies. Document in adjustment glossary context. - [ ] **PDF rendering limitations** — Wide tables cause overflow in generated PDFs. The agent should recommend Excel/CSV for data-heavy exports or warn about column-width limitations proactively. ### P3 — Data access gaps - [ ] **GraphQL auth context for restricted queries** — Several queries (NR contributor search, `orchAdminUsers`, `topSoundRecordings`) require a `profileUUID` header the agent can't provide, resulting in 500 errors. Catalog these queries and either add auth context threading or mark them as unavailable in the glossary. - [ ] **Transaction type reference data** — `abacusReferencePaymentTypes` returns 500. No standalone "list all transaction types" root query exists. Add a working Snowflake fallback or fix the GraphQL endpoint. Users frequently ask about service fees and transaction type IDs. - [ ] **Sales file processing totals** — Users want "total USD of sales files processed in period X." Data exists via `abacusSalesFile` but is access-restricted. Document the restriction or add a Snowflake alternative. - [ ] **Tax document retrieval** — Users ask for VAT invoices and 1042-S forms. The agent can't generate or retrieve these. Add clear routing instructions ("contact your account manager" or link to self-service portal). --- ## AI Response Transparency - [ ] **Query sources field** — When the AI executes a GraphQL or Snowflake query, expose the actual query in a dedicated `sources` field in the response (rather than plain-text links). Requires design decision: new frontend-rendered structured block (like `chart`) vs. inline code in markdown. Design Q: should it be a fenced `source` block the frontend renders as a collapsible panel, or plain markdown code blocks? --- ## Ephemeral Environment Jenkins Trigger **Context:** `Jenkinsfile.ephemeral` exists but comment triggers (`ephemeral deploy/destroy/list`) don't fire. **Root causes found:** 1. `jenkins-config/pipeline.theorchard.io/jobs/orgFolder.groovy` uses `scriptPath('Jenkinsfile')` only — no job is ever created for `Jenkinsfile.ephemeral`. 2. `Jenkinsfile.ephemeral` uses `env.CHANGE_ID` to get the PR number, which is only injected in Multibranch Pipeline jobs, not standalone pipeline jobs. **How jobs are auto-created:** Jenkins GitHub Organization Folder (configured in `theorchard/jenkins-config`) scans all repos and creates Multibranch Pipeline jobs for files named exactly `Jenkinsfile`. Non-standard filenames require explicit entries in `standalonePipelines.groovy`. **Options:** - [ ] **Option A — PR to `jenkins-config` (recommended):** Add a project-scoped entry to `pipeline.theorchard.io/jobs/standalonePipelines.groovy` pointing at `theorchard/ows-coda` with `scriptPath: 'Jenkinsfile.ephemeral'`. Also update `Jenkinsfile.ephemeral:93` to fall back to `env.ghprbPullId` when `CHANGE_ID` is absent (GHPRB plugin variable in standalone pipeline context — confirm with platform team). This is a small PR, follows the established pattern, and scopes the job to ows-coda only. - [ ] **Option B — Incorporate into existing `Jenkinsfile` (fully self-contained):** Extend the `issueCommentTrigger` pattern in `Jenkinsfile` to also match `ephemeral` comments and delegate to the ephemeral logic. `CHANGE_ID` is already available. Downside: couples CI and ephemeral lifecycle into one file. **Ruled out:** - Adding a second `workflowMultiBranchProjectFactory` to `orgFolder.groovy` — org-level, would scan all 100+ repos for `Jenkinsfile.ephemeral`. - GitHub Actions — doesn't integrate with the existing Jenkins infrastructure. --- ## Google Suite Integration - [ ] **Google Sheets integration** — Agent tool for reading/writing Google Sheets. Requires user-scoped OAuth credential flow (Google OAuth2 with refresh tokens). The runner service reserves a `sheets` executor type and defers the credential flow — design needed for token acquisition, storage, and per-user scoping. - [ ] **3P OAuth credential management** — Shared infrastructure for acquiring, storing, and refreshing user-scoped OAuth tokens for third-party services (Google, etc.). Needed before Google Sheets or any Google Suite integration can land. Design questions: token storage (DB vs. Secrets Manager), refresh flow (server-side background vs. on-demand), consent UI in client, multi-tenant scoping. - [ ] **Broader Google Suite tools** — Once OAuth infra and Sheets are in place, evaluate Google Drive (file access), Google Docs (document read/write), and Google Calendar as additional agent tools. --- ## Other Project Items - [ ] **S3 cleanup before GDPR user hard-deletion** — DataSource rows now cascade on user deletion (migration `20260420000000`), but `cachedResultKey` S3 objects are not cleaned up by FK cascades. Before issuing `DELETE FROM users`, application code must: (1) list the user's data sources, (2) delete their S3 cached results via `BlobStore.delete()`, (3) then delete the user row. Implement as a `GdprErasureService` or pre-delete hook. Blocked on: GDPR erasure flow design. - [ ] **SQS visibility heartbeat in runner consumer** — `apps/runner/src/worker/consumer.ts` has no `ChangeMessageVisibility` heartbeat. Currently relies on a 600s SQS visibility timeout (set in Terraform). Add a periodic heartbeat (~60s) using `ChangeMessageVisibilityCommand` during pipeline execution to decouple infra timeout from app behavior and prevent duplicate execution if job duration ever exceeds the timeout. - [ ] **Model capabilities to DB** — Move `ALLOWED_MODELS` + `supportsThinking` from constants to DB `models` table. - [ ] **Docker smoke tests** — Run smoke tests after Docker Compose local environment implementation. - [ ] **Archive Notion pages** — Add "moved to repo" banners to legacy Notion docs. - [ ] **Move core pillars to committed docs** — Promote `docs/superpowers/specs/2026-05-12-core-pillars-design.md` to a durable location (e.g., `docs/architecture/pillars.md` or a section in `docs/architecture/overview.md`). Four pillars: Discovery, Knowledge, Memory, Execution. Platform as cross-cutting. See spec for full definitions, package mappings, investment directions, and pillar relationships. - [ ] **[ACC-10043] Explore gRPC/proto pattern for `api/`** — The search service uses ConnectRPC (proto as source of truth, types generated and committed). Evaluate whether `api/` (the server's REST/SSE contract) should adopt the same pattern. If yes, proto definitions would live in `packages/server-api/`. Ticket: [ACC-10043](https://theorchard.atlassian.net/browse/ACC-10043) - [ ] **Shared rate limiting in `common/`** — Extract rate limiting from `server/` into `common/` with a `Throttler` strategy interface. Two strategies: sliding window (current) + token bucket. Each strategy owns its own Redis Lua scripts with in-memory fallback. Both `server/` and `search/` use a thin `express-rate-limit` Store adapter. Server's `RateLimitStore` interface loses rate limiting methods (back to pure cache). Approach agreed: strategy pattern in `common/`, `express-rate-limit` middleware, each strategy self-contained with Lua scripts + ioredis client. - [ ] **Standardize on Zod v4 across the project** — `apps/server`, `apps/search`, `apps/runner`, and `packages/core-api` use `zod/mini`; `apps/platform` uses `zod` (standard). Converge on one entry point. `zod/mini` is preferred for new code (functional composition, smaller bundle). Migration: rewrite each `import { z } from "zod"` to `import * as z from "zod/mini"`, replace method chaining (`.string().optional().default()`) with functional composition (`z.pipe()` + `z.transform()`). See `apps/server/src/utils/env.ts` for the established helpers (`str`, `optInt`, `optBool`, `optEnum`). - [ ] **Surface rate limit info in DataClient** — `ApiClient` now returns `ApiResponse` with `rateLimit: { limit, remaining, reset }` on every response, but `DataClient` unwraps `.body` and discards the rate limit info. Add an `onRateLimit` callback to `DataClient` that fires after every API call, so the React frontend can display remaining quota or back off proactively. 3P consumers use `ApiClient` directly and already get full rate limit visibility. - [ ] **Client-side circuit breaker for RPC clients** — `SearchClient`, `AccessClient`, and `RunnerClient` have no circuit breaker or throttling. When internal services are near capacity, the server fires all outbound calls simultaneously — they all fail with `resource_exhausted` and 3P requests get silently degraded. Wrap `callRpc` (or each client) with a `CircuitBreaker` from `@coda/async` so the server backs off when a downstream service is struggling, returns fast failures instead of waiting for timeouts, and recovers automatically when the service is healthy. Different from HTTP rate limiting — this is internal back-pressure, not quota enforcement. - [ ] **Search hardening** — IP rate limiting (using shared `common/` rate limiter), ReportUsage hardening (Redis-backed query_id validation: caller IP match, subset check, dedup), audit logging (structured Pino: source IP, RPC name, query_id, timestamp), raise client timeout 500ms→3000ms, server-side search tool fallback tests. Requires connecting search service to shared Redis. - [ ] **RDS Proxy for connection pooling** — Provision RDS Proxy (`terraform-rds-proxy` or custom resource in `terraform-infra/qa/ows-coda/`) for Aurora MySQL connection multiplexing. Trigger: multiple services sharing the same Aurora cluster (server, search, runner, platform) with parallel query patterns increasing per-request connection usage. Benefits: absorbs Fargate task churn, transparent Aurora failover, optional IAM auth. Design documented in `docs/operations/platform-infrastructure.md` (Future: RDS Proxy section). ~$22/month for db.r6g.large. - [ ] **Blue-green deployments (CodeDeploy)** — Start with search service (custom Terraform, not behind Fargate module). Add `deployment_controller { type = "CODE_DEPLOY" }` to ECS service, CodeDeploy Application + Deployment Group, IAM service role, AppSpec YAML. Gate behind `var.enable_search_codedeploy_blue_green` (default false). Jenkins: replace `fargateDeploy` with `aws codedeploy create-deployment` + status polling. Main Fargate service blocked on `terraform-fargate` module (v6.4.2 has no CodeDeploy support — needs module upgrade or fork). Key risk: GPU warm-up (30-120s) during traffic shift; use AllAtOnce strategy, not canary. Full analysis: see conversation notes from 2026-03-26. --- ## Greenfield Reviews - [ ] **AI agent loop** — `apps/server/src/ai/`: tool dispatch, context building, streaming, conversation loop - [ ] **Search subsystem (round 2)** — 2026-04-28: 15 prioritized findings across abstractions, structure, performance, and test quality. Spec: `docs/superpowers/specs/greenfield-review-search-service.md` - [ ] **Sandbox/Runner** — `packages/sandbox/` + `apps/runner/`: isolated-vm, code execution pipeline - [ ] **Client** — `apps/client/`: React components, state management, admin UI --- ## Cache Greenfield Review Follow-ups ### Open - [ ] **Pipeline sequential Redis round-trips** — `setConversation` (2 sequential `set` calls), metadata create/update (`hset` + `expire`), and the full `appendMessages` flow have unnecessary round-trips. Requires either adding pipeline/mset to `CacheStore` (re-couples to Redis semantics) or internal `RedisCacheStore` optimization. Performance follow-up. - [ ] **Register schema caches on AppLocals** — `SnowflakeSchemaCache` and `GraphQLSchemaCache` are module-level singletons invisible to the DI container. Move to `AppLocals` typed fields, remove free-function wrappers. Improves test isolation and per-request swappability. --- --- ## Search Service Greenfield Review (2026-04-28) Source: [greenfield review](superpowers/specs/greenfield-review-search-service.md) ### Immediate - [ ] **`matchGlossary` parameter to `ReadonlySet`** — One-line fix. `packages/search/src/glossary.ts:74-75` builds `new Set(candidateIds)` on every query, but both callers already pass a `Set`. O(n) allocation on the hot path, called twice per search. Change parameter type, delete the copy. - [ ] **`vectorCount`/`documentCount` getters on `HybridSearch`** — `search-engine.ts:519-526` calls `toSnapshot()` after every refresh just to recount docs/vectors, allocating three corpus-sized arrays. `QuantizedHnswIndex.size` already provides O(1) vector count. Expose getters, replace the counting block. - [ ] **Test `allDown` circuit-breaker → 503 health branch** — `server.ts:129` returns 503 when both embed and query circuit breakers are open, but no test covers this path. Highest-impact coverage gap. Two new test cases in `server.test.ts`. ### Next iteration - [ ] **Split `SnowflakeAdapter` into focused classes** — Implements 3 interfaces simultaneously (`DocumentTransformer`, `GraphBuilder`, `GlossaryProvider`), holding mutable FK graph state alongside stateless projection. Split into `SnowflakeTransformer` (transform), `SnowflakeFkGraphBuilder` (graph + state), promote `SnowflakeDocumentBuilder` as glossary provider. Mechanical split, small effort. - [ ] **Unify `searchGraphQLHandler` neighborhood handling** — `search-graphql.ts:35` casts `SchemaIndex` → `SearchEngine` and calls `getGraph()` directly, bypassing pipeline `graphDepth`/`graphMaxRelated` config that Snowflake uses via `related`. Add neighborhoods to `SearchPipelineResult`, remove the cast, unify both handlers. Medium effort. - [ ] **Batch Snowflake column IN clauses** — `sql-builders.ts:81-101` generates unbounded `(TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME) IN (...)` tuples. A schema migration touching 200+ tables produces a 200-tuple IN clause. Batch into chunks of ~500 or fall back to database-scoped query for large diffs. Low effort. - [ ] **Extract shared `GlossaryIndex`** — `GlossaryExpander` and `GlossaryMatchStage` independently maintain identical glossary entries + regex caches, and both call `matchGlossary()` separately per query. Extract a `GlossaryIndex` value object owning the compiled state, run matching once per query via `StageContext`. Medium effort. - [ ] **Test `engine-factory.ts`** — Entirely untested. Key risk paths: partial engine failure (`Promise.allSettled` swallows errors), skip-when-absent (no `snowflakeAccount`), snapshot key prefix construction. ~3 hours. - [ ] **Complete `introspect.ts` refactor** — Does HTTP fetch, schema state building, AND re-exports symbols from `schema-parser`/`schema-diff`/`types` "for backward compatibility." Delete re-exports, extract `buildSchemaState` to `schema-builder.ts`. Small effort. ### Improvement - [ ] **Split `schema-loader.ts`** — 377 lines owning type definitions, raw SQL row types, pure row parsers, async orchestration, and re-exports. Split into `types.ts`, `row-parsers.ts`, keep `schema-loader.ts` for orchestration only. Medium effort. - [ ] **Replace `SearchServiceContext` parallel maps with `EngineRegistry`** — Five parallel `Map` keyed by engine name (`engines`, `fetchers`, `transformers`, `glossaryProviders`, `snapshotManagers`). Replace with single `EngineRegistry` holding `EngineRecord` bundles. Medium effort. - [ ] **Multi-source BFS in `computeJoinPaths`** — `signals/joinPath.ts:36-99` runs one independent BFS per result node (up to 40 BFS runs). `computeProximitySignal` already uses multi-source BFS. Refactor to seed all `fromIds` simultaneously — one O(V+E) traversal instead of O(|fromIds| × (V+E)). Medium effort. - [ ] **Add retry to `_embedPhase2`** — If phase-2 embedding fails, `degraded` stays `true` forever with no retry. The `degraded` flag conflates "still computing" and "permanently failed." Add `embeddingFailed` flag and simple retry with backoff. Medium effort. - [ ] **Remove `EngineBundle.pollingHandle` mutable field** — Lifecycle concern mixed into a data object. Have `startEnginePolling` return a handle that `main()` stores in a local map. Low effort. - [ ] **Test admin handler stubs and error paths** — `listSnapshots`, `rollbackSnapshot`, `updateGlossary` have no dedicated tests. ~3 hours for four new test files. - [ ] **Delete `health-lifecycle.integration.test.ts`** — Near-identical to `server.test.ts`. Port its one unique assertion (`body.circuitBreakers`), then delete. - [ ] **Test `introspectWithHash` HTTP error paths** — No tests for HTTP 500, network failure, or malformed JSON. ~1 hour. - [ ] **Test `SnowflakeFetcher.diff()` before `fetch()`** — Undefined behavior path (internal `_state` is null). ~20 min. - [ ] **Test `exec-sql` `pool.acquire` failure** — No test for pool exhaustion. ~20 min. - [ ] **Test `GraphQLFetcher` concurrent `diff()` and abort signal** — ~45 min. - [ ] **`getKeywords()` re-tokenizes pre-computed fields** — `snowflake-adapter.ts:78-128` re-splits name/db/schema/comment on every index add, ignoring pre-computed `doc.keywords`. Use the pre-computed field for base tokens. Low effort. - [ ] **Strip dead validation from `updateGlossary` stub** — Validates source existence and entry count, then unconditionally throws `Unimplemented`. Remove the validation or implement the handler (the engine refresh path already supports glossary updates). Trivial. - [ ] **Fix `stream-events.test.ts` off-by-one** — Comment says "events 0 and 1 were dropped" but assertion checks `op-3`. Add explicit `expect(remaining.length).toBe(999)` guard. ~15 min. --- ## API & CLI - [ ] **Pull upstream master into `COD-92_phase_2` and resolve conflicts** — spec written at `docs/superpowers/specs/2026-04-13-api-cli-design.md`, user review pending before implementation planning - [ ] **User review of API/CLI design spec** — review spec, then proceed to implementation plan via `superpowers:writing-plans` --- ## MCP Server (from PR 199 review, 2026-05-10) Source: [comparison doc](reference/mcp-server-comparison.md) Round 7 review ### Remaining review items - [ ] **MCP-layer rate limiting** — The MCP spec says servers MUST rate-limit tool invocations. Express has rate limiting downstream (100/min) but the MCP process has no protection. Use `TokenBucketThrottler` from `@coda/async` (already in monorepo). Add `RateLimiter` interface to `ToolExecutionContext` (same DI pattern as `ConcurrencyLimiter`), `tryAcquire()` before semaphore in `handleToolCall`, reject immediately with `[RATE_LIMITED]` error code + `retryAfterMs` hint. New env var `CODA_MCP_RATE_LIMIT` (default 60 calls/min). ~2-3h. - [ ] **Chunked response body size cap** — 50MB `Content-Length` guard in `http-client.ts` but chunked responses without `Content-Length` bypass the check. Add streaming accumulation limit. ~1-2h. - [ ] **Permission-aware MCP tool filtering via platform service** — MCP server exposes a flat tool list to all users. Use the platform service (auth, tenancy, RBAC) to resolve the user's effective permissions at MCP startup, then filter the tool list so users only see tools they're authorized to use. Server-side enforcement remains as a backstop. First use case: expose `AdminService.traceQuery` (search service) as an admin-gated MCP tool. Relates to enterprise permissions design (Phases 1-3 merged, remaining work in greenfield WS-2/3/6). --- ## Evals (from search quality audit, 2026-05-15) Source: eval query analysis of 76 Lorelai questions, reranker bug discovery, agent hallucination debugging ### Layer 1: Search Ranking Evals - [x] **Automated search ranking regression tests** — `apps/search/benchmarks/regression.ts`. Connects to live QA search service via `TraceQuery` admin RPC, runs golden queries, asserts `mustRank`/`mustInclude`/`mustNotInclude` constraints, computes MRR, and fails if regression exceeds threshold (default 0.05). Supports `--tag`, `--url`, `--threshold`, `--verbose` flags. Next steps: wire into Jenkins as a post-deploy gate (conditional on `SEARCH_CHANGED`), add nightly cron schedule. - [ ] **Admin UI regression dashboard** — Surface regression test results in the search admin UI. Display per-query pass/fail, MRR trends over time (line chart), rank delta heatmap, and deep-link to `TraceQuery` details for failed queries. Data source: store regression run results (MRR, per-query ranks, timestamp) in S3 or a DB table after each CI run. Admin page reads the latest N results and renders the dashboard. ### Layer 2: Agent End-to-End Evals - [ ] **Agent behavior evals via Langfuse datasets** — highest ROI. Define (question, expected behavior) pairs using the 76 eval CSV questions as seed data. Each eval checks: `mustCallTools` (right tools called), `mustNotSay` (no hallucinated errors like "not accessible"), `responseContains` (answer has relevant data), `sqlMustReference` (correct tables in generated SQL). Use Langfuse's dataset/eval feature (already deployed) or a custom harness that sends questions to the chat API and asserts against the tool-call chain. - [ ] **LLM-as-judge for answer quality** — for subjective quality ("is this answer helpful?"), use a second LLM to grade the agent's response against the question + ground truth data from the eval CSVs. ### Layer 3: Regression Detection - [ ] **Datadog monitor: query.rerank.uniform** — alert on WARN-level `query.rerank.uniform` events. Should never appear after the reranker fix; recurrence means the reranker is broken again. - [ ] **Glossary target freshness check** — periodic diff of glossary targets against `SNOWFLAKE.ACCOUNT_USAGE.TABLES`. The manual check we ran (2026-05-15) found 4 stale DBT\_ targets. Automate as a scheduled job or pre-deploy check. - [ ] **Smoke tests on nightly schedule** — wire `RUN_SMOKE_TESTS` Jenkinsfile param into a nightly cron job. Currently requires manual trigger. _Last updated: 2026-05-17 (auto-naming rate limit, retry circuit breaker, metadata guards, tool discovery fix, regression test script)_