# File Splits & Deduplication -- TRD

## Status

**Implemented.** All four tasks from the plan have been merged (commits `e8b2c01`, `7fcb582`, `5762718`, `e14374a`). Three file splits were evaluated and explicitly deferred. This TRD documents the rationale, design, and outcomes.

## Overview

Targeted refactoring to improve codebase maintainability: extract a shared `tokenize` utility (eliminating duplicate code in two packages), split an oversized 507-line `schema-index.ts` into three focused modules, replace O(n^2) tool-result scans with O(1) Map lookups in the orchestrator hot path, and extract a `useConversationMutations` hook from `chat-provider.tsx`.

The codebase review on 2026-03-20 identified five files exceeding 400 lines and two categories of duplicate infrastructure. The plan scoped four tasks for immediate action and explicitly deferred three file splits where the cost/benefit ratio was unfavorable.

## Goals

1. **Reduce file sizes toward a 300-line target.** Large files impose a cognitive tax on both human reviewers and AI-assisted development tools. Files over 400 lines require scrolling to understand, make diffs harder to review, and increase the odds of merge conflicts when multiple developers touch the same file.

2. **Eliminate duplicate code.** Two independent `STOP_WORDS` constant definitions and two `tokenize()` function implementations drifted apart over time (one supported camelCase splitting, the other did not). A single source of truth prevents future drift.

3. **Improve hot-path performance.** The orchestrator's per-round tool-result matching used `.find()` inside a loop -- O(n \* m) where n = tool uses and m = tool results. With up to 5 parallel tool calls per round and 15 rounds per request, this compounds.

4. **Preserve all existing behavior.** Every change is a pure refactor or performance optimization. No public API changes, no new features, no behavioral changes.

## Architecture

### Before (2026-03-20 codebase review snapshot)

```
server/src/ai/tools/
  catalog.ts .................. 210 lines (included STOP_WORDS + tokenize)
  graphql/
    schema-index.ts ........... 507 lines (introspection + search + state + lifecycle)

server/src/ai/
  orchestrator.ts ............. ~490 lines (toolResults.find() at line 406)
  auto-batch.ts ............... ~87 lines  (toolResults.find() at line 66)

server/src/routes/
  stream-handler.ts ........... ~348 lines (toolResults.find() at line 232)

client/src/providers/
  chat-provider.tsx ........... 427 lines  (mutations inline)
```

### After (current)

```
server/src/ai/utils/
  tokenize.ts ................. 119 lines  (shared STOP_WORDS + tokenize)
  __tests__/tokenize.test.ts .. dedicated test suite

server/src/ai/tools/
  catalog.ts .................. 140 lines  (imports from shared tokenize)
  graphql/
    introspect.ts ............. 308 lines  (schema fetching + index building)
    search.ts ................. 288 lines  (scoring, type lookup, hybrid search)
    schema-index.ts ........... 215 lines  (thin facade: state + lifecycle + re-exports)

server/src/ai/
  orchestrator.ts ............. 490 lines  (Map lookup at line 401)
  auto-batch.ts ............... 87 lines   (Map lookup at line 62)

server/src/routes/
  stream-handler.ts ........... 348 lines  (Map lookup at line 238)

client/src/providers/
  chat-provider.tsx ........... 384 lines  (delegates to mutations hook)
  use-conversation-mutations.ts 144 lines  (extracted mutation logic)
```

### Line count summary

| File (before)       | Lines before | File(s) after                                         | Lines after                                            | Delta                                              |
| ------------------- | ------------ | ----------------------------------------------------- | ------------------------------------------------------ | -------------------------------------------------- |
| `schema-index.ts`   | 507          | `introspect.ts` + `search.ts` + `schema-index.ts`     | 308 + 288 + 215 = 811 total (largest single file: 308) | Largest file reduced by 199 lines (39%)            |
| `catalog.ts`        | ~210         | `catalog.ts` + `tokenize.ts` (shared)                 | 140 + 119 = 259 total (largest: 140)                   | Reduced by 70 lines (33%)                          |
| `chat-provider.tsx` | 427          | `chat-provider.tsx` + `use-conversation-mutations.ts` | 384 + 144 = 528 total (largest: 384)                   | Reduced by 43 lines (10%); mutation logic isolated |

Note: total line counts increase when splitting files because of added imports, exports, type re-exports, and module documentation headers. The metric that matters is the **largest single file** a developer must hold in working memory, which decreased in every case.

## Detailed Design

### Task 1: Shared tokenize utility

**Problem.** `catalog.ts` and `schema-index.ts` each defined their own `STOP_WORDS` set and `tokenize()` function. The schema-index version included camelCase boundary splitting (`replace(/([a-z])([A-Z])/g, "$1 $2")`); the catalog version did not. The STOP_WORDS sets had minor differences.

**Solution.** Extracted to `server/src/ai/utils/tokenize.ts`:

- Merged both STOP_WORDS sets into their union (superset -- no words removed).
- Adopted the camelCase-splitting variant as the canonical implementation. This is strictly additive: queries like `"abacusContract"` now produce tokens `["abacus", "contract"]` in the catalog too, improving recall without harming precision for multi-token queries.
- Both consumers (`catalog.ts`, `schema-index.ts`) now import from the shared module.

**Why a separate `utils/` directory?** The tokenize function is domain-agnostic text processing. It does not belong in either the tool catalog or the GraphQL schema index. Placing it in `ai/utils/` makes it available to any future search/scoring module (and it is already consumed by the Snowflake schema index as well).

### Task 2: schema-index.ts split

**Problem.** At 507 lines, `schema-index.ts` combined four distinct responsibilities: GraphQL introspection (HTTP fetch + schema parsing), index building (transforming schema types into searchable entries), search scoring (keyword matching + ranking), and module lifecycle (state management, warm-up, lazy init).

**Solution.** Split along natural responsibility boundaries:

- **`introspect.ts` (308 lines)** -- All type interfaces (`QueryFieldEntry`, `TypeEntry`, `TypeDetail`, `FieldDetail`, `QueryFieldArg`), schema parsing helpers (`formatType`, `formatArgSignature`, `buildQueryFieldEntry`, `buildTypeEntry`, `fieldToDetail`, `inputFieldToDetail`, `buildTypeDetail`), and `introspectAndIndex()`. Returns a `SchemaState` object rather than writing to module-level variables -- the caller owns state.

- **`search.ts` (288 lines)** -- Pure search functions that accept state as parameters: `searchSchema(queryFieldIndex, typeIndex, query, limit)`, `getTypeInfo(typeMap, typeName)`. Also contains the hybrid search adapter (glossary loading, embedding index building, graph construction). No module-level mutable state beyond the glossary cache and search indexes.

- **`schema-index.ts` (215 lines)** -- Thin facade that owns the five module-level state variables (`queryFieldIndex`, `typeIndex`, `typeMap`, `loaded`, `initPromise`) and lifecycle functions (`warmUpSchema`, `ensureSchemaLoaded`, `isSchemaLoaded`, polling). Delegates to `introspect.ts` and `search.ts`. Re-exports type interfaces so all existing `import { ... } from "./schema-index"` statements continue to work with zero consumer changes.

**Key design decision:** The facade pattern was chosen over a getter/setter injection pattern (considered and rejected in the plan) because it preserves the existing public API exactly. No mock changes were needed in `handlers.test.ts`.

### Task 3: Map lookup optimization

**Problem.** Three call sites matched tool results to tool uses via `toolResults.find(r => r.toolUseId === toolUseId)` inside a loop over tool uses. This is O(n \* m) per round.

**Solution.** At each call site, build a `Map<string, ToolResult>` from the results array once, then use `.get()` for O(1) lookups:

```ts
const resultsByToolUseId = new Map(allToolResults.map((r) => [r.toolUseId, r]));
// Inside loop:
const result = resultsByToolUseId.get(tu.toolUseId);
```

Applied in:

- `orchestrator.ts:401` -- main orchestration loop
- `auto-batch.ts:62` -- auto-batch trigger detection
- `stream-handler.ts:238` -- persistence callback

**Note:** One remaining `.find()` call exists in `selection-emitter.ts:50`. This was not in scope because (a) it was not identified in the original review, and (b) the loop iterates over only the search-type tool uses (at most 2 entries in `SEARCH_TOOL_META`), so the O(n) scan is negligible.

### Task 4: useConversationMutations extraction

**Problem.** `chat-provider.tsx` at 427 lines contained ~120 lines of conversation mutation functions (`deleteConversationFn`, `renameConversation`, `toggleStarConversation`, `bulkDeleteConversationsFn`) that are logically independent of the provider's core responsibilities (context creation, message state, streaming orchestration).

**Solution.** Extracted to `client/src/providers/use-conversation-mutations.ts` (144 lines). The hook accepts its dependencies as a parameter object and returns the four mutation functions. `chat-provider.tsx` calls the hook and passes the returned functions into the context value.

## Alternatives Explored

### Why not a full module rewrite?

The codebase review identified these as P2 issues -- code quality improvements, not correctness bugs. A full rewrite would risk introducing regressions in production code that is currently working correctly. Targeted splits along existing responsibility boundaries minimize risk.

### Why these files first?

The plan prioritized files where (a) the split follows obvious responsibility boundaries, (b) the public API can remain unchanged, and (c) existing tests provide a safety net. `schema-index.ts` at 507 lines was the clearest win: three distinct responsibilities, pure functions that can be extracted without shared mutable state.

### Why defer three splits?

Three files from the codebase review's "Files to Split" table were explicitly deferred:

1. **`api/src/client.ts` (744 lines)** -- Despite its size, this file is a single well-organized class where methods are thin wrappers around fetch calls. Splitting into `ApiClient`, `StreamQueryRunner`, and `RetryFetcher` would add three files and cross-file imports with no reduction in cognitive load per individual method. The class is the natural unit of understanding.

2. **`client/src/hooks/use-coda-orchestrator.ts` (608 lines)** -- Tight coupling between session state and callbacks makes extraction risky. The `handleSend` closure captures 15+ dependencies; splitting would require either a complex shared-state interface between hooks or prop-drilling that is worse than the current single file.

3. **`client/src/offline/coda-store.ts` (472 lines)** -- IndexedDB transactions span stores; splitting into `MessageStore`, `ChatStore`, `BlobStore` would require careful transaction coordination to avoid breaking atomicity guarantees. The risk is disproportionate to the benefit.

## Cost Analysis

| Category            | Cost                                                                |
| ------------------- | ------------------------------------------------------------------- |
| Engineering effort  | ~2-3 hours implementation (4 tasks)                                 |
| Infrastructure cost | Zero -- pure refactoring, no new services or dependencies           |
| Review burden       | Low -- each task is a self-contained commit with clear before/after |
| Migration cost      | Zero -- no public API changes, no consumer updates required         |

**Benefits:**

- Reduced cognitive load: largest server-side file dropped from 507 to 308 lines (39% reduction)
- Eliminated 2 instances of duplicate infrastructure (STOP_WORDS, tokenize)
- Improved hot-path performance in the orchestrator loop
- Better testability: `introspect.ts` and `search.ts` export pure functions that can be tested without mocking module state

## Performance Analysis

### Map lookup optimization

The `.find()` to `Map.get()` change affects the orchestrator's per-round performance:

| Metric                               | Before (`.find()`)                         | After (`Map.get()`)                                     |
| ------------------------------------ | ------------------------------------------ | ------------------------------------------------------- |
| Time complexity per round            | O(n \* m) where n = tool uses, m = results | O(m) for Map construction + O(n) for lookups = O(n + m) |
| Typical n                            | 1-5 tool uses per round                    | Same                                                    |
| Typical m                            | 1-5 tool results per round                 | Same                                                    |
| Worst case (15 rounds, 5 tools each) | 75 \* 75 = 5,625 comparisons               | 75 Map insertions + 75 lookups = 150 operations         |

At typical scale (1-5 tools), the absolute time savings are negligible (sub-microsecond). The value is in correctness of algorithmic complexity -- if tool counts grow (e.g., parallel tool execution expansion), the Map approach scales linearly rather than quadratically.

### Tokenize unification

Adding camelCase splitting to the catalog's tokenize path adds one `.replace()` call per tokenization. This is a negligible cost (single regex pass over short strings) offset by improved search recall.

## Scaling Characteristics

**Smaller files improve AI-assisted development.** LLM context windows and code-generation tools perform better with focused, single-responsibility files. A 300-line file fits entirely in a single prompt context; a 500+ line file may require chunking or selective reading, reducing comprehension accuracy.

**Smaller files reduce merge conflicts.** With multiple developers working on the codebase, files that combine unrelated responsibilities are merge-conflict magnets. After splitting `schema-index.ts`, changes to introspection logic no longer conflict with changes to search scoring.

**Shared utilities prevent drift.** The duplicated `tokenize()` implementations had already diverged (camelCase splitting present in one, absent in the other). A single source of truth ensures future improvements benefit all consumers.

## Breakdown Points & Mitigations

| Risk                                                | Likelihood | Impact | Mitigation                                                                                                                                                    |
| --------------------------------------------------- | ---------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Import path changes breaking consumers              | Low        | Medium | Facade re-exports all types and functions -- existing imports from `"./schema-index"` continue to work unchanged                                              |
| Circular dependencies after split                   | Low        | High   | Dependency flows one direction: `schema-index` -> `introspect` + `search`. No back-imports. `search.ts` imports types from `introspect.ts` but not vice versa |
| Tight coupling preventing clean extraction          | Medium     | Low    | Addressed by the facade pattern: `schema-index.ts` owns state, passes it to pure functions. No shared mutable state between `introspect.ts` and `search.ts`   |
| camelCase splitting changing catalog search results | Low        | Low    | Strictly additive -- produces a superset of tokens. Multi-token queries may see improved recall; no results are removed                                       |
| Merged STOP_WORDS set filtering too aggressively    | Low        | Low    | Union of both sets -- no new words added that were not already filtered in at least one consumer                                                              |

## Decision Log

| Decision                                        | Rationale                                                                                        | Date       |
| ----------------------------------------------- | ------------------------------------------------------------------------------------------------ | ---------- |
| Defer `api/src/client.ts` split                 | Single well-organized class; splitting adds indirection with no behavioral benefit               | 2026-03-20 |
| Defer `use-coda-orchestrator.ts` split          | `handleSend` closure captures 15+ dependencies; splitting creates complex shared-state interface | 2026-03-20 |
| Defer `coda-store.ts` split                     | IndexedDB transactions span stores; splitting risks breaking atomicity                           | 2026-03-20 |
| Use facade pattern for schema-index             | Preserves public API, avoids consumer changes, no mock updates in tests                          | 2026-03-20 |
| Adopt camelCase splitting in shared tokenize    | Strictly better recall; schema-index already had it, catalog gains it                            | 2026-03-20 |
| Do not convert `selection-emitter.ts` `.find()` | Loop iterates at most 2 entries; optimization is unnecessary                                     | 2026-03-20 |

## Dependencies

- No new runtime dependencies introduced.
- No infrastructure changes required.
- The shared `tokenize.ts` module is consumed by both the GraphQL and Snowflake schema indexes, plus the tool catalog. Changes to tokenization behavior affect all three.

## Testing Strategy

All changes are pure refactors. The testing strategy is:

1. **New unit tests** for the extracted `tokenize.ts` module (`server/src/ai/utils/__tests__/tokenize.test.ts`) covering whitespace/underscore/hyphen/slash splitting, camelCase boundary detection, lowercase normalization, single-char filtering, stop-word filtering, and empty input.

2. **Existing test suites must pass unchanged.** No test modifications were required because:
   - The `schema-index.ts` facade re-exports the same public API.
   - `handlers.test.ts` mocks `"../schema-index"` which still exports `searchSchema`, `getTypeInfo`, `ensureSchemaLoaded`, etc.
   - The Map optimization changes internal implementation without affecting return values.
   - The `useConversationMutations` hook is called inside `chat-provider.tsx`, so the context value shape is unchanged.

3. **No behavioral changes to verify.** These are structural refactors -- the before and after behavior is identical by design.

## Rollout Plan

All four tasks have been implemented and merged:

| Task | Commit    | Description                                                       |
| ---- | --------- | ----------------------------------------------------------------- |
| 1    | `e8b2c01` | Extract shared tokenize utility from catalog and schema-index     |
| 2    | `7fcb582` | Split schema-index.ts into introspect, search, and facade modules |
| 3    | `5762718` | Replace O(n^2) toolResults.find() with Map lookups                |
| 4    | `e14374a` | Extract useConversationMutations hook from chat-provider          |

No feature flags, gradual rollout, or rollback plan needed -- these are pure refactors with no behavioral changes.

## Open Questions

1. **Should `selection-emitter.ts:50` also use a Map lookup?** Currently deferred because the loop body iterates at most 2 tool types. If `SEARCH_TOOL_META` grows to include more entity types, this should be revisited.

2. **Should the deferred splits be re-evaluated?** The three deferred files (`client.ts` at 744 lines, `use-coda-orchestrator.ts` at 608 lines, `coda-store.ts` at 472 lines) remain above the 300-line target. If any of these files grow further or if coupling is reduced through other refactors, they should be reconsidered.

3. **Should `schema-index.ts` facade (215 lines) be further reduced?** The facade grew from the original plan's estimate of ~80 lines to 215 lines due to polling logic, hybrid index building, and additional re-exports added during the semantic search feature. This is still well within the 300-line target.