# 01 — Keyword Search

The minimal setup: fetch data, transform it for indexing, and search with BM25.

## What you'll learn

- **SchemaFetcher** — provides raw data to the engine (`fetch()` for full load, `diff()` for incremental)
- **DocumentTransformer** — adapts raw records into searchable documents with weighted keyword fields
- **SearchEngine** — orchestrates the full lifecycle: init, search, destroy
- **SnapshotPersistence** — stores index state between restarts (in-memory here, S3 in production)
- **tokenize()** — the built-in NLP pipeline (camelCase split, lowercase, stop words, Porter stemming)

## Key concepts

### BM25 scoring

BM25 (Okapi Best Match 25) is the standard keyword ranking algorithm. It scores documents by:

1. **Term frequency** — repeated query terms boost the score, with diminishing returns
2. **Inverse document frequency** — rare terms count more than common ones
3. **Length normalization** — longer documents are penalized slightly

No embedding model, no GPU, no async warmup. Keyword search is available immediately after `init()`.

### Keyword fields with boost

`getKeywords()` returns an array of `KeywordField` objects. Each field has tokens and an optional `boost` multiplier:

```ts
{ tokens: tokenize(doc.name), weight: 3 }    // table name — strong signal
{ tokens: tokenize(doc.comment), weight: 2 }  // description — moderate signal
{ tokens: doc.columns.flatMap(c => tokenize(c)) }  // columns — base weight (1)
```

Higher boost means matches in that field contribute more to the BM25 score.

## Running

```bash
npx tsx examples/01-keyword-search/main.ts
```

## Expected output

```
Indexed 5 tables

Query: "revenue"
  0.0769  ANALYTICS.PUBLIC.REVENUE_SUMMARY
  0.0370  ANALYTICS.PUBLIC.ARTIST_CATALOG
  0.0357  ROYALTY.PUBLIC.CONTRACT

Query: "artist genre"
  0.0755  ANALYTICS.PUBLIC.ARTIST_CATALOG
  0.0385  ANALYTICS.PUBLIC.REVENUE_SUMMARY
  0.0357  ROYALTY.PUBLIC.CONTRACT

Query: "payment account"
  0.0742  ANALYTICS.PUBLIC.REVENUE_SUMMARY
  0.0729  ROYALTY.PUBLIC.PAYMENT
  0.0728  ROYALTY.PUBLIC.CONTRACT
```