# 05 — Filters and Pagination

Builds on [01-keyword-search](../01-keyword-search/) by adding query-time filtering and cursor-based pagination.

## What's new

- **SearchFilter** — include/exclude patterns and arbitrary predicates applied at query time
- **Keyset pagination** — iterate through results page-by-page using a `(score, id)` cursor

## Key concepts

### SearchFilter

Filters are applied _after_ scoring but _before_ returning results. They prune the ranked list without affecting scores — a filtered-out document doesn't change the ranking of remaining documents.

Three filter modes, composable:

```ts
// Prefix pattern — include only ANALYTICS tables
{
  include: ["ANALYTICS.*"];
}

// Prefix pattern — hide all DEV tables
{
  exclude: ["DEV.*"];
}

// Arbitrary logic — only tables with > 1,000 rows
{
  predicate: (doc) => (doc as Table).rowCount > 1_000;
}
```

Patterns use `.*` for prefix matching (e.g., `"ANALYTICS.*"` matches `"ANALYTICS.REVENUE_SUMMARY"`). Without `.*`, it's an exact match.

Pass the filter as the 4th argument to `engine.search()`:

```ts
const results = await engine.search("revenue", { limit: 10, filter });
```

### Keyset pagination

The engine uses cursor-based (keyset) pagination, not offset-based. Pass the last result's score and ID as the `after` cursor:

```ts
let cursor: { score: number; id: string } | undefined;

while (true) {
  const { results, scores } = await engine.search("query", {
    limit: pageSize,
    after: cursor,
  });
  if (results.length === 0) break;

  // Process this page...

  // Advance cursor to last item
  const last = results.length - 1;
  cursor = { score: scores[last]!, id: results[last]!.fqn };
}
```

Keyset pagination is stable — concurrent index changes don't cause items to shift between pages. It's also O(1) to skip to any page (no re-scanning earlier results).

## Running

```bash
npx tsx examples/05-filters-and-pagination/main.ts
```

## Expected output

```
Indexed 8 tables

=== Unfiltered: "revenue" ===
  0.0755  ANALYTICS.REVENUE_SUMMARY
  0.0697  DEV.TEST_REVENUE
  0.0370  ANALYTICS.ARTIST_CATALOG
  0.0357  ANALYTICS.TERRITORY_MAP
  0.0345  ROYALTY.CONTRACT

=== Include ANALYTICS.*: "revenue" ===
  0.0755  ANALYTICS.REVENUE_SUMMARY
  0.0370  ANALYTICS.ARTIST_CATALOG
  0.0357  ANALYTICS.TERRITORY_MAP

=== Exclude DEV.*: "revenue" ===
  0.0755  ANALYTICS.REVENUE_SUMMARY
  0.0370  ANALYTICS.ARTIST_CATALOG
  0.0357  ANALYTICS.TERRITORY_MAP
  0.0345  ROYALTY.CONTRACT
  0.0333  ROYALTY.PAYMENT

=== Predicate (rowCount > 1000): "account" ===
  0.0729  ANALYTICS.REVENUE_SUMMARY  (50,000 rows)
  0.0718  ROYALTY.PAYMENT  (200,000 rows)
  0.0702  ROYALTY.CONTRACT  (5,000 rows)
  0.0370  ANALYTICS.ARTIST_CATALOG  (10,000 rows)
  0.0323  ROYALTY.STATEMENT  (30,000 rows)

=== Pagination: "account" (page size = 2) ===
  Page 1:
    0.0729  ANALYTICS.REVENUE_SUMMARY
    0.0718  ROYALTY.PAYMENT
  Page 2:
    0.0702  ROYALTY.CONTRACT
    0.0683  DEV.TEST_REVENUE
  Page 3:
    0.0370  ANALYTICS.ARTIST_CATALOG
    0.0357  ANALYTICS.TERRITORY_MAP
  Page 4:
    0.0323  ROYALTY.STATEMENT
    0.0303  DEV.SCRATCH_PAD
```