# LLM Wiki Pattern Comparison & Applicable Ideas ## Overview This document records our analysis of [Karpathy's LLM Wiki pattern](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) (2026) against our schema discovery search service. The LLM Wiki is an architectural pattern where LLMs incrementally build and maintain persistent wikis rather than performing retrieval-augmented generation on-demand. We evaluated the pattern to identify transferable techniques, as we did with [LightRAG](../search/lightrag-comparison.md), [ChromaFs](../search/chromafs-comparison.md), and [RAG-Anything](../search/raganything-comparison.md). **Last updated**: 2026-05-10 (refreshed with live GitHub data. nashsu/llm_wiki at 6,605 stars (up from 6,586), v0.4.7 released May 6 -- rapid feature cadence with knowledge graph filters, SerpApi web search, Louvain community detection. SamurAIGPT/llm-wiki-agent at 2,546 stars (up from 2,543). @coda/search post-PR-198 with engine decomposition, graph exploration RPCs, glossary linting, progressive examples, cookbook.) --- ## How the LLM Wiki works Three-layer architecture: ### Layer 1 -- Raw sources Immutable curated documents. The LLM reads but never modifies them. ### Layer 2 -- The wiki LLM-generated markdown files: summaries, entity pages, concept pages, comparisons, synthesis documents. The LLM owns this layer entirely. ### Layer 3 -- The schema A configuration document (e.g., `CLAUDE.md`) instructing the LLM on structure, conventions, and workflows. ### Core operations - **Ingestion**: Source added -> LLM writes summary page, updates index, updates entity/concept pages (touches 10-15 pages per source). - **Querying**: LLM reads `index.md` to find relevant pages, drills in, synthesizes answer. Valuable analysis can be filed back. - **Linting**: Periodic health checks identify contradictions, staleness, orphans, missing cross-references. --- ## How our system differs | Aspect | LLM Wiki | @coda/search | | -------------------------- | --------------------------------------- | ----------------------------------------------------------------- | | **Knowledge model** | Pre-compiled (LLM synthesizes once) | On-demand (search retrieves raw schema, LLM interprets per query) | | **Corpus** | Unstructured documents | Structured schema metadata | | **Index source** | LLM-generated markdown pages | Raw schema items tokenized + embedded | | **Query latency** | Seconds (LLM reads pages + synthesizes) | ~10-15ms (in-memory BM25 + HNSW + RRF) | | **Knowledge accumulation** | Explicit: analysis filed back as pages | None: each query re-derives meaning from raw schema | | **Evaluation** | Manual review | Benchmark suite: 82 tables, 40 golden queries, NDCG/MRR/Recall | ### Why we can't adopt the pattern wholesale 1. **Latency incompatible** -- LLM Wiki assumes seconds per query; we need 10-15ms. 2. **Our corpus is already structured** -- the compilation step would re-derive existing structure. 3. **LLM-per-source cost at poll frequency** -- $0.25-0.75 per 5-minute poll cycle vs $0. 4. **We need hybrid search, not wiki lookup** -- BM25 + HNSW + RRF fusion is load-bearing. --- ## The core insight worth exploring > Knowledge should be compiled once and kept current, not re-derived on every query. Today, when the agent searches for "royalty payments," it gets raw `SnowflakeTableEntry` objects. The agent must figure out what each table _means_, which columns matter, and what joins make sense -- every time, re-deriving the same understanding. Our glossary partially addresses this but is hand-curated, static, and thin. The LLM Wiki insight suggests we should enrich what the search layer indexes and returns more aggressively. --- ## Transferable ideas ### 1. Glossary enrichment -- wiki-style entity context (low effort, medium value) Expand glossary entries with `relationships`, `gotchas`, and `related_concepts` fields. ### 2. Glossary linting -- wiki-style health checks -- IMPLEMENTED `lintGlossary()` is now exported from `@coda/search` with 5 automated check types: | Check | Description | Severity | | ------------------ | -------------------------------------------- | -------- | | **stale_target** | Glossary targets not in current index | Error | | **stale_related** | Related IDs not in current index | Warning | | **orphan_entry** | All targets missing | Error | | **coverage_gap** | High-degree documents with no glossary entry | Warning | | **duplicate_term** | Same term appears in multiple entries | Warning | ### 3. Query knowledge capture (medium effort, medium value) Wire up the `ReportUsage` stub to persist `(query, selected_ids, timestamp)` tuples. Phase 2: analyze which tables are selected together, which queries return unused results. ### 4. Pre-query catalog (low effort, low-medium value) A cached `GetCatalog` endpoint returning a structured summary of databases, tables, and domains. ### 5. Compiled entity summaries (high effort, high value) LLM or rule-based templates generating per-table summaries. Only pursue when per-query interpretation is a measured bottleneck. --- ## What we could adopt from LLM-Wiki ### Incremental knowledge layering The wiki's three-layer architecture maps onto: raw schema (Layer 1) -> enriched glossary + entity summaries (Layer 2) -> search configuration + signal weights (Layer 3). Layer 2 is the gap the LLM Wiki most clearly highlights. ### Self-healing through structured linting -- IMPLEMENTED `lintGlossary()` directly implements the LLM Wiki-style health checks proposed in this comparison. Dedicated tests in `glossary-lint.test.ts`. ### Compounding knowledge via feedback loops Our system has no feedback loop: ReportUsage is a stub, query outcomes are not captured. This is the single largest gap the LLM Wiki pattern reveals. --- ## Enterprise readiness comparison | Dimension | LLM Wiki (pattern) | nashsu/llm_wiki v0.4.7 | @coda/search | | ------------------------ | ---------------------------------------- | ------------------------------------------- | ----------------------------------------------------------------- | | **Multi-tenancy** | Single-user; no isolation | Single knowledge base per instance | Per-tenant glossary + search filters; schema-level access control | | **State management** | Stateless files; no lifecycle guarantees | Persistent ingest queue with crash recovery | State machine with two-phase init and warm-start snapshots | | **Graceful degradation** | Binary: works or doesn't | Binary: LLM available or not | Keyword-only fallback, circuit breaker, reranker fallback | | **Observability** | log.md text file | Activity panel with progress bars | 28 typed event types; OTel-compatible envelopes | | **Testing** | None built-in | Basic test coverage; no benchmark suite | ~1,386 tests; benchmark suite with NDCG/MRR regression guards | | **Cost predictability** | $0.25-0.75 per poll cycle (LLM calls) | Per-ingest LLM cost (variable) | $0 at search time (local ONNX embedding) | --- ## Developer experience comparison | Dimension | LLM Wiki (pattern) | nashsu/llm_wiki v0.4.7 | @coda/search | | ------------------ | ------------------------------------------- | ------------------------------- | --------------------------------------------------------------- | | **Setup** | Copy CLAUDE.md, create directory; 5 minutes | Download desktop app; 2 minutes | Implement 4-5 interfaces, configure engine; ~1 hour | | **Extensibility** | Edit the schema document | Scenario templates, purpose.md | Plug interfaces: SearchStage, Signal, Fusion, Expander, Fetcher | | **Learning curve** | Low -- write markdown, prompt LLM | Low -- GUI with drag-and-drop | Medium -- understand IR concepts, signals, pipeline, fusion | | **Type safety** | None (markdown + LLM reasoning) | TypeScript (Electron app) | Full TypeScript generics | --- ## Abstraction quality comparison **LLM Wiki: convention-based** -- The schema document defines naming rules, page templates, workflow steps. The LLM interprets dynamically. Powerful for flexibility, fragile for reliability. **nashsu/llm_wiki: application-layered** -- Wraps the pattern in a desktop application with a knowledge graph (4-signal relevance model: direct links 3x, source overlap 4x, Adamic-Adar 1.5x, type affinity 1x), Louvain community detection, vector search (LanceDB), and a two-step chain-of-thought ingest pipeline. Fixed-weight model, not pluggable N-signal RRF fusion. **@coda/search: interface-based** -- 12 TypeScript interfaces with single-responsibility boundaries. Enforced at compile time. Each interface has 1-3 methods. --- ## Documentation & examples comparison | Dimension | LLM Wiki (pattern) | nashsu/llm_wiki v0.4.7 | @coda/search | | ------------------ | ------------------------------------------- | -------------------------------------------- | --------------------------------------------------------------------- | | **Primary docs** | Single gist (~2,000 words) + example schema | README (~3,000 words) + Chinese translation | Architecture doc, API doc, code-style guide, comparison series | | **Usage examples** | The gist itself is the example | Desktop app with built-in scenario templates | 4 progressive examples + 8 cookbook recipes + golden query benchmarks | | **API reference** | N/A | None (desktop app, no library API) | Exported types fully documented | --- ## Scope & limitations ### LLM Wiki limitations 1. Scale ceiling at ~100 sources / hundreds of pages 2. No evaluation framework 3. Single-user assumption 4. Cost scales with corpus ($72-216/day at our scale vs $0) 5. Latency fundamentally LLM-bound ### @coda/search limitations 1. No knowledge accumulation -- each query re-derives understanding (the gap LLM Wiki exposes) 2. Glossary is static and hand-curated -- no automated evolution (though `lintGlossary()` now detects staleness and gaps) 3. No compiled context layer 4. Graph is structurally derived only -- misses implicit usage-based relationships --- ## If we started fresh, what would we adopt from this project? 1. **The knowledge compilation philosophy.** The core thesis -- compile knowledge once, keep it current -- is sound. We would adopt a richer glossary layer (Layer 2) from day one, with automated enrichment from usage patterns. 2. **The 4-signal knowledge graph from nashsu/llm_wiki.** Their relevance model (direct links + source overlap + Adamic-Adar + type affinity) is a useful complement. Adamic-Adar (shared-neighbor weighting) has been adopted as `AdamicAdarSignal`. 3. **Louvain community detection.** Automatic clustering of schema items by link topology could surface domain groupings without manual glossary curation. 4. **Graph insights (surprising connections, knowledge gaps).** Our `lintGlossary()` coverage gap check is a simpler version of this. ## What are we doing better? 1. **Retrieval precision.** 8-signal RRF fusion with NDCG=0.887, MRR=0.943. 2. **Latency.** 10-15ms vs seconds. Two orders of magnitude faster. 3. **Cost.** $0 per query and $0 per index build. 4. **Composability.** 12 pluggable interfaces, 9 sub-path exports. 5. **Enterprise infrastructure.** State machine lifecycle, two-phase init, graceful degradation, 28 typed event types, S3 snapshots, circuit breakers. 6. **Determinism.** Same query, same results. ## What are we missing for enterprise-level production readiness? 1. **Knowledge feedback loops.** ReportUsage is still a stub. 2. **Automated glossary evolution.** nashsu/llm_wiki's LLM-driven page generation suggests automated enrichment is feasible. 3. **Community/cluster detection.** No automatic grouping of schema items by structural or semantic affinity. 4. **Cross-session context.** The agent forgets between sessions. --- ## Priority and sequencing | Idea | Effort | Value | When | | ---------------------------- | ------ | ------- | --------------------------------------------------- | | 1. Glossary enrichment | Low | Medium | Now | | 2. Glossary linting | Low | Medium | **DONE** -- `lintGlossary()` exported with 5 checks | | 4. Pre-query catalog | Low | Low-Med | Soon | | 3. Query knowledge capture | Medium | Medium | After ReportUsage is wired up | | 5. Compiled entity summaries | High | High | When interpretation cost is measured | --- ## Ecosystem (May 2026) The LLM Wiki pattern has gone from a philosophical gist to a **mainstream movement** with 10+ implementations and 25K+ combined stars: | Project | Stars | Language | Description | | ------------------------------ | ----- | ---------- | ------------------------------------------------------------------------------- | | **nashsu/llm_wiki** | 6,605 | TypeScript | Cross-platform desktop app. v0.4.7. 4-signal KG, Louvain, SerpApi. Very active. | | **SamurAIGPT/llm-wiki-agent** | 2,546 | Python | Self-building personal knowledge base. Supports Claude Code, Codex, Gemini CLI. | | **sdyckjq-lab/llm-wiki-skill** | 1,374 | TypeScript | Claude Code skill for personal knowledge bases. | | **llm-wiki-compiler** | 1,083 | TypeScript | "The knowledge compiler. Raw sources in, interlinked wiki out." | | **Ar9av/obsidian-wiki** | 1,074 | TypeScript | Obsidian framework for LLM Wiki. | | **lucasastorian/llmwiki** | 835 | Python | Open source implementation with MCP/Supabase integration. | --- ## Verdict & recommendation **Adopt the philosophy, not the implementation.** The LLM Wiki pattern addresses a real gap -- knowledge accumulation and feedback loops -- but its execution model is incompatible with our constraints. The concrete path forward: 1. **Short term**: Enrich the glossary (idea #1) and wire up ReportUsage (idea #3). 2. **Medium term**: ~~Evaluate Adamic-Adar as a custom `StaticSignal`~~ DONE (`AdamicAdarSignal`). Louvain community detection remains. 3. **Long term**: Compiled entity summaries (idea #5) if per-query interpretation cost becomes a bottleneck. --- ## References - Karpathy, A.: [LLM Wiki](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f) (2026) - nashsu: [llm_wiki](https://github.com/nashsu/llm_wiki) (TypeScript, 6,605 stars, v0.4.7, 2026) - SamurAIGPT: [llm-wiki-agent](https://github.com/SamurAIGPT/llm-wiki-agent) (Python, 2,546 stars, 2026) - LightRAG comparison: [lightrag-comparison.md](../search/lightrag-comparison.md) - RAG-Anything comparison: [raganything-comparison.md](../search/raganything-comparison.md)