The user wants to create a code knowledge graph for a repository that can be passed as context to an AI agent when making code changes. They want: 1. An architecture design 2. Detailed explanation 3. Prioritization of open-source libraries Let me design a comprehensive system for this. I need to think about: 1. **What is a code knowledge graph?** - Nodes: Files, Classes, Functions, Modules, Variables, External Libraries, APIs, etc. - Edges: Calls, Imports, Inherits, Uses, Depends On, etc. - Relationships: Both syntactic (code-level) and semantic (concept-level) 2. **What context is useful for an AI agent?** - File structure - Function/class definitions and their locations - Call graphs (who calls what) - Import dependencies - Data flow - Control flow - Documentation/docstrings - Type information - Recent changes/git history - Tests - Business logic patterns 3. **Architecture components:** - Parsers/Analyzers (static analysis) - Graph database/storage - Embedding/Vector store - Query/Retrieval layer - Context packer (formats for LLM) 4. **Open-source libraries to use:** - **Tree-sitter** - Multi-language AST parser (fast, incremental) - **Language servers** (LSP) - For semantic understanding - **Neo4j / Memgraph** - Graph databases (Neo4j has free community edition) - **Apache AGE** - Graph extension for PostgreSQL - **DuckDB** - Embedded analytical database - **NetworkX** - Python graph library - **rustworkx** - Faster Python graph library by IBM - **LightRAG** - Lightweight RAG with graph support - **Microsoft GraphRAG** - Graph-based RAG - **LlamaIndex** - Has graph integration - **LangChain** - For orchestration - **Sentence-transformers** - For embeddings - **ChromaDB / Qdrant / FAISS** - Vector stores - **scikit-learn** - For clustering/community detection - **pygments** - For code tokenization - **radon** - For Python code metrics - **jedi / pylsp** - For Python - **clangd** - For C/C++ - **gopls** - For Go - **typescript-language-server** - For TypeScript - **SourceGraph's SCIP** or **LSIF** - Pre-computed code intelligence - **Chunkr / Chonkie** - Code chunking - **pycparser, javalang, go-parser** - Various parsers - **Semgrep** - For semantic grep / pattern matching - **Joern** - For code property graphs (security focus) - **Sourcetrail** - Now open-sourced? (Actually discontinued) - **Aider's repomap** - Simple but effective approach - **RepoCoder-style** approach 5. **The output should be a context graph** that can be: - Passed as text (JSON, YAML, custom format) - Used for retrieval - Compressed/summarized for LLM context window Let me think about the most practical architecture: **Layered Architecture:** 1. **Ingestion Layer** - Git clone - File traversal - Language detection 2. **Analysis Layer** - Tree-sitter for AST parsing - Language-specific analyzers - LSP integration for semantic info (symbols, references, types) - Git history mining 3. **Graph Construction Layer** - Build nodes (entities) - Build edges (relationships) - Use NetworkX/rustworkx in memory or Neo4j for persistence 4. **Embedding & Indexing Layer** - Chunk code semantically - Generate embeddings (sentence-transformers) - Store in vector DB - Create BM25/text index 5. **Retrieval Layer** - Hybrid search (vector + graph traversal) - Community detection for summarization - Path-based retrieval 6. **Context Packaging Layer** - Token-aware packing - Format as text/JSON for LLM - Include relevant snippets, signatures, relationships 7. **API/CLI Layer** - Query interface - Update mechanism Now let me think about the best open-source stack: **Recommended Stack:** 1. **Parsing**: - `tree-sitter` (multi-language, fast, used by GitHub) - `tree-sitter-language-pack` for many languages - `scip-python`, `scip-typescript`, `scip-go` for cross-language SCIP - Or use LSP via `pygls`, `multilspy` (newer), or `eclipse-lsp4j` 2. **Graph Storage**: - `Neo4j` (Community Edition) for persistent large graphs - `Kùzu` (open-source, embeddable graph DB, very fast) - `DuckDB` with graph extensions for analytical workloads - `NetworkX` for in-memory simple use - `rustworkx` (Rust-based, much faster than NetworkX) 3. **Embeddings**: - `sentence-transformers` (HuggingFace) - Models: `BAAI/bge-code-base`, `Salesforce/codet5-base`, or purpose-trained code models 4. **Vector Store**: - `Qdrant`, `ChromaDB`, `Weaviate`, or `FAISS` 5. **Graph RAG**: - `Microsoft GraphRAG` - Excellent for community detection and hierarchical summarization - `LightRAG` - Lightweight alternative - `Neo4j LLM Graph Builder` 6. **Code Analysis**: - `semgrep` for pattern-based analysis - `bandit` for security issues - `radon` for Python complexity 7. **Chunking**: - `chonkie` (open source, modern, semantic chunking) - `langchain` text splitters - `tree-sitter` based chunkers Let me design a comprehensive architecture diagram and explanation. I'll structure this as: 1. High-level architecture 2. Component breakdown 3. Data flow 4. Output format (the "context graph" that gets passed to the agent) 5. Open-source library recommendations with rationale 6. Implementation phases 7. Example schema Let me create a really detailed and well-structured response. I should also mention: - The graph should be **token-efficient** when passed to LLM - Should support **incremental updates** - Should support **multiple query modes** (symbol search, semantic search, graph traversal) - Should produce **ranked context** (most relevant first) Let me think about the **output format** for the agent. When the agent asks "how do I change X?", the system should return: - Target file(s) and their content/summary - Symbols affected - Callers/callees - Related tests - Type definitions - Recent changes - Documentation This can be formatted as: - JSON with structured fields - A natural language summary + structured data - A "code map" similar to Aider's repomap Now let me write the detailed response. I'll include: - Mermaid diagrams for architecture - Code snippets showing structure - Library comparison tables - Implementation steps - Schema design Let me draft this comprehensively but not too overwhelming. The user wants architecture + explanation, so I should be thorough but organized. # Code Knowledge Graph: Architecture & Detailed Explanation ## 1. Why a Code Knowledge Graph? A code knowledge graph transforms a repository into a queryable, semantic structure where: - **Nodes** = files, classes, functions, variables, modules, packages, tests, docs, external APIs, git commits, authors - **Edges** = imports, calls, inherits, implements, owns, references, depends_on, modifies, documents, tests This lets an AI agent answer questions like: - *"What functions call `process_payment()` and would be affected if I change its signature?"* - *"Show me all the entry points and the data flow for user signup."* - *"What are the high-level architectural boundaries of this codebase?"* --- ## 2. High-Level Architecture ```mermaid flowchart TB subgraph Ingestion["1. Ingestion Layer"] A[Git Clone / Local Repo] --> B[File Scanner + Language Detector] end subgraph Analysis["2. Static Analysis Layer"] B --> C[Tree-sitter Parser] B --> D[LSP Servers - semantic] B --> E[Git History Miner] C --> F[AST + Symbol Extraction] D --> G[Reference Resolution + Types] E --> H[Commit/Author Graph] end subgraph Graph["3. Graph Construction Layer"] F --> I[Entity Builder] G --> I H --> I I --> J[Code Property Graph] J --> K[(Kùzu / Neo4j
Graph DB)] end subgraph Indexing["4. Indexing Layer"] F --> L[Semantic Chunker
Chonkie] L --> M[Embedder
bge-code-base] M --> N[(Qdrant / ChromaDB
Vector Store)] F --> O[BM25 Index
SQLite FTS5] end subgraph Enrichment["5. Enrichment Layer"] J --> P[Community Detection
Leiden Algorithm] P --> Q[Hierarchical Summarizer
LLM] Q --> R[Architecture Map] end subgraph Retrieval["6. Retrieval & Packaging Layer"] S[Agent Query] --> T[Query Router] T --> U[Hybrid Retriever
Vector + Graph + BM25] K --> U N --> U O --> U R --> U U --> V[Token-Aware Packer] V --> W[Context Graph
JSON / Markdown] end W --> X[AI Agent] ``` --- ## 3. Component-by-Component Breakdown ### 3.1 Ingestion Layer **Purpose**: Get a clean, language-aware view of the repository. | Task | Tool | Why | |------|------|-----| | Clone/fetch | `gitoxide` (pure Rust) or `GitPython` | Fast, safe | | File scanning | `pathspec` + custom walker | Respects `.gitignore` | | Language detection | `linguist`-style heuristics or `enry` | Used by GitHub | **Output**: List of `File` nodes with paths, sizes, language, last modified. --- ### 3.2 Static Analysis Layer (the heart of the system) This is where you decide between two complementary approaches: #### Approach A: **Tree-sitter** (syntactic, fast, complete) - **Library**: [`tree-sitter`](https://tree-sitter.github.io/tree-sitter/) + [`tree-sitter-language-pack`](https://pypi.org/project/tree-sitter-language-pack/) - Covers 100+ languages, used by GitHub, Neovim, Helix - Incremental, blazing fast - Gives you raw AST → you build your own symbol/relation extraction #### Approach B: **Language Server Protocol (LSP)** (semantic, slower, smarter) - **Library**: [`multilspy`](https://github.com/microsoft/multilspy) or `pygls` - Get real **type information, cross-file references, hover docs** - Example: it knows that `user.name` refers to `User.name` even if it's a dynamic attribute - Supports Python (`pylsp`), TS/JS (`typescript-language-server`), Go (`gopls`), Rust (`rust-analyzer`), Java (`jdtls`) **Recommended Hybrid**: 1. Use **Tree-sitter** for fast bulk extraction of definitions (functions, classes, imports). 2. Use **LSP** (in background workers) for high-value files to get references, types, and docs. #### Bonus: **SCIP** (pre-computed code intelligence protocol) - **Library**: [`scip-python`](https://github.com/sourcegraph/scip-python), `scip-typescript`, `scip-go` - Outputs a single `.scip` file with all symbols + references - **Best option if your language has a SCIP indexer** — much faster than running LSP from scratch #### Other useful open-source tools: - **`semgrep`** — pattern-based semantic analysis (security, anti-patterns) - **`radon`** / [`xenon`](https://github.com/rubik/xenon) — Python complexity metrics - **`Joern`** — Code Property Graphs (CPG) with data flow (overkill for most) --- ### 3.3 Graph Construction Layer **Entities (Nodes)** to extract: ``` Repository └── Module / Package ├── File │ ├── Class │ │ ├── Method │ │ └── Attribute │ ├── Function │ ├── Variable (module-level) │ └── Import ├── Test └── DocString (embedded in symbols) ExternalPackage └── ExternalSymbol Commit └── Author ``` **Relationships (Edges)**: | Edge | From → To | Source | |------|-----------|--------| | `CONTAINS` | Module → File | Tree-sitter | | `DEFINES` | File → Class/Function | Tree-sitter | | `IMPORTS` | File → ExternalPackage | Tree-sitter | | `CALLS` | Function → Function | Tree-sitter + LSP | | `INHERITS` | Class → Class | Tree-sitter + LSP | | `IMPLEMENTS` | Class → Interface | Tree-sitter + LSP | | `USES_TYPE` | Function → Class | LSP only | | `TESTS` | TestFile → Class/Function | Naming convention + AST | | `MODIFIES` | Commit → File | Git log | | `AUTHORED_BY` | Commit → Author | Git log | | `DOCUMENTS` | DocString → Symbol | Tree-sitter | | `SIMILAR_TO` | Chunk → Chunk (vector) | Embeddings | **Graph Storage — pick one**: | Option | Type | Best For | |--------|------|----------| | **[Kùzu](https://github.com/kuzudb/kuzu)** | Embedded, columnar, Cypher | ⭐ **Recommended** — modern, fast, no server, single-file DB | | **Neo4j Community** | Server, Cypher | Large teams, persistence, Bloom UI | | **Apache AGE** | Postgres extension | If you already use Postgres | | **DuckDB + graph joins** | Embedded, SQL | Analytical queries on the graph | | **rustworkx / NetworkX** | In-memory Python | Prototyping, small repos | I recommend **Kùzu** for an open-source, embeddable, fast graph DB. It supports **Cypher** (same as Neo4j), is written in C++, and is ~10x faster than NetworkX. --- ### 3.4 Indexing Layer (Hybrid Search) The graph alone is not enough — you also need semantic and lexical search. #### Chunking - **Tool**: [`chonkie`](https://github.com/chonkie-ai/chonkie) — modern, open-source, semantic chunker - **Strategy**: chunk by code structure (function = chunk), not by character count - Each chunk has metadata: `file`, `start_line`, `end_line`, `symbol_id`, `language`, `docstring` #### Embeddings - **Library**: [`sentence-transformers`](https://www.sbert.net/) - **Models** (open-source, in priority order): 1. `BAAI/bge-code-base-v1` — code-specific, excellent 2. `Salesforce/codet5-base` — encoder for retrieval 3. `nomic-embed-text-v1.5` — general, works surprisingly well for code 4. `jinaai/jina-embeddings-v2-base-code` — purpose-built for code #### Vector Store - **[Qdrant](https://qdrant.tech/)** (open-source, Rust, fast) ⭐ recommended - Alternative: `ChromaDB` (easiest), `Weaviate`, `FAISS` (no metadata), `LanceDB` #### Lexical Index - **SQLite FTS5** — built into Python, no extra server - Index code tokens + identifiers + docstrings --- ### 3.5 Enrichment Layer: Community Detection & Summarization This is what makes the graph **agent-friendly** — it produces an "architecture map" that fits in a few hundred tokens. #### Community Detection - **Algorithm**: **Leiden** (preferred over Louvain — no disconnected communities) - **Library**: [`leidenalg`](https://github.com/vtraag/leidenalg) + [`igraph`](https://igraph.org/python/) - Run on the **call graph** + **import graph** to find natural module boundaries. #### Hierarchical Summarization - **Library**: [`graphrag`](https://github.com/microsoft/graphrag) by Microsoft ⭐ - Implements Leiden + LLM-based community summarization - Produces a **hierarchical report** of the codebase at multiple levels (file → module → subsystem) - Open-source, MIT, very mature - **Alternative**: [`LightRAG`](https://github.com/HKUDS/LightRAG) — faster, simpler, less storage This is the killer feature: when the agent asks *"what is this codebase about?"*, you can return a token-efficient summary built from the graph's natural structure. --- ### 3.6 Retrieval & Packaging Layer This is what actually **serves the agent's request**. #### Query Router The agent's query can be: - **Symbolic** ("Where is `process_payment` defined?") → graph lookup - **Semantic** ("How is user authentication handled?") → vector search - **Structural** ("Show me the call graph of `OrderService`") → graph traversal - **Hybrid** ("What should I know before changing `db.py`?") → all three #### Hybrid Retriever ```python def retrieve(query: str, repo_id: str) -> ContextGraph: # 1. Vector search → find semantically similar chunks chunks = vector_search(query, k=20) # 2. Graph expansion → walk the graph from those chunks symbols = [chunk.symbol_id for chunk in chunks] expanded_nodes = graph_expand(symbols, hops=2) # 3. BM25 → catch exact identifier matches bm25_hits = bm25_search(query, k=10) # 4. Community context → add module-level summaries communities = get_communities_for(symbols) # 5. Reciprocal Rank Fusion to merge final = rrf([chunks, expanded_nodes, bm25_hits]) return ContextGraph(final, communities) ``` **Library**: [`rank-bm25`](https://github.com/dorianbrown/rank_bm25) for BM25, custom RRF logic. #### Token-Aware Packer - **Library**: [`tiktoken`](https://github.com/openai/tiktoken) for token counting - **Strategy**: allocate budget across: - High-level architecture summary (always included, 500 tokens) - Target symbol + signature (always) - Code body (target symbol, max 1500 tokens) - Callers/callees (signatures only, max 800 tokens) - Tests (signatures + 1 example, max 500 tokens) - Related docs (max 300 tokens) - Snippets from vector search (max 1000 tokens) #### Output Format: **The "Context Graph"** A JSON document the agent consumes. Example: ```json { "repo": "my-app", "query": "change process_payment to support crypto", "context_graph": { "summary": "Monorepo with FastAPI backend, React frontend, Postgres DB.", "target": { "symbol": "process_payment", "qualified_name": "billing.services.process_payment", "file": "src/billing/services.py", "lines": "42-87", "signature": "def process_payment(order_id: str, method: str) -> Receipt", "docstring": "Charges the order via the configured payment gateway...", "body": "..." }, "callers": [ {"symbol": "checkout_endpoint", "file": "src/api/checkout.py", "line": 23} ], "callees": [ {"symbol": "stripe_gateway.charge", "external": true}, {"symbol": "save_receipt", "file": "src/billing/repo.py"} ], "tests": [ {"file": "tests/billing/test_services.py", "covers": "process_payment"} ], "related_chunks": [ {"file": "docs/billing.md", "text": "...", "score": 0.87} ], "affected_modules": [ {"name": "billing", "summary": "Handles payments, invoices, and refunds."} ] }, "token_count": 2847 } ``` --- ## 4. Recommended Open-Source Stack (Priority Order) | Layer | Primary Choice | License | Why | |-------|----------------|---------|-----| | **AST Parser** | `tree-sitter` + `tree-sitter-language-pack` | MIT | Battle-tested, fast, complete | | **Semantic Layer** | `multilspy` (MS) or `scip-*` | MIT / Apache | Real type info, cross-file refs | | **Git Mining** | `pygit2` (libgit2 bindings) | GPL→MIT | Fast, safe, parallelizable | | **Graph DB** | **Kùzu** | MIT | Embedded, Cypher, very fast | | **Vector DB** | **Qdrant** | Apache 2.0 | Rust-fast, metadata-rich | | **Embeddings** | `sentence-transformers` + `bge-code-base` | Apache 2.0 | SOTA code retrieval | | **Chunking** | **Chonkie** | MIT | Semantic, code-aware | | **Lexical Search** | SQLite FTS5 | Public Domain | Built-in, zero setup | | **Community Detection** | `leidenalg` + `igraph` | GPL/MIT | Best algorithm | | **GraphRAG** | **Microsoft GraphRAG** | MIT | Hierarchical summaries | | **Token Counting** | `tiktoken` | MIT | Standard | | **Orchestration** | Custom (lightweight) or `LlamaIndex` | MIT | Optional | --- ## 5. Data Flow Example **Scenario**: Agent is asked to *"Add a retry mechanism to the email sending function."* ``` 1. Agent sends query: "add retry mechanism to email sending" 2. System runs: a. Vector search → finds send_email, EmailService.send b. BM25 → matches identifier "email" c. Graph walk → finds callers (UserNotifier, DigestJob, ...) d. Community detection → email module summary retrieved 3. Packer assembles context within token budget 4. Returns ContextGraph with: - email module summary - send_email signature + body - 5 callers (signature only) - 1 relevant test - docstring of the function 5. Agent uses this to plan and execute the change ``` --- ## 6. Implementation Phases **Phase 1 — MVP (1-2 weeks)** - Tree-sitter for 3-5 languages - Kùzu for graph storage - In-memory `NetworkX` if Kùzu is overkill - Basic symbol + import + call extraction - Output: JSON graph dump **Phase 2 — Smart Search (1-2 weeks)** - Add vector store + embeddings - Add Chonkie chunking - Build hybrid retriever - Output: ranked context snippets **Phase 3 — Semantic Depth (1-2 weeks)** - Add LSP for type info (optional, slow) - Add git history mining - Add Leiden community detection - Output: architecture maps + change-impact analysis **Phase 4 — Agent Integration (1 week)** - Token-aware packer - CLI + REST API - Hook into agent's context window - Output: drop-in context for any agent --- ## 7. Schema (Cypher for Kùzu/Neo4j) ```cypher // Create constraints CREATE NODE TABLE Repository (name STRING, url STRING, PRIMARY KEY(name)); CREATE NODE TABLE File (path STRING, language STRING, loc INT, PRIMARY KEY(path)); CREATE NODE TABLE Class (qualified_name STRING, name STRING, file STRING, line INT, docstring STRING, PRIMARY KEY(qualified_name)); CREATE NODE TABLE Function (qualified_name STRING, name STRING, file STRING, line INT, signature STRING, docstring STRING, complexity DOUBLE, PRIMARY KEY(qualified_name)); CREATE NODE TABLE Module (name STRING, path STRING, summary STRING, PRIMARY KEY(name)); CREATE NODE TABLE ExternalPackage (name STRING, version STRING, PRIMARY KEY(name)); CREATE NODE TABLE Commit (sha STRING, author STRING, date DATETIME, message STRING, PRIMARY KEY(sha)); // Relationships CREATE REL TABLE CONTAINS (FROM Repository TO File); CREATE REL TABLE DEFINES (FROM File TO Class); CREATE REL TABLE DEFINES (FROM File TO Function); CREATE REL TABLE IMPORTS (FROM File TO ExternalPackage); CREATE REL TABLE CALLS (FROM Function TO Function); CREATE REL TABLE INHERITS (FROM Class TO Class); CREATE REL TABLE PART_OF (FROM Class TO Module); CREATE REL TABLE MODIFIES (FROM Commit TO File); CREATE REL TABLE TESTS (FROM File TO Function); ``` --- ## 8. Key Design Principles 1. **Open-source first** — no vendor lock-in, all components MIT/Apache. 2. **Embeddable** — runs in a single Docker container or process, no external services required (except optional LLM for summarization). 3. **Incremental** — supports `git pull` and re-indexes only changed files. 4. **Token-budget aware** — never returns more context than the LLM can handle; always returns the *most* relevant context. 5. **Language-agnostic core** — Tree-sitter + LSP abstraction lets you add new languages with minimal effort. 6. **Agent-friendly output** — structured JSON, not raw text, so agents can navigate it programmatically. --- ## 9. Quick-Start Repo Structure ``` code-knowledge-graph/ ├── src/ │ ├── ingestion/ # Git, file scanning │ ├── parsers/ # Tree-sitter wrappers │ ├── lsp/ # Multilspy clients │ ├── graph/ # Kùzu schema + builders │ ├── embeddings/ # Sentence-transformers │ ├── retrieval/ # Hybrid search │ ├── enrichment/ # Leiden + GraphRAG │ ├── packer/ # Token-aware context builder │ └── api/ # FastAPI / CLI ├── data/ │ ├── graph.kuzu/ # Embedded graph DB │ ├── vectors.qdrant/ # Vector store │ └── index.sqlite # FTS5 ├── tests/ └── pyproject.toml ``` This architecture gives you a **production-grade, fully open-source** code knowledge graph that can be incrementally updated, queried in multiple ways, and packaged into a compact context for any AI agent. The key insight: **combine a graph DB (Kùzu), a vector DB (Qdrant), and hierarchical summarization (GraphRAG)** — that trio covers structural, semantic, and architectural understanding of any codebase.

i want to create graph for some code repo to be passed on a...