Document Chunking Strategies & Vector Indexing
Split long-form documents into semantic chunks with overlap, index vector embeddings in pgvector, and retrieve relevant context top-K passages.
RAG Ingestion Architecture
### RAG Architecture Stages
1. **Document Processing**: Parse Markdown/PDF into text passages.
2. **Chunking**: Split text into fixed-size chunks (e.g., 500 tokens) with 50-token overlap to preserve boundary context.
3. **Embedding**: Compute vector embeddings for each chunk.
4. **Vector Retrieval**: Perform k-Nearest Neighbors ($k$-NN) query matching user query embedding.
