RAG Architecture

Current Design in the Project Notes

User question
  -> pipeline.ask(question, session_id)
  -> router.classify_intent(question): intent
  -> retriever.search(query, top_k=3): kb_query or complex only
  -> router.route(question, context, intent): ModelConfig
  -> robot.ask(question, history, model_config)
  -> LLM answer

Component	Implementation recorded in the notes	Role
Vector store	ChromaDB	Local persistent storage
Retrieval framework	LlamaIndex	Reads Markdown and performs vector search
Embedding model	qwen/qwen3-embedding-8b	Chinese-oriented model called through OpenRouter
Index build	Full rebuild on deployment	Clears and recreates the previous index
Document parsing	MarkdownNodeParser	chunk_size=1024, chunk_overlap=100
Retrieval	Similarity top-K	top_k=3 by default

Embedding settings lived in router.py; search lived in retriever.py. The index path was /srv/kb_index through INDEX_PATH, and the Markdown knowledge-base path was /srv/knowledge_base through KB_PATH.

Completed Change: Chinese-Oriented Embeddings

The notes marked the move to qwen/qwen3-embedding-8b as complete. It was called through OpenRouter, so no local model deployment was needed. The index had to be rebuilt before a model change could take effect.

The document suggested checking the effect in three ways: compare similarity scores for the same questions, inspect whether retrieved passages matched the relevant sections, and test informal questions or synonyms. Examples included a question about JVM garbage collection and a colloquial question about improving Redis performance.

It also projected a 15–30% increase in retrieval recall and a 0.1–0.2 increase in average similarity scores. These were expected improvements in the original plan, not measured results.

Completed Change: Markdown-Aware Chunking

The notes marked a custom chunking strategy as implemented. MarkdownNodeParser split content along Markdown headings, with a default chunk size of 1,024 characters and 100 characters of overlap. The settings could be adjusted through environment variables. Overlap helped keep context at chunk boundaries; preserving source metadata helped identify the originating file.

For a technical document of 100–500 lines, the source estimated roughly 4–20 chunks, depending on length. A full rebuild ran during deployment; the notes estimated it took less than one minute for about 50 documents. That is an estimate from the project notes, not a benchmark verified here. Parameter changes also required rebuilding the index.

Proposed Improvement: Incremental Indexing

The current process embedded the entire knowledge base on deployment. An incremental process could detect added, modified, and deleted files using hashes or modification times, remove stale vectors, and embed only changed content. The goal was to reduce index-build work when the knowledge base changed. The source did not mark this as implemented.

Proposed Improvement: Reranking

Top-K similarity search can drift toward semantically related but less useful passages. A reranker, such as bge-reranker-large or an LLM-based reranker, could retrieve the top ten candidates and rerank them to the best three. This was a proposed change.

Proposed Improvement: Hybrid Search

Pure vector search may miss exact keywords. The proposed hybrid approach would combine BM25 keyword search with vector search, then fuse rankings through weighting or reciprocal rank fusion. This was also pending.

Proposed Improvement: Evaluation

The notes stated there was no quantitative retrieval evaluation yet. They proposed a test set of question-answer pairs and comparison using MRR, NDCG, and Recall@K. That would allow embedding, chunking, and ranking changes to be evaluated against a stable set rather than judged only by impression.

Iteration Record

The plan was to update the architecture notes after each change with the code or configuration changed, comparative results, and problems encountered. External references in the original notes were the LlamaIndex and ChromaDB documentation.