Retrieval Mechanics: Chunking, Hybrid Search, and Reranking

What this chapter covers

Retrieval is not one decision but four: how documents are split, how candidates are found, how they are reordered, and how much survives into the prompt. This chapter is the engineering layer under every context strategy — structure-aware chunking, hybrid dense and sparse search, reciprocal rank fusion, and cross-encoder reranking — with the metrics that tell you which stage is actually broken.

Key takeaways

• Why fixed-size chunking quietly destroys accuracy on policies, contracts, and code
• How hybrid search recovers the identifiers and rare terms that embeddings miss
• The wide-then-narrow pattern: retrieve one hundred, rerank, keep six
• Why raising top-k makes answers worse, more expensive, and slower at the same time
• How to measure retrieval recall and precision so tuning stops being an opinion