Studying the impact of different rerankers and hybrid search in a RAG pipeline for Educational documents.

The challenge

A client developing a search engine for college students needed to evaluate the effectiveness of different rerankers and hybrid search approaches within a tight timeline.

My solution

I approached this project through several key steps:

• Developed a comprehensive testing set with carefully crafted queries designed to evaluate different aspects of the search system.
• Created an automated testing program that could process these queries and output detailed results in Excel format.
• Established a rigorous methodology for manual review and validation of query results.
• Implemented state-of-the-art (SOTA) rerankers and sparse embedding search to enhance keyword matching capabilities.
• Generated detailed search performance reports for each system configuration.

The outcome

After thorough analysis of the test results, we discovered that implementing a specific reranker improved search accuracy by 22% without any performance regressions. Based on these findings, we deployed the enhanced configuration to production.