Hybrid Search & Cross-Encoder Re-Ranking

Executive Summary

Standard RAG pipelines frequently suffer from retrieval dilution. Dense vector embedding searches (like cosine similarity) excel at matching general semantic concepts, but often miss exact keyword qualifiers (serial numbers, names, specific product SKUs). Conversely, sparse keyword searches (BM25) lack contextual awareness.

To resolve this for a enterprise customer managing over 50 million product documentation chunks, Arvento implemented a Sparse/Dense Hybrid Search network paired with a secondary neural Cross-Encoder Re-ranking model. This optimization pushed retrieval hit accuracy (Hit Rate @ 10) from 71% to 94.2%, eliminating downstream LLM context hallucinations.

1. System Architecture

The retrieval workflow combines dense vector collections inside Qdrant and sparse tokens indexing within Elasticsearch, combining outputs using Reciprocal Rank Fusion before feeding candidates to the re-ranking layer.

Pipeline Stage	Technological Implementation	Latencies	Retrieval Hit Rate
Dense Indexing	Qdrant HNSW vector index (1536-dim Cohere embeddings)	15ms	78.4% (Recall@10)
Sparse Indexing	Elasticsearch BM25 inverted term collection	8ms	68.1% (Recall@10)
Fusion & Re-rank	Reciprocal Rank Fusion (RRF) + Cohere Rerank v3 filter	22ms	94.2% (Hit Rate@10)

2. Sparse/Dense Hybrid Fusion Mechanics

Because vector similarity scores and BM25 scores utilize completely different mathematical distributions, they cannot be added together directly. Instead, we use the Reciprocal Rank Fusion (RRF) algorithm:

For each document d present in both dense and sparse retrieval queues, the RRF score is calculated as:

RRF(d) = Σ_{m ∈ M} ( 1 / (k + r_m(d)) )

Where r_m(d) is the rank position of document d inside retrieval queue m (dense or sparse), and k is a smoothing constant (typically set to 60). This mathematical approach assigns weight to candidates that appear near the top of both results queues without requiring score scale normalizations.

3. Cross-Encoder Re-ranking Pipeline

While Bi-Encoders encode documents and queries separately into vector coordinates (allowing fast indexing), they cannot model semantic relationships between words directly.

To solve this, the top 50 fused candidates from the RRF calculation are fed to a Cross-Encoder re-ranker. The Cross-Encoder processes the query and document candidates simultaneously through self-attention layers, mapping deep relevance scoring.

hybrid_fusion_rank.py

import numpy as np

def reciprocal_rank_fusion(dense_results, sparse_results, k=60):
    # Dict to store calculated fusion values
    rrf_scores = {}
    
    # Process dense vector rank array
    for rank, doc_id in enumerate(dense_results):
        rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
        
    # Process sparse keyword rank array
    for rank, doc_id in enumerate(sparse_results):
        rrf_scores[doc_id] = rrf_scores.get(doc_id, 0.0) + 1.0 / (k + rank + 1)
        
    # Sort candidates by descending RRF score
    sorted_docs = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
    return [doc[0] for doc in sorted_docs[:50]]

4. Accuracy Improvements Comparison

Evaluating system tests over a baseline validation suite of 5,000 queries demonstrated the impact of adding hybrid index layers and re-ranking filters:

Retrieval Hit Rate @ k Comparison

Conclusion

A single model context format cannot resolve complex enterprise documentation requests alone. By merging high-speed keyword arrays and neural embeddings indexes through reciprocal rankings and Cross-Encoder attention filters, Arvento's systems provide highly accurate pipelines suitable for automation deployments.