End-to-End Benchmarking Methodology for Vector Search Systems under Multi-Modal, Multi-Query Workloads

• Hamza Qureshi

  Department of Computer Science, University of Gujrat, Jalalpur Jattan Road, Gujrat 50700, Pakistan

  Author

• Imran Siddiq

  Department of Information Technology, The University of Haripur, Hattar Road, Haripur 22620, Pakistan

  Author

Vector search has become a common substrate for retrieval in applications that combine text, images, audio, and structured metadata. In practice, systems are evaluated under heterogeneous workloads that include nearest-neighbor lookups, filtered retrieval, hybrid sparse-dense ranking, and re-ranking stages that depend on model inference. Benchmarking these systems end-to-end is difficult because measured outcomes couple embedding quality, index construction, storage layout, distributed execution, caching, and concurrency control, while user-facing objectives trade latency, throughput, recall, and operational cost. This paper proposes an end-to-end benchmarking methodology for vector search systems under multi-modal, multi-query workloads. The methodology defines workload families that mix modalities and query operators, prescribes dataset construction that preserves cross-modal alignment and temporal drift, and introduces a harness that measures tail latency, recall at fixed budgets, stability under load, and cost-normalized performance across deployment topologies. It provides guidance for controlling confounders such as warmup effects, background compaction, and replica imbalance, and for reporting uncertainty using statistically grounded confidence intervals and sensitivity analyses. The methodology emphasizes reproducibility by specifying parameter disclosures, seeds, hardware and kernel settings, and failure-mode reporting. The result is a structured approach that makes benchmark outcomes interpretable as system-level trade-offs rather than isolated micro-metrics, and that supports comparative analysis across approximate nearest-neighbor algorithms, storage engines, and multi-stage retrieval pipelines without relying on workload simplifications that omit common production behaviors.

• PDF