feat: search quality eval

[Orbital-Web]

Description

• Created run_search_eval.py and generate_search_queries.py in onyx/backend/test/regression/search_quality
• Added other dependencies too

What does it do?

• run_search_eval.py runs a bunch of queries locally and compares the results from the search and reranker

• it evaluates the search quality based on how closely it aligns with the reranker (assuming the reranker works well)

• it is mostly used as a tool for quickly testing and tuning search parameters such as hybrid alpha, decay, etc. Can also be used to test other factors that affect searching, such as the prompt, embedding model, quantization, etc.

• unlike answer_quality/run_qa, it doesn't need a ground truth label (enables quick and easy testing, and testing of queries without clear "ground truth" orderings)

• It also makes sure the query doesn't switch around every time (since normally, the query is modified before going into the search pipeline) to enable fair comparisons

• generate_search_queries is a helper tool to convert queries and save them, so the evaluation script can reuse the same modified queries

How Has This Been Tested?

• tested in single tenant environment, not intended to be used in multi tenant currently, though could be modified in the future for users to run their own tests on their own documents to tune their parameters
• tested with different llm and reranker models

Backporting (check the box to trigger backport action)

Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.

• This PR should be backported (make sure to check that the backport attempt succeeds)
• [Optional] Override Linear Check

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
internal-search	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	May 15, 2025 10:07pm

[greptile-apps]

PR Summary

This PR introduces a comprehensive search quality evaluation framework with scripts and configuration files to assess and tune search parameters by comparing search results against reranked results.

• Added run_search_eval.py implementing metrics like Jaccard similarity and rank changes to evaluate search quality against reranker output
• Added generate_search_queries.py to ensure consistent query modification across test runs using LLM and search tool interface
• Implemented score-adjusted evaluation metrics in run_search_eval.py to handle varying relevance thresholds
• Added detailed configuration templates (search_eval_config.yaml.template) for customizing search and evaluation parameters
• Enhanced error handling in /backend/onyx/context/search/utils.py for stop word removal and logging

_{💡 (2/5) Greptile learns from your feedback when you react with 👍/👎!}

_{6 file(s) reviewed, 10 comment(s)
Edit PR Review Bot Settings | Greptile}

[evan-onyx]

A few nits, LGTM!