Feat: Search Eval Testing Overhaul (provide ground truth, categorize query, etc.) #4739
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
PR Summary
This PR introduces a comprehensive overhaul of the search evaluation testing framework, adding structured test queries with ground truth validation and category-based analysis.
- The new
test_queries.json.templatehas a syntax error (missing comma) and incomplete second test query that needs to be fixed search_eval_config.yaml.templatereducesNUM_RETURNED_HITSfrom 200 to 50 andEVAL_TOPKfrom 20 to 5, which may impact evaluation quality- New metrics system in
util_eval.pycompares against both ground truth and reranked results but needs better edge case handling - Modular code structure with separate utility files improves maintainability but some error handling gaps exist
- Query keyword generation in
util_data.pyshould persist generated keywords to prevent result inconsistency across runs
💡 (5/5) You can turn off certain types of comments like style here!
9 file(s) reviewed, 17 comment(s)
Edit PR Review Bot Settings | Greptile
backend/tests/regression/search_quality/test_queries.json.template
Outdated
Show resolved
Hide resolved
| pass | ||
|
|
||
|
|
||
| warned = False |
There was a problem hiding this comment.
style: Using global state for warning flag could cause issues in concurrent test execution
ferdinandl007
pushed a commit
to ferdinandl007/onyx
that referenced
this pull request
May 27, 2025
…query, etc.) (onyx-dot-app#4739) * fix: autoflake & import order * docs: readme * fix: mypy * feat: eval * docs: readme * fix: oops forgot to remove comment * fix: typo * fix: rename var * updated default config * fix: config issue * oops * fix: black * fix: eval and config * feat: non tool calling query mod
aronszanto
pushed a commit
to aronszanto/onyx
that referenced
this pull request
May 27, 2025
…query, etc.) (onyx-dot-app#4739) * fix: autoflake & import order * docs: readme * fix: mypy * feat: eval * docs: readme * fix: oops forgot to remove comment * fix: typo * fix: rename var * updated default config * fix: config issue * oops * fix: black * fix: eval and config * feat: non tool calling query mod
ZhipengHe
pushed a commit
to ZhipengHe/onyx
that referenced
this pull request
Jun 6, 2025
…query, etc.) (onyx-dot-app#4739) * fix: autoflake & import order * docs: readme * fix: mypy * feat: eval * docs: readme * fix: oops forgot to remove comment * fix: typo * fix: rename var * updated default config * fix: config issue * oops * fix: black * fix: eval and config * feat: non tool calling query mod
AnkitTukatek
pushed a commit
to TukaTek/onyx
that referenced
this pull request
Sep 23, 2025
…query, etc.) (onyx-dot-app#4739) * fix: autoflake & import order * docs: readme * fix: mypy * feat: eval * docs: readme * fix: oops forgot to remove comment * fix: typo * fix: rename var * updated default config * fix: config issue * oops * fix: black * fix: eval and config * feat: non tool calling query mod
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Greatly overhauled search eval testing script to take in a list of relevant search results (optional). Also more seamlessly generates the modified queries.
Also added question categories and category-based aggregation to see what type of search results are performing poorly.
How Has This Been Tested?
Ran script, manually computed metrics to ensure correctness. Looked at soft truth set to validate that too.
Backporting (check the box to trigger backport action)
Note: You have to check that the action passes, otherwise resolve the conflicts manually and tag the patches.