[WIP] Quake Entry for the Streaming Track

[sarda-devesh]

This PR adds a submission for Quake for the streaming track. This PR is still a WIP as we are still tuning some of the parameters

[magdalendobson]

Thank you for working on this! I recently tried this out on simple_runbook.yaml using dataset random-xs.

In order to install the Dockerfile, I had to add the line RUN python3 -m pip install --upgrade pip==22.3.1 before line 48. After that, the installation proceeded correctly, but I found an error when actually running the code. It appears to break on line 200 during the deletion phase in Step 3 of the runbook, and there is no associated error message. I am wondering if this could be something to do with the Python or pip version? I am using python3.10 as that is what is typically used to run the other entries.

[sarda-devesh]

Hey,

Thank you so much for reaching out. The Python/pip version shouldn't matter as all of the logic lives in the C++ code and we use pybind to expose these functionalities as a python library. I will try to rerun the benchmark on my end with the changes you made and see if I can recreate the error on my end.

[magdalendobson]

Thanks for looking into it! I realized I could also help sanity check by running one of the runbooks that doesn't include any deletions, since insert and query appeared to work fine. When I ran using runbook clustered_runbook.yaml and dataset random-xs-clustered I got the following result:

Does this match with what you expect?

[sarda-devesh]

Hey,
Thank you so much for checking this. I think that matches what I expect because we are seeing this low recall behaviour in other workloads and are trying to debug right now. We are currently priortizing performance on the "final" streaming task but as a sanity check I will rerun all of the streaming runbooks to verify before submitting

[magdalendobson]

Ok, good to hear that result is what you expect to see, at least at the moment. I assume I could increase recall by increasing the number of probes during search, and maybe the refinement radius/number of refinement iterations? Any other parameters you would suggest tweaking?

[sarda-devesh]

I think the primary parameter would be to increase the number of probes. I am actually seeing similar behaviours on the final_runbook.yaml so I am debugging that. I will take a look at this workload after that

[sarda-devesh]

Hey, I just pushed a couple of updates to this PR to hopefully resolve all of the issues. I have tested:

• The random-xs dataset on simple_runbook.yaml
• The random-xs-clustered dataset on clustered_runbook.yaml
• The msturing-30M-clustered dataset on the final_runbook.yaml

I have run these experiment on the Azure Standard D8lds v5 as specified on the competition specs. Please let me know if you run into issues on your end

[magdalendobson]

Thanks for running these! I ran your code on my end (although not on the competition spec machine) and found the following results:

Random-xs, simple_runbook.yaml (note the scale here):

Random-xs-clustered, clustered_runbook.yaml:

MSTuring-30M-clustered, final_runbook.yaml (this completed within an hour on 8 threads):

If this looks fine to you, I am happy to merge this PR as verified to be functional. If you would like to appear on the leaderboard, I can rent a competition-spec machine and verify that your run on the competition runbook completes within the time and memory constraints.

[sarda-devesh]

Hey,

Thank you so much for verifying. We would like to have Quake appear on the leaderboard, so it would be great if you could run it on the competition spec machines. Do let me know if you run into any issues.

[magdalendobson]

Sounds good. I checked out a competition-spec machine this morning and ran your algorithm. It completed within the hour timeframe with average recall 85.4%. If you're happy with that, go ahead and mark this PR as ready for review, I can merge it and then I'll submit a PR updating the leaderboard.

One quick request before publishing--the word "maintenance" seems to be misspelled throughout the entry. Would you mind correcting it before we merge?

[magdalendobson]

Hey, just wanted to give a quick ping on this as I have an upcoming leave starting Nov 26, so it would be great to close before then. I just need validation from you that you're happy with the recall number that I'd be adding to the leaderboard and quick correction of typos, then we can merge.

[sarda-devesh]

Hey,

Sorry about the delayed response - I can quickly correct the typos but I think we would like to optimize our recall numbers before we add it to the leaderboard. Is Nov 26 a hard deadline?

[magdalendobson]

Unfortunately it is a hard deadline for me to run the code. You would have to get another maintainer to verify after that, and it might take a while. I suggest if you aren't ready for the leaderboard by then, we could still go ahead and merge since the PR is functional, and you could submit a PR with a new config file/edits when you're ready to be an entry on the leaderboard.