Skip to content

[POP-2951] Slow-but-perfect KNN in plaintext #1676

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 18 commits into from
Oct 3, 2025
Merged

[POP-2951] Slow-but-perfect KNN in plaintext #1676

merged 18 commits into from
Oct 3, 2025

Conversation

mcalancea
Copy link
Collaborator

@mcalancea mcalancea commented Sep 23, 2025 •
edited
Loading

Usage:

cargo run --release --bin generate_ideal_neighborhoods -- --num-irises 10000 --k 320 --results-file results.txt --num-threads 4

bgillesp
bgillesp reviewed Sep 23, 2025
View reviewed changes
Copy link
Collaborator

@bgillesp bgillesp left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like a good first pass, thanks! Let's see how the performance looks when we get up to around the 1M-2M size dataset and see if it's still enough to finish in a couple of days. If not (esp. when we add in the min-fhd computation), there might be some tricks we can apply to improve cache locality -- e.g. processing iris codes in blocks, restructuring memory layout, etc.

@mcalancea mcalancea force-pushed the mihai/ideal-neighborhoods-gen branch from c878f9c to b32449d Compare September 25, 2025 12:55
@mcalancea mcalancea marked this pull request as ready for review September 25, 2025 13:35
@mcalancea mcalancea changed the base branch from main to dev September 26, 2025 12:00
semgrep-code-worldcoin[bot]
semgrep-code-worldcoin bot reviewed Sep 26, 2025
View reviewed changes
semgrep-code-worldcoin[bot]
semgrep-code-worldcoin bot reviewed Sep 26, 2025
View reviewed changes
@mcalancea mcalancea changed the title mihai/ideal-neighborhoods-gen [POP-2951] Slow-but-perfect KNN in plaintext Sep 29, 2025
@mcalancea mcalancea requested a review from bgillesp September 29, 2025 13:40
@sdwoodbury
[POP-2929] add graceful shutdown to the networking stack (#1685)
9844546 
* squash

* dont try to reconnect if sessions are dropped

* add more shutdown handlers

* add cancellation token to the connection builder

* reduce logs

* address code review

* omit reconnect failure due to cancellation

* return error if message fails to deserialize
bgillesp
bgillesp reviewed Sep 30, 2025
View reviewed changes
Copy link
Collaborator

@bgillesp bgillesp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really great binary! Extremely usable and resilient. I had just one larger comment about the serialization format, but otherwise it's looking good. 👍

mcalancea and others added 4 commits October 2, 2025 11:53
@mcalancea
u32 instead of usize and stream.take
1cf5cbd
@naure @carlomazzaferro
@dependabot @kszarek @eaypek-tfh @wojciechsromek @iliailia
Merge main -> dev (#1705)
e160325 
* do not run against dev (#1695)

* Bump actions/checkout from 4.2.2 to 5.0.0 (#1651)

Bumps [actions/checkout](https://github.com/actions/checkout) from 4.2.2 to 5.0.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v4.2.2...08c6903)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: 5.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com>
Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com>

* Bump tj-actions/changed-files from 46.0.5 to 47.0.0 (#1656)

Bumps [tj-actions/changed-files](https://github.com/tj-actions/changed-files) from 46.0.5 to 47.0.0.
- [Release notes](https://github.com/tj-actions/changed-files/releases)
- [Changelog](https://github.com/tj-actions/changed-files/blob/main/HISTORY.md)
- [Commits](tj-actions/changed-files@ed68ef8...24d32ff)

---
updated-dependencies:
- dependency-name: tj-actions/changed-files
  dependency-version: 47.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump docker/setup-buildx-action from 3.10.0 to 3.11.1 (#1652)

Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3.10.0 to 3.11.1.
- [Release notes](https://github.com/docker/setup-buildx-action/releases)
- [Commits](docker/setup-buildx-action@v3.10.0...e468171)

---
updated-dependencies:
- dependency-name: docker/setup-buildx-action
  dependency-version: 3.11.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump docker/build-push-action from 6.15.0 to 6.18.0 (#1650)

Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6.15.0 to 6.18.0.
- [Release notes](https://github.com/docker/build-push-action/releases)
- [Commits](docker/build-push-action@v6.15.0...2634353)

---
updated-dependencies:
- dependency-name: docker/build-push-action
  dependency-version: 6.18.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com>

* Ability to disable anon stats per-batch, from SQS messages (#1696)

* enable the ability to disable anon stats per-batch, from SQS messages

* basic tests for e2e anon stats disabled

* test fix

* probabilistic disablement of anon stats in tests

* remove unnecessary if statement

* enable reauth in e2e test (#1694)

e2e: enable reauth

* Oblivious swap network (#1682)

Add oblivious swap network

* clamp counter to actual buffer size (#1703)

* clamp counter to actual buffer size

* warning log if clamping

* improve logging for anon stats disabling

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com>
Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com>
Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com>
Co-authored-by: iliailia <iliailiashenko@gmail.com>
@mcalancea
Merge branch 'dev' into mihai/ideal-neighborhoods-gen
bb22b9a
@mcalancea
clippy
d90538b
@naure naure deleted the branch dev October 2, 2025 13:28
@naure naure closed this Oct 2, 2025
@bgillesp
Copy link
Collaborator

bgillesp commented Oct 2, 2025

Closed accidentally because of changes made to dev -- re-opening now.

@bgillesp bgillesp reopened this Oct 2, 2025
@bgillesp bgillesp requested review from philsippl and a team as code owners October 2, 2025 18:44
bgillesp
bgillesp approved these changes Oct 2, 2025
View reviewed changes
Copy link
Collaborator

@bgillesp bgillesp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking great, thanks for the revisions! 🙏

@mcalancea mcalancea merged commit 542574d into dev Oct 3, 2025
16 of 18 checks passed
@mcalancea mcalancea deleted the mihai/ideal-neighborhoods-gen branch October 3, 2025 07:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bgillesp bgillesp bgillesp approved these changes

@philsippl philsippl Awaiting requested review from philsippl

+1 more reviewer

@semgrep-code-worldcoin semgrep-code-worldcoin[bot] semgrep-code-worldcoin[bot] left review comments

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@mcalancea @bgillesp @naure @sdwoodbury