[POP-2929] add graceful shutdown to the networking stack #1685
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
switched from new() to default() to resolve conflicts in tcp/mod.rs
naure
approved these changes
Sep 29, 2025
Merged
naure
added a commit
that referenced
this pull request
Oct 2, 2025
* dev: deploy main * dev: Increase timeout to 30min to support batch_size=32 * [POP-2929] add graceful shutdown to the networking stack (#1685) * squash * dont try to reconnect if sessions are dropped * add more shutdown handlers * add cancellation token to the connection builder * reduce logs * address code review * omit reconnect failure due to cancellation * return error if message fails to deserialize --------- Co-authored-by: Mihai <mihai.calancea@gmail.com> Co-authored-by: sdwoodbury <stuart@inversed.tech>
mcalancea
added a commit
that referenced
this pull request
Oct 3, 2025
* draft naive-knn * Release memory correctly add checkpoints and safeguards * clippy * fixes * change floats to fractions * dev: deploy main * dev: Increase timeout to 30min to support batch_size=32 * [POP-2929] add graceful shutdown to the networking stack (#1685) * squash * dont try to reconnect if sessions are dropped * add more shutdown handlers * add cancellation token to the connection builder * reduce logs * address code review * omit reconnect failure due to cancellation * return error if message fails to deserialize * u32 instead of usize and stream.take * Merge main -> dev (#1705) * do not run against dev (#1695) * Bump actions/checkout from 4.2.2 to 5.0.0 (#1651) Bumps [actions/checkout](https://github.com/actions/checkout) from 4.2.2 to 5.0.0. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4.2.2...08c6903) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 5.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> * Bump tj-actions/changed-files from 46.0.5 to 47.0.0 (#1656) Bumps [tj-actions/changed-files](https://github.com/tj-actions/changed-files) from 46.0.5 to 47.0.0. - [Release notes](https://github.com/tj-actions/changed-files/releases) - [Changelog](https://github.com/tj-actions/changed-files/blob/main/HISTORY.md) - [Commits](tj-actions/changed-files@ed68ef8...24d32ff) --- updated-dependencies: - dependency-name: tj-actions/changed-files dependency-version: 47.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump docker/setup-buildx-action from 3.10.0 to 3.11.1 (#1652) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3.10.0 to 3.11.1. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](docker/setup-buildx-action@v3.10.0...e468171) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-version: 3.11.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump docker/build-push-action from 6.15.0 to 6.18.0 (#1650) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6.15.0 to 6.18.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](docker/build-push-action@v6.15.0...2634353) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-version: 6.18.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> * Ability to disable anon stats per-batch, from SQS messages (#1696) * enable the ability to disable anon stats per-batch, from SQS messages * basic tests for e2e anon stats disabled * test fix * probabilistic disablement of anon stats in tests * remove unnecessary if statement * enable reauth in e2e test (#1694) e2e: enable reauth * Oblivious swap network (#1682) Add oblivious swap network * clamp counter to actual buffer size (#1703) * clamp counter to actual buffer size * warning log if clamping * improve logging for anon stats disabling --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: iliailia <iliailiashenko@gmail.com> * clippy --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Aurel <naure@users.noreply.github.com> Co-authored-by: sdwoodbury <stuart@inversed.tech> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: iliailia <iliailiashenko@gmail.com> Co-authored-by: Bryan Gillespie <bgillesp@users.noreply.github.com>
naure
added a commit
that referenced
this pull request
Oct 29, 2025
* au/deploy-dev: Deploy latest main (#1706) * [POP-2951] Slow-but-perfect KNN in plaintext (#1676) * draft naive-knn * Release memory correctly add checkpoints and safeguards * clippy * fixes * change floats to fractions * dev: deploy main * dev: Increase timeout to 30min to support batch_size=32 * [POP-2929] add graceful shutdown to the networking stack (#1685) * squash * dont try to reconnect if sessions are dropped * add more shutdown handlers * add cancellation token to the connection builder * reduce logs * address code review * omit reconnect failure due to cancellation * return error if message fails to deserialize * u32 instead of usize and stream.take * Merge main -> dev (#1705) * do not run against dev (#1695) * Bump actions/checkout from 4.2.2 to 5.0.0 (#1651) Bumps [actions/checkout](https://github.com/actions/checkout) from 4.2.2 to 5.0.0. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4.2.2...08c6903) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 5.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> * Bump tj-actions/changed-files from 46.0.5 to 47.0.0 (#1656) Bumps [tj-actions/changed-files](https://github.com/tj-actions/changed-files) from 46.0.5 to 47.0.0. - [Release notes](https://github.com/tj-actions/changed-files/releases) - [Changelog](https://github.com/tj-actions/changed-files/blob/main/HISTORY.md) - [Commits](tj-actions/changed-files@ed68ef8...24d32ff) --- updated-dependencies: - dependency-name: tj-actions/changed-files dependency-version: 47.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump docker/setup-buildx-action from 3.10.0 to 3.11.1 (#1652) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3.10.0 to 3.11.1. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](docker/setup-buildx-action@v3.10.0...e468171) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-version: 3.11.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump docker/build-push-action from 6.15.0 to 6.18.0 (#1650) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6.15.0 to 6.18.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](docker/build-push-action@v6.15.0...2634353) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-version: 6.18.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> * Ability to disable anon stats per-batch, from SQS messages (#1696) * enable the ability to disable anon stats per-batch, from SQS messages * basic tests for e2e anon stats disabled * test fix * probabilistic disablement of anon stats in tests * remove unnecessary if statement * enable reauth in e2e test (#1694) e2e: enable reauth * Oblivious swap network (#1682) Add oblivious swap network * clamp counter to actual buffer size (#1703) * clamp counter to actual buffer size * warning log if clamping * improve logging for anon stats disabling --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: iliailia <iliailiashenko@gmail.com> * clippy --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Aurel <naure@users.noreply.github.com> Co-authored-by: sdwoodbury <stuart@inversed.tech> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: iliailia <iliailiashenko@gmail.com> Co-authored-by: Bryan Gillespie <bgillesp@users.noreply.github.com> * [POP-2966] Oblivious minimum distance (#1709) Add oblivious minimum tree * [POP-2867] Graph diff functionality (#1655) * duplicate files with new types * start changing files * prune migrate calls * add graph_diff.rs * rename aliases to include ref * merge type change * draft differ * refactor traits * detailed jaccard * extend compare_to_db * fix some indices * refactors * Simplify and format * wip * change abstractions * adjustments * dev: deploy main * dev: Increase timeout to 30min to support batch_size=32 * change abstraction; include options in CLI * clippy * feat: mod e2e deploy templates (#1693) Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> * Revert changes to deploy/e2e/ for merge into dev --------- Co-authored-by: Aurel <naure@users.noreply.github.com> Co-authored-by: Bartosz Jasinski <32450522+jazzbee@users.noreply.github.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: Bryan Gillespie <bgillesp@users.noreply.github.com> Co-authored-by: Bryan Gillespie <bryan@inversed.tech> * [POP-2916] Change `intra_batch_rule` to `AND` (#1713) * change intra-batch rule to AND * modify unit test * Implement construct-graph-ptxt binary (#1711) PR implements a small binary to construct an HNSW graph from plaintext iris code input, with output serialized to file using the standard "single-graph" binary output format. This binary is to be used primarily for upcoming data analysis tasks, and may later be integrated with related graph-building utilities. * [POP-2956] rotation aware trick dot (#1707) * add new variants of trick dot * add unit tests and a benchmark * [POP-2962] batch rotation aware dot products (#1714) * add batch dot product * Chore/dev use arm x8gs 1 (#1724) use arm instance * [POP-2965] Compute minimal distance over rotations (#1716) * Add generic minimal distance metric * Minimal-fhd in MPC and formatting * Change test * Add rotation-aware dot product logic to minimal fraction distance computation * [POP-2983] Batch oblivious array minimum (#1722) Batch computation of several minimum trees * [POP-2996] improve trick_dot benchmark (#1734) * simulate a batch * update redundant benchmark group names * implement Quickselect (#1741) add quickselect.rs * Dev sync (#1727) * feat: mod e2e deploy templates (#1693) Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> * Bump docker/login-action from 3.4.0 to 3.6.0 (#1698) Bumps [docker/login-action](https://github.com/docker/login-action) from 3.4.0 to 3.6.0. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@74a5d14...5e57cd1) --- updated-dependencies: - dependency-name: docker/login-action dependency-version: 3.6.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> * fix: E2E config typo (#1710) * [DO NOT MERGE] release 0.21.0 (#1708) release 0.21.0 Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> * [POP-2991] Test Utils: Revert port of generate-benchmark-data (#1717) * 1. Reverted iris-mpc-utils/bin/write_plaintext_store.rs port of iris-mpc-cpu/bin/generate_benchmark_data.rs. * 1. Removed obsolete types. * Add pre-commit hooks and gitleaks pipeline (#1719) * Add pre-commit hooks and gitleaks pipeline * Add permissions to github actions to avoid security alerts * run latest * fix readme * remove cargo check * Bump actions/cache from 4.2.4 to 4.3.0 (#1699) Bumps [actions/cache](https://github.com/actions/cache) from 4.2.4 to 4.3.0. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@0400d5f...0057852) --- updated-dependencies: - dependency-name: actions/cache dependency-version: 4.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Exclude sql files from precommit checks (#1730) * Bump foundry-rs/foundry-toolchain from 1.4.0 to 1.5.0 (#1715) Bumps [foundry-rs/foundry-toolchain](https://github.com/foundry-rs/foundry-toolchain) from 1.4.0 to 1.5.0. - [Release notes](https://github.com/foundry-rs/foundry-toolchain/releases) - [Changelog](https://github.com/foundry-rs/foundry-toolchain/blob/master/RELEASE.md) - [Commits](foundry-rs/foundry-toolchain@82dee4b...50d5a89) --- updated-dependencies: - dependency-name: foundry-rs/foundry-toolchain dependency-version: 1.5.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [POP-2993] Test Utils: Revert port of graph-mem-cli (#1718) * 1. Reverted iris-mpc-utils/bin/write_plaintext_store.rs port of iris-mpc-cpu/bin/generate_benchmark_data.rs. * 1. Removed obsolete types. * 1. Reverted iris-mpc-utils/bin/graph_mem_cli.rs port of iris-mpc-cpu/bin/graph_mem_cli.rs. * 1. Reverted iris-mpc-utils/bin/initialise_test_dbs.rs port of iris-mpc-cpu/bin/init_test_dbs.rs. * 1. Reverted iris-mpc-utils/bin/initialise_test_dbs.rs port of iris-mpc-cpu/bin/init_test_dbs.rs. * 1. Reverting work pushed to a followup PR. * 1. Reverting work pushed to a followup PR. * [POP-2995] Revert port of initialise-test-dbs (#1721) * 1. Reverted iris-mpc-utils/bin/initialise_test_dbs.rs port of iris-mpc-cpu/bin/init_test_dbs.rs. * 1. Synced with previous PR branch. * [POP-2998] Test Utils: Revert port of helper functions (#1725) * 1. Reverting port of py-bindings. * 1. Reverting port of py-bindings. * 1. Reverting port of genesis and graph utils. * 1. Reverting port of iris utils. * 1. Reverting port of iris utils. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Bartosz Jasinski <32450522+jazzbee@users.noreply.github.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> Co-authored-by: Mark A. Conway-Greenslade <siajasl-0@pm.me> * dev: build ARM64 * (bugfix): TlsConfig (#1744) fix clap for TlsConfig * [POP-2952] (Ideal KNN for `minFHD`) (#1720) Extends ideal KNN binary to support min-fhd metric. Additionally refactors the supporting functionality to simplify integration of additional metrics in the future. * scale dev up (#1746) * au/disable-rotations: Refactor rotation support (POP-2967) (#1728) au/disable-rotations: Refactor VecRots into VecRotationSupport without default * [POP-2997] parallelize batch trick_dot() (#1742) * rename benchmarks * use worker pool to parallelize dot product batch * (improvement): add ability to debug failed genesis runs (#1751) * improve genesis * use tracing instead of println * remove unnecessary changes * remove unnecessary change to service ports, thanks to use of the cancellation token * [POP-3034] Implement HNSW layer search with batched node openings (#1738) The existing implementation of batched HNSW layer search provides batching of all basic distance comparison operations (e.g. filtering of visited nodes against the current worst candidate node, and insertion of filtered nodes into the candidate neighborhood), but processes node openings one at a time. This means that the distances of neighbors of individual candidate nodes are computed in a batch, but that there is no mechanism for batching together the openings of multiple candidate nodes at once. This is more of an issue now because the distance evaluation phase makes up a majority of the overall execution round complexity. This PR implements a new version of batched HNSW layer search which handles batching at the node opening level: as before, an ongoing estimate of the "rate of insertions of visited nodes into the candidate neighborhood" is maintained, and multiple unopened nodes are opened as a batch, the number of which is chosen so that the batch is expected to insert a fixed constant number of nodes per iteration. This simplifies the overall procedure quite significantly, as individual queues are not required for the separate filtering and opening phases, since the selection of a number of nodes to open already provides rate calibration. This additionally eliminates the "cleanup" phase of operation in the previous version. See `iris_mpc_cpu::hnsw::searcher::layer_search_batched_v2` for additional details. * [POP-3033] Add search layer strategy via linear scan for top layers of HNSW graphs (#1736) * Add linear search as initialization strategy for HNSW search * Add oblivious argmin to Aby3Store * Integrate linear scan into searcher * [POP-3078] refactor networking setup code (#1760) * wip: refactor make_sessions and setup for network_handle * compiles * fix tests * clippy * rename trait function * add make_sessions * use parallelize * Sw/iris mpc bin (#1763) * make separate crate for binaries * use cargo shear * lock * refactor docker build steps * remove dead code --------- Co-authored-by: Stuart Woodbury <stuart@inversed.tech> * use public ecr (#1766) * use public ecr * compose parallel limits * login to the public ecr * same change in e2e genesis * us-east-1 * remove compose parallel limits --------- Co-authored-by: Wojciech Sromek <wojciech.sromek@toolsforhumanity.com> * (bugfix): update the scripts/ to use iris-mpc-bins (#1765) * update test scripts to build binaries using iris-mpc-bins crate * update documentation and test scripts to use -p iris-mpc-bins and move the data directory from iris-mpc-cpu to iris-mpc-bins * fix documentation --------- Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> * dev: Deploy latest --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: mcalancea <mihai.calancea@gmail.com> Co-authored-by: sdwoodbury <stuart@inversed.tech> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: iliailia <iliailiashenko@gmail.com> Co-authored-by: Bryan Gillespie <bgillesp@users.noreply.github.com> Co-authored-by: Bartosz Jasinski <32450522+jazzbee@users.noreply.github.com> Co-authored-by: Bryan Gillespie <bryan@inversed.tech> Co-authored-by: Mark A. Conway-Greenslade <siajasl-0@pm.me> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@toolsforhumanity.com> Co-authored-by: Wojciech Sromek <wojciech.sromek@toolsforhumanity.com>
carlomazzaferro
added a commit
that referenced
this pull request
Oct 29, 2025
* feat: re-randomization binary * feat: implement a tripartite DH based on pairings. * address clippy error * clippy * fix comment * bench: add re-randomize benchmark * bench: add MT bench * test: unit tests for rerandomization * feat: local run of rerandomization script + crosscheck * feat: add range parameters to allow handling of sub-ranges * feat: use tripartite DH in rerandomize binary, with AWS integrations to store private and public keys * au/deploy-dev: Deploy latest main (#1706) * [POP-2951] Slow-but-perfect KNN in plaintext (#1676) * draft naive-knn * Release memory correctly add checkpoints and safeguards * clippy * fixes * change floats to fractions * dev: deploy main * dev: Increase timeout to 30min to support batch_size=32 * [POP-2929] add graceful shutdown to the networking stack (#1685) * squash * dont try to reconnect if sessions are dropped * add more shutdown handlers * add cancellation token to the connection builder * reduce logs * address code review * omit reconnect failure due to cancellation * return error if message fails to deserialize * u32 instead of usize and stream.take * Merge main -> dev (#1705) * do not run against dev (#1695) * Bump actions/checkout from 4.2.2 to 5.0.0 (#1651) Bumps [actions/checkout](https://github.com/actions/checkout) from 4.2.2 to 5.0.0. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4.2.2...08c6903) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 5.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> * Bump tj-actions/changed-files from 46.0.5 to 47.0.0 (#1656) Bumps [tj-actions/changed-files](https://github.com/tj-actions/changed-files) from 46.0.5 to 47.0.0. - [Release notes](https://github.com/tj-actions/changed-files/releases) - [Changelog](https://github.com/tj-actions/changed-files/blob/main/HISTORY.md) - [Commits](tj-actions/changed-files@ed68ef8...24d32ff) --- updated-dependencies: - dependency-name: tj-actions/changed-files dependency-version: 47.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump docker/setup-buildx-action from 3.10.0 to 3.11.1 (#1652) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3.10.0 to 3.11.1. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](docker/setup-buildx-action@v3.10.0...e468171) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-version: 3.11.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump docker/build-push-action from 6.15.0 to 6.18.0 (#1650) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6.15.0 to 6.18.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](docker/build-push-action@v6.15.0...2634353) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-version: 6.18.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> * Ability to disable anon stats per-batch, from SQS messages (#1696) * enable the ability to disable anon stats per-batch, from SQS messages * basic tests for e2e anon stats disabled * test fix * probabilistic disablement of anon stats in tests * remove unnecessary if statement * enable reauth in e2e test (#1694) e2e: enable reauth * Oblivious swap network (#1682) Add oblivious swap network * clamp counter to actual buffer size (#1703) * clamp counter to actual buffer size * warning log if clamping * improve logging for anon stats disabling --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: iliailia <iliailiashenko@gmail.com> * clippy --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Aurel <naure@users.noreply.github.com> Co-authored-by: sdwoodbury <stuart@inversed.tech> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: iliailia <iliailiashenko@gmail.com> Co-authored-by: Bryan Gillespie <bgillesp@users.noreply.github.com> * [POP-2966] Oblivious minimum distance (#1709) Add oblivious minimum tree * [POP-2867] Graph diff functionality (#1655) * duplicate files with new types * start changing files * prune migrate calls * add graph_diff.rs * rename aliases to include ref * merge type change * draft differ * refactor traits * detailed jaccard * extend compare_to_db * fix some indices * refactors * Simplify and format * wip * change abstractions * adjustments * dev: deploy main * dev: Increase timeout to 30min to support batch_size=32 * change abstraction; include options in CLI * clippy * feat: mod e2e deploy templates (#1693) Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> * Revert changes to deploy/e2e/ for merge into dev --------- Co-authored-by: Aurel <naure@users.noreply.github.com> Co-authored-by: Bartosz Jasinski <32450522+jazzbee@users.noreply.github.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: Bryan Gillespie <bgillesp@users.noreply.github.com> Co-authored-by: Bryan Gillespie <bryan@inversed.tech> * [POP-2916] Change `intra_batch_rule` to `AND` (#1713) * change intra-batch rule to AND * modify unit test * Implement construct-graph-ptxt binary (#1711) PR implements a small binary to construct an HNSW graph from plaintext iris code input, with output serialized to file using the standard "single-graph" binary output format. This binary is to be used primarily for upcoming data analysis tasks, and may later be integrated with related graph-building utilities. * [POP-2956] rotation aware trick dot (#1707) * add new variants of trick dot * add unit tests and a benchmark * [POP-2962] batch rotation aware dot products (#1714) * add batch dot product * Chore/dev use arm x8gs 1 (#1724) use arm instance * [POP-2965] Compute minimal distance over rotations (#1716) * Add generic minimal distance metric * Minimal-fhd in MPC and formatting * Change test * Add rotation-aware dot product logic to minimal fraction distance computation * [POP-2983] Batch oblivious array minimum (#1722) Batch computation of several minimum trees * merge * docker image and flow for shares re-randomization * flow name and branches * [POP-2996] improve trick_dot benchmark (#1734) * simulate a batch * update redundant benchmark group names * implement Quickselect (#1741) add quickselect.rs * Dev sync (#1727) * feat: mod e2e deploy templates (#1693) Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> * Bump docker/login-action from 3.4.0 to 3.6.0 (#1698) Bumps [docker/login-action](https://github.com/docker/login-action) from 3.4.0 to 3.6.0. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@74a5d14...5e57cd1) --- updated-dependencies: - dependency-name: docker/login-action dependency-version: 3.6.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> * fix: E2E config typo (#1710) * [DO NOT MERGE] release 0.21.0 (#1708) release 0.21.0 Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> * [POP-2991] Test Utils: Revert port of generate-benchmark-data (#1717) * 1. Reverted iris-mpc-utils/bin/write_plaintext_store.rs port of iris-mpc-cpu/bin/generate_benchmark_data.rs. * 1. Removed obsolete types. * Add pre-commit hooks and gitleaks pipeline (#1719) * Add pre-commit hooks and gitleaks pipeline * Add permissions to github actions to avoid security alerts * run latest * fix readme * remove cargo check * Bump actions/cache from 4.2.4 to 4.3.0 (#1699) Bumps [actions/cache](https://github.com/actions/cache) from 4.2.4 to 4.3.0. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@0400d5f...0057852) --- updated-dependencies: - dependency-name: actions/cache dependency-version: 4.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Exclude sql files from precommit checks (#1730) * Bump foundry-rs/foundry-toolchain from 1.4.0 to 1.5.0 (#1715) Bumps [foundry-rs/foundry-toolchain](https://github.com/foundry-rs/foundry-toolchain) from 1.4.0 to 1.5.0. - [Release notes](https://github.com/foundry-rs/foundry-toolchain/releases) - [Changelog](https://github.com/foundry-rs/foundry-toolchain/blob/master/RELEASE.md) - [Commits](foundry-rs/foundry-toolchain@82dee4b...50d5a89) --- updated-dependencies: - dependency-name: foundry-rs/foundry-toolchain dependency-version: 1.5.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [POP-2993] Test Utils: Revert port of graph-mem-cli (#1718) * 1. Reverted iris-mpc-utils/bin/write_plaintext_store.rs port of iris-mpc-cpu/bin/generate_benchmark_data.rs. * 1. Removed obsolete types. * 1. Reverted iris-mpc-utils/bin/graph_mem_cli.rs port of iris-mpc-cpu/bin/graph_mem_cli.rs. * 1. Reverted iris-mpc-utils/bin/initialise_test_dbs.rs port of iris-mpc-cpu/bin/init_test_dbs.rs. * 1. Reverted iris-mpc-utils/bin/initialise_test_dbs.rs port of iris-mpc-cpu/bin/init_test_dbs.rs. * 1. Reverting work pushed to a followup PR. * 1. Reverting work pushed to a followup PR. * [POP-2995] Revert port of initialise-test-dbs (#1721) * 1. Reverted iris-mpc-utils/bin/initialise_test_dbs.rs port of iris-mpc-cpu/bin/init_test_dbs.rs. * 1. Synced with previous PR branch. * [POP-2998] Test Utils: Revert port of helper functions (#1725) * 1. Reverting port of py-bindings. * 1. Reverting port of py-bindings. * 1. Reverting port of genesis and graph utils. * 1. Reverting port of iris utils. * 1. Reverting port of iris utils. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Bartosz Jasinski <32450522+jazzbee@users.noreply.github.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> Co-authored-by: Mark A. Conway-Greenslade <siajasl-0@pm.me> * dev: build ARM64 * (bugfix): TlsConfig (#1744) fix clap for TlsConfig * [POP-2952] (Ideal KNN for `minFHD`) (#1720) Extends ideal KNN binary to support min-fhd metric. Additionally refactors the supporting functionality to simplify integration of additional metrics in the future. * scale dev up (#1746) * au/disable-rotations: Refactor rotation support (POP-2967) (#1728) au/disable-rotations: Refactor VecRots into VecRotationSupport without default * [POP-2997] parallelize batch trick_dot() (#1742) * rename benchmarks * use worker pool to parallelize dot product batch * (improvement): add ability to debug failed genesis runs (#1751) * improve genesis * use tracing instead of println * remove unnecessary changes * remove unnecessary change to service ports, thanks to use of the cancellation token * [POP-3034] Implement HNSW layer search with batched node openings (#1738) The existing implementation of batched HNSW layer search provides batching of all basic distance comparison operations (e.g. filtering of visited nodes against the current worst candidate node, and insertion of filtered nodes into the candidate neighborhood), but processes node openings one at a time. This means that the distances of neighbors of individual candidate nodes are computed in a batch, but that there is no mechanism for batching together the openings of multiple candidate nodes at once. This is more of an issue now because the distance evaluation phase makes up a majority of the overall execution round complexity. This PR implements a new version of batched HNSW layer search which handles batching at the node opening level: as before, an ongoing estimate of the "rate of insertions of visited nodes into the candidate neighborhood" is maintained, and multiple unopened nodes are opened as a batch, the number of which is chosen so that the batch is expected to insert a fixed constant number of nodes per iteration. This simplifies the overall procedure quite significantly, as individual queues are not required for the separate filtering and opening phases, since the selection of a number of nodes to open already provides rate calibration. This additionally eliminates the "cleanup" phase of operation in the previous version. See `iris_mpc_cpu::hnsw::searcher::layer_search_batched_v2` for additional details. * [POP-3033] Add search layer strategy via linear scan for top layers of HNSW graphs (#1736) * Add linear search as initialization strategy for HNSW search * Add oblivious argmin to Aby3Store * Integrate linear scan into searcher * [POP-3078] refactor networking setup code (#1760) * wip: refactor make_sessions and setup for network_handle * compiles * fix tests * clippy * rename trait function * add make_sessions * use parallelize * Sw/iris mpc bin (#1763) * make separate crate for binaries * use cargo shear * lock * refactor docker build steps * remove dead code --------- Co-authored-by: Stuart Woodbury <stuart@inversed.tech> * use public ecr (#1766) * use public ecr * compose parallel limits * login to the public ecr * same change in e2e genesis * us-east-1 * remove compose parallel limits --------- Co-authored-by: Wojciech Sromek <wojciech.sromek@toolsforhumanity.com> * (bugfix): update the scripts/ to use iris-mpc-bins (#1765) * update test scripts to build binaries using iris-mpc-bins crate * update documentation and test scripts to use -p iris-mpc-bins and move the data directory from iris-mpc-cpu to iris-mpc-bins * fix documentation --------- Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> * isolate deps * fix dockerfile * download pk from s3 * value for iris db rerandomization --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: naure <naure@users.noreply.github.com> Co-authored-by: mcalancea <mihai.calancea@gmail.com> Co-authored-by: sdwoodbury <stuart@inversed.tech> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: iliailia <iliailiashenko@gmail.com> Co-authored-by: Bryan Gillespie <bgillesp@users.noreply.github.com> Co-authored-by: Bartosz Jasinski <32450522+jazzbee@users.noreply.github.com> Co-authored-by: Bryan Gillespie <bryan@inversed.tech> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@toolsforhumanity.com> Co-authored-by: Mark A. Conway-Greenslade <siajasl-0@pm.me> Co-authored-by: Wojciech Sromek <wojciech.sromek@toolsforhumanity.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Additionally does the following:
build_network_handle()