Skip to content

Commit 2036d17

Browse files
dkalesnauremcalanceasdwoodburycarlomazzaferro
authored
Implement protocol to rerandomize database with zero shares (#1628)
* feat: re-randomization binary * feat: implement a tripartite DH based on pairings. * address clippy error * clippy * fix comment * bench: add re-randomize benchmark * bench: add MT bench * test: unit tests for rerandomization * feat: local run of rerandomization script + crosscheck * feat: add range parameters to allow handling of sub-ranges * feat: use tripartite DH in rerandomize binary, with AWS integrations to store private and public keys * au/deploy-dev: Deploy latest main (#1706) * [POP-2951] Slow-but-perfect KNN in plaintext (#1676) * draft naive-knn * Release memory correctly add checkpoints and safeguards * clippy * fixes * change floats to fractions * dev: deploy main * dev: Increase timeout to 30min to support batch_size=32 * [POP-2929] add graceful shutdown to the networking stack (#1685) * squash * dont try to reconnect if sessions are dropped * add more shutdown handlers * add cancellation token to the connection builder * reduce logs * address code review * omit reconnect failure due to cancellation * return error if message fails to deserialize * u32 instead of usize and stream.take * Merge main -> dev (#1705) * do not run against dev (#1695) * Bump actions/checkout from 4.2.2 to 5.0.0 (#1651) Bumps [actions/checkout](https://github.com/actions/checkout) from 4.2.2 to 5.0.0. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@v4.2.2...08c6903) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 5.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> * Bump tj-actions/changed-files from 46.0.5 to 47.0.0 (#1656) Bumps [tj-actions/changed-files](https://github.com/tj-actions/changed-files) from 46.0.5 to 47.0.0. - [Release notes](https://github.com/tj-actions/changed-files/releases) - [Changelog](https://github.com/tj-actions/changed-files/blob/main/HISTORY.md) - [Commits](tj-actions/changed-files@ed68ef8...24d32ff) --- updated-dependencies: - dependency-name: tj-actions/changed-files dependency-version: 47.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump docker/setup-buildx-action from 3.10.0 to 3.11.1 (#1652) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3.10.0 to 3.11.1. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](docker/setup-buildx-action@v3.10.0...e468171) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-version: 3.11.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump docker/build-push-action from 6.15.0 to 6.18.0 (#1650) Bumps [docker/build-push-action](https://github.com/docker/build-push-action) from 6.15.0 to 6.18.0. - [Release notes](https://github.com/docker/build-push-action/releases) - [Commits](docker/build-push-action@v6.15.0...2634353) --- updated-dependencies: - dependency-name: docker/build-push-action dependency-version: 6.18.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> * Ability to disable anon stats per-batch, from SQS messages (#1696) * enable the ability to disable anon stats per-batch, from SQS messages * basic tests for e2e anon stats disabled * test fix * probabilistic disablement of anon stats in tests * remove unnecessary if statement * enable reauth in e2e test (#1694) e2e: enable reauth * Oblivious swap network (#1682) Add oblivious swap network * clamp counter to actual buffer size (#1703) * clamp counter to actual buffer size * warning log if clamping * improve logging for anon stats disabling --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: iliailia <iliailiashenko@gmail.com> * clippy --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Aurel <naure@users.noreply.github.com> Co-authored-by: sdwoodbury <stuart@inversed.tech> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: iliailia <iliailiashenko@gmail.com> Co-authored-by: Bryan Gillespie <bgillesp@users.noreply.github.com> * [POP-2966] Oblivious minimum distance (#1709) Add oblivious minimum tree * [POP-2867] Graph diff functionality (#1655) * duplicate files with new types * start changing files * prune migrate calls * add graph_diff.rs * rename aliases to include ref * merge type change * draft differ * refactor traits * detailed jaccard * extend compare_to_db * fix some indices * refactors * Simplify and format * wip * change abstractions * adjustments * dev: deploy main * dev: Increase timeout to 30min to support batch_size=32 * change abstraction; include options in CLI * clippy * feat: mod e2e deploy templates (#1693) Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> * Revert changes to deploy/e2e/ for merge into dev --------- Co-authored-by: Aurel <naure@users.noreply.github.com> Co-authored-by: Bartosz Jasinski <32450522+jazzbee@users.noreply.github.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: Bryan Gillespie <bgillesp@users.noreply.github.com> Co-authored-by: Bryan Gillespie <bryan@inversed.tech> * [POP-2916] Change `intra_batch_rule` to `AND` (#1713) * change intra-batch rule to AND * modify unit test * Implement construct-graph-ptxt binary (#1711) PR implements a small binary to construct an HNSW graph from plaintext iris code input, with output serialized to file using the standard "single-graph" binary output format. This binary is to be used primarily for upcoming data analysis tasks, and may later be integrated with related graph-building utilities. * [POP-2956] rotation aware trick dot (#1707) * add new variants of trick dot * add unit tests and a benchmark * [POP-2962] batch rotation aware dot products (#1714) * add batch dot product * Chore/dev use arm x8gs 1 (#1724) use arm instance * [POP-2965] Compute minimal distance over rotations (#1716) * Add generic minimal distance metric * Minimal-fhd in MPC and formatting * Change test * Add rotation-aware dot product logic to minimal fraction distance computation * [POP-2983] Batch oblivious array minimum (#1722) Batch computation of several minimum trees * merge * docker image and flow for shares re-randomization * flow name and branches * [POP-2996] improve trick_dot benchmark (#1734) * simulate a batch * update redundant benchmark group names * implement Quickselect (#1741) add quickselect.rs * Dev sync (#1727) * feat: mod e2e deploy templates (#1693) Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> * Bump docker/login-action from 3.4.0 to 3.6.0 (#1698) Bumps [docker/login-action](https://github.com/docker/login-action) from 3.4.0 to 3.6.0. - [Release notes](https://github.com/docker/login-action/releases) - [Commits](docker/login-action@74a5d14...5e57cd1) --- updated-dependencies: - dependency-name: docker/login-action dependency-version: 3.6.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> * fix: E2E config typo (#1710) * [DO NOT MERGE] release 0.21.0 (#1708) release 0.21.0 Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> * [POP-2991] Test Utils: Revert port of generate-benchmark-data (#1717) * 1. Reverted iris-mpc-utils/bin/write_plaintext_store.rs port of iris-mpc-cpu/bin/generate_benchmark_data.rs. * 1. Removed obsolete types. * Add pre-commit hooks and gitleaks pipeline (#1719) * Add pre-commit hooks and gitleaks pipeline * Add permissions to github actions to avoid security alerts * run latest * fix readme * remove cargo check * Bump actions/cache from 4.2.4 to 4.3.0 (#1699) Bumps [actions/cache](https://github.com/actions/cache) from 4.2.4 to 4.3.0. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@0400d5f...0057852) --- updated-dependencies: - dependency-name: actions/cache dependency-version: 4.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Exclude sql files from precommit checks (#1730) * Bump foundry-rs/foundry-toolchain from 1.4.0 to 1.5.0 (#1715) Bumps [foundry-rs/foundry-toolchain](https://github.com/foundry-rs/foundry-toolchain) from 1.4.0 to 1.5.0. - [Release notes](https://github.com/foundry-rs/foundry-toolchain/releases) - [Changelog](https://github.com/foundry-rs/foundry-toolchain/blob/master/RELEASE.md) - [Commits](foundry-rs/foundry-toolchain@82dee4b...50d5a89) --- updated-dependencies: - dependency-name: foundry-rs/foundry-toolchain dependency-version: 1.5.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [POP-2993] Test Utils: Revert port of graph-mem-cli (#1718) * 1. Reverted iris-mpc-utils/bin/write_plaintext_store.rs port of iris-mpc-cpu/bin/generate_benchmark_data.rs. * 1. Removed obsolete types. * 1. Reverted iris-mpc-utils/bin/graph_mem_cli.rs port of iris-mpc-cpu/bin/graph_mem_cli.rs. * 1. Reverted iris-mpc-utils/bin/initialise_test_dbs.rs port of iris-mpc-cpu/bin/init_test_dbs.rs. * 1. Reverted iris-mpc-utils/bin/initialise_test_dbs.rs port of iris-mpc-cpu/bin/init_test_dbs.rs. * 1. Reverting work pushed to a followup PR. * 1. Reverting work pushed to a followup PR. * [POP-2995] Revert port of initialise-test-dbs (#1721) * 1. Reverted iris-mpc-utils/bin/initialise_test_dbs.rs port of iris-mpc-cpu/bin/init_test_dbs.rs. * 1. Synced with previous PR branch. * [POP-2998] Test Utils: Revert port of helper functions (#1725) * 1. Reverting port of py-bindings. * 1. Reverting port of py-bindings. * 1. Reverting port of genesis and graph utils. * 1. Reverting port of iris utils. * 1. Reverting port of iris utils. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Bartosz Jasinski <32450522+jazzbee@users.noreply.github.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> Co-authored-by: Mark A. Conway-Greenslade <siajasl-0@pm.me> * dev: build ARM64 * (bugfix): TlsConfig (#1744) fix clap for TlsConfig * [POP-2952] (Ideal KNN for `minFHD`) (#1720) Extends ideal KNN binary to support min-fhd metric. Additionally refactors the supporting functionality to simplify integration of additional metrics in the future. * scale dev up (#1746) * au/disable-rotations: Refactor rotation support (POP-2967) (#1728) au/disable-rotations: Refactor VecRots into VecRotationSupport without default * [POP-2997] parallelize batch trick_dot() (#1742) * rename benchmarks * use worker pool to parallelize dot product batch * (improvement): add ability to debug failed genesis runs (#1751) * improve genesis * use tracing instead of println * remove unnecessary changes * remove unnecessary change to service ports, thanks to use of the cancellation token * [POP-3034] Implement HNSW layer search with batched node openings (#1738) The existing implementation of batched HNSW layer search provides batching of all basic distance comparison operations (e.g. filtering of visited nodes against the current worst candidate node, and insertion of filtered nodes into the candidate neighborhood), but processes node openings one at a time. This means that the distances of neighbors of individual candidate nodes are computed in a batch, but that there is no mechanism for batching together the openings of multiple candidate nodes at once. This is more of an issue now because the distance evaluation phase makes up a majority of the overall execution round complexity. This PR implements a new version of batched HNSW layer search which handles batching at the node opening level: as before, an ongoing estimate of the "rate of insertions of visited nodes into the candidate neighborhood" is maintained, and multiple unopened nodes are opened as a batch, the number of which is chosen so that the batch is expected to insert a fixed constant number of nodes per iteration. This simplifies the overall procedure quite significantly, as individual queues are not required for the separate filtering and opening phases, since the selection of a number of nodes to open already provides rate calibration. This additionally eliminates the "cleanup" phase of operation in the previous version. See `iris_mpc_cpu::hnsw::searcher::layer_search_batched_v2` for additional details. * [POP-3033] Add search layer strategy via linear scan for top layers of HNSW graphs (#1736) * Add linear search as initialization strategy for HNSW search * Add oblivious argmin to Aby3Store * Integrate linear scan into searcher * [POP-3078] refactor networking setup code (#1760) * wip: refactor make_sessions and setup for network_handle * compiles * fix tests * clippy * rename trait function * add make_sessions * use parallelize * Sw/iris mpc bin (#1763) * make separate crate for binaries * use cargo shear * lock * refactor docker build steps * remove dead code --------- Co-authored-by: Stuart Woodbury <stuart@inversed.tech> * use public ecr (#1766) * use public ecr * compose parallel limits * login to the public ecr * same change in e2e genesis * us-east-1 * remove compose parallel limits --------- Co-authored-by: Wojciech Sromek <wojciech.sromek@toolsforhumanity.com> * (bugfix): update the scripts/ to use iris-mpc-bins (#1765) * update test scripts to build binaries using iris-mpc-bins crate * update documentation and test scripts to use -p iris-mpc-bins and move the data directory from iris-mpc-cpu to iris-mpc-bins * fix documentation --------- Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> * isolate deps * fix dockerfile * download pk from s3 * value for iris db rerandomization --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: naure <naure@users.noreply.github.com> Co-authored-by: mcalancea <mihai.calancea@gmail.com> Co-authored-by: sdwoodbury <stuart@inversed.tech> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Krzysztof Szarek <kszarek@users.noreply.github.com> Co-authored-by: Ertugrul Aypek <ertugrul.aypek@toolsforhumanity.com> Co-authored-by: Wojciech Sromek <157375010+wojciechsromek@users.noreply.github.com> Co-authored-by: iliailia <iliailiashenko@gmail.com> Co-authored-by: Bryan Gillespie <bgillesp@users.noreply.github.com> Co-authored-by: Bartosz Jasinski <32450522+jazzbee@users.noreply.github.com> Co-authored-by: Bryan Gillespie <bryan@inversed.tech> Co-authored-by: Carlo Mazzaferro <carlo.mazzaferro@toolsforhumanity.com> Co-authored-by: Mark A. Conway-Greenslade <siajasl-0@pm.me> Co-authored-by: Wojciech Sromek <wojciech.sromek@toolsforhumanity.com>
1 parent 95f1cfa commit 2036d17

20 files changed

+1811
-5
lines changed
Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
name: Build and push image for rerandomization protocol
2+
3+
on:
4+
push:
5+
branches:
6+
- main
7+
- "dk/re_randomize_db"
8+
paths:
9+
- Dockerfile.shares-re-randomization
10+
- iris-mpc-upgrade/**
11+
- .github/workflows/build-and-push-rerandomization-protocol.yaml
12+
13+
release:
14+
types:
15+
- 'published'
16+
17+
workflow_dispatch:
18+
19+
concurrency:
20+
group: '${{ github.workflow }} @ ${{ github.event.pull_request.head.label || github.head_ref || github.ref }}'
21+
cancel-in-progress: true
22+
23+
env:
24+
REGISTRY: ghcr.io
25+
IMAGE_NAME: "rerandomization-protocol"
26+
27+
jobs:
28+
docker:
29+
timeout-minutes: 40
30+
runs-on:
31+
labels: ubuntu-22.04-16core
32+
permissions:
33+
packages: write
34+
contents: read
35+
attestations: write
36+
id-token: write
37+
steps:
38+
- name: Checkout
39+
uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8
40+
- name: Set up QEMU
41+
uses: docker/setup-qemu-action@29109295f81e9208d7d86ff1c6c12d2833863392
42+
- name: Set up Docker Buildx
43+
uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435
44+
- name: Log in to the Container registry
45+
uses: docker/login-action@5e57cd118135c172c3672efd75eb46360885c0ef
46+
with:
47+
registry: ${{ env.REGISTRY }}
48+
username: ${{ github.repository_owner }}
49+
password: ${{ secrets.GITHUB_TOKEN }}
50+
- name: Build and Push
51+
uses: docker/build-push-action@263435318d21b8e681c14492fe198d362a7d2c83
52+
with:
53+
context: .
54+
push: true
55+
tags: |
56+
${{ env.REGISTRY }}/worldcoin/${{ env.IMAGE_NAME }}:${{ github.sha }}
57+
${{ github.event_name == 'release' && format('{0}/worldcoin/{1}:{2}', env.REGISTRY, env.IMAGE_NAME, github.event.release.tag_name) || '' }}
58+
platforms: linux/amd64
59+
cache-from: type=gha
60+
cache-to: type=gha,mode=max
61+
file: Dockerfile.shares-re-randomization

Cargo.lock

Lines changed: 214 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Dockerfile.shares-re-randomization

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
FROM public.ecr.aws/ubuntu/ubuntu:22.04 AS build-image
2+
3+
WORKDIR /src
4+
RUN apt-get update && apt-get install -y \
5+
curl \
6+
build-essential \
7+
libssl-dev \
8+
texinfo \
9+
libcap2-bin \
10+
pkg-config \
11+
git \
12+
devscripts \
13+
debhelper \
14+
ca-certificates \
15+
protobuf-compiler \
16+
wget
17+
18+
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y
19+
ENV PATH "/root/.cargo/bin:${PATH}"
20+
ENV RUSTUP_HOME "/root/.rustup"
21+
ENV CARGO_HOME "/root/.cargo"
22+
RUN rustup toolchain install 1.85.0
23+
RUN rustup default 1.85.0
24+
RUN rustup component add cargo
25+
RUN cargo install cargo-build-deps \
26+
&& cargo install cargo-edit --version 0.13.6 --locked
27+
28+
FROM build-image AS build-app
29+
WORKDIR /src/iris-mpc
30+
COPY . .
31+
RUN cargo build -p iris-mpc-bins --release --bin rerandomize-db --bin rerandomize-check --bin seed-v2-dbs
32+
33+
FROM ubuntu:22.04
34+
ENV DEBIAN_FRONTEND=noninteractive
35+
36+
COPY --from=build-app /src/iris-mpc/target/release/rerandomize-db /bin/rerandomize-db
37+
COPY --from=build-app /src/iris-mpc/target/release/seed-v2-dbs /bin/seed-v2-dbs
38+
39+
USER 65534
40+
ENTRYPOINT ["/bin/rerandomize-db"]

0 commit comments

Comments
 (0)