Skip to content

Conversation

ninjaasmoke
Copy link
Contributor

  • Does this PR have a descriptive title that could go in our release notes?
  • Does this PR add any new dependencies?
  • Does this PR modify any existing APIs?
    • Is the change to the API backwards compatible?
  • Should this result in any changes to our documentation, either updating existing docs or adding new ones?

Reference Issues/PRs

What does this implement/fix? Briefly explain your changes.

Fixes a potential crash in partition_with_ram_budget function when k_base parameter exceeds the number of partitions created.

Issue:
When the partitioning algorithm creates fewer partitions than the requested k_base value, the code attempts to access cluster centers that don't exist, leading to memory access violations and crashes.

Example error (intermittent):

Processing global k-means (kmeans_partitioning Step)
Residuals unchanged: 175.863 becomes 175.863. Early termination.
ERROR: k (5) > num_center(3)
unknown location(0): fatal error: memory access violation at address: 0x20ebdbf40: no mapping at fault address

This occurs when k_base=5 is passed but the k-means algorithm converges to only 3 clusters, causing out-of-bounds access in the cluster size estimation step. The issue is intermittent due to the non-deterministic nature of k-means clustering.

Solution:
Added a bounds check to ensure k_base never exceeds the actual number of partitions:

k_base = std::min(k_base, static_cast<size_t>(num_parts));

This prevents out-of-bounds access during cluster size estimation while preserving the intended functionality.

Any other comments?

This is a minimal, safe fix that maintains backward compatibility. The change ensures robust behavior when the partitioning algorithm determines that fewer partitions are needed than initially requested via k_base.

@ninjaasmoke
Copy link
Contributor Author

@gopalrs Review please?

@gopalrs gopalrs merged commit 1f08328 into microsoft:main Jul 7, 2025
9 of 32 checks passed
Danbaiwq pushed a commit to Danbaiwq/DiskANN that referenced this pull request Sep 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants