Skip to content

Exception when importing more than 1 bucket via BucketBytes #16

@DeLoooping

Description

@DeLoooping

When importing more than 1 bucket via BucketBytes, the full buckets cannot be mapped to a BitArray because the byte array to import is too long (by 1 byte). The problem already arises in FilterMemory.ExportToBytes(), because the last 7 bits of the full bucket are mapped to the 268435456th byte of the result array, which is 1 byte more than can be mapped to the BitArray in void Import(IList<byte[]> bucketBytes).

Example code to reproduce the issue:

        long expectedElements = 100_000_000;
        double errorRate = 0.0000001;
        var hashMethod = HashMethod.XXHash128;

        var bf = (FilterMemory)FilterBuilder.Build(expectedElements, errorRate, hashMethod);        
        var bytes = bf.ExportToBytes();

        var bf2 = (FilterMemory)FilterBuilder.Build(new FilterMemoryOptions()
        {
            Method = hashMethod,
            ExpectedElements = expectedElements,
            ErrorRate = errorRate,
            BucketBytes = bytes
        });

To fix this issue, Filter.MaxInt could be set to 2147483640 (see pull request #15).
Another fix could be to handle this special case by creating the BitArray at max size, copy all but the last byte into it and and map the last 7 bits manually.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions