-
Notifications
You must be signed in to change notification settings - Fork 25.3k
Open
Labels
:Distributed Indexing/EngineAnything around managing Lucene and the Translog in an open shard.Anything around managing Lucene and the Translog in an open shard.Team:Distributed IndexingMeta label for Distributed Indexing teamMeta label for Distributed Indexing team
Description
Lucene 10.2 increased the floor segment size from 2MB to 16MB via apache/lucene#14189. I'll copy the PR description here to explain the motivation:
My motivation is that such small segment sizes don't make index structures actually helpful vs. linear scans, so we should avoid them. Furthermore, there has been progress on merging rules for segments below the floor size, in particular merge policies no longer perform quadratic merging (apache/lucene#900) so this change will not make indexing/merging absurdly slow if an application flushes tiny segments.
Finally this likely helps vector search, which likes fewer segments better.
I believe that it would make sense for Elasticsearch to update MergePolicyConfig
to align its defaults with Lucene?
Metadata
Metadata
Assignees
Labels
:Distributed Indexing/EngineAnything around managing Lucene and the Translog in an open shard.Anything around managing Lucene and the Translog in an open shard.Team:Distributed IndexingMeta label for Distributed Indexing teamMeta label for Distributed Indexing team