ArrayIndexOutOfBoundsException in ES819TSDBDocValuesProducer

[JonasKunz]

Elasticsearch Version

Current snapshot / main

Installed Plugins

No response

Java Version

bundled

OS Version

MacBook M1 Pro on macOS 15.7.2

Problem Description

Trying to load 2 million exponential_histograms via ES|QL crashed in the blockloader for the BinaryDocValues used within that field type.
The query partially fails with an ArrayIndexOutOfBoundsException:

java.lang.ArrayIndexOutOfBoundsException: Index 256000 out of bounds for length 1025
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesProducer$BinaryDecoder.decodeBulk(ES819TSDBDocValuesProducer.java:635)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.index.codec.tsdb.es819.ES819TSDBDocValuesProducer$5.tryRead(ES819TSDBDocValuesProducer.java:414)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.index.mapper.blockloader.docvalues.BytesRefsFromBinaryBlockLoader$BytesRefsFromBinary.read(BytesRefsFromBinaryBlockLoader.java:59)
	at org.elasticsearch.xpack.exponentialhistogram.ExponentialHistogramFieldMapper$ExponentialHistogramFieldType$2$1.read(ExponentialHistogramFieldMapper.java:412)
	at org.elasticsearch.compute.lucene.read.ValuesFromSingleReader.loadFromSingleLeaf(ValuesFromSingleReader.java:115)
	at org.elasticsearch.compute.lucene.read.ValuesFromSingleReader.load(ValuesFromSingleReader.java:58)
	at org.elasticsearch.compute.lucene.read.ValuesReader.next(ValuesReader.java:38)
	at org.elasticsearch.compute.lucene.read.ValuesReader.next(ValuesReader.java:18)
	at org.elasticsearch.compute.operator.AbstractPageMappingToIteratorOperator$AppendBlocksIterator.next(AbstractPageMappingToIteratorOperator.java:345)
	at org.elasticsearch.compute.operator.AbstractPageMappingToIteratorOperator$AppendBlocksIterator.next(AbstractPageMappingToIteratorOperator.java:321)
	at org.elasticsearch.compute.operator.AbstractPageMappingToIteratorOperator$RuntimeTrackingIterator.next(AbstractPageMappingToIteratorOperator.java:190)
	at org.elasticsearch.compute.operator.AbstractPageMappingToIteratorOperator$RuntimeTrackingIterator.next(AbstractPageMappingToIteratorOperator.java:175)
	at org.elasticsearch.compute.operator.AbstractPageMappingToIteratorOperator.getOutput(AbstractPageMappingToIteratorOperator.java:149)
	at org.elasticsearch.compute.operator.Driver.runSingleLoopIteration(Driver.java:283)
	at org.elasticsearch.compute.operator.Driver.run(Driver.java:191)
	at org.elasticsearch.compute.operator.Driver$1.doRun(Driver.java:431)
	at org.elasticsearch.server@9.3.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at org.elasticsearch.compute.operator.DriverScheduler$1.doRun(DriverScheduler.java:57)
	... 7 more

Steps to Reproduce

• Start ES on this commit via ./gradlew run run -Dtests.es.xpack.security.enabled=false -Drun.license_type=trial -Dtests.heap.size=4G -Dtests.jvm.argline="-da -dsa"
  That commit enhances the built-in OTLP endpoint to ingest metrics using the new exponential_histogram field type
• Checkout this repo and run the data-generator. That will ingest two million documents with exponential_histograms to the locally running ES instance
• Run the following ES|QL query: FROM metrics-* | STATS AVG(my_histogram). That will fail with an ArrayIndexOutOfBoundsException.
  I did this using a local Kibana snapshot instance running in Docker.

For me, this approach has reliably reproduced the problem. The exact line number of the exception seems to vary though.

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)