Get ItemReader reading a Stream, particularly RepositoryItemReader [BATCH-2681]

[spring-projects-issues]

Philippe opened BATCH-2681 and commented

Today there is no reader component to be configured to read data from a Stream object.

It would be especially usefull that RepositoryItemReader could be configured to read data from a method whose return type is Stream<T> (today we get an exception if you try to do it, it seems to support only Page<T> types).

The reason is that with JPA those method use cursor, if the database support it, and so we can consequently get good performance results when batching treatment.

In the other case, we have to use the HibernateCursorItemReader, which is Hibernate specific because it require a SessionFactory, to read data with cursor.

---

No further details from BATCH-2681

[spring-projects-issues]

Michael Minella commented

Thanks for raising this. We are considering this. In the meantime, there is a work around that you could do. Use the IteratorItemReader and use a FactoryBean to provide the Stream to its constructor.

[spring-projects-issues]

Philippe commented

Did you mean get a Collection

from the Stream with a stream.collect(Collectors.toList()) to set it in the IteratorItemReader
?

But this does not fit because with Spring Data, setting Stream return type permit to get a cursor.

If you make stream.collect(), I think the cursor is parsing until the last item and so you loose the benefit of working with cursor, in performances terms...

The problem is I don't see how to extract stream's items one by one outside Stream's methods, so you can fit in the ItemReader logic...

[spring-projects-issues]

Michael Minella commented

No. I mean Stream.iterator() and set that as the argument to the IteratorItemReader. That, I believe (since an Iterator can be lazily loaded), should allow a cursor effect.

[spring-projects-issues]

Philippe commented

Ok I try it and tell you soon. Thx

[spring-projects-issues]

Philippe commented

Ok I confirm that it works. Thx !

[spring-projects-issues]

Philippe commented

I spoke a little bit too soon... this works, but there is a problem with the transation management.

With HibernateCursorItemReader you have 2 separate transactions : 1 for the reader (and the processor I think), which is read-only, and the other one for the writer, dedicated for write-operations.

This is not the case if you're using stream from repository. Everything is done in the same transaction, which duration is all the batch.

How is it possible to get the same transaction management than in other batch configuration, with separate transactions between reader and writer and with multiple transactions for writer ?