oaipmh-opener not stream based

[dr0i]

From metafacture/metafacture-playground#208:

Unfortunately 4bf31e7 does not prevent the behavior described in metafacture/metafacture-playground#159 (comment)

    This big JSON file was created by processing a specific OAI endpoint with >50k records. I can reproduce this behavior with this same endpoint. Lots of requests (per resumptionToken) are being made while the result file is still empty. At some point in time the requests come to a halt and the result file is written very fast.

I assume that the oaipmh opener does not stream data through but buffer until all data is fetched, @dr0i can you confirm this?

Yes, I think so.

[TobiasNx]

@dr0i what should we discuss here?

[dr0i]

We should make it stream-based. We would have to discuss then:

• do we introduce a parameter to be able to toggle between both variations and keep the old behaviour as default (do we?), maybe mark it as deprecated
• see how we want to serve the data to the flux command chain (i.e. do we need a harvest element ... and so on)