Split documents by page

[aleksitukiainen]

Hi, I think one natural extension of the PreProcessor.split(split_by: str) method would be to also enable splitting a document by page. Often a single page of a document contains a specific set of content that is about the same subtopic and thus splitting by page would be a great feature. Unsure how much others might be needing this, but given document splitting is one of the key ways of making sizable chunks for retrievers and readers, I feel like it will be a useful addition.

I'm currently needing this, but will probably build a manual work-around.

Thanks!

[bogdankostic]

Hi @aleksitukiainen! I think this feature would be a nice extension of #2932 where we added the page number as metadata to Documents. It would be nice if you could raise a feature request issue for this. Also, would you be interested in making a contribution to Haystack with this feature? Otherwise, we will put it in our backlog and work on this ourselves.

[aleksitukiainen]

Hi, @bogdankostic - sounds good. I'll raise a feature request issue for this and let's see. On quick look, it looks pretty easy to at least hack it to work for now, but I'm not quite sure when I'll get time to integrate things properly with tests etc.

Let's see who gets to it first! :)