Is there anyway to use s3 as the document store itself ?

[asharm0662]

Hi,

Is there anyway Haystack is able to use s3 buckets in aws as the document store itself, without zipping up files and reading them from the bucket? Ideally I would like for a way to directly upload my documents to s3 without the zipping portion.

I have successfully been able to retrieve the zipped files as per Tutorial 1 for Haystack here and I can read in the zip file that is specified in the tutorial, no problem on my local setup: s3_url = "https://s3.eu-central-1.amazonaws.com/deepset.ai-farm-qa/datasets/documents/wiki_gameofthrones_txt.zip"

[tstadel]

Hey @asharm0662,

thanks for reaching out! :-)
Currently there is no DocumentStore implementation using s3. Main reason for that is, that it is crucial for the latency of a search system how fast data can be retrieved from the DocumentStore and thus how the data is organized within the DocumentStore. That's why special data structures like inverted indexes (e.g. in Elasticsearch), vector databases (e.g. faiss, mulvus or weaviate) and relational databases (e.g. postgres) are used. As far as I know, most S3 implementations do not provide such advanced indexing mechanisms.

[asharm0662]

Amazing thank you! @tstadel

[AdrianB-sovo]

A DocumentStore with metadata in – for example – PostgreSQL and content in S3 could be a good compromise, no?
Fast indexing/filtering, but with the cheaper prices of S3.