Haystack Inference only (farm-haystack[inference]) very heavy and consumes a lot of RAM

[gooseillo]

Hi everyone,

I'm trying to deploy a haystack RAG pipeline to EBS or EC2 on one of the lower config machines (2CPU - 2GB RAM and 2CPU 4GB RAM). While building the container, I used pip install 'farm-haystack[inference]' ## installs torch, sentence-transformers, sentencepiece, and huggingface-hub.

I was wondering, since I only intend to do inference on the CPU, why does it have to install torch and other heavier libraries. Is there a lighter install?

Otherwise, is there a cheaper deployment pipeline for a RAG solution?

[vblagoje]

Hey @gooseillo don't use inference - but only "preprocessing". Have a look at how we build our lightweight Docker image

[vblagoje]

Perhaps it would be useful to see how we make Docker images at https://github.com/deepset-ai/haystack/tree/main/docker and pay attention to cpu-remote-inference in our bake file