How to evaluate a pipeline on entire data as context?

[tobias-mack]

Hi 👋 , is there a way to evaluate a pipeline on FAQ data without the given context in the SQuAD format, so that it tests how well the retrieval actually works with the entire data written to the documentstore and not just a small piece of context?

Since the evaluation gave the same results for a prewritten and empty documentstore (adding eval data with document_store.add_eval_data() ), the retrieval is only based on the given context field it seems.

Do I have to put the entire text corpus into every single context field or is there a simpler way to achieve this ?

So i basically want to use the context field in the SQUaD like dataset as reference for comparison and not as a basis for retrieval:
A question is given -> the retriever gets context pieces -> context pieces are compared with given context -> reader takes top k -> reader generates answer -> generated answer is compared with given answer

I am thankful for any help!