Discussion on EvaluationHarness

[TuanaCelik]

Result: This experiment is dropped

This is the discussion board for EvaluationHarness, an experimental component which allows you to orchestrate evaluation for a pipeline with various parameters and for a number of metrics.

The experimental feature includes the RAGEvaluationHarness designed for standard RAG pipelines.

📚 Full Documentation

🧑‍🍳 Cookbook here

Let us know what you think, and if you have any questions about this feature.

Note: Experimental features live in this repository for a fixed period of time. We don't guarantee that we will continue maintaining experimental features. But if they are successful, stable and if you like it, we will move the feature to the core Haystack package.

PS: The first version of this component was implemented by @shadeMe 🚀

[bilgeyucel]

📢 More about EvaluationHarness

Check out our latest cookbook that showcases how to effectively use EvaluationHarness with its override and comparison features: Evaluating AI with Haystack

[d-kleine]

As a feedback, I have noticed that the metrics returned by score_report() will be returned in a unsorted order, which makes it difficult to compare across multiple runs. I think it would be better if the metrics' order in score_report() would match with the defined order in metrics in RAGEvaluationHarness.

pipeline_eval_harness = RAGEvaluationHarness(
    rag_pipeline=basic_rag,
    rag_components=DefaultRAGArchitecture.GENERATION_WITH_EMBEDDING_RETRIEVAL,
    metrics={
        RAGEvaluationMetric.SEMANTIC_ANSWER_SIMILARITY,
        RAGEvaluationMetric.FAITHFULNESS,
        RAGEvaluationMetric.CONTEXT_RELEVANCE,
    }
)

Also, I think the metric_ prefix in the output is redundant, and therefore can be removed for better readability:

From

                    metrics     score
0  metric_context_relevance  0.266667
1                metric_sas  0.537721
2       metric_faithfulness  0.747778

to

                    metrics     score
0         context_relevance  0.266667
1                       sas  0.537721
2              faithfulness  0.747778

[bilgeyucel]

Great feedback, thanks @d-kleine!

[d-kleine]

I just saw that this suggestion would also match with the naming structure in the EvaluationRunResult output

[d-kleine]

Following up on my previous feedback about the score_report(), here are some additional thoughts about the EvaluationHarness for Custom RAG Pipelines:

• The documentation would benefit from explicitly specifying which parameters are required versus optional in RAGExpectedComponentMetadata. Currently, there's ambiguity around whether name, input_mapping, and output_mapping are mandatory parameters for the implementation.

• When implementing custom RAG pipelines, it would be helpful to have a clear overview of commonly used meta identifiers (like retrieved_documents, generated_output) and their typical usage patterns, even though they aren't strictly required.

• For hybrid pipelines, the documentation nicely explains that components can serve multiple roles, but more examples of complex setups would be beneficial. For instance, showing how the same component can act as both a query processor and document retriever with different meta identifiers. The current limitation of "Expects one input that contains the query string" creates challenges when working with hybrid pipelines, e.g. when defining two processors of the same kind of the query, only the last will be taken, but there is no error/warning message indicating that only one processor type (e.g. QUERY_PROCESSOR) can be used. It would be also more intuitive to have the ability to map multiple query inputs to their respective components simultaneously. This would streamline the setup process for complex pipeline architectures

[julian-risch]

We decided to not continue this experiment.