In evaluation branch, mirixAgent is hardcoded to use GoogleGenAIEmbedding, ignoring YAML configuration

[YiLu09]

Bug Description

I've identified a critical design flaw in the public_evaluations branch that prevents running evaluations with custom embedding models (e.g., OpenAI). The mirixAgent class is hardcoded to initialize GoogleGenAIEmbedding for memory/embedding tasks, completely ignoring the embedding_provider and related settings in the YAML configuration file.

This makes it impossible to correctly run the locomo benchmark as described in the README if one intends to use a unified provider (like OpenAI) for both chat and embedding functionalities.

Symptom Progression

The issue manifests in a series of misleading steps:

1. Initial Attempt: Following the README.md, I configured a custom_openai.yaml to use OpenAI as the model_provider and ran the evaluation.
2. Illusion of API Success: My API proxy logs showed successful calls to the OpenAI chat completion endpoint (e.g., gpt-4.1-mini).
3. Incorrect Result File: Despite the successful chat API calls, the generated results.json contained "ERROR_RESPONSE_FAILED" for all answers.
4. Unit Test Reveals Truth: A minimal test script revealed that the program was crashing before even calling the chat API. The error log pointed directly to a failure in initializing GoogleGenAIEmbedding due to a missing Google API Key.
5. Configuration Changes Ineffective: Explicitly adding embedding_provider: openai and embedding_model_name to the YAML file had no effect. The error persisted, proving that these configuration fields are being ignored by the code.

Root Cause Analysis

The root of the problem lies in the __init__ method of the mirixAgent class, located in mirix/agent/agent_wrapper.py.

• The code correctly parses the model_provider for the Chat functionality from the YAML config.
• However, it lacks any logic to read the embedding_provider or embedding_model_name fields for the Memory/Embedding functionality.

Due to the missing configuration intake, the code falls back to its hardcoded default, which is GoogleGenAIEmbedding. This forces the requirement for a Google API Key and causes the crash when it's not present.

Suggested Solution

To properly fix this bug, the __init__ method of the mirixAgent class needs to be modified to:

1. Add logic to correctly parse the embedding_provider and embedding_model_name fields from the YAML configuration.
2. Based on this configuration, initialize the corresponding embedding client (e.g., OpenAIEmbedding when openai is specified).

Thank you for developing this incredible framework. I'm looking forward to getting the evaluation running correctly and continuing my research based on it. Let me know if any more information is needed!

[wangyu-ustc]

Hi just to clarify, are you using the branch public_evaluation or the branch evals?
In the branch public_evaluation I fixed the default embedding config to be text-embedding-3-small as shown here: https://github.com/Mirix-AI/MIRIX/blob/public_evaluation/mirix/agent/agent_wrapper.py#L69
So in the branch public_evaluation should only need OpenAI api key.

For the branch evals we fixed using gemini as shown here
https://github.com/Mirix-AI/MIRIX/blob/evals/mirix/agent/agent_wrapper.py#L121

We are aware that currently in main and evals we haven't allowed using specified embeddings, we are rewriting the whole framework recently and these configurations will be set in the new version. We apologize for the current inconviniences. Based on what you described, I think you can switch to the branch public_evalution for the evaluation of MIRIX on LoCoMo. For the branch evals you can simply uncomment line 120 and comment out line 121. Please let me know if you have any other questions!