最新版本是否有LoCoMo的测试代码

[YueAWu]

我在测试之前branch中的public_evaluations时会报告各种错误。

[wangyu-ustc]

What model were you using here?

[YueAWu]

I used the default model: gpt-4.1-mini

[wangyu-ustc]

Wired, can you try using the evaluation code in branch evals? We have been using this branch to run evaluations on other benchmarks.

[3210448723]

Wired, can you try using the evaluation code in branch evals? We have been using this branch to run evaluations on other benchmarks.

Hello, I used the evals branch to execute the python main.py --agent_name mirix --dataset LOCOMO --num_exp 1 command and found that there was a delay of at least 5 minutes between each request to the large model, resulting in slow evaluation. May I ask what the reason is?

[Mirix-AI]

Ah this has never happened to me before. If you simply run from openai import OpenAI and then call gpt-4.1-mini will it be as slow as 5 minutes per request?

[3210448723]

Ah this has never happened to me before. If you simply run from openai import OpenAI and then call gpt-4.1-mini will it be as slow as 5 minutes per request?

我可能知道原因了：#33 (comment)
也是embedding model调用的是google api，连接超时了
public_evaluations/logs/mirix_LOCOMO-modelgpt-4.1-mini/0_subsetNone_cksizeNone/logs/child_20251005_142406.log日志：

Processing LOCOMO queries and answers:   0%|          | 0/10 [00:00<?, ?item/s]
Processing LOCOMO queries and answers: 100%|██████████| 10/10 [00:00<00:00, 7033.88item/s]
Creating engine postgresql+pg8000://user:admin_203@localhost:5432/mirix
[Mirix.AgentWrapper.mirix] INFO: Setting timezone to: UTC (UTC+00:00)
[Mirix.AgentWrapper.mirix] INFO: (UTC+00:00)
🔄 Initializing model: gpt-4.1-mini

Ingesting chunks:   0%|          | 0/19 [00:00<?, ?it/s]/home/user/abc/MIRIX/public_evaluations/../mirix/llm_api/helpers.py:287: UserWarning: Overwriting existing inner monologue () with kwarg (The conversation between Caroline and Melanie focuses on Caroline attending a LGBTQ support group, the impact it had on her, her interest in education and career options related to counseling and mental health, and Melanie's painting as a form of self-expression and relaxation. The main topics are LGBTQ support group experience, career aspirations in counseling/mental health, and painting as a creative outlet.)
  warnings.warn(f"Overwriting existing inner monologue ({new_choice.message.content}) with kwarg ({inner_thoughts})")
[Mirix.Agent.meta_memory_agent] ERROR: step() failed
messages = [Message(created_by_id=None, last_updated_by_id=None, created_at=datetime.datetime(2025, 10, 6, 15, 0, 55, 68645, tzinfo=datetime.timezone.utc), updated_at=None, id='message-1a7483d6-82c7-4058-8ddd-b8b31cb9a046', organization_id=None, user_id=None, agent_id='agent-54187bc6-eca8-43ed-9b0b-08cda21228a9', model=None, role=<MessageRole.user: 'user'>, content=[TextContent(type=<MessageContentType.text: 'text'>, text='The following are text messages from the user:'), TextContent(type=<MessageContentType.text: 'text'>, text='Timestamp: 2025-10-06 15:00:54 Text:\nYou have access to memories from two speakers in a conversation. These memories contain timestamped information that may be relevant to answering the question.\n\nInstructions:\n\n1. Carefully analyze all utterances from both speakers.\n2. The conversation has a timestamp, but the events mentioned in the conversation may have different timestamps. You have to extract the exact date of the mentioned events. Remember that "mentioned at" is not the same as "occurred at" so this has to be noted in the memories.\n3. If there is a question about time references (like "last year", "two months ago", etc.), calculate the actual date based on the memory timestamp. For example, if a memory from 4 May 2022 mentions "went to India last year," then the trip occurred in 2021.\n4. Always convert relative time references to specific dates, months, or years. For example, convert "last year" to "2022" or "two months ago" to "March 2023" based on the conversation timestamp. \n5. Focus only on the content of the memories from both speakers. Do not confuse character names mentioned in memories with the actual users who created those memories.\n6. You are supposed to extract the event/fact/semantic knowledge from the conversation. For example, if the conversation happens at 2023 and the conversation says that "John went to India last year", then you should save the fact that "John went to India in 2022". Similarly for all other kinds of memories.\n7. Make sure to extract the facts about the characters, such as their name, age, gender, occupation, hometown, etc.\n\nThe conversation is shown below (the conversation is timestamped at 1:56 pm on 8 May, 2023):\n\nCaroline: Hey Mel! Good to see you! How have you been?\nMelanie: Hey Caroline! Good to see you! I\'m swamped with the kids & work. What\'s up with you? Anything new?\nCaroline: I went to a LGBTQ support group yesterday and it was so powerful.\nMelanie: Wow, that\'s cool, Caroline! What happened that was so awesome? Did you hear any inspiring stories?\nCaroline: The transgender stories were so inspiring! I was so happy and thankful for all the support.\nMelanie: Wow, love that painting! So cool you found such a helpful group. What\'s it done for you?\nCaroline: The support group has made me feel accepted and given me courage to embrace myself.\nMelanie: That\'s really cool. You\'ve got guts. What now?\nCaroline: Gonna continue my edu and check out career options, which is pretty exciting!\nMelanie: Wow, Caroline! What kinda jobs are you thinkin\' of? Anything that stands out?\nCaroline: I\'m keen on counseling or working in mental health - I\'d love to support those with similar issues.\nMelanie: You\'d be a great counselor! Your empathy and understanding will really help the people you work with. By the way, take a look at this.\nCaroline: Thanks, Melanie! That\'s really sweet. Is this your own painting?\nMelanie: Yeah, I painted that lake sunrise last year! It\'s special to me.\nCaroline: Wow, Melanie! The colors really blend nicely. Painting looks like a great outlet for expressing yourself.\nMelanie: Thanks, Caroline! Painting\'s a fun way to express my feelings and get creative. It\'s a great way to relax after a long day.\nCaroline: Totally agree, Mel. Relaxing and expressing ourselves is key. Well, I\'m off to go do some research.\nMelanie: Yep, Caroline. Taking care of ourselves is vital. I\'m off to go swimming with the kids. Talk to you soon!\n'), TextContent(type=<MessageContentType.text: 'text'>, text='[System Message] As the meta memory manager, analyze the provided content. Based on the content, determine what memories need to be updated (episodic, procedural, knowledge vault, semantic, core, and resource)')], name=None, tool_calls=None, tool_call_id=None, step_id=None, otid=None, tool_returns=None, group_id=None, sender_id=None)]
error = [Errno 110] Connection timed out
[Mirix.Agent.meta_memory_agent] ERROR: step() failed with an unrecognized exception: '[Errno 110] Connection timed out'
[Mirix] ERROR: Error in server._step: [Errno 110] Connection timed out
Traceback (most recent call last):
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpx/_transports/default.py", line 101, in map_httpcore_exceptions
    yield
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpx/_transports/default.py", line 250, in handle_request
    resp = self._pool.handle_request(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 256, in handle_request
    raise exc from None
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpcore/_sync/connection_pool.py", line 236, in handle_request
    response = connection.handle_request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
    raise exc
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 78, in handle_request
    stream = self._connect(request)
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpcore/_sync/connection.py", line 124, in _connect
    stream = self._network_backend.connect_tcp(**kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpcore/_backends/sync.py", line 207, in connect_tcp
    with map_exceptions(exc_map):
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions
    raise to_exc(exc) from exc
httpcore.ConnectTimeout: [Errno 110] Connection timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/abc/MIRIX/public_evaluations/../mirix/server/server.py", line 588, in _step
    usage_stats = mirix_agent.step(
                  ^^^^^^^^^^^^^^^^^
  File "/home/user/abc/MIRIX/public_evaluations/../mirix/agent/agent.py", line 1129, in step
    step_response = self.inner_step(
                    ^^^^^^^^^^^^^^^^
  File "/home/user/abc/MIRIX/public_evaluations/../mirix/agent/agent.py", line 1682, in inner_step
    raise e
  File "/home/user/abc/MIRIX/public_evaluations/../mirix/agent/agent.py", line 1476, in inner_step
    complete_system_prompt, retrieved_memories = self.build_system_prompt_with_memories(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/abc/MIRIX/public_evaluations/../mirix/agent/agent.py", line 1242, in build_system_prompt_with_memories
    embedded_text = embedding_model(self.agent_state.embedding_config).get_text_embedding(key_words)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/llama_index_instrumentation/dispatcher.py", line 335, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/llama_index/core/base/embeddings/base.py", line 301, in get_text_embedding
    text_embedding = self._get_text_embedding(text)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/llama_index_instrumentation/dispatcher.py", line 335, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/llama_index/embeddings/google_genai/base.py", line 306, in _get_text_embedding
    return self._embed_texts([text], task_type="RETRIEVAL_DOCUMENT")[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/llama_index/embeddings/google_genai/base.py", line 267, in _embed_texts
    return retryable_embed()
           ^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/tenacity/__init__.py", line 338, in wrapped_f
    return copy(f, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/tenacity/__init__.py", line 400, in <lambda>
    self._add_action_func(lambda rs: rs.outcome.result())
                                     ^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/llama_index/embeddings/google_genai/base.py", line 251, in embed_with_client
    results = self._client.models.embed_content(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/google/genai/models.py", line 5559, in embed_content
    response = self._api_client.request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/google/genai/_api_client.py", line 1290, in request
    response = self._request(http_request, http_options, stream=False)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/google/genai/_api_client.py", line 1126, in _request
    return self._retry(self._request_once, http_request, stream)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__
    do = self.iter(retry_state=retry_state)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter
    result = action(retry_state)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/tenacity/__init__.py", line 420, in exc_check
    raise retry_exc.reraise()
          ^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/tenacity/__init__.py", line 187, in reraise
    raise self.last_attempt.result()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__
    result = fn(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/google/genai/_api_client.py", line 1096, in _request_once
    response = self._httpx_client.request(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpx/_client.py", line 825, in request
    return self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
    response = self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
    response = self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
    response = self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpx/_client.py", line 1014, in _send_single_request
    response = transport.handle_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpx/_transports/default.py", line 249, in handle_request
    with map_httpcore_exceptions():
         ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/home/user/anaconda3/envs/yjm_mirix/lib/python3.12/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout: [Errno 110] Connection timed out
None

请问论文https://arxiv.org/abs/2507.07957中Mirix的评测结果用的是什么embedding model？好像文中没有说明