Regarding tutorial '25_Customizing_Agent' (2)

[dkbs12]

Hello,
I'd like to ask three things about the tutorial '25_Customizing_Agent' as below.

I found an odd answer using Agent when I made a query as "How tall was the Rhodes statue comparing to the modern Statue of Liberty?"
I used the Colab in the tutorial and everything was as same as written in the Colab.
The final answer to the above query was "The Rhodes statue was taller than the modern Statue of Liberty.".
But, the context in the concerned file was "According to most contemporary descriptions, the Colossus stood approximately 70 cubits, or 33 metres (108 feet) high – approximately the height of the modern Statue of Liberty from feet to crown – making it the tallest statue in the ancient world."
So I don't think there is any confident evidence for the final answer by inferring from the context.
What do you think it bring the Agent to make such an answer?

Second, I have been seeing two type s of answer when the Agent generate insufficient or ambiguous answer.
One of them is "inconclusive" and the other is "The AI Agent does not have any information or tools related to A".
I'd like to know what is the difference between both of them and what makes this kind of difference.

Third, they used "text-davinci-003" for creating a generative QA pipeline but, used "gpt-3.5-turbo" when initialize PromptNode for the Agent.
As you know, both of them is model of OpenAI but, I'd like to know there is any special reason for using different model for each process.
What if I adopt the same model for each process?

I always appreciate your help!

[anakin87]

Hello, @dkbs12!

1. In this use case, we are using two non-deterministic LLMs (one for the Generative QA Pipeline and one for the Agent), that can sometimes make errors in ways that are not easily reproducible.
   With the same setup, I get: "The Colossus of Rhodes was approximately 33 meters (108 feet) tall, while the Statue of Liberty is approximately 46 meters (151 feet) tall."

2. This Agent can use a tool: seven_wonders_search (the Generative QA Pipeline), which is "useful for when you need to answer questions about the seven wonders of the world".

• "The AI Agent does not have any information or tools related to A": currently, I am not able to reproduce this answer, but the idea is that the available Tools are not suitable to ask about a certain topic
• "inconclusive": the Agent used the Tools, but could not find a specific answer.

3. text-davinci-003 is considered legacy, but it is a very good language model. gpt-3.5-turbo is optimized for chat at 1/10th the cost of text-davinci-003 (source: OpenAI docs). In practice, you can try to use only gpt-3.5-turbo, but the quality of reasoning/generation seems a bit lower.

[bilgeyucel]

Hey @dkbs12, in addition to what @anakin87 wrote, I'd like to mention some points:

1. I got "The Rhodes statue was as tall as the modern Statue of Liberty" from the same passage you shared which is a good answer in my opinion. So, I can say that the generative QA pipeline works fine whereas Agent is still very undeterministic.
2. I can easily reproduce "inconclusive" but not "The AI Agent does not have any information or tools related to A". Do you have an example prompt to reproduce it?
3. You can try to use gpt-3.5-turbo. I used it for the generative pipeline as it generates better answers than text-davinci-003. However, in my experience, especially with Wikipedia examples like this, the Agent with gpt-3.5-turbo tends to use its own knowledge rather than using the given Tool to answer the question. So I'd say, depending on your use case, you can try different models, mix and match. e.g. I find gpt-4 really powerful for this task.

[dkbs12]

Hi,
Thank you for your quick and kind answer.
All of your answers was very useful, but I still have the same problem.

As for No. 1, the answers made by Agent chatting are very confused to me.
'Yes' and 'No' answers is existing at the same time.
Is there any special reason for such an inconsistency in answers even though everything is in the same condition?
I tried the question "Was Rhodes Statue taller than the modern Statue of Liberty?" and tried it again the other day.
Finally, I found the contrasting answer from the same question and same Agent as below;

Question : "Was Rhodes Statue taller than the modern Statue of Liberty?"

Answer No.1
Agent custom-at-query-time started with {'query': 'Was Rhodes Statue taller than the modern Statue of Liberty?', 'params': None}
The AI Agent should use the "seven_wonders_search" tool to find information about the height of the Colossus of Rhodes and compare it to the height of the modern Statue of Liberty.

Tool: seven_wonders_search
Tool Input: Was the Colossus of Rhodes taller than the modern Statue of Liberty?

Observation: Yes, the Colossus of Rhodes was approximately 33 metres (108 feet) high, making it taller than the modern Statue of Liberty which is 93 metres (305 feet) tall.
Thought: Based on the information from the "seven_wonders_search" tool, the Colossus of Rhodes was taller than the modern Statue of Liberty.

Final Answer: Yes, the Colossus of Rhodes was taller than the modern Statue of Liberty.{'query': 'Was Rhodes Statue taller than the modern Statue of Liberty?',
'answers': [<Answer {'answer': 'Yes, the Colossus of Rhodes was taller than the modern Statue of Liberty.', 'type': 'generative', 'score': None, 'context': None, 'offsets_in_document': None, 'offsets_in_context': None, 'document_ids': None, 'meta': {}}>],
'transcript': 'The AI Agent should use the "seven_wonders_search" tool to find information about the height of the Colossus of Rhodes and compare it to the height of the modern Statue of Liberty.\n\nTool: seven_wonders_search\nTool Input: Was the Colossus of Rhodes taller than the modern Statue of Liberty?\nObservation: Yes, the Colossus of Rhodes was approximately 33 metres (108 feet) high, making it taller than the modern Statue of Liberty which is 93 metres (305 feet) tall.\nThought:Based on the information from the "seven_wonders_search" tool, the Colossus of Rhodes was taller than the modern Statue of Liberty.\n\nFinal Answer: Yes, the Colossus of Rhodes was taller than the modern Statue of Liberty.'}

Answer No.2
Agent custom-at-query-time started with {'query': 'Was Rhodes Statue taller than the modern Statue of Liberty?', 'params': None}
The AI Agent will compare the heights of the Colossus of Rhodes and the Statue of Liberty to determine which one is taller.

Tool: seven_wonders_search
Tool Input: What was the height of the Colossus of Rhodes?
Observation: Approximately 70 cubits, or 33 metres (108 feet) high - approximately the height of the modern Statue of Liberty from feet to crown.
Thought: The height of the Colossus of Rhodes is approximately the same as the height of the modern Statue of Liberty.

Final Answer: No, the Rhodes Statue was not taller than the modern Statue of Liberty.{'query': 'Was Rhodes Statue taller than the modern Statue of Liberty?',
'answers': [<Answer {'answer': 'No, the Rhodes Statue was not taller than the modern Statue of Liberty.', 'type': 'generative', 'score': None, 'context': None, 'offsets_in_document': None, 'offsets_in_context': None, 'document_ids': None, 'meta': {}}>],
'transcript': 'The AI Agent will compare the heights of the Colossus of Rhodes and the Statue of Liberty to determine which one is taller.\n\nTool: seven_wonders_search\nTool Input: What was the height of the Colossus of Rhodes?\nObservation: Approximately 70 cubits, or 33 metres (108 feet) high - approximately the height of the modern Statue of Liberty from feet to crown.\nThought:The height of the Colossus of Rhodes is approximately the same as the height of the modern Statue of Liberty. \n\nFinal Answer: No, the Rhodes Statue was not taller than the modern Statue of Liberty.'}

Moreover, I have some errors when I process another multi-hop question.
In this case, AI tried to use internet search, even though it is directed to only use the given documents by the prompt template (of course, the related context is existing and we can find the answer from that context.)
I guess there may be some defects in handling the multi-hop question using Agent.
The examples are as below;

Question : "When was the largest Egyptian pyramid built?"

Answer
Agent custom-at-query-time started with {'query': 'When was the largest Egyptian pyramid built?', 'params': None}
The AI Agent should use a different approach to find the answer since the "seven_wonders_search" tool does not provide information about the construction time of the largest Egyptian pyramid.

Tool: Internet Search
Tool Input: "When was the largest Egyptian pyramid built?"
Tool: Internet Search
Tool Input: "When was the largest Egyptian pyramid built?"Tool: Internet Search
Tool Input: "When was the largest Egyptian pyramid built?"

KeyError Traceback (most recent call last)
in <cell line: 1>()
----> 1 conversational_agent.run("When was the largest Egyptian pyramid built?")

8 frames
/usr/local/lib/python3.10/dist-packages/haystack/nodes/prompt/invocation_layer/chatgpt.py in _extract_token(self, event_data)
84
85 def _extract_token(self, event_data: Dict[str, Any]):
---> 86 delta = event_data["choices"][0]["delta"]
87 if "content" in delta:
88 return delta["content"]

KeyError: 'choices'

How can I avoid such an inconsistency and get the correct answer without any errors?
Thank you for your help in advance!

[azachar]

KeyError Concern:

Shouldn't there be a provision in the code to handle situations when 'choices' isn't available in the event_data? Perhaps by simply returning None or some other fail-safe method?

Thank you for your assistance, and I'm looking forward to a resolution.

[dkbs12]

Hello, @dkbs12!

1. In this use case, we are using two non-deterministic LLMs (one for the Generative QA Pipeline and one for the Agent), that can sometimes make errors in ways that are not easily reproducible.
   With the same setup, I get: "The Colossus of Rhodes was approximately 33 meters (108 feet) tall, while the Statue of Liberty is approximately 46 meters (151 feet) tall."
2. This Agent can use a tool: seven_wonders_search (the Generative QA Pipeline), which is "useful for when you need to answer questions about the seven wonders of the world".

• "The AI Agent does not have any information or tools related to A": currently, I am not able to reproduce this answer, but the idea is that the available Tools are not suitable to ask about a certain topic
• "inconclusive": the Agent used the Tools, but could not find a specific answer.

3. text-davinci-003 is considered legacy, but it is a very good language model. gpt-3.5-turbo is optimized for chat at 1/10th the cost of text-davinci-003 (source: OpenAI docs). In practice, you can try to use only gpt-3.5-turbo, but the quality of reasoning/generation seems a bit lower.

[dkbs12]

Hi,
Thank you for your quick and kind reply.

As for No.2 question, the clause "The AI Agent does not have any information or tools related to A" was actually answer made by generative_pipeline which used the google/flan-t5 model.
I know the prompt template which is adopted by google/flan-t5 and text-davinci-003 is different, but I'not sure it comes from the difference between the prompt templates.
In addition, I found another answer type as "The AI Agent doesn't have any information about how Michael Jackson looks like."
It came from the answer for the question "How does Michael Jackson look like?" made by Agent using the "seven_wonders_search" tool.
Do you have any idea whether this kind of variance comes from the difference of prompt template or not?

Question : "How does Michael Jackson look like?"

Answer
Agent custom-at-query-time started with {'query': 'How does Michael Jackson look like?', 'params': None}
The AI Agent doesn't have any information about how Michael Jackson looks like.Final Answer: The AI Agent doesn't have any information about how Michael Jackson looks like.{'query': 'How does Michael Jackson look like?',
'answers': [<Answer {'answer': "The AI Agent doesn't have any information about how Michael Jackson looks like.", 'type': 'generative', 'score': None, 'context': None, 'offsets_in_document': None, 'offsets_in_context': None, 'document_ids': None, 'meta': {}}>],
'transcript': "The AI Agent doesn't have any information about how Michael Jackson looks like.Final Answer: The AI Agent doesn't have any information about how Michael Jackson looks like."}