Gemini 1.5 Flash: Candidate was blocked due to RECITATION when responseMimeType is json

[marian2js]

Description of the bug:

When responseMimeType: 'application/json', a request is failing with error:
[GoogleGenerativeAI Error]: Candidate was blocked due to RECITATION.

However, without responseMimeType, the same prompt works (returns a markdown with json).

The exact same instructions and prompt work on the AI Studio, even with output in JSON on.

// The error happens even if safety settings are set to block none.
const safetySettings = [
  {
    category: HarmCategory.HARM_CATEGORY_HARASSMENT,
    threshold: HarmBlockThreshold.BLOCK_NONE,
  },
  {
    category: HarmCategory.HARM_CATEGORY_HATE_SPEECH,
    threshold: HarmBlockThreshold.BLOCK_NONE,
  },
  {
    category: HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT,
    threshold: HarmBlockThreshold.BLOCK_NONE,
  },
  {
    category: HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,
    threshold: HarmBlockThreshold.BLOCK_NONE,
  },
]

const model = this.genAI.getGenerativeModel({
  model: 'gemini-1.5-flash-latest',
  systemInstruction: instructions,
  safetySettings,
})

const generationConfig = {
  temperature: 0,
  topP: 0.95,
  topK: 64,
  maxOutputTokens: 8192,
  responseMimeType: 'application/json', // fails only if this option is sent. 
}

const chatSession = model.startChat({
  generationConfig,
})

const result = await chatSession.sendMessage(prompt)
const text = result.response.text() // throws [GoogleGenerativeAI Error]: Candidate was blocked due to RECITATION.

Actual vs expected behavior:

Actual: Throws [GoogleGenerativeAI Error]: Candidate was blocked due to RECITATION
Expected: Return the same result as in the AI Studio.

Any other information you'd like to share?

No response

[ryanwilson]

Hi @marian2js, sorry for the troubles. Does every prompt cause this, or only specific prompts? If it's a specific prompt, are you able to share the prompt (and system instructions) so we can try to reproduce it?

[marian2js]

Hi @ryanwilson. The issue happens only with a very specific prompt that I cannot share publicly. I've been trying to remove the personal data from it, but as soon as I do, it starts working.

I noticed that when I remove the responseMimeType, the JSON returned in the markdown is invalid as it has a js variable: { "key": value }. However, with a different prompt the model returned the invalid json, so I don't know if the issue is related to that.

I am sorry for not being of more help.

[ryanwilson]

No worries! Out of curiousity, do you run into the same issue if you use sendMessageStream instead of sendMessage? That could be the difference with AI Studio, where the response is streamed.

[marian2js]

Hi @ryanwilson, the bug doesn't happen with sendMessageStream. I made changes to my prompt, so this bug is not triggered anymore for me. But I can confirm the bug is still happening with my old prompt and sendMessage.

[florian583]

Hi guys,

I'm experiencing the exact same problem as @marian2js. We're using Gemini with Vertex to extract structured data (as JSON) from job offer listing PDFs.

Here's what I've tried so far:

• Using the 1.5 Pro model instead of the Flash model
• Switching to sendMessageStream from sendMessage
• Using 1.0 Pro (works, but the output is non-usable)
• Changing region, originally using asia-east2, switched to europe-west9, same result

Oddly enough, the issue seems to only happen when processing non-English documents. When I upload PDFs in French, I always run into the RECITATION problem.

On the flip side, if I use Google Vision for OCR on the PDF, and then use the Vercel AI SDK with chat and Gemini 1.5 Flash, it works perfectly with the same prompt, but on the OCR data (string) instead of the inline PDF.

Hope this info helps in figuring out the issue!

[GuyVivedus]

Hi folks,

Running into the same problem as @marian2js (with almost identical API call settings but with Golang - model conf, generationConfig, SafetyConfig all the same). Again parsing a plaintext file with a big prompt to output JSON (where the plaintext was originally converted from PDF).

Setting model.GenerationConfig.ResponseMIMEType = "application/json" hits the "blocked: candidate: FinishReasonRecitation" resp with no PromptFeedback value, annoyingly, so flying a bit blind as to the cause.

If it's helpful the content is senior school syllabus so shouldn't be anywhere close to hitting any of the harm response thresholds either way.

Commenting ResponseMIMEType with a slight prompt modification gets a usable (yet obviously inconsistent) result.

Much appreciated!

[KlimentP]

Hi all,
Running into the same issue, model: 'gemini-1.5-flash-latest' , responseMimeType: 'application/json'.

I am parsing 50 text documents by passing a json schema for the desired output, and more than half of them fail, but always the same ones.

Setting the response type to stream does not fix it and the error occurs always in the same chunk.

Prompt is f"""You will be provided content in the form of html. Using this content, return a valid json object that is based entirely on information from the content, not your guess. The content should satisfy the following json schema: {schema} """

[chanmathew]

Experiencing same issue as well, same setup as others above.

Also get it if i run generateContent:

const generationConfig = {
temperature: 1,
topP: 0.95,
topK: 64,
maxOutputTokens: 8192,
responseMimeType: 'application/json'
}

const safetySettings = [
{
	category: HarmCategory.HARM_CATEGORY_HARASSMENT,
	threshold: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE
},
{
	category: HarmCategory.HARM_CATEGORY_HATE_SPEECH,
	threshold: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE
},
{
	category: HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT,
	threshold: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE
},
{
	category: HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,
	threshold: HarmBlockThreshold.BLOCK_MEDIUM_AND_ABOVE
}
]
const model = genAI.getGenerativeModel({
		model: 'gemini-1.5-flash-latest',
		generationConfig,
		safetySettings,
		systemInstruction:
			'My prompt here.'
	})

const result = await model.generateContent(content)

const response = result.response
const text = response.text()

[ShivQumis]

Changing to stream mode worked for me as well:

const safetySettings = [
   {
     category: HarmCategory.HARM_CATEGORY_HARASSMENT,
     threshold: HarmBlockThreshold.BLOCK_NONE,
   },
   {
     category: HarmCategory.HARM_CATEGORY_HATE_SPEECH,
     threshold: HarmBlockThreshold.BLOCK_NONE,
   },
   {
     category: HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT,
     threshold: HarmBlockThreshold.BLOCK_NONE,
   },
   {
     category: HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT,
     threshold: HarmBlockThreshold.BLOCK_NONE,
   },
];


try {
           const chatSession = this.model.startChat({
               generationConfig: {
                   temperature: 1,
                   topP: 0.95,
                   topK: 64,
                   maxOutputTokens: 8192,
                   responseMimeType: "text/plain",
               },
               history: history,
               systemInstruction: systemMessage,
               safetySettings: safetySettings
           });
           
           var stream = true;
           if (stream) {
               const result = await chatSession.sendMessageStream(currentMessage);
             for await (const item of result.stream) {
                   console.log("Stream chunk: ", item.candidates[0].content.parts[0].text);
               } 
               const aggregatedResponse = await result.response;
               return aggregatedResponse.text();
           } else {
               const result = await chatSession.sendMessage(currentMessage);
               return result.response.text();
           }
       } catch (error) {
           console.error("Error during chat session:", error);
           throw error; // Re-throw the error after logging or handle it as needed
       }

[yharaskrik]

I started to receive this today as well, didn't get it at all yesterday and I made hundreds of calls, now today a significant number of them are receiving the error.

[jsomeara]

Also getting this error.

[Kashi-Datum]

Any updates on this? As it stands at the current moment, using Gemini Pro 1.5 for RAG-base Q/A with chat history is incredibly unreliable.

Simply ask 2-3 questions about a document with some overlap, and you will encounter a RECITATION error almost guranteed.

I.e, using LangChain:

const chatTemplate = ChatPromptTemplate.fromMessages([['system', 'You are a helpful assistant. Answer all questions to the best of your ability.'], new MessagesPlaceholder('history'), new MessagesPlaceholder('input')]);
const chain = chatTemplate.pipe(pdfModel).pipe(new StringOutputParser());

const chainWithHistory = new RunnableWithMessageHistory({
		runnable: chain,
		getMessageHistory: () => chatHistory,
		inputMessagesKey: 'input',
		historyMessagesKey: 'history',
		config: { configurable: { sessionId: decodedToken.uid } }
	});

const response = await chainWithHistory.invoke({ input: input });

If you simply ask the model to recite a specific section of a file/PDF twice in a row, you will get a Recitation error GURANTEED (whether you pass the input File in once at the beginning of the conversation, only in the latest message, or as part of every message input is irrelevant).