Can't figure out how to get a final answer out of a multi-turn generateText

[austinm911]

Here's some pseudocode of what I'm trying to do. I've been using gemini flash 8b to test.

I basically have an array of objects with content that I want the model to iterate over, analyze each object of data, and then return a final answer based on all of the analysis.

However, i will pass in 5 objects of data and at the end it will return the text generation of the 5th object analysis. I can't figure out how to force the answer tool to be the last tool call to be executed which uses all of the generated intermediate steps of text to produce the final structured output.

export async function aiExtraction(
	params: Data,
	options: { markdownBlocks: ParsedMarkdown[]; model: AllModelValues },
) {
	const prompt = `
	<Summary>
	You will be given access to markdown blocks containing information about a company and its leader. Your task is to extract specific data points from the provided information.
	</Summary>

	<Task>
	Find the data listed below for the company named ${companyName} and its leader ${leaderName}.
	</Task>

	<Data to find>
	- Products
	- Services
	- Company description
	- Company website
	</Data to find>


	<Instructions>
	- One by one, analyze each markdown block using the parseMarkdownBlocks tool and find the required information.
	- At a minimum, you must analyze 2 blocks.
	- You are required to sequentially return a tool call and generate a tool call response for each block before moving on to the next.
	- Provide a brief explanation of your findings from each block.
	- If no valid information is found after checking the blocks, return null for the missing fields.
	</Instructions>

	<Final Instruction>
	At the end of your analysis, you must use the 'answer' tool to provide the final structured response, even if no relevant information was found. This will be the last tool call you make.
	</Final Instruction>
  `

	const aiResponse = await generateText({
		model: registry.languageModel(options.model),
		prompt,
		maxRetries: 3,
		system: 'You are a precise assistant specialized in extracting data from structured content.',
		tools: {
			parseMarkdownBlocks: parseMarkdownBlocksTool(markdownBlocks),
			answer: tool({
				description: 'A tool for providing the final answer.',
				parameters: z.object({
					steps: z.array(
						z.object({
							reasoning: z.string(),
							blockNumber: z.number(),
						}),
					),
					answer: scrapedSchema,
				}),
			}),
		},
		toolChoice: 'auto',
		temperature: 0,
		experimental_telemetry: { isEnabled: true, metadata: { model: options.model } },
		maxSteps: Math.min(markdownBlocks.length) + 3,
	})

	// Find the answer tool call
	const answerToolCall = rest.toolCalls.find((call) => call.toolName === 'answer')

	if (answerToolCall) {
		const finalAnswer = answerToolCall.args

		// Remove the 'steps' property before returning
		const { steps, ...scrapedData } = finalAnswer
		return { response: aiResponse, result: scrapedData }
	}
	return { response: aiResponse, result: null }
}

Here this image, Langfuse maybe has a UI bug, but the last generation is the last object the model analyzes with the arrow pointing to it.

Here are the steps produced. The first index produces the desired results, but I still want the model to continue checking. On one hand I can't get the model to only check 2 indexes (maybe just small model issue)

I have no problems if i prompt the model to stop and return an answer as soon as it gets the info from the 1st block. But when I edit my prompt to try to have the model "iterate further" im running into this issue.

🚀 ~ file:  ~ response: [
  {
    text: "",
    finishReason: "tool-calls",
    isContinued: false,
    stepType: "initial",
    warnings: [],
    toolCalls: [
      {
        type: "tool-call",
        toolCallId: "tool_0_parseMarkdownBlocks",
        toolName: "parseMarkdownBlocks",
        args: {
          index: 0,
        },
      }
    ],
    toolResults: [
      {
        toolCallId: "tool_0_parseMarkdownBlocks",
        toolName: "parseMarkdownBlocks",
        args: {
          index: 0,
        },
        result: {...},
      }
    ],
  }, {
    text: "Block 0 contains Company's information.... The source website is https://....com/.\n\n",
    finishReason: "tool-calls",
    isContinued: false,
    stepType: "tool-result",
    warnings: [],
    toolCalls: [
      {
        type: "tool-call",
        toolCallId: "tool_0_parseMarkdownBlocks",
        toolName: "parseMarkdownBlocks",
        args: {
          index: 1,
        },
      }
    ],
    toolResults: [
      {
        toolCallId: "tool_0_parseMarkdownBlocks",
        toolName: "parseMarkdownBlocks",
        args: {
          index: 1,
        },
        result: {... },
      }
    ],
  }, {
    text: "Block 1 contains a similar biography to Block 0, but no new information.  Therefore, the information from Block 0 is preferred.\n",
    finishReason: "tool-calls",
    isContinued: false,
    stepType: "tool-result",
    warnings: [],
    toolCalls: [
      {
        type: "tool-call",
        toolCallId: "tool_0_parseMarkdownBlocks",
        toolName: "parseMarkdownBlocks",
        args: {
          index: 2,
        },
      }
    ],
    toolResults: [
      {
        toolCallId: "tool_0_parseMarkdownBlocks",
        toolName: "parseMarkdownBlocks",
        args: {
          index: 2,
        },
        result: {...},
      }
    ],
  }, {
    text: "Block 2 is the .... No additional information is found here.\n\n",
    finishReason: "tool-calls",
    isContinued: false,
    stepType: "tool-result",
    warnings: [],
    toolCalls: [
      {
        type: "tool-call",
        toolCallId: "tool_0_parseMarkdownBlocks",
        toolName: "parseMarkdownBlocks",
        args: {
          index: 3,
        },
      }
    ],
    toolResults: [
      {
        toolCallId: "tool_0_parseMarkdownBlocks",
        toolName: "parseMarkdownBlocks",
        args: {
          index: 3,
        },
        result: {...},
      }
    ],
  }, {
    text: "Block 3 is ... However, I will stick with the information already extracted from Block 0 as it contains the most relevant information.\n\n",
    finishReason: "tool-calls",
    isContinued: false,
    stepType: "tool-result",
    warnings: [],
    toolCalls: [
      {
        type: "tool-call",
        toolCallId: "tool_0_parseMarkdownBlocks",
        toolName: "parseMarkdownBlocks",
        args: {
          index: 4,
        },
      }
    ],
    toolResults: [
      {
        toolCallId: "tool_0_parseMarkdownBlocks",
        toolName: "parseMarkdownBlocks",
        args: {
          index: 4,
        },
        result: {... },
      }
    ],
  }, {
    text: "Block 4 is from the....  Again, the information from Block 0 remains the most complete and relevant.\n\n",
    finishReason: "unknown",
    isContinued: false,
    stepType: "tool-result",
    warnings: [],
    toolCalls: [],
    toolResults: [],
  }
]

Ultimately I expect that in this trace, the model uses all of the text generated to produce a final result, mostly relying on tool call 0 in this example.

I'm referencing the https://sdk.vercel.ai/docs/ai-sdk-core/agents#structured-answers docs here and cannot get the model to terminate like the docs suggest 😔

[austinm911]

I realize this may be an issue with a) using a weaker model and b) the docs using OpenAIs model which has the strucutredOutput setting to true which is not available on other models. So for now, I'm getting around it by just using a 2nd generateObject function to take the previous text responses from the steps to try to return an object.

However, the docs seem to suggest the expected behavior of using an answer tool would provide a structured response with other models, since you're using the model to generate the arguments. Not sure if I'm following this correct

[lgrammel]

Can you try forcing tool calls with toolChoice: required and potentially tweak the prompt? https://sdk.vercel.ai/docs/ai-sdk-core/tools-and-tool-calling#tool-choice

[austinm911]

@lgrammel I was able to get closer to a solution using gpt-4o mini and keeping everything in one generateText function. Generally was noticing that with the same prompt and required set, I was running into the issues described.