Handling tool results larger than the model’s context window

[anmol098]

Hi team,

I’m working with the Vercel AI SDK and using tool calls successfully. The issue I’m running into is with tool results that are larger than the model’s context window.

• The model makes the tool call as expected.
• The tool responds, but the result is very large (sometimes exceeding the model’s context limit).
• Since the SDK submits the entire tool result back into the conversation, this can cause the context window to overflow.

This is the error which can be seen

Error [AI_APICallError]: This model's maximum context length is 128000 tokens. However, your messages resulted in 209275 tokens (208360 in the messages, 915 in the functions). Please reduce the length of the messages or functions.

I’m curious about the recommended patterns for handling this scenario within the SDK or in general how this scenario is handled.

Any guidance would be super helpful. Thanks!

[lgrammel]

You can use toModelMessage to control what is being sent to the model in AI SDK 5.

[devanshg03]

Does this also handle context length? The only docs I found are for this https://ai-sdk.dev/docs/reference/ai-sdk-ui/convert-to-model-messages, but seem to be more tailored for passing in generated images to the mode

[github-actions[bot]]

This discussion was automatically locked because it has not been updated in over 30 days. If you still have questions about this topic, please ask us at community.vercel.com/ai-sdk