Configurable RAG Response Format and/or Structured Output for knowledge_search tool

[onmete]

🚀 Describe the new functionality needed

Currently, the knowledge_search tool in llama-stack produces a text-based response format with:

• Header (hardcoded): "knowledge_search tool found N chunks:\nBEGIN of knowledge_search tool results.\n"
• Chunks (configurable via RAGQueryConfig.chunk_template): Default: "Result {index}\nContent: {chunk.content}\nMetadata: {metadata}\n"
• Footer (hardcoded): "END of knowledge_search tool results.\n"

Source: llama_stack/providers/inline/tool_runtime/rag/memory.py

Proposed improvements (in order of preference):

1. Best: Provide structured output (JSON) as an option, returning parsed chunk objects with content and metadata directly accessible without text parsing. Ideally, also expose the relevance/similarity score from the vector search, which is currently not included in the response.

2. Good: Allow full client-side control of the response format, including:

   • Custom header template (or ability to omit)
   • Custom footer template (or ability to omit)
   • Custom chunk template (already supported)

💡 Why is this needed? What if we don't build it?

Why it's needed:

• Clients currently must implement brittle regex-based parsing to extract individual RAG chunks from the text response
• The hardcoded header/footer and default chunk format create tight coupling between llama-stack's internal implementation and client-side parsing logic
• Any future changes to the format (even whitespace changes) can silently break client parsers

Benefits of structured output:

• Clean API contract with explicit fields
• No text parsing required on client side
• Type-safe access to chunk content and metadata
• Opportunity to expose additional data (e.g., relevance scores) that are currently discarded
• Easier to maintain compatibility across versions

Other thoughts

No response

[franciscojavierarceo]

@onmete can you elaborate why you want the chunks directly? Are you using the responses API?

CC @mattf

[mattf]

@onmete can you elaborate why you want the chunks directly? Are you using the responses API?

please include example client code where you're doing the regex.

generally, tool output is expected to be input to an llm.

[onmete]

We want to have chunks directly, as we don't need to parse the blob when storing those as conversation metadata then. This serves different purposes, for example post-analysis of document relevance.
We use Agents API.
Relevant code: https://github.com/lightspeed-core/lightspeed-stack/blob/main/src/utils/types.py#L96-L117 (note the llama_stack_client.types.tool_execution_step.ToolExecutionStep)