Fine-Tuning Multi-Turn RL Search Agents on Amazon SageMaker AI

Building autonomous search agents capable of navigating complex information retrieval tasks requires striking a difficult balance between multi-turn reasoning capabilities, overall operational latency, and execution cost. While frontier models such as Claude 3.5 Sonnet or DeepSeek-V3 demonstrate remarkable zero-shot tool-use accuracy, relying on them for every hop in a multi-turn search trajectory rapidly inflates API costs and introduces unacceptable latency overheads. Developers can evaluate initial baselines using high-availability, low-latency LLM routers like n1n.ai, but scaling thousands of iterative tool invocations per second often calls for specialized, self-hosted models.

Fine-tuning a lightweight model (such as Qwen2.5-7B or Llama-3.1-8B) with Multi-Turn Reinforcement Learning (MTRL) allows developers to distill frontier-level decision-making directly into an open-weights model fine-tuned for a target API environment. In this technical deep dive, we walk through the process of fine-tuning a multi-turn search agent using Group Relative Policy Optimization (GRPO) on Amazon SageMaker AI. We will review environment design, policy trajectory formulations, custom reward shaping, and deployment strategies.

The Shift from Single-Turn RAG to Multi-Turn RL Agents

Standard Retrieval-Augmented Generation (RAG) relies on a single-step pipeline: user query $\\rightarrow$ dense retriever $\\rightarrow$ context injection $\\rightarrow$ generator response. However, realistic information gathering often demands dynamic, multi-hop interaction loops:

1. Generating an initial sub-query search API payload.
2. Parsing unstructured or semi-structured search JSON returns.
3. Evaluating result relevancy and detecting missing contextual nodes.
4. Refining parameters or reformulating search terms for subsequent calls.
5. Synthesizing evidence across multiple turns into a verifiable final answer.

+----------------+      1. Formulate Query      +--------------------+
|                | ---------------------------> |                    |
|  Search Agent  |                              |  Search Tool API   |
|   (Policy LLM) | <--------------------------- |  (Vector Database) |
|                |      2. Returned Evidence    +--------------------+
+----------------+                                        |
        |                                                 |
        | 3. Evaluate Sufficiency                         |
        v                                                 |
  [ Decision ] ---> (Incomplete) -> Loop to Step 1 -------+
        |
        +---------> (Complete)   -> 4. Emit Final Answer

Supervised Fine-Tuning (SFT) on static trajectory datasets frequently falls short because small open-source models overfit to specific tool formats and fail gracefully when search APIs yield empty, unexpected, or noisy responses.

MTRL resolves this limitation by exposing the policy to simulated tool interactions during training. Using algorithms like GRPO (or multi-turn PPO), the agent explores various interaction pathways and learns optimal decision policies through reward signals assigned over full step trajectories.

Multi-Turn RL Framework Mechanics

In a multi-turn setup, the environment is framed as a Markov Decision Process (MDP):

• State Space ($S_t$): The complete dialogue history, including system prompts, previous search queries, search execution outputs, and generated chain-of-thought tokens up to turn $t$.
• Action Space ($A_t$): The token output sequence emitted by the policy model, representing either a structured tool call (e.g., `{"action": "search