Karpathy nanochat GRPO RL Loop Deep Dive

Andre Karpathy recently released nanochat, a project that demonstrates how to train a ChatGPT-like model in just four hours on an 8×H100 node for roughly $100. While the headline was the speedrun, the most fascinating component is tucked away in scripts/chat_rl.py. This 300-line script implements a version of GRPO (Group Relative Policy Optimization), the algorithm that powered DeepSeek's R1 and DeepSeek-V3 models.

In the world of LLM development, accessing high-performance models for testing is crucial. Platforms like n1n.ai provide the necessary infrastructure to experiment with these cutting-edge architectures. This article breaks down why Karpathy's implementation is a masterclass in simplicity and what it reveals about the current state of Reinforcement Learning (RL) in Large Language Models.

The Gap Between SFT and Reasoning

Standard Supervised Fine-Tuning (SFT) allows a model to learn the 'vibes' of a conversation. After SFT, a model can follow tool-use grammar and maintain a persona. However, it often fails at 'last-mile' accuracy in logic-heavy tasks. For instance, given a GSM8K word problem, an SFT model might set up the equation perfectly but fail the basic arithmetic.

The RL step in nanochat targets this specific gap. It doesn't aim to teach the model how to speak; it aims to teach the model how to be right. By using a machine-checkable reward system, Karpathy bypasses the need for complex human preference labels or expensive reward models.

The Simplified GRPO Recipe

DeepSeek introduced GRPO in the DeepSeekMath paper to eliminate the need for a separate 'critic' model, which is traditionally required in PPO (Proximal Policy Optimization). Karpathy’s chat_rl.py takes this simplification even further.

The core logic revolves around the GSM8K dataset. Because every answer in GSM8K ends with a specific marker (e.g., #### 42), the reward function is simply a regular expression and a string comparison.

# The Reward Function in nanochat
GSM_RE = re.compile(r"#### (\-?[0-9\.\,]+)")

def extract_answer(completion):
    match = GSM_RE.search(completion)
    if match:
        return match.group(1).strip().replace(",", "")
    return None

If the model's output matches the ground truth, the reward is 1; otherwise, it is 0. This unambiguous signal is the bedrock of the training loop.

How the Training Loop Works

The training loop follows three critical steps for each problem:

1. Tokenization: Render the prompt through the tokenizer up to the assistant's turn.
2. Batch Generation: Use Engine.generate_batch to produce 16 completions (the group).
3. Advantage Calculation: Score each completion, subtract the group mean, and use the result as the advantage.

# Advantage logic in nanochat
rewards = torch.tensor(rewards, dtype=torch.float, device=device)
mu = rewards.mean()
advantages = rewards - mu

In this setup, if all 16 completions are correct (mean = 1) or all are incorrect (mean = 0), the advantage becomes 0, and no update occurs. The model only learns when there is a variance in performance within the group. This 'group' is doing the work that a value function (critic) normally does in PPO.

nanochat vs. Full GRPO: What's Missing?

Karpathy's implementation is what he calls "REINFORCE with a mean baseline." It strips away several 'safety rails' found in the full DeepSeek recipe to prioritize readability and speed. For developers using n1n.ai to build production-grade RAG or reasoning systems, understanding these trade-offs is vital.

By removing the KL penalty and clipping, Karpathy reduces the complexity of the code. However, this relies on the reward being 'clean' (like a regex). If you were using human preference labels, the model would likely collapse or 'exploit' the reward model without these constraints.

The Gradient Update

The objective function is the classic REINFORCE update from the 1990s, implemented in modern PyTorch. It calculates the log-likelihood of the tokens, multiplies by the advantage, and backpropagates.

# The core optimization block
logp = -model(inputs, targets, loss_reduction='none').view_as(inputs) # \{B, T\}
pg_obj = (logp * advantages.unsqueeze(-1)).sum()
num_valid = (targets >= 0).sum().clamp(min=1)
pg_obj = pg_obj / (num_valid * num_passes * examples_per_rank)
loss = -pg_obj
loss.backward()

This is the purest form of policy gradient. There is no KL term and no ratio. It proves that for machine-checkable tasks, simplicity often beats complexity.

Why Developers Should Care

The release of DeepSeek-V3 and OpenAI o3 has sparked a gold rush in 'reasoning' models. Many projects claim to use GRPO, but the complexity of the implementation varies wildly. Karpathy's code shows that the 'intelligence' isn't just in the algorithm—it's in the reward design.

If you are building a system where the output can be verified (code execution, math, structured JSON), you don't need a 10,000-line RL framework. You need a clean reward function and a group-based advantage. For those looking to integrate these reasoning capabilities into their own apps, n1n.ai offers a unified API to access the world's most powerful LLMs, including those fine-tuned with GRPO.

Professional Tips for Implementing RL

1. Start with SFT: RL is not a substitute for supervised learning. It is a 'last-mile' optimization. Ensure your model already understands the basic format before applying RL.
2. Monitor Pass@k: In nanochat, the evaluation uses pass@k. This measures if any of the k generated samples are correct. It is a much more robust metric for reasoning than simple accuracy.
3. Reward Engineering: If your reward function has a 'leak' (e.g., the model finds a way to get a high score without solving the problem), the model will find it. This is why machine-checkable rewards are superior to LLM-as-a-judge for logic tasks.

Conclusion

Karpathy's nanochat is a reminder that the fundamentals of AI—like REINFORCE—are still incredibly powerful when applied correctly. By stripping GRPO down to its 300-line essence, he has demystified the 'magic' behind modern reasoning models. Whether you are a solo developer or an enterprise, the path to better LLM performance lies in unambiguous rewards and efficient training loops.

Get a free API key at n1n.ai