INTRODUCING PRL / AGENTIC RL INFRASTRUCTURE
Program howagents learn.
One Python function. Three ways to shape experience.
PRL opens a programmable interface between rollout and advantage computation. Decide how much to learn, what to learn, and where to explore again.
REWRITE
REROLLYour idea.
Your transform.
RL INFRA, IN MOTION
A new place to put your ideas.
01 / WHY PRL
The rollout ends. Your ideas begin.
A successful trajectory can still contain wrong turns. A failed attempt can still contain useful steps. The final reward alone leaves many choices about how to use that experience.
PRL (Programmable Reinforcement Learning) is an Agentic RL framework built on Miles. It exposes trajectory processing as a Python function. Read the task, reward, reasoning, commands, and terminal output; then decide how that experience should enter training. The framework connects your decisions to the training loop.
02 / A PROGRAMMABLE INTERFACE
Put your idea inside the loop.
Keep trajectory interventions together in fn(traj). PRL translates your decisions into token-level loss masks, advantage coefficients, and new samples, then passes them into GRPO, PPO, or REINFORCE++.
Need a teacher or judge? Call ask from your function. The framework handles retries, rate limits, timeouts, fallback models, and call records.
Complete and score a trajectory
Reweight · Rewrite · Reroll
GRPO / PPO / REINFORCE++
Three ways to work with a trajectory.
Adjust weights, change context, or explore another continuation. Switch between the operations to see what the interface makes possible.
Reweight / How much should this step count?
Keep the trajectory text and change how steps contribute to the loss. Retain a useful step, mask an irrelevant one, or apply a negative weight to a bad step. You can also target a text span within a step.
step.mask() · step.weight = -1.0ATLAS-style transforms are one starting point: select critical steps, or distinguish useful actions from mistakes within a successful trajectory.
Rewrite / What should the policy learn from?
Edit reasoning, commands, or terminal output, or remove an entire turn. PRL re-encodes the changed parts. Teacher-written text can serve as context without directly contributing to the loss.
step.edit_thinking(old, new)Explore how teacher feedback enters context, how edits affect later steps, and which text should contribute to the training objective.
Reroll / Where should exploration restart?
Choose a step, replay its prefix in a fresh sandbox, and let the policy continue. The new branches are tested, scored, and included in training. Branch selection, sampling allocation, and credit assignment are open to experimentation.
step.reroll(n=2, credit="member")The illustration uses member credit. Subgroup credit is currently supported with GRPO only.
Illustrative diagrams; these are not recorded training trajectories.
03 / ONE FUNCTION
A method can start as one function.
This minimal ATLAS-style example asks a judge to identify critical steps and masks the rest. Save it as transforms/focus.py and select focus at runtime. Existing transforms provide starting points for more elaborate interventions.
04 / BUILT TO EXPLORE
Build on the examples. Try your own approach.
PRL is infrastructure for experimenting with Agentic RL. Use an existing transform, modify it, or chain several in order. Explore a new kind of feedback, a different credit scheme, or another way for a teacher and policy to work together.
Use feedback more precisely
Move from whole-step selection to the specific span that went wrong. Explore what should be reinforced, ignored, or pushed back on.
Rethink the teacher’s role
Let a teacher supply context, revise reasoning, or choose where the policy should try again. Study how the policy uses that feedback.
Change where exploration happens
Try different branch selectors, sample counts, and credit rules. The branch template exposes replaceable components.
TRANSFORM=rewrite+atlasChain existing transforms, or replace them with your own.
05 / EARLY EXPERIMENTS
Early experiments.
We ran several transforms end to end with Qwen3.5-9B, training on OpenThoughts-Agent-v1-RL and evaluating on OpenThoughts-TB-dev. These runs provide initial examples of what can be implemented through the interface.
Different transforms in the same framework.
Reweight, Rewrite, and Reroll have all run through the full training pipeline. Reweight performed best in these initial runs, while the teacher-editing and reroll configurations leave room for further exploration. The transforms and templates are available as starting points for your own experiments.
Similar training reward, different held-out results.
06 / START SMALL
Write a function. Run an experiment.
Start with the 8 recorded training trajectories included in the repository. Check a transform without a GPU, inspect its decisions and model calls, then connect it to cluster training.
Local checks need Python 3.10+ and a tokenizer. Model calls and cluster training need additional setup.
Run from the PRL repository root · Bash
Training jobs run a preflight check too. A transform error, timeout, or all-skipped result prevents training from starting.
RESOURCES
Explore PRL
Continue with the code, interfaces, and examples.