INTRODUCING PRL / AGENTIC RL INFRASTRUCTURE

Program howagents learn.

One Python function. Three ways to shape experience.

PRL opens a programmable interface between rollout and advantage computation. Decide how much to learn, what to learn, and where to explore again.

RL INFRA, IN MOTION

A new place to put your ideas.

01 / WHY PRL

The rollout ends. Your ideas begin.

A successful trajectory can still contain wrong turns. A failed attempt can still contain useful steps. The final reward alone leaves many choices about how to use that experience.

PRL (Programmable Reinforcement Learning) is an Agentic RL framework built on Miles. It exposes trajectory processing as a Python function. Read the task, reward, reasoning, commands, and terminal output; then decide how that experience should enter training. The framework connects your decisions to the training loop.

02 / A PROGRAMMABLE INTERFACE

Put your idea inside the loop.

Keep trajectory interventions together in fn(traj). PRL translates your decisions into token-level loss masks, advantage coefficients, and new samples, then passes them into GRPO, PPO, or REINFORCE++.

Need a teacher or judge? Call ask from your function. The framework handles retries, rate limits, timeouts, fallback models, and call records.

01Rollout

Complete and score a trajectory

02 / YOUR FUNCTION↗ ask
fn(traj)

Reweight · Rewrite · Reroll

03Policy update

GRPO / PPO / REINFORCE++

FIG. 01 / Transform a completed trajectory before computing advantages.

Three ways to work with a trajectory.

Adjust weights, change context, or explore another continuation. Switch between the operations to see what the interface makes possible.

Reweight / How much should this step count?

Keep the trajectory text and change how steps contribute to the loss. Retain a useful step, mask an irrelevant one, or apply a negative weight to a bad step. You can also target a text span within a step.

ONE TRAJECTORY / DIFFERENT STEP WEIGHTS
01+1
02mask
03+1
04−1
05mask
06+1
step.mask() · step.weight = -1.0

ATLAS-style transforms are one starting point: select critical steps, or distinguish useful actions from mistakes within a successful trajectory.

Illustrative diagrams; these are not recorded training trajectories.

03 / ONE FUNCTION

A method can start as one function.

This minimal ATLAS-style example asks a judge to identify critical steps and masks the rest. Save it as transforms/focus.py and select focus at runtime. Existing transforms provide starting points for more elaborate interventions.

transforms/focus.py

04 / BUILT TO EXPLORE

Build on the examples. Try your own approach.

PRL is infrastructure for experimenting with Agentic RL. Use an existing transform, modify it, or chain several in order. Explore a new kind of feedback, a different credit scheme, or another way for a teacher and policy to work together.

01 / FEEDBACK

Use feedback more precisely

Move from whole-step selection to the specific span that went wrong. Explore what should be reinforced, ignored, or pushed back on.

02 / TEACHER

Rethink the teacher’s role

Let a teacher supply context, revise reasoning, or choose where the policy should try again. Study how the policy uses that feedback.

03 / EXPLORATION

Change where exploration happens

Try different branch selectors, sample counts, and credit rules. The branch template exposes replaceable components.

COMPOSABLE BY DESIGNTRANSFORM=rewrite+atlas

Chain existing transforms, or replace them with your own.

05 / EARLY EXPERIMENTS

Early experiments.

We ran several transforms end to end with Qwen3.5-9B, training on OpenThoughts-Agent-v1-RL and evaluating on OpenThoughts-TB-dev. These runs provide initial examples of what can be implemented through the interface.

FIG. 02 / TB-dev (%)

Different transforms in the same framework.

Baseline / standard RLReweightRewriteReroll
Model: Qwen3.5-9B. Training: OpenThoughts-Agent-v1-RL. Evaluation: OpenThoughts-TB-dev.

Reweight, Rewrite, and Reroll have all run through the full training pipeline. Reweight performed best in these initial runs, while the teacher-editing and reroll configurations leave room for further exploration. The transforms and templates are available as starting points for your own experiments.

FIG. 03 / Training reward (%)

Similar training reward, different held-out results.

GRPOsigned ATLAS
Faint lines: training reward on each of 68 updates (32 tasks × 8 attempts). Solid lines: the mean of the last 8. Both runs rise alike, from about 57% in the first quarter of training to 67% in the last; the gap appears on held-out TB-dev, 58.09% versus 54.32%.

06 / START SMALL

Write a function. Run an experiment.

Start with the 8 recorded training trajectories included in the repository. Check a transform without a GPU, inspect its decisions and model calls, then connect it to cluster training.

Local checks need Python 3.10+ and a tokenizer. Model calls and cluster training need additional setup.

Run from the PRL repository root · Bash

Training jobs run a preflight check too. A transform error, timeout, or all-skipped result prevents training from starting.

PROGRAM WHAT COMES NEXT.

Your next idea. Your next transform.

Explore PRL

RESOURCES

Explore PRL

Continue with the code, interfaces, and examples.