question

what do we know about RL environments?

short answer

we now have two useful sources, and one of them is directly about RL environment design for tool-using agents in production debugging.

current kb understanding

1) environment design pattern: hierarchical subagents (HUD case study)

from raw/articles/2026-04-15-debugging-rl-environment.md (user-provided article text):
- flat setup with 104 tools to one model did not work well.
- they switched to hierarchical architecture:
- orchestrator agent gets a small tool surface (about 6 tools),
- each tool maps to a specialized subagent (Sentry, Supabase, Kubernetes, etc.),
- each subagent is its own RL environment with its own scenarios/tools/reward.
- training order recommendation: train subagents first, then train orchestrator.

2) environment/task construction principles (from same HUD source)

• choose domains with verifiable outcomes.
• build tasks from real historical production failures.
• use automatic verification when possible (binary checks are ideal).
• treat environment as both eval harness and training substrate (rollouts -> trajectories -> fine-tuning loop).

3) concrete training/eval datapoints (HUD source)

• Sentry subagent trained on 24 real tasks with explicit verification criteria.
• OpenAI RFT (o4-mini), ~13 hours, 3,000+ traces.
• reported result: trained model 13% vs base 6.3% at 15-step cap (~2x on their harder Sentry task set).

4) RL system performance engineering (Finbarr source)

from raw/articles/2026-04-15-making-rl-fast.md:
- sync -> async RL transition for scaling.
- throughput levers: continuous batching, inflight updates, better thread synchronization.
- tradeoff: more async can increase actor/learner policy lag (off-policy risk).

evidence

sources consulted:
- raw/articles/2026-04-15-debugging-rl-environment.md
- raw/articles/2026-04-15-making-rl-fast.md
- wiki/concepts/2026-04-15-debugging-rl-environment.md

uncertainty

• confidence: medium.
• caveat: HUD claims are from a single org case study; cross-validation against independent benchmarks would strengthen confidence.