AstraFlow

§What AstraFlow supports out of the box

AstraFlow is an open-source, dataflow-oriented reinforcement learning system built for flexibility and scale. It natively supports the following for LLM RL training — without any feature-specific system engineering:

• Fully-async multi-policy collaborative RL
• Elastic heterogeneous cross-region rollouts
• Substitutable rollout and trainer service
• Composable data algorithms

§Dataflow-oriented RL: Separate control, not just compute

1.Dataflow Abstraction: Data algorithms become plug-ins

RaaS nodes pull rollout tasks from the data layer and push completed trajectories back, while trainers independently pull batches. The layer exposes programmable dataflow policies: selective rollout, curriculum scheduling, filtering, sampling, replay, mixing, and staleness correction, without requiring changes to trainers, RaaS nodes, or orchestration. It also regulates autonomous components by throttling slow rollouts, prioritizing fresh trajectories, blocking unsuitable batches via backpressure, and routing multi-policy data using metadata such as policy, model version, timestamp, reward, and task type.

2.RaaS Abstraction: Rollout becomes a service contract

The RaaS contract makes rollout execution substitutable: any efficient agent-serving runtime plugs in. The runtime doesn't need to know how trajectories are sampled, filtered, or assigned to trainers; the trainer doesn't need to know which runtime produced a trajectory. AstraFlow can reuse specialized agent-serving systems as backends instead of re-implementing their internal logic.

RaaS also makes capacity elastic. Adding capacity is just launching more nodes connected to the same dataflow and weight interfaces; removing nodes, slow workers, or failures only affect the rate trajectories arrive, not the trainer loop. Heterogeneous and cross-region settings — where rollout services have different latency, throughput, bandwidth — fall out of the same contract.

3.Trainer Abstraction: Training backends become substitutable

A trainer consumes batches and publishes updated weights via a trainer-side interface. To the trainer, AstraFlow is just a streaming corpus plus a weight publication target: it does not manage rollout workers, serve weights, or coordinate with other trainers. This makes training backends interchangeable: any RL, SFT, or fault-tolerant backend can participate by consuming batches and publishing weights. For multi-policy training, each policy has its own trainer and weight stream while the dataflow layer routes trajectories.

Weight transfer is decoupled from training: model versions are stored and exposed to RaaS nodes, which pull updates when appropriate. The same interface supports full-model transfer, sparse deltas, and version-aware refresh without adding trainer-side complexity.

§BibTeX

@article{zheng2026astraflow,
  title   = {AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs},
  author  = {Zheng, Haizhong and Di, Yizhuo and Wang, Jiahui and Jin, Shuowei
             and Liu, Xueshen and Wu, Yongji and Mao, Z. Morley and Stoica, Ion
             and Zhao, Jiawei and Chen, Beidi},
  journal = {arXiv preprint},
  year    = {2026}
}