Skip to content

Xaytune

Xaytune is an agent-native experiment control plane for model post-training and adaptation.

It records what an experiment is testing, compiles each candidate into a plan a trainer can run, submits that plan to a runtime, and keeps a durable record of everything that happened. That record is what lets a restarted controller pick up running work instead of losing it or starting it twice. It is built to answer: what should this training experiment do next?

Xaytune controls the experiment. Trainer integrations compile training intent. Runtime integrations execute it. Infrastructure schedules and runs the workloads.

Alpha

1.0.0a2 is the second pre-release of the control plane: declare an experiment, and it trains, evaluates, decides, plans and branches to the next candidate on its own, on a durable record. It is a pre-release: install it with pip install "xaytune==1.0.0a2" (a plain pip install xaytune still installs 0.6.0), and expect the API to change before 1.0.0. The legacy trainer API is still included.

Start here

What Xaytune owns, and what it delegates

Xaytune owns Today
Experiment lifecycle and durable controller state Available
Candidate identity and scientific lineage Available
Training orchestration across trainer backends Available: Native and TRL, on a local runtime
Evaluation orchestration Available: the built-in native evaluator, and lm-eval benchmark tasks pinned at submission (on main, unreleased)
Decisions and branching Decisions available: deterministic thresholds on the objective; branching planned
Resilience policy and semantic recovery Planned
Policy gates, budgets and agent-driven control Budgets available: runs, failures and parallel runs, on a durable ledger (on main, unreleased); policy gates and agent control planned

Xaytune delegates tensor execution, distributed training, worker management and second-scale worker recovery, scheduling, and infrastructure admission to trainer, runtime and platform integrations. The boundary is experiment topology, not GPU count.

Available today

Status as of 1.0.0a2:

  • A durable record: SQLite persistence, versioned migrations, atomic state + event + outbox transactions, a RuntimeOperation journal that records every external effect before it is attempted, and cancellation as a recorded Action.
  • Candidates with identities: CandidateSpec and a versioned CandidateFingerprint.
  • Compilation: NativeCompiler and TRLCompiler compile full-parameter SFT into a serializable TrainingExecutionSpec, and refuse any candidate that leaves a training-relevant value to a trainer default.
  • A restart-safe local runtime: LocalRuntime, with idempotent submit_or_get and versioned telemetry.
  • An embedded controller: EmbeddedControllerHost and ExperimentHandle (submit, status, wait, cancel, events, attach). A restarted process adopts running work and never submits it twice.
  • Durable evaluation, with a built-in evaluator: evaluation runs, attempts, results and cycles, recorded and restart-safe. The native evaluator measures next-token loss, perplexity and token accuracy on a local held-out file pinned by its content digest.
  • Durable decisions: an evaluated candidate is decided from the record -- the objective's target and constraints against the recorded results -- and the decision is written with the transitions it causes, in one commit. A candidate that cannot be decided (no target, a missing metric) stays DECIDING, with the reason recorded.
  • The adaptive loop: a missed target can branch; a RuleBasedPlanner proposes the next candidate, branching records it, and the controller trains it, from one submission and after any restart.
  • Budgets, policy and approval: run, failure, parallelism and wall-time limits on an append-only ledger; typed actions judged by a PolicyEngine, with human approval.
  • Incidents, checkpoints and recovery: durable incidents, exact checkpoints, governed CUDA OOM and numerical recovery.
  • A local daemon and CLI with a durable controller lease.
  • Ray, KubeRay, Ray Tune and TorchFT behind the runtime, search and resilience boundaries (TorchFT as a checked request, not yet hosted).
  • An audited LLM planner whose action proposals are escalated, not carried out.
  • Studio: a read-only HTTP API and a React client, with loss charts.
  • Agent harness building blocks, local and in-process only.
  • Upgrade safety: a build refuses a database a newer Xaytune migrated.

Planned

Durable harness execution, evaluation and search; budgets on GPU-hours, tokens and cost; decisions that compare candidates; hosting TorchFT; and Training Hub. None of these exist yet.

Architecture

The diagram shows the status after PR-015: solid green boxes were implemented then, dashed boxes planned; Available today above is current. The runtime feedback path is separate from the path that submits work.

Xaytune target architecture: experiment control plane, trainer compilation, capability resolution, runtime backends, and infrastructure, with implementation status after PR-015.

The architecture specification defines the contracts and dependency boundaries, and the implementation plan the order the remaining work lands in.

Roadmap

Architectural band Status
A — domain foundation Complete
B — persistence and control records Complete
C — compile/execute, local runtime, runtime reconciliation Complete
D — durable evaluation and decisioning Complete: lifecycle, native evaluator, threshold decisions; lm-eval evaluator on main
E — policy and budget over the Action substrate Complete on main: budget ledger, typed actions, policy and approval; carrying actions out is band F
F — checkpoints, semantic recovery, interventions Planned
G — rule-based planner and experiment branching Planned
H — daemon hosting and whole-controller restart Planned
I — LLM planner Planned
J — Ray / TorchFT / Training Hub integrations Planned

Legacy trainer API

Xaytune began as an opinionated PyTorch training library. That library is what PyPI's 0.6.0 contains, and 1.0.0a2 still includes it: SFT with LoRA/QLoRA, DPO and GRPO alignment, evaluation, export, multi-stage pipelines, YAML configuration, callbacks and logging integrations. Its trainer is what NativeCompiler runs, behind the compile/execute boundary.

import xaytune

state = xaytune.finetune(
    model="meta-llama/Llama-3.1-8B",
    dataset="data/train.jsonl",
    method="lora",
    format="alpaca",
    num_epochs=3,
)

Additional alignment methods, FSDP, Studio and data preparation are experimental. DeepSpeed is experimental and partial: optimizer/scheduler and checkpoint ownership are still tracked by TASK-029. The trainer's own evaluate() and pipeline evaluation stages are separate from the control plane's durable evaluation lifecycle.

See Getting Started, Recipes, CLI Reference and Examples.