Track runs and run in the cloud

Plural runs every Job from your machine. It never runs a Job on its own servers. Two independent choices decide what happens around that:

The two combine freely: a recorded run of an Environment on Daytona starts each Trial in a cloud sandbox, and records the Job in the web app as it runs.

Record a run in the web app

Connect the project folder to a private hosted project once. Run this inside the folder; it registers the project as it is and overwrites nothing:

plural project init my-eval --push

Then add --record to any run:

plural run --benchmark support --agent careful --record

The run still happens here. --record first pushes anything the run needs that the hosted project doesn't hold, such as an edited Task, as its next revision. Then it opens the hosted Job and reports each Trial as it starts, while it runs, and when it is scored. Open Jobs in the web app to watch it.

If the connection drops mid-run, the local Job still finishes, and plural job record uploads what was missed. Pushing a recorded Job again records nothing twice. Pushing never makes anything public.

In the web app, Run a Job on the Jobs page and Rerun on a Job build the exact command for you to copy. A Job can be rerun only from the checkout that ran it, since that's where its record lives.

Bind a Benchmark run to an evaluation track

A Benchmark can declare evaluation tracks, versioned rules that make results comparable, such as a fixed number of attempts. Only a Job bound to a track can be ranked on a leaderboard and published. Name the track by id:

plural run --benchmark support --agent careful --record --evaluation-track controlled

The run uses the track's attempts. An Agent that breaks the track's rules, such as a Harness the track doesn't allow, is refused before anything runs. See Benchmarks.

Run Trials in cloud sandboxes

An Environment's runtime decides where its Trials execute:

runtime:  provider: daytona

A Daytona Runtime starts its sandboxes on Plural's Daytona account unless you give it your own key, so you don't need a Daytona account. Sandboxes are billed to your credits by the vCPU, memory, and disk they use, and Usage shows that compute separately from model calls. The provider's API key never reaches your machine. A sandbox idle for 30 minutes is stopped, and one running for 6 hours is ended.

A long run that uses up your credits pauses rather than fails: model calls wait until you add credits, and sandboxes keep their state until the balance is $5 below zero. See Out of credits.

To size sandboxes, use your own Daytona account, or let an organization decide which providers members may use, set up a project Runtime. See Runtimes.

Supply credentials and secrets

Apps declare the names of the values they need, and a run fills them in. No value ever reaches the model or its Harness.

• A credential belongs to whoever runs the Agent, such as their own GitHub token. Store yours once with plural credential set NAME. Each Trial records whose credential acted.
• A secret belongs to the project, for setup such as a reset command or a Verifier's database password. Store it in the hosted project with plural secret set NAME, and a run fills it in for every member.

plural run names any value that's missing before it starts. See Apps for how an App declares them.

Where this goes next

• Push and pull explains revisions, plural.lock, and sharing resources with your team.
• Publish to the Hub shares a Benchmark and its leaderboard.
• Jobs covers attempts, concurrency, and retries.