# AgentBeats Leaderboard Sync

The public AgentBeats leaderboard should use pinned task-set versions, not a live pointer to the latest `skillsbench/main` tasks.

Use `.github/workflows/agentbeats-leaderboard-sync.yml` to generate a review PR into `benchflow-ai/skillsbench-leaderboard` whenever SkillsBench task definitions change.

## Behavior

- Pushes to `main` under task/config paths generate or update a candidate `skillsbench-v1.1` PR.
- Manual runs can generate a named version such as `skillsbench-v1.2`.
- The workflow refuses to write `standard-v1`; that version is frozen.
- The workflow writes `task_sets/<task_set>.json` and `task_sets/<task_set>.source.json`.
- It includes `prebuilt_images/<task_set>.json` only when manually pointed at a completed task-env image artifact.
- It never directly pushes to the leaderboard default branch.

## Required Secret

Set this secret in `benchflow-ai/skillsbench`:

- `LEADERBOARD_SYNC_TOKEN`: token with contents and pull-request write access to `benchflow-ai/skillsbench-leaderboard`.

## Normal Flow

1. SkillsBench task files change.
2. The sync workflow opens or updates a leaderboard PR for `skillsbench-v1.1`.
3. Review the task-set diff.
4. Build/publish task env images separately with `AgentBeats Task Env Images`.
5. Re-run the sync workflow manually with:
   - `target_task_set`: next public version, for example `skillsbench-v1.2`
   - `include_prebuilt_images`: `true`
   - `task_env_artifact_run_id`: the completed image-build workflow run id
6. Merge the leaderboard PR only after digest coverage and representative smoke/shard checks pass.

This keeps AgentBeats scores reproducible: old result rows stay tied to the task-set version they actually ran against.