35 lines
1.7 KiBLFS
Markdown
35 lines
1.7 KiBLFS
Markdown
# AgentBeats Leaderboard Sync
|
|
|
|
The public AgentBeats leaderboard should use pinned task-set versions, not a live pointer to the latest `skillsbench/main` tasks.
|
|
|
|
Use `.github/workflows/agentbeats-leaderboard-sync.yml` to generate a review PR into `benchflow-ai/skillsbench-leaderboard` whenever SkillsBench task definitions change.
|
|
|
|
## Behavior
|
|
|
|
- Pushes to `main` under task/config paths generate or update a candidate `skillsbench-v1.1` PR.
|
|
- Manual runs can generate a named version such as `skillsbench-v1.2`.
|
|
- The workflow refuses to write `standard-v1`; that version is frozen.
|
|
- The workflow writes `task_sets/<task_set>.json` and `task_sets/<task_set>.source.json`.
|
|
- It includes `prebuilt_images/<task_set>.json` only when manually pointed at a completed task-env image artifact.
|
|
- It never directly pushes to the leaderboard default branch.
|
|
|
|
## Required Secret
|
|
|
|
Set this secret in `benchflow-ai/skillsbench`:
|
|
|
|
- `LEADERBOARD_SYNC_TOKEN`: token with contents and pull-request write access to `benchflow-ai/skillsbench-leaderboard`.
|
|
|
|
## Normal Flow
|
|
|
|
1. SkillsBench task files change.
|
|
2. The sync workflow opens or updates a leaderboard PR for `skillsbench-v1.1`.
|
|
3. Review the task-set diff.
|
|
4. Build/publish task env images separately with `AgentBeats Task Env Images`.
|
|
5. Re-run the sync workflow manually with:
|
|
- `target_task_set`: next public version, for example `skillsbench-v1.2`
|
|
- `include_prebuilt_images`: `true`
|
|
- `task_env_artifact_run_id`: the completed image-build workflow run id
|
|
6. Merge the leaderboard PR only after digest coverage and representative smoke/shard checks pass.
|
|
|
|
This keeps AgentBeats scores reproducible: old result rows stay tied to the task-set version they actually ran against.
|