SkillCompiler/data/skills-bench/docs/agentbeats-leaderboard-sync.md

AgentBeats Leaderboard Sync

The public AgentBeats leaderboard should use pinned task-set versions, not a live pointer to the latest skillsbench/main tasks.

Use .github/workflows/agentbeats-leaderboard-sync.yml to generate a review PR into benchflow-ai/skillsbench-leaderboard whenever SkillsBench task definitions change.

Behavior

• Pushes to main under task/config paths generate or update a candidate skillsbench-v1.1 PR.
• Manual runs can generate a named version such as skillsbench-v1.2.
• The workflow refuses to write standard-v1; that version is frozen.
• The workflow writes task_sets/<task_set>.json and task_sets/<task_set>.source.json.
• It includes prebuilt_images/<task_set>.json only when manually pointed at a completed task-env image artifact.
• It never directly pushes to the leaderboard default branch.

Required Secret

Set this secret in benchflow-ai/skillsbench:

• LEADERBOARD_SYNC_TOKEN: token with contents and pull-request write access to benchflow-ai/skillsbench-leaderboard.

Normal Flow

1. SkillsBench task files change.
2. The sync workflow opens or updates a leaderboard PR for skillsbench-v1.1.
3. Review the task-set diff.
4. Build/publish task env images separately with AgentBeats Task Env Images.
5. Re-run the sync workflow manually with:
   • target_task_set: next public version, for example skillsbench-v1.2
   • include_prebuilt_images: true
   • task_env_artifact_run_id: the completed image-build workflow run id
6. Merge the leaderboard PR only after digest coverage and representative smoke/shard checks pass.

This keeps AgentBeats scores reproducible: old result rows stay tied to the task-set version they actually ran against.