25 lines
850 BLFS
Markdown
25 lines
850 BLFS
Markdown
# Human Evaluation of Instructions
|
|
- Step1: Select a task and convert it into AMTI-friendly format:
|
|
```terminal
|
|
> python 1.amti_create_input_data.py --file [NAME] --eval_count [COUNT]
|
|
```
|
|
For example:
|
|
```terminal
|
|
> python 1.amti_create_input_data.py --file task156_codah_classification_adversarial.json --eval_count 10
|
|
```
|
|
- Step2: follow the steps in `2.run_eval.sh`
|
|
- Step3: evaluate
|
|
|
|
|
|
To disqualify a certain user, add them to `genie_disqualification/blocklist.txt` and then:
|
|
|
|
amti associate-qual --file genie_disqualification/blocklist.txt --qual 3090SA10WM5MIHCWNTON1VROMP4CN3 --live
|
|
|
|
Or disqualify a particular user:
|
|
|
|
amti associate-qual WorkerId --qual 3090SA10WM5MIHCWNTON1VROMP4CN3 --live
|
|
|
|
To list workers that are disqualified:
|
|
|
|
amti associated --qual 3090SA10WM5MIHCWNTON1VROMP4CN3 --status Granted --live
|