Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Oe-eval ci test 5 #301

Closed
wants to merge 6 commits into from
Closed

Oe-eval ci test 5 #301

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8d26654
Oe-eval ci test 5
vwxyzjn Aug 27, 2024
77b0309
test changes
vwxyzjn Aug 27, 2024
27233d7
push changes
vwxyzjn Aug 27, 2024
a16cf82
support oe-eval revision
vwxyzjn Aug 27, 2024
6cb2a95
please
vwxyzjn Aug 27, 2024
4c793c5
minor fix
vwxyzjn Aug 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 10 additions & 9 deletions .github/workflows/push-image-olmo.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,15 @@ on:
- '.github/workflows/push-image-olmo.yml'
# Note, add .olmo dockerfile + requirements if adding auto build to those
branches: [main]
# pull_request: # note, comment this out for running on every push
# # Also run on PRs that update the files in the image's directory (other than README).
# branches: [main]
# paths:
# - 'open_instruct/**'
# - '!open_instruct/README.md'
# - 'requirements-olmo.txt'
# - 'Dockerfile.olmo'
pull_request: # note, comment this out for running on every push
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just need to make sure to comment this out before merging :)

# Also run on PRs that update the files in the image's directory (other than README).
branches: [main, oe-eval-ci-test5]
paths:
- 'open_instruct/**'
- '!open_instruct/README.md'
- 'requirements-olmo.txt'
- 'Dockerfile.olmo'
- '.github/workflows/push-image-olmo.yml'
workflow_dispatch: # This allows us to manually trigger a build through the GitHub UI.

env:
Expand All @@ -47,7 +48,7 @@ jobs:

- uses: actions/checkout@v3
with:
repository: vwxyzjn/learn-fsdp2 # dummy private repo to test out deploy keys
repository: allenai/oe-eval-internal
path: './oe-eval-internal'
ssh-key: ${{ secrets.OE_EVAL_GIT_CLONE_ACCESS_PRIVATE_SSH_DEPLOY_KEY }}

Expand Down
19 changes: 10 additions & 9 deletions .github/workflows/push-image.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,14 +23,15 @@ on:
- '.github/workflows/push-image.yml'
# Note, add .olmo dockerfile + requirements if adding auto build to those
branches: [main]
# pull_request: # note, comment this out for running on every push
# # Also run on PRs that update the files in the image's directory (other than README).
# branches: [main]
# paths:
# - 'open_instruct/**'
# - '!open_instruct/README.md'
# - 'requirements.txt'
# - 'Dockerfile'
pull_request: # note, comment this out for running on every push
# Also run on PRs that update the files in the image's directory (other than README).
branches: [main, oe-eval-ci-test5]
paths:
- 'open_instruct/**'
- '!open_instruct/README.md'
- 'requirements-olmo.txt'
- 'Dockerfile.olmo'
- '.github/workflows/push-image.yml'
workflow_dispatch: # This allows us to manually trigger a build through the GitHub UI.

env:
Expand All @@ -47,7 +48,7 @@ jobs:

- uses: actions/checkout@v3
with:
repository: vwxyzjn/learn-fsdp2 # dummy private repo to test out deploy keys
repository: allenai/oe-eval-internal
path: './oe-eval-internal'
ssh-key: ${{ secrets.OE_EVAL_GIT_CLONE_ACCESS_PRIVATE_SSH_DEPLOY_KEY }}

Expand Down
14 changes: 12 additions & 2 deletions scripts/eval/oe-eval.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ set -ex

# Function to print usage
usage() {
echo "Usage: $0 --model-name MODEL_NAME --model-location MODEL_LOCATION [--hf-upload]"
echo "Usage: $0 --model-name MODEL_NAME --model-location MODEL_LOCATION [--revision REVISION] [--hf-upload]"
exit 1
}

Expand All @@ -41,6 +41,7 @@ while [[ "$#" -gt 0 ]]; do
case $1 in
--model-name) MODEL_NAME="$2"; shift ;;
--model-location) MODEL_LOCATION="$2"; shift ;;
--revision) REVISION="$2"; shift ;;
--hf-upload) HF_UPLOAD="true" ;;
*) echo "Unknown parameter passed: $1"; usage ;;
esac
Expand All @@ -58,6 +59,7 @@ MODEL_NAME_SAFE=${MODEL_NAME//\//_}

# Set defaults for optional arguments
HF_UPLOAD="${HF_UPLOAD:-false}"
REVISION="${REVISION:-main}"

# Set HF_UPLOAD_ARG if HF_UPLOAD is true
if [ "$HF_UPLOAD" == "true" ]; then
Expand All @@ -83,5 +85,13 @@ for TASK in "${TASKS[@]}"; do
else
BATCH_SIZE=$BATCH_SIZE_VLLM
fi
python oe-eval-internal/oe_eval/launch.py --model "$MODEL_NAME" --beaker-workspace "ai2/tulu-3-results" --beaker-budget ai2/oe-adapt --task "$TASK" $MODEL_TYPE --batch-size "$BATCH_SIZE" --model-args {\"model_path\":\"${MODEL_LOCATION}\"} ${HF_UPLOAD_ARG} --gpus "$GPU_COUNT"
python oe-eval-internal/oe_eval/launch.py \
--revision "$REVISION" \
--model "$MODEL_NAME" \
--beaker-workspace "ai2/tulu-3-results" \
--beaker-budget ai2/oe-adapt \
--task "$TASK" $MODEL_TYPE \
--batch-size "$BATCH_SIZE" \
--model-args "{\"model_path\":\"${MODEL_LOCATION}\",\"revision\":\"${REVISION}\",\"tokenizer_revision\":\"${REVISION}\"}" ${HF_UPLOAD_ARG} \
--gpus "$GPU_COUNT"
done
2 changes: 2 additions & 0 deletions scripts/submit_eval_jobs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -579,6 +579,8 @@ def adjust_gpus(task_spec, experiment_group, model_name, gpu_multiplier):
oe_eval_cmd += f" --model-location {model_info[1]}"
else:
oe_eval_cmd += f" --model-location beaker://{model_info[1]}"
if args.hf_revision:
oe_eval_cmd += f" --revision {args.hf_revision}"
subprocess.Popen(oe_eval_cmd, shell=True)

# create an experiment that runs the safety eval tasks
Expand Down
Loading