Skip to content
This repository has been archived by the owner on Jun 9, 2024. It is now read-only.

fixing backslashes, adding basic metrics #89

Merged
merged 13 commits into from
Jul 12, 2023
Merged

fixing backslashes, adding basic metrics #89

merged 13 commits into from
Jul 12, 2023

Conversation

SilenNaihin
Copy link
Contributor

Background

Metrics are super important, making things better.

Changes

  • default of 3 to get added to regression test, 1 to get removed (mocks don't affect either)
  • fixing backslashes
  • adding metrics for individual challenges (run_time, success_%), is_regression, highest_difficulty you beat overall,
{
    "command": "agbenchmark start --mock",
    "completion_time": "2023-07-11-21:02",
    "metrics": {
        "run_time": "0.93 seconds",
        "highest_difficulty": "advanced: 5"
    },
    "tests": {
        "TestDebugSimpleTypoWithGuidance": {
            "data_path": "agbenchmark/challenges/code/d1",
            "is_regression": false,
            "metrics": {
                "difficulty": "basic",
                "success": false,
                "fail_reason": "assert 1 in [0.0]",
                "success_%": 0,
                "run_time": "0.465 seconds"
            }
        },

PR Quality Checklist

  • I have run the following commands against my code to ensure it passes our linters:
    black .
    isort .
    mypy .
    autoflake --remove-all-unused-imports --recursive --ignore-init-module-imports --ignore-pass-after-docstring --in-place agbenchmark

Copy link

@ai-maintainer ai-maintainer bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution! 

This pull request is too large for automated review.

Reviewed with AI Maintainer

waynehamadi
waynehamadi previously approved these changes Jul 12, 2023
@SilenNaihin SilenNaihin merged commit 8d0c517 into master Jul 12, 2023
5 checks passed
@SilenNaihin SilenNaihin deleted the feat/metrics branch July 12, 2023 05:38
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants