-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ion-python-benchmark-cli's threshold value need to be more accurate. #265
Comments
Some thoughts on this. In the short term, we should increase the warm-up and iteration numbers to make sure the results become stable. in the long term,
|
Few TODOs:
|
The benchmark-cli's performance detection workflow fails if the performance regression exceeds the threshold value. Currently, the threshold is set to 1, meaning that the workflow only fails when the new commit is twice as slow as the original one.
Need a better strategy to calculate the performance gap between the original commit and the new commit to generate a desired and accurate threshold value.
Related issue - amazon-ion/ion-java-benchmark-cli#53
The text was updated successfully, but these errors were encountered: