Fixed crashes on high-performance servers. Stabilization of variation tests. #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A timeout is required to prevent freezing and some crashes.
Problems: 1. Using a random number as a thrashhold is an extremely unstable way to estimate. Example: take 1,000,000 numbers and for the first iteration add up all to a random number which turned out to be 10, and on the second it turned out to be 10,000, respectively, and the time on the iteration will differ and variation per iterations jumps 2. Using 1 number to compose an array takes place. But since we operate with large numbers and the largest in this chain do not have time to pass in 1 second. It will be more convenient and efficient to measure the average time. that is, to evaluate not on one number, but to use the range for the trashholds at once and evaluate the average of them. We took it depending on the numbers from 1% for little to 50% for huge ones. 3. Shuffle destabilizes the program only because when adding values, they are stored side by side. But by mixing them, he mixes them in memory, so each time we have a different distance between the necessary cells in memory. That would not remove the possibility of mixing numbers. On the setup, after mixing, we added a pass to stabilize the values in memory, so that they would be consistently next to each other. 4. Make a setup at each iteration on which each time both the array shuffle and the threshhold change, see the description above - this is not advisable from the point of view of variation, which turns out to be more than 100% in many cases. Accordingly, the setup before all iterations reduced the variation several times. 5. Some cases remained unchanged because they are not variable.
1. For some benchmarks, there was too little warm-up 2. Adding a random number strongly affects the variation. Therefore, we generate an array in advance in the setup, and then simply add elements from this array to other structures. 3. We increased the value and slightly reduced the execution time for blocked structures, since there were regular falls.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hello
I'm from Azul Systems
We liked your benchmark, but some of tests crashed and were extremely unstable for a number of reasons.
In my commits I describe some of the problems. I also brought the .log files in which I indicated the current variations for Intel Xeon Gold 5222 CPU 3.60GHz (4 cores), Cascade Lake Arch, RAM 48Gb, OS CentOS 8 hosts
There are still some unstable tests, you can still work with them please see unstavle.log, but we use this version of them in testing right now.
It will be great if you add our changes to you, we would like to recommend this benchmark to other companies after.