Enable reproducible comparing tests on jdk21+ macos #3844
Conversation
sophia-guo
force-pushed
the
mac
branch
5 times, most recently
from
5dd3e43
to
4e6a29f
Compare
|
I have tried both static mac and orka. Both no jq installed. https://ci.adoptium.net/view/Test_grinder/job/Grinder/10327/console Might be easy to install jq on static mac and orka? |
|
https://ci.adoptium.net/view/Test_grinder/job/Grinder/10427/ Does this mean we can't do reproducing on this environment https://ci.adoptium.net/computer/test-macstadium-macos1014-x64-1/ due to the Xcode version not matching. @steelhead31 ? |
Yup, that machines OS I think is too old for Xcode 15.2, and I think it has to be the same version of Xcode for reproducibility to work, @andrew-m-leonard can confirm this I think :) |
Yes, correct Xcode 15.2 requires MacOS 13.5 minimum : https://developer.apple.com/support/xcode/ |
|
So we need extra label for macos to ensure the reproducing tests run on correct environment. Ether xcode15.2 or sw.os.osx.13_5. Two Mac static agents wont work for reproducing tests. https://ci.adoptium.net/label/ci.role.test&&hw.arch.x86&&%28sw.os.osx%7C%7Csw.os.mac%29/ Current orka ones are label as xcode15.0.1&&build&&mac&&x64 so can't work for reproducing either. Should we open an issue in infra to see if it's possible to update orka's os version? |
|
Interesting orka ones with label |
@sophia-guo so the Orka nodes are using Xcode 15.2, the 15.0.1 label has not been changed just because of the hassle involved! should have been called Xcode15 ! |
|
orka updated with jq rerun adoptium/infrastructure#3593 https://ci.adoptium.net/view/Test_grinder/job/Grinder/10470/console |
parsing issue as SBOM --configure-args value format updated with double quotes. Parsing in mac script also need to update. |
sophia-guo
force-pushed
the
mac
branch
2 times, most recently
from
a8cf660
to
524ffcb
Compare
|
New grinder with latest and label https://ci.adoptium.net/label/ci.role.test&&sw.os.osx&&orka&&hw.arch.x86/ to make sure running on orka https://ci.adoptium.net/view/Test_grinder/job/Grinder/10769/ - waiting |
sophia-guo
force-pushed
the
mac
branch
3 times, most recently
from
0bc4005
to
3132741
Compare
|
Running with orka, issue message complains the Xcode can't be found. Test is running on orka node , which should have Xcode 15.2 available #3844 (comment) |
|
@steelhead31 If it's related with how the Xcode installation is checked? #3844 (comment) |
|
rerun on build orka node https://ci.adoptium.net/label/xcode15.0.1&&build&&mac&&x64/ got different error from running on test orka node https://ci.adoptium.net/label/ci.role.test&&sw.os.osx&&orka&&hw.arch.x86/ https://ci.adoptium.net/view/Test_grinder/job/Grinder/10775/console |
|
@sophia-guo I've put some debugging in, and it does appear the script is checking correctly, but not finding an Xcode installation on that Orka node..
|
|
@sophia-guo I've run a grinder here https://ci.adoptium.net/view/Test_grinder/job/Grinder/10782/console with these labels.. ( borrowed from a build job )
So this issue is down to the orka test nodes not having Xcode installed... I'll take a look at the other issue now |
|
@sophia-guo I've fixed the issue with the repro script, and it now attempts to build, but its complaining about a missing SPEC.mk file..
@andrew-m-leonard may also be able to help in my absence. See the grinder https://ci.adoptium.net/view/Test_grinder/job/Grinder/10791/console |
sophia-guo
force-pushed
the
mac
branch
3 times, most recently
from
5a2bd7f
to
2b72104
Compare
|
@andrew-m-leonard what should we do with option |
|
https://ci.adoptium.net/view/Test_grinder/job/Grinder/10922/console @andrew-m-leonard . This is the results by reproduced on arm64 by comparing with repro_compare.sh. Anything wrong? Regarding to run this x64 tests on arm64, I haven't got a good solution for this. Right now we can enable as part of dev.system and set a special rule like dev.system on mac only run on arm64, as there is no other tests in this group right now. But going forward this is not correct. |
@sophia-guo It looks like it's not picking up a good JDK for running the Java compare classes: |
@andrew-m-leonard I don't quite understand this. Source jdk is copied from build https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk23/job/jdk23-mac-x64-temurin/16/artifact/workspace/target/OpenJDK23U-jdk_x64_mac_hotspot_23_37-ea.tar.gz. We can successfully rebuild the compared one. Both have this issue But the compare results is exactly same as when we use the independent jenkins job (thought it's jdk21) https://ci.adoptium.net/view/ReproducibleBuild/job/jdk21u-mac-x64-temurin_reproduce_compare/74/consoleFull So I'm thinking this might not be issues from test view? |
The problem is occuring invoking "javap" to process the System*.class files, here: The System*.class will be compiled with the given JDK internal class version, so for jdk-23 that is "67". I would guess the "javap" being picked up off PATH by the test is not jdk-23, so it will not know how to read a jdk-23(67) class file. The reason it works for jdk21u is probably because the test is picking up a jdk-21 jdk in the PATH? To resolve, I think we need to put a copy of the source or target JDK in the PATH, to resolve to a version of "javap" that knows how to process class files of the same internal version. |
|
For cross compiling issue discussion is here adoptium/TKG#618 |
|
Enable aarch64_mac https://ci.adoptium.net/view/Test_grinder/job/Grinder/11050/console #3972 Interesting , test will archive the reproduced jdk and compare result. For other platform it's around 260M, but Mac_aarch64 is around 4.57G. Will download to see what's the difference and cause such a large aritfact. Caused by wrong workdir, updated. |
|
Update all javap with testJDK. Also there are 150 dylib differences, which seems removeMacOSNonComparableData() ( remove the macos dylib non-compararable data) was not called at all. Updated to call removeMacOSNonComparableData() and removeWindowsNonComparableData() before the final patchManifests step. @andrew-m-leonard removeMacOSNonComparableData() get the error, what's the issue here? https://ci.adoptium.net/view/Test_grinder/job/Grinder/11059/console |
|
@andrew-m-leonard Build without call removeMacOSNonComparableData New failures with "${JDK_DIR}"/bin/javap -v -sysinfo -l -p -c -s -constants "$ff" > "$f.javap.tmp" |
|
One question is if it is reproducible and comparable between cross-compiled x64 mac64 and native x64 mac. I've tried locally with xcode 16.1. Using sbom file generated by cross-compiled x64 mac64 . Got following failures, @andrew-m-leonard do you think the failure is caused by using cross-compiled x64 mac64 with aarch64 to compile on x64? |
@sophia-guo mmm not sure, however it maybe Xcode 16.1 issue, as we currently build using Xcode 15.2 |
|
I've installed xcode 15.2 instead of 16.1. Using cross-compiled x64 mac jdk on arm64 mac SBOM https://github.com/adoptium/temurin21-binaries/releases/download/jdk-21.0.5%2B11/OpenJDK21U-sbom_x64_linux_hotspot_21.0.5_11.json did reproducing on my local x64 mac and did a compare. Result shows except the dylib differences ( known issue #3989) it's indentical. So go back to initial question if it makes sense to do the reproduce a cross compiled x64 mac on aarch64 and compare with native complied jdk on x64 mac?
|
|
i'd hope make no difference, but we'd need to check |
|
With #4025 x64 mac reproducible can be enabled on aarch64 mac, which requires mac x64 special.system only have reproducible comparing tests https://github.com/adoptium/ci-jenkins-pipelines/compare/master...sophia-guo:mac64?expand=1 |
|
Temporarily enable on macx64 adoptium/ci-jenkins-pipelines#1136 |
github-actions
bot
added
the
jenkins
Depends adoptium/ci-jenkins-pipelines#1136
#4026