adoptium-packages-linux-pipeline_new does not work across nodes with different workspace directory

[andrew-m-leonard]

Some ci.adoptium.net nodes use /home/jenkins and some (Azure Cloud VMs) use /home/adoptopenjdk.

The problem is not actually the VM, it's due to the Artifactory installer upload pipeline incorrectly combining "Declarative" and "Scripted" Jenkins pipeline syntax: https://github.com/adoptium/installer/blob/master/linux/Jenkinsfile
As stated in the Jfrog-cli doc: https://jfrog.com/help/r/jfrog-integrations-documentation/use-declarative-or-scripted-syntax-with-pipelines

When working with the Jenkins Artifactory plugin, be sure to choose either scripted or declarative.
In other words, do not use declarative and scripted steps within a single pipeline.
This will not work.

The jfrog-cli Declarative tools definition sets up the pipeline environment for the agent the top-level pipeline runs on, the pipeline then switches to "scripted" mode and switches context using node(), the jfrog-cli will then not necessarily work in that context.

The https://github.com/adoptium/installer/blob/master/linux/Jenkinsfile needs re-working as either a pure "Declarative" pipeline using the jfrog plugin, or as a "Scripted" pipeline (will need server config and credentials adding...).

There is also a secondary issue in that the installer jenkins file locks two "dockerBuild&&linux&&x64" nodes, which inevitably causes a deadlock given there are only 2 of these nodes and in the situation where 2 jobs have been queued.
The top-level agent lock is not required to remain when calling the stages, and is an unfortunate consequence of using a top-level "Declarative" pipeline which then calls "scripted" pipeline that then uses node() locking...

I think the easiest way to refactor this is to split this job into two, the top-level one being a "Scripted" pipeline that does the "what we need to build and where" logic, and a second "Declarative" upload pipeline job which it calls. As it is only the second "upload" job that needs to use the jfrog-cli plugin. The second job then be called multiple times by the top-level job with the appropriate parameters.
Important note: the top-level job does not need to reserve a agent node while calling the sub-jobs...

[andrew-m-leonard]

Another issue found, it seems if a publish installer job is aborted, and then another one started, the upload seems to upload from an existing workspace that has not been "cleaned". The following job for jdk21 riscv uploaded jdk17 as well which was aborted due to "node deadlock":
https://ci.adoptium.net/job/adoptium-packages-linux-pipeline_new/1694/console

[adoptium-packages-linux-pipeline_new] $ /home/jenkins/tools/io.jenkins.plugins.jfrog.JfrogInstallation/jfrog-cli/jf rt u **/build/ospackage/temurin-*${BUILDARCH}.deb deb/pool/main/t/temurin-${VERSION}/ --target-props=${DISTROLIST}deb.component=main;deb.architecture=${BUILDARCH} --flat=true
10:17:47 [Info] [Thread 2] Uploading: jdk/debian/build/ospackage/temurin-21-jdk_21.0.5.0.0+11-1_riscv64.deb
10:17:47 [Info] [Thread 0] Uploading: jdk/debian/build/ospackage/temurin-17-jdk_17.0.13.0.0+11_riscv64.deb
{
  "status": "success",
  "totals": {
    "success": 2,
    "failure": 0
  }
}