Managing versions and compilation

[SorooshMani-NOAA]

This ticket is related to the fact that different system components are kept align in terms of versioning through a manual checking process. We'd like to be able to switch between system versions based on the commit version, and without too many manual checks. This will help in finding the sources of bugs introduced during development and also comparing the results of the system between different states.

Some initial thoughts:

If flow idempotency key is used during registration and it is equal to commit hash, and then we also tag Docker images with commit hash, then for the system to be updated there MUST be a new commit to get new flow versions.
There's also the group uuid for flows. Although it seems to be used for grouping flows of the same name into version groups. It could lead to some issues if removed afterwards, also the create_flow_run will find multiple active flows!
We can even append -dev to the names of the resources/flows/etc. in the code based on whether the code is in main branch or not! On Python we can use Dunamai, but does this work for terraform?

On using singularity for ParallelWorks:

There was an issue with the Singularity image not finding MPI and getting stuck due to the missing package. To fix the issue use the base image centos:centos7.8.2003 which is closer PW cluster's OS. In addition to that, add the package yum -y groupinstall "development tools"

[SorooshMani-NOAA]

The issue with cmake getting stuck on finding MPI also occur on the PW node directly too. This is resolved by installing the same packages: yum -y groupinstall "development tools"
This is not correct, the issue was due to incorrect compilation flags and invalid locale set