Decide on what the new "Launch" task should accomplish

[sk8forether]

Description

The rnr-refresh software needs to reproduce at least some of the base functionality of the UFS-RNR software to be able to launch a Cylc workflow (in v8, previously called a "suite" in Cylc v7). A minimum viable product (MVP) of the launch task to demonstrate how rnr-refresh will work constitutes being able to accomplish the launch task on at least 2 platforms: 1 HPC system (Hera), and 1 Parallel Works cloud (Azure). This issue is to capture the decision-making process of what we will include in the new launch task as we compare and work off of the UFS-RNR version.

Background

In UFS-RNR, the main python runscript is called with an experiment YAML input, and the launch task itself is accomplished across several files. The main runscript (via the run function in the CylcRun class) does the following:

• collects CLI args
• collects vars from YAML
• "builds" cylc suite
• launches cylc suite (via a call to the launch_cylc function, which in turn calls the run function in cyclutil.cylc_interface.CylcLauncher)

The cylcutil Class is imported from ush by the main runscript. The run function in CylcLauncher does the following:

• calls register_suite function, which
  • adds to log and error files in specific directories
  • sends a bash command via subprocess that calls the cylc_app class to register the cylc suite with the YAML object for the experiment, the suite path, and run dir
• calls the run_suite function, which
  • adds to the same log and error files
  • sends another bash command via subprocess that calls cylc_app again with the run command and the YAML experiment object

Those Cylc commands in turn initiate the Cylc suite(s) including the runtime suite (where the launch task is actually defined), which then:

• initiates a bash script for the job, which then
• runs a python script for the job, which then
  • creates a launcher from ufs_rnr.launcher.Launcher function
  • calls the run function for the Launcher

The actual Launcher Class run function then calls other functions to:

• build directories for the experiment
• create a json file for other tasks to reference
• calls run function from the class UFSRNRCylcStatus, which then does some more Singularity-related tasks (not including a full list here)

Relevant files in UFS-RNR

These links are stored here as a reference for mapping out the Launch task functionality present in UFS-RNR, and to help facilitate decisions about what to bring forward (and how).

Starting

The launch task is a core task that is used in all runs of UFS-RNR, thus the best path to follow is starting from a baseline experiment. The experiments are initiated by the runscript with a YAML path input. The configuration files and the runscript can be found in the cylc directory as follows:

• Main run script for the UFS-RNR software: cylc_run_ufsrnr.py
• experiments directory contains YAML files for configuring experiments
  • We are typically looking at the 3DVAR baseline experiment which is defined in RDHPCS-Hera.3DVAR.1p0.coupled.yaml
  • Also see the LETKF baseline experiment defined in RDHPCS-Hera.LETKF_HYBGAIN.1p0.coupled.yaml
  • Note: Also see Tim's versions of these experiments for comparison as we work on this.

Python cylcutil Class

The cylcutil class is imported by the main run script, which launches the cylc suite.

• cylcutil directory
  • cylcutil class init
  • cylc_interface.py
  • datetime_interface.py
  • exception_interface.py
  • logger_interface.py
  • yaml_interface.py

Cylc Suite Files

Several Cylc suites are relevant because there is a main suite for the experiment combined with other general runtime suites.

• Main Cylc suite file that is called: UFSRNR.ATMOS_OCEAN_ICE.gsi_soca_3dvar_initconds_fetch.rc
• UFS-RNR/cylc/parm
  • environment.rc
  • suite.rc
• UFS-RNR/cylc/runtime
  • depends.UFSRNRv2.ufsp7c.yaml
  • runtime.UFSRNRv2.ufsp7c.rc
    • Note: This is where the launch task is actually defined for Cylc

Environment Files

Environment files that are used in the bash job are platform-specific, even for the task itself.

• UFS-RNR/modulefiles/platforms/rdhpcs-hera
  • Platform-specific environment file: platform.rdhpcs_hera.env
  • Platform-specific task environment file: launch.env

Job Files

• Main bash job script for the Launch task: UFS-RNR/jobs/JRNR_LAUNCHER
• Associated python script that actually performs the job: UFS-RNR/scripts/exrnr_launcher.py

Python ufs_rnr Class

• Class directory: UFS-RNR/ush/ufs_rnr
  • init.py
    • UFSRNRCylcStatus Class
  • launcher.py

Looking Forward

Basically, the question how much do we want to reproduce and/or bring forward?

[sk8forether]

Adding note per our meeting on 9/12/23 (with @timothyas and @frolovsa)

For the MVP

We likely only want to do the following:

• create directories
• store inputs to json (in case we would like to store to a relational database going forward)

This task should also:

• Be tested and confirmed to work on 2 platforms
• Take advantage of new Cylc 8 features instead of rebuilding with python (where possible). This means we need to better understand:
  • options for platform config
  • how cylc 8 uses "workflows" and new file version (instead of "suites" and .rc files)

Left to decide in future

• What should the python look like?
  • How much automation will we need?
  • 1 class for launching? Or a launch family?
  • Where to store error and logs?
  • How much flexibility do we move up to the configuration level?