Add the option to rerun failed tasks

[nweires]

Adds the option to use the --missingonly flag to only run the tasks that don't already have results present.

Notes:

• This checks the output directory for results_job{TASK_ID}.json.gz files, and runs the tasks for which that file is missing. This means you can also trigger reruns by deleting those files.
• This assumes that you're rerunning with the same project file. If you change it, the behavior is undefined. (Some types of changes would be ignored, others could cause wrong results.)

Testing:

• I deleted some results files from a previous job, then ran with --missingonly and confirmed that only the missing tasks were rerun.
• I also compared the new results file to the old one, to ensure that the correct set of simulations were run.

[github-actions]

File	Coverage
All files	86%	✅
base.py	91%	✅
exc.py	57%	✅
hpc.py	78%	✅
local.py	70%	✅
postprocessing.py	84%	✅
utils.py	91%	✅
cloud/docker_base.py	79%	✅
sampler/base.py	79%	✅
sampler/downselect.py	33%	✅
sampler/precomputed.py	93%	✅
sampler/residential_quota.py	61%	✅
test/shared_testing_stuff.py	85%	✅
test/test_docker.py	33%	✅
test/test_local.py	97%	✅
test/test_validation.py	97%	✅
workflow_generator/base.py	90%	✅
workflow_generator/commercial.py	53%	✅
workflow_generator/residential_hpxml.py	86%	✅

Minimum allowed coverage is 33%

Generated by 🐒 cobertura-action against a941448

[mfathollahzadeh]

Thanks, Natalie! This is great! Just added a few minor comments

[mfathollahzadeh]

I think it would be useful to add print(f"Missing task ID: {task_id}") so that we can keep track of them

[mfathollahzadeh]

also if the job count becomes 0, should we print something like all expected task results are present?

[mfathollahzadeh]

also if the job_count becomes zero, we should skip the post process, right?

[nweires]

Added logging of the list of tasks. In gcp.py (where this is called), we raise an error and quit if there's nothing to retry.

[mfathollahzadeh]

not sure how long this is taking right now (probably not that long) but what do you think about using a set comprehension for faster lookups? something like or something similar?

fp = re.compile(".*results_job(\\d*).json.gz$")
done_tasks = {int(m.group(1)) for f in fs.ls(f"{self.results_dir}/simulation_output/")
                      if (m := fp.match(f))}

[nweires]

I think a loop is more readable than a comprehension, plus it avoids evaluating the regex twice (e.g. {int(fp.match(f).group(1)) for f in files if fp.match(f)}), but I will switch to a set instead of a list.

[mfathollahzadeh]

This means the job identifier needs to be changed for --missingonly flag, right?

[mfathollahzadeh]

Thanks, Natalie! Looks good to me!