FCS_GX on dev failed

[gbdias]

• In an initial run of the pipeline (main branch 23f2522) FCS_GX ran for 5 days and was terminated because of walltime limit.
• I then moved to the dev branch (19de7b6) to make use of the ramdisk update and resumed the pipeline but the process still fails. The command.log is this:

INFO:    Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred
INFO:    Environment variable SINGULARITYENV_NXF_DEBUG is set, but APPTAINERENV_NXF_DEBUG is preferred
INFO:    Environment variable SINGULARITYENV_SNIC_TMP is set, but APPTAINERENV_SNIC_TMP is preferred
2024/03/28 09:06:50 NOTICE: Config file "/home/guibo205/.config/rclone/rclone.conf" not found - using defaults
2024/03/28 09:06:51 ERROR : all.manifest: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.manifest: operation not permitted
2024/03/28 09:06:51 ERROR : all.meta.jsonl: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.meta.jsonl: operation not permitted
2024/03/28 09:06:51 ERROR : all.README.txt: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.README.txt: operation not permitted
2024/03/28 09:06:51 ERROR : all.taxa.tsv: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.taxa.tsv: operation not permitted
2024/03/28 09:06:51 ERROR : all.blast_div.tsv.gz: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.blast_div.tsv.gz: operation not permitted
2024/03/28 09:06:51 ERROR : all.assemblies.tsv: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.assemblies.tsv: operation not permitted
2024/03/28 09:06:51 ERROR : all.seq_info.tsv.gz: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.seq_info.tsv.gz: operation not permitted
2024/03/28 09:15:01 ERROR : all.gxs: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.gxs: operation not permitted
2024/03/28 09:21:07 ERROR : all.gxi: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.gxi: operation not permitted
2024/03/28 09:21:07 ERROR : Attempt 1/3 failed with 9 errors and: chtimes /dev/shm/gxdb/1/all.gxi: operation not permitted
2024/03/28 09:21:07 ERROR : all.meta.jsonl: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.meta.jsonl: operation not permitted
2024/03/28 09:21:07 ERROR : all.README.txt: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.README.txt: operation not permitted
2024/03/28 09:21:07 ERROR : all.manifest: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.manifest: operation not permitted
2024/03/28 09:21:07 ERROR : all.taxa.tsv: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.taxa.tsv: operation not permitted
2024/03/28 09:21:07 ERROR : all.blast_div.tsv.gz: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.blast_div.tsv.gz: operation not permitted
2024/03/28 09:21:07 ERROR : all.assemblies.tsv: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.assemblies.tsv: operation not permitted
2024/03/28 09:21:08 ERROR : all.seq_info.tsv.gz: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.seq_info.tsv.gz: operation not permitted
2024/03/28 09:29:07 ERROR : all.gxs: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.gxs: operation not permitted
2024/03/28 09:35:24 ERROR : all.gxi: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.gxi: operation not permitted
2024/03/28 09:35:24 ERROR : Attempt 2/3 failed with 9 errors and: chtimes /dev/shm/gxdb/1/all.gxi: operation not permitted
2024/03/28 09:35:24 ERROR : all.README.txt: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.README.txt: operation not permitted
2024/03/28 09:35:24 ERROR : all.manifest: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.manifest: operation not permitted
2024/03/28 09:35:24 ERROR : all.meta.jsonl: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.meta.jsonl: operation not permitted
2024/03/28 09:35:24 ERROR : all.taxa.tsv: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.taxa.tsv: operation not permitted
2024/03/28 09:35:24 ERROR : all.blast_div.tsv.gz: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.blast_div.tsv.gz: operation not permitted
2024/03/28 09:35:25 ERROR : all.assemblies.tsv: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.assemblies.tsv: operation not permitted
2024/03/28 09:35:25 ERROR : all.seq_info.tsv.gz: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.seq_info.tsv.gz: operation not permitted
2024/03/28 09:43:26 ERROR : all.gxs: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.gxs: operation not permitted
2024/03/28 09:49:41 ERROR : all.gxi: Failed to set modification time: chtimes /dev/shm/gxdb/1/all.gxi: operation not permitted
2024/03/28 09:49:41 ERROR : Attempt 3/3 failed with 9 errors and: chtimes /dev/shm/gxdb/1/all.gxi: operation not permitted
2024/03/28 09:49:41 Failed to copy with 9 errors: last error was: chtimes /dev/shm/gxdb/1/all.gxi: operation not permitted

[mahesh-panchal]

Thanks. That helps a lot. I think it's time to drop this tool for now. I'll try an update the workflow to skip this step as soon as possible.

[mahesh-panchal]

My own run seems to have gone through this time within 15mins, but I also have rclone setup on my system.
Anyway, priority is skipping this step for now.

[MartinPippel]

For me it ran also through as well in 12-15 minutes (4 times by now)

[mahesh-panchal]

Did you get the usage metrics?

[MartinPippel]

I think it did not work.

[mahesh-panchal]

It used the cores though that time, compared to Estelle's run.

[mahesh-panchal]

Is this working more consistently now?