Incorporate use of datasets-cli into genome prepare pipeline

[ens-LCampbell]

Aim::
Make use of container with datasets-cli tool to pull both metadata and data file from NCBI.

Changes:

• Addition of bespoke nextflow module (download_asm_with_datasets) process to work with datasets-cli, retained old module process (download_asm_data) for now.
• Update nextflow config to make use of container in process with use of withlabel: 'datasets_cli'
• Various updates to genome_metadata/extend and seq_region/prepare modules to align with change in assembly report meta keys
• Added the latest (ok it was, now is semi latest 16.17.1) definition file for datasets-cli SIF creation
• Update the test suite of genome_prepare pipeline, including data differences and MD5 checksums

@MatBarba I have tested this over, if you can think of anything else let me know thanks

• inputs with and without annotation
• with and without brc_mode enabled
• using '-stub'

[JAlvarezJarreta]

Great improvement to GenomIO 👍 Good to see datasets is in good shape to be used for production.

Since v16.17.3 is already out, I'm suggesting we use the latest to avoid having to do another update at a later stage (the definition file will need to be renamed as well).

[MatBarba]

One important typo, and need to check the output of seq_region.json (the new version includes some empty values, and has seemingly lost another)

[MatBarba]

The content of the file has significantly changed. Here's a diff:

        "synonyms": [                                                   "synonyms": [
            {                                                               {
                "name": "CM029948.1",                                           "name": "CM029948.1",
                "source": "GenBank"                                             "source": "GenBank"
            },                                                              },
            {                                                               {
                "name": "Mitochondrion",                      |                 "name": "MT",
                "source": "INSDC"                                               "source": "INSDC"
            },                                                              },
            {                                                               {
                "name": "",                                   |                 "name": "HcG217B07",
                "source": "INSDC_submitted_name"                                "source": "INSDC_submitted_name"
            },                                                <
            {                                                 <
                "name": "",                                   <
                "source": "RefSeq"                            <
            }                                                               }

[MatBarba]

Another general comment: we're generating an assembly report file derived from the jsonl file, but it has different attributes than the usual one that we get from ftp. Then would it not make more sense to parse the json file directly (and leave the current seq_region/extend.py as deprecated/legacy to parse assembly report from ftp)?

[ens-LCampbell]

Another general comment: we're generating an assembly report file derived from the jsonl file, but it has different attributes than the usual one that we get from ftp. Then would it not make more sense to parse the json file directly (and leave the current seq_region/extend.py as deprecated/legacy to parse assembly report from ftp)?

This would make more sense Indeed ! Basically remove the extra step of converting to TSV from seq_region.jsonl. However it does work albeit slightly extra processing overhead, the issue is loss of INSDC_submitted_name in the datasets world. If Im not mistake, this is incoming soonish is that right @JAlvarezJarreta ?

[JAlvarezJarreta]

Yes, although Nuala has not provided a timeline so if this elements is critical for our system we may refrain from merging this PR until that time.

We have also to be careful about legacy elements and where they sit (although this one, whilst legacy, its presence is useful and justifiable).