conda envs across sessions

[PatricioSolis-Urra]

Hello,

I'm attempting to utilize heudiconv with the Neurodesk app on my local Mac (By module loading containers). After creating the conda environment by following these instructions: https://www.neurodesk.org/tutorials-examples/tutorials/programming/conda/, and using "pip install heudiconv", the program works. However, as mentioned in the provided link, the environment and configuration do not persist across sessions. This is inconvenient, as I need to reinstall everything every time I restart the session. Does anyone have any suggestions for a better approach to setting this up? Specifically, I'm looking for a way to maintain the conda environment and any necessary setups across sessions.

Thanks in advance,
best
Patricio

[stebo85]

Dear @PatricioSolis-Urra,

There are two options to preserve changes across neurodesk sessions:

1. Everything you store in /neurodesktop-storage will be stored outside of the neurodesk session (in this case on your Mac). So, what you could do is, to install a miniconda manually in this location and when you need it, you activate that environment. However, this comes with a performance penalty, because there will be a translation between Mac/Linux file systems. If you make sure that your Docker settings are set to 'VirtioFS' this could work:

2. Another option is to contribute the heudiconv workflow back to Neurodesk, then we include it in a container that you could load via the module system. We already have a container with various BIDStools and it would be very easy to add heudiconv to that: https://github.com/NeuroDesk/neurocontainers/blob/master/recipes/bidstools/README.md

You would just need to create a pull request to this and add heudiconv in that line and add some documentation in the README.md:
https://github.com/NeuroDesk/neurocontainers/blob/a462944dd10a1a56227326024f7f6a502d5aa02f/recipes/bidstools/build.sh#L24

The advantages about the second option are:

1. It's fast
2. It's reproducible
3. It contributes back to the community and everyone else can profit from it

Thank you
Steffen

[PatricioSolis-Urra]

Hello @stebo85,

The second option is definitely better.
I have created a pull request. I hope this works
Thank you very much.
Best regards.

[stebo85]

Dear @PatricioSolis-Urra ,

this container is now available on Neurodesk:

jovyan@neurodesktop:~$ ml bidstools
jovyan@neurodesktop:~$ ml

Currently Loaded Modules:
  1) bidstools/1.0.1

jovyan@neurodesktop:~$ heudiconv
WARNING: Nothing to be done - displaying usage help
usage: heudiconv [-h] [--version] [-d DICOM_DIR_TEMPLATE | --files [FILES ...]] [-s [SUBJS ...]] [-c {dcm2niix,none}] [-o OUTDIR] [-l LOCATOR] [-a CONV_OUTDIR] [--anon-cmd ANON_CMD] [-f HEURISTIC] [-p] [-ss SESSION] [-b [BIDSOPTION1 [BIDSOPTION2 ...]]]
                 [--overwrite] [--datalad] [--dbg] [--command {heuristics,heuristic-info,ls,populate-templates,sanitize-jsons,treat-jsons,populate-intended-for}] [-g {studyUID,accession_number,all,custom}] [--minmeta] [--random-seed RANDOM_SEED]
                 [--dcmconfig DCMCONFIG] [-q {SLURM,None}] [--queue-args QUEUE_ARGS]

Example: heudiconv -d 'rawdata/{subject}' -o . -f heuristic.py -s s1 s2 s3

options:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -d DICOM_DIR_TEMPLATE, --dicom_dir_template DICOM_DIR_TEMPLATE
                        Location of dicomdir that can be indexed with subject id {subject} and session {session}. Tarballs (can be compressed) are supported in addition to directory. All matching tarballs for a subject are extracted and their content processed
                        in a single pass. If multiple tarballs are found, each is assumed to be a separate session and the --ses argument is ignored. Note that you might need to surround the value with quotes to avoid {...} being considered by shell
  --files [FILES ...]   Files (tarballs, dicoms) or directories containing files to process. Cannot be provided if using --dicom_dir_template.
  -s [SUBJS ...], --subjects [SUBJS ...]
                        List of subjects - required for dicom template. If not provided, DICOMS would first be "sorted" and subject IDs deduced by the heuristic.
  -c {dcm2niix,none}, --converter {dcm2niix,none}
                        Tool to use for DICOM conversion. Setting to "none" disables the actual conversion step -- useful for testing heuristics.
  -o OUTDIR, --outdir OUTDIR
                        Output directory for conversion setup (for further customization and future reference. This directory will refer to non-anonymized subject IDs.
  -l LOCATOR, --locator LOCATOR
                        Study path under outdir. If provided, it overloads the value provided by the heuristic. If --datalad is enabled, every directory within locator becomes a super-dataset thus establishing a hierarchy. Setting to "unknown" will skip that
                        dataset.
  -a CONV_OUTDIR, --conv-outdir CONV_OUTDIR
                        Output directory for converted files. By default this is identical to --outdir. This option is most useful in combination with --anon-cmd.
  --anon-cmd ANON_CMD   Command to run to convert subject IDs used for DICOMs to anonymized IDs. Such command must take a single argument and return a single anonymized ID. Also see --conv-outdir.
  -f HEURISTIC, --heuristic HEURISTIC
                        Name of a known heuristic or path to the Python script containing heuristic.
  -p, --with-prov       Store additional provenance information. Requires python-rdflib.
  -ss SESSION, --ses SESSION
                        Session for longitudinal study_sessions. Default is None.
  -b [BIDSOPTION1 [BIDSOPTION2 ...]], --bids [BIDSOPTION1 [BIDSOPTION2 ...]]
                        Flag for output into BIDS structure. Can also take BIDS-specific options, e.g., --bids notop. The only currently supported options is "notop", which skips creation of top-level BIDS files. This is useful when running in batch mode to
                        prevent possible race conditions.
  --overwrite           Overwrite existing converted files.
  --datalad             Store the entire collection as DataLad dataset(s). Small files will be committed directly to git, while large to annex. New version (6) of annex repositories will be used in a "thin" mode so it would look to mortals as just any other
                        regular directory (i.e. no symlinks to under .git/annex). For now just for BIDS mode.
  --dbg                 Do not catch exceptions and show exception traceback.
  --command {heuristics,heuristic-info,ls,populate-templates,sanitize-jsons,treat-jsons,populate-intended-for}
                        Custom action to be performed on provided files instead of regular operation.
  -g {studyUID,accession_number,all,custom}, --grouping {studyUID,accession_number,all,custom}
                        How to group dicoms (default: by studyUID).
  --minmeta             Exclude dcmstack meta information in sidecar jsons.
  --random-seed RANDOM_SEED
                        Random seed to initialize RNG.
  --dcmconfig DCMCONFIG
                        JSON file for additional dcm2niix configuration.

Conversion submission options:
  -q {SLURM,None}, --queue {SLURM,None}
                        Batch system to submit jobs in parallel.
  --queue-args QUEUE_ARGS
                        Additional queue arguments passed as a single string of space-separated Argument=Value pairs.

Can you test it to make sure everything works?

[PatricioSolis-Urra]

Hello @stebo85,
I tested it, and it works perfectly.
Thank you very much!
Patricio

[stebo85]

Wonderful to hear :) If you are keen to contribute a jupyter notebook example for bids conversion using heudiconv in Neurodesk that would be super cool.