Skip to content

Scripts and documentation for dealing with open datasets

Notifications You must be signed in to change notification settings

waigwamacha/open-data

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

41 Commits
 
 
 
 

Repository files navigation

Open Data

Scripts and documentation for open datasets


Open Dataset Descriptions

Dataset fMRI availability
ABCD - The Adolescent Brain Cognitive Development Study Resting + Task
HBN - Child Mind Institute: healthy brain network Resting + Nat Viewing
HCP-D - HCP-Development Resting + Task
PNC - Philadelphia neurodevelopmental cohort Resting + Task
NKI - Enhanced Nathan Kline Institute - Rockland Sample Resting
PING - Pediatric Imaging, Neurocognition, and Genetics Resting
ABIDE - Autism Brain Imaging Data Exchange Resting
CoRR - Consortium for Reliability and Reproducibility Resting
ADHD-200 - International Neuroimaging Datasharing Initiative (INDI), the ADHD-200 Sample Resting
NIHPD - The NIH MRI study of normal brain development NA

Running fMRI prep on Compute Canada / ACENET

TAR & Upload Site BIDS Data (from PBIL)

From the PBIL, tar and upload raw bids dataset onto shared project drive. Each site (ie, /ADHD200/Brown/) is done separately to limit size of tar-archives, transfer speeds, and job-management on ComputeCanada. Use the tar_and_rsync_to_CC.sh script from the PBIL, /shared/uher/FORBOW/OpenDatasets/ADHD200/. This script will tar an entire site's raw BIDS dataset, then rsync up to cedar into a Tar_BIDS folder under the Project space (eg, ~/projects/def-ruher/fmri/ADHD200/Tar_BIDS/Brown.tar)

1) Unpack BIDS TAR Archive into ~/scratch/fmri/ (on Cedar)

SSH into cedar.computecanada.ca, [email protected] then use the script below to upack the site BIDS tar file from ~/projects/def-ruher/PROJECT/TarBIDS/ into ~/scratch/fmri/PROJECT/SITE/BIDS/. Specify project and site.

cd ~/projects/def-ruher/fmri/
./scripts/fmri_0_unpack_site_bids_archive.sh -p ADHD200 -s Brown

Another option to execute script: bash script-name-here.sh -arg1 -arg2

2) Run Fmriprep (on Cedar)

Then run every subject individually as one fmriprep-slurm job. First pass, add the '--debug' flag to ensure job-scripts are created without any errors. Then remove the flag to submit to slurm-scheduler. This script by default uses NCPUS=8, MemPerCPU=1024MB, OMP-THREADS=NCPUS, with Slurm-WallTime=12hrs. After jobs are submitted use 'sq' command to check the queue status.

./scripts/fmri_1_slurm_fmriprep_batch.sh -p ADHD200 -s Brown -e [email protected] --debug

3) Restart TimeOut-Cancelled Fmriprep Jobs (on Cedar)

Depending on resources used, quality of data, and ComputeCanada workload, a certain amount of fmriprep jobs will not finish within the specified WallTime limit (12hrs). These jobs can be re-started easily, from scratch with original outputs deleted, with another script as described below. Each of these jobs will use NCPUS=12. Use '--debug' on the first try to ensure cancelled jobs are properly found and reported. Once satisfied use the '--submit' flag when ready to run. This script safely removes the incomplete results then calls the fmri_1_slurm_fmriprep_batch.sh above to submit the jobs.

./scripts/fmri_2_restart_cancelled_jobs.sh -p ADHD200 -s Brown -e [email protected] --debug

4) TAR Fmriprep Derivatives (on Cedar)

Once fmriprep has completed for all subjects in a site, use this script to archive the derivatives from user ~/scratch/fmri/PROJECT/ (slower-SSD) back over to group-shared tape-drive ~/projects/def-ruher/fmri/PROJECT/Tar_DERIVS/SITE.tar:

./scripts/fmri_3_archive_site_derivs.sh -p ADHD200 -s Brown

Download Tar Derivatives from Cedar to PBIL (from PBIL)

From the PBIL, download the completed site derivatives Tar file.

cd /shared/uher/FORBOW/OpenDatasets/ADHD200/ ./download_tar_derivs.sh -p ADHD200 -s Brown

ADHD200 > HBN > ABIDE > Rockland > Corr

About

Scripts and documentation for dealing with open datasets

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Shell 100.0%