Examples of analysis of MITgcm LLC simulations of NASA Pleiades supercomputer using Pangeo tools (xarray, dask, xmitgcm).
These examples will not teach you to use Python if you are a novice. In order to take advantage of the tools we have developed for the LLC simulations, you must have a basic understanding of Xarray, the python library we use for analysis of multidimensional arrays. If you have not used Xarray before, I recommend the following online lectures:
You will also probably need to know something about Dask, the library we use for parallel computation. Although Dask primarily operates in the background, it's still important to have some idea of how it works under the hood in order to achieve good performance. You should have some basic familiarity with dask-jobqueue, which allows us to launch Dask clusters on Pleaides.
These tools require a highly customized python environment. To set up the proper environment, follow these steps.
From the Pleiades command line, run:
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
# incase the intializer does not do anything (happened to me)
source <path to conda>/bin/activate
conda update -y conda
# if you use a different shell, replace "bash" with your shell (e.g. "csh")
conda init bash # just gave a message saying no action taken.
A suitable enviroment file is contained in this repository at pangeo_pleiades_environment.yaml. Rather than copy and paste, you can get it by cloning this repository
git clone https://github.com/rabernat/pleiades_llc_recipes
cd pleiades_llc_recipes
conda env create -f pangeo_pleiades_environment.yaml
sh install_jupyter_extensions.sh # unable to run this. a) compained about nodejs, after installing that
# then complains: ValueError: "@jupyter-widgets/jupyterlab-manager" is not a valid extension
The above command created about 3.5GB of files in your home directory at $HOME/miniconda3.
You only have 8GB of home storage space on Pleiades, so it is wise to clean this up a bit.
conda clean -tipsy
rm -rf $HOME/miniconda3/pkgs/*
This should get the size of the environment down to about 500MB.
Dask needs special configuration files. Install them by running
cp dask_config/* $HOME/.config/dask/
There is a test script located in this repository. From the command line, run
source activate pangeo
py.test -v test_llcreader_pleiades.py # the test failed, probably because llc has moved around.
The easiest way to use this environment is from a head node (pfe). However, you should probably not do any heavy computing on a head node, and almost everything involving the LLC data is heavy.
Nevertheless, if you wish to do so, you can just do
source activate pangeo
ipython
and you will be on your way.
Before using jupyterlab, you need to set a password. Run
jupyter notebook password
from the command line.
This repo contains an example job script for launching jupyterlab on a compute node
called launch_jupyter.sh. You need to modify it to include your GID.
Submit the script to the queue:
qsub launch_jupyter.sh
and monitor it with qstat. Once it runs, you will see something like.
$ qstat -n -u $USER
Req'd Elap
JobID User Queue Jobname TSK Nds wallt S wallt Eff
-------------- -------- ------ ------- --- --- ----- - ----- ---
6846900.pbspl1 rpaberna normal jupyter 20 1 07:59 R 00:09 0%
r435i0n8/0*20
The server name is r435i0n8. We need to create an SSH tunel to this host.
From our local machine, run the command
ssh -L 8877:r435i0n8:8877 pfe
and connect to your server at http://localhost:8877
You can see an example notebook in notebooks/pleiades_llc_examples.ipynb
TODO
It's pretty simple: just write a python file with some code in it and submit it as a batch job.
TODO
Need to tune the optimal dask-jobqueue settings for Pleiades.