materials/course_setup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## tldr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Expressed as concisely as possible (see Figure below) we will be working on a jupyter notebook server for the duration of this course.\n",
    "We will access this notebook server through our local web browser. The server will be running inside a singularity container on a biowulf compute node.\n",
    "The singularity container will be run from a persistent tmux session.\n",
    "In order to access the jupyter notebook server from our local browser we will need to use ssh to forward the appropriate port from the compute node back to our local computer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "![](./course_setup_overview.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Course setup overview"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to work remotely with python we will use a jupyter notebook server on the NIH HPC systems. This allows us to keep track of the commands we enter in a notebook. The notebook server also allows us to edit text, run terminals, and conveniently upload and download files with a graphical interface. In addition we could easily switch this setup to work with  R, Julia, or [many other](https://github.com/jupyter/jupyter/wiki/Jupyter-kernels) programming languages using the same interface.\n",
    "\n",
    "In order to provide a consistent environment for all students we will run the jupyter notebook server from within a [singularity](http://singularity.lbl.gov) container. Container technologies (Docker and singularity) allow us to isolate a software stack from the host operating system. It means that using the same container we can rerun our analysis on most operating systems, and computers. There are limits to this portability which you will learn about during the course but overall containers are a very effective means of rapidly sharing a distinct environment in which to run an analysis.\n",
    "\n",
    "A universal solution for a course setup to run the required software - for employees within NIH, as well as those with guest accounts from outside - is to run the aforementioned singularity container on a [biowulf](https://hpc.nih.gov/systems/) computing node. It utilizes the superb resources available from the NIH HPC as well as avoiding the problems of attempting to work reproducibly across many personal computers with varying degrees of installation privileges.\n",
    "\n",
    "As we are working remotely we can be vulnerable to internet connectivity issues. To reduce the effect of drops in connectivity we will use [tmux](http://www.hamvocke.com/blog/a-quick-and-easy-guide-to-tmux/) with its session management, which allows us to maintain a persistent shell session on the compute node independent of an internet connection. Without this software, we would lose our work each time our internet connectivity dropped. Once the tmux session is setup, we will only need to reconnect to it if we need to change something with  our launched singularity container/jupyter notebook server (hopefully this won't happen). If connective does drop, we simply need to reopen a connection (with the appropriate port forwarding to our compute node).\n",
    "\n",
    "The following setup will work through the steps required for obtaining persistent access to a computing node in the biowulf cluster and running our jupyter notebook from the singularity container."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Course setup (with all the details)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to run the jupyter notebook we must:\n",
    "+ Set up ssh keys\n",
    "+ Set up a persistent tmux session on the biowulf headnode\n",
    "+ Request the resources of a compute node\n",
    "+ Run the jupyter notebook server in a singularity container.\n",
    "+ Setup an ssh tunnel with port forwarding to the compute node"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting up ssh keys to work on the NIH cluster."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** For windows users will need to use putty to connect remotely to the hpc systems as described [here](https://docs.google.com/document/d/11Byl0wZ5FSqaj3lhMPlDmwaFUP-xQ8Cm8EgNEaBjmgw/edit#).\n",
    "\n",
    "The setup for this course requires a number of hops using ssh in order for us to access the biowulf compute node. For unix-like OSes we can streamline this process to avoid typing in our password for each jump we make. We will achieve this by using ssh-agent to manage an encryption key. This process can be roughly broken down into the following steps:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ Generating a keypair\n",
    "+ Transferring the public key to .ssh on the target system.\n",
    "+ Use ssh-agent to manage the private key.\n",
    "+ Use a config file to configure our ssh behaviour"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Generating a keypair"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have not used ssh from your computer before you will have some extra steps to carry out. The first of these is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mkdir ~/.ssh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to create an ssh encryption key pair we type:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cd ~/.ssh\n",
    "ssh-keygen -f nih_ssh_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Follow the prompts and be sure to create a passphrase."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Transferring the public key to .ssh on the target system.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Before we transfer our key we must check that the .ssh directory exists for our hpc account. \n",
    "In the below commands you will need to change 'USER' to the username you have on the NIH hpc systems."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ssh USER@helix.nih.gov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When connected to helix check if the directory exists by typing (it will display nothing if the directory does not exist):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ls ~/.ssh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the directory does not exist you must make it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "mkdir ~/.ssh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know we have a .ssh directory for our account we can transfer our public key across to helix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "scp nih_ssh_key.pub USER@helix.nih.gov:~/.ssh/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copy the key to the authorized_keys file on the target system (once again if authorized_keys does not exist you can create it first with \"touch authorized_keys\"):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ssh USER@helix.nih.gov\n",
    "    cd .ssh\n",
    "    cat nih_ssh_key.pub >> authorized_keys\n",
    "    chmod go-rwx authorized_keys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use ssh-agent to manage the private key."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Back on our local system we can use ssh-agent for management of our private key. Type the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ssh-add nih_ssh_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To check it worked:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ssh-add -l "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Use a config file to configure our ssh behaviour"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can edit the file ~/.ssh/config (create one if it doesn’t already exist) in order to configure how we ssh to different hosts (helix, felix, biowulf), which can save lots of time. Forwarding X11 and ssh keys are useful so we will put this in our config files along with an entry for helix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ForwardX11 yes\n",
    "ForwardAgent yes\n",
    "\n",
    "Host helix\n",
    "    HostName helix.nih.gov\n",
    "    User USER\n",
    "    IdentityFile ~/.ssh/nih_ssh_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting up a persistent tmux session on the biowulf headnode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "First ssh to helix, and from there, ssh to the biowulf head node:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ssh helix\n",
    "    ssh biowulf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we are on node biowulf head we should set up a persistent tmux session. \n",
    "A basic overview of tmux commands can be found [here](https://tmuxcheatsheet.com).\n",
    "We need to first load tmux before we can use it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "module load tmux\n",
    "tmux new -s jupyter_server"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we are running a tmux session everything we do will be persistent \n",
    "i.e. we can reconnect to it even if we close/lose our connection to biowulf.\n",
    "We will now request resources from the cluster for our analysis..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Request the resources of a compute node"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The command below requests 2 cpus and 10GB of RAM for 2000 minutes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "sinteractive --mem=10g --cpus-per-task=2 -t 2000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This command will provide a terminal on an interactive node on biowulf. Take note of your nodename in the command prompt. For example:\n",
    "\n",
    "[username@cn3092 ~]$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run the jupyter notebook server in a singularity container."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can start the server by typing the following command (specify an available port, the port to use has been added next to your name on https://github.com/nih-fmrif/nimh_repro_wrkshpAug2017/blob/master/participants.txt ): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "module load singularity\n",
    "singularity exec /data/classes/RepNeurSci/images/nih-workshop-2017-latest.img jupyter notebook --no-browser --port=[put your port number here]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup an ssh tunnel with port forwarding to the compute node"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the server running and taking note of the port number that the server is running on,\n",
    "we now need to create a connection that allows the notebook server communicate with own computer.\n",
    "Without closing the old terminal window we type the following into a new window \n",
    "(you’ll need to use the node name and port number relevant to you):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "scp helix:/data/DSST/scripts/connect_with_port .\n",
    "bash connect_with_port [your node name] [your port number]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It’ll look like: bash connect_with_port cn3092 9090"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to the jupyter notebook session from our local machine"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, go back to the original terminal where we started the server. \n",
    "Copy the link displayed and paste it into a local web browser.\n",
    "Save this link because this is what you will need to connect to the server later too."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Subsequent reconnection to the jupyter server"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assuming all of the steps detailed in the course setup have been followed the jupyter notebook server will be accessible\n",
    "for the duration that the compute node resources are allocated to you.\n",
    "Because of this, if you need to connect to the server after being disconnected is\n",
    "\n",
    "a) Reconnect to the node with the ssh port forwarding in step 3.5\n",
    "\n",
    "b) You can refresh the localhost web-page you had open connecting to the server,\n",
    "or select \"Kernel > Reconnect\" from the notebook menu.\n",
    "\n",
    "Note: If you have closed your browser you will have to paste the link you previously saved in step 3.5 into your browser."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  },
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "103px",
    "width": "251px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}