AWS Configuration

This semester, we will be using an Amazon Web Service (AWS)-based compute cluster for our homework and project work. This will give us a uniform environment in which to work. Although we will start with a single computer, once we start our more substantial work, the size and capabilities of the cluster will be increased. And, we will eventually shift over to the OU supercomputer.

Compute Cluster Software

The compute cluster consists of a set of virtual Linux machines. We will use ssh to connect to a command-line shell, through which you will do some of your work. See this Linux command-line cheat sheet for typical commands. Common tools available on this compute cluster include:

python (version 2 and 3)
- scikit-learn
- tensorflow
Editors:
- vi
- emacs
- gedit
Development environments:
- jupyter

Other tools can be installed, as needed.

Compute Cluster Machines

The compute cluster machines are named mlfdsX.cs.ou.edu, where X is a number (0, 1, ...). All of the machines share the same accounts and user file system (all user accounts are located in /home2). In general, user accounts are configured so that only the user and the instructor have access to the account files. Common materials are placed in the /home2/ubuntu directory and is readable by all.

Status: mlfds0 is up

Hostname: mlfds0.cs.ou.edu

Status: mlfds1 is NOT up

Hostname: mlfds1.cs.ou.edu

SSH Access

We are using key-based authentication to the compute cluster. This means that access will be linked to specific machines and accounts that you will be accessing the cluster from. Also, you will not use a password for access (unless your local private key is encrypted).

SSH Installation

Linux: installed by default
OSX: installed by default
Windows: install Ubuntu for Windows. This will give you a set of useful tools, including bash and the ssh tool set.

Create SSH Keys

If you already have a .ssh/id_rsa.pub in your home (user) directory, then you are all set and should skip the generation step.

Generate a public/private key pair on your local machine:
- At the shell, type:
  ssh-keygen
  ssh-keygen will prompt you for a passphrase; it is okay to use an empty passphrase, but doing so means that your private key is unencrypted (this is often okay, since it is stored on your local machine only).
Email your public key (.ssh/id_rsa.pub) to fagg@ou.edu Do not email the private key or you will compromise your key pair.
I will create your account and confirm this in an email.
After you have access to the cluster, you can add other machines to the cluster access list (e.g., another laptop). This is done by appending the contents of other id_rsa.pub files to the .ssh/authorized_keys file on your cluster account.

Cluster Access

Once your account has been confirmed, you may open up an ssh connection to one of the cluster nodes (one of the host names above).

From the shell, type: ssh UID@HOSTNAME
where HOSTNAME is mlfds0.cs.ou.edu and UID is your 4x4

This gives you terminal access to the node, which allows you to list/view/edit files within your home directory (and list/view files in some other directories).

Jupyter

Jupyter is an interactive environment for writing and executing python (and Julia and R) code. Here is a few different references:

High-level tutorial (note that we have already done all of the installation for you).
Notebook tutorial (including plotting in-line)
Full Documentation

Jupyter in the Cluster

One of the benefits of Jupyter is that the user interface executes in your browser. This means that the Jupyter server can be executing on one of our cluster machines and the interface itself can be on your laptop or desktop. Here is the procedure:

Initial configuration - do this once on mlfds0:

Execute the following lines in the shell:
source activate python3
jupyter notebook --generate-config
Edit .jupyter/jupyter_notebook_config.py (use nano, emacs or vi)
- Find the line containing c.NotebookApp.port
- Remove the comment symbol at the front of the line (the #) and set the port to 9XXX, where XXX are three digits that have been assigned to you.
- Save the file and exit the editor

Every time you wish to use Jupyter:

In your shell on your laptop, execute:
ssh -L 9XXX:127.0.0.1:9XXX UID@HOSTNAME
where XXX are your assigned digits, UID is your cluster user name and HOSTNAME is the name of the cluster machine that you will be executing your jupyter server on. This sets up an encrypted tunnel from your local machine to the port that the Jupyter server is listening on.
Note: if you are using a non-standard ssh key file name or location, then you must tell ssh where your private key is located:
ssh -i PRIVATE_KEY_LOCATION -L 9XXX:127.0.0.1:9XXX UID@HOSTNAME
where PRIVATE_KEY_LOCATION contains the directory and file name of your private key.
On the cluster machine: execute the following lines in the shell:
lab
This will start the Jupyter server and result in a URL being printed in your shell
Point your browser to this URL
When you are done using Jupyter, make sure that everything is saved. Then, stop the Jupyter server (^C in the shell in which the server was started, and answer "y").

References

Tutorial on ssh and scp

andrewhfagg@gmail.com

Last modified: Thu Feb 14 22:45:15 2019