CS 5043: HW2

Objectives

• Implement a simple, shallow neural network that solves a brain-machine interface prediction problem

• Incorporate TF/Keras performance metrics into the model evaluation process

• Practice Holistic Cross-Validation for evaluating models

• Use SLURM to perform a large batch of experiments

• Implement code that brings the results from a large batch of experiments into a single analysis

Assignment Notes

• Deadline: Friday, February 14th @11:59pm.

• Hand-in procedure: submit PDF to the HW2 dropbox on Gradescope, and the Jupyter notebook/Python files to Canvas.

• This work is to be done on your own. While general discussion about Python, TensorFlow and Keras is okay, sharing solution-specific code is inappropriate. Likewise, you may not download code solutions to this problem from the network.

Data Set

/home/fagg/ml_datasets/bmi/bmi_dataset.pkl

This is a 203MB file - please do not make local copies of the file (you don't need to). You are welcome to copy the file to other machines, if you wish. Two requirements:

• Large data transfers to/from OSCER should be done through the host: dtn2.oscer.ou.edu

• Please do not share this data set with others, and please delete copies after the semester is over

The data set contains both neural and arm movement data (the latter being, theta, dtheta, ddtheta and torque). In addition, there is a "time" channel that is a time stamp for each sample. Arm movements are two degrees of freedom, corresponding to the shoulder and elbow, respectively. Each sample of the neural data already contains the history of each neuron over a 1 second period (20 samples at 50ms/sample).

The data are already partitioned into 20 folds for us. Each fold contains multiple blocks of contiguous-time samples. So, if one were to plot theta as a function of time, you would see the motion of the arm over time (with gaps). Across the folds, it is safe to assume that the data are independent of one-another.

Provided Code

• hw2_base.py. This is the heart of my BMI model implementation. As such, it has a lot of features and is rather complicated. You may use it in its entirety, adapt it to your needs, or ignore it all together.

  Features include:

  • Accepts many different arguments for conducting an experiment, including accepting many hyper-parameters. Some hyper-parameters can be set by SLURM, allowing us to perform a large number of experiments with a single invocation of sbatch

  • From the BMI data set file, it will generate training/validation/testing data sets for a given rotation

  • High-level code for conducting an individual experiment and saving the results

• symbiotic_metrics.py: proper Keras implementation of the fraction-of-variance-accounted-for metric (FVAF). This implementation computes FVAF for each dimension independently

Part 1: Network

• Implement a function that constructs a neural network that can predict arm state as a function of neural state. The network should be relatively shallow (you don't need a lot of non-linearities to do reasonably well, here). But, note that quantities such as torque or velocity can be positive or negative

• Do not take steps to address over-fitting (we will work on this HW 3)

• Arm state can be scalar (e.g., shoulder orientation) or a vector (e.g., shoulder and elbow velocity)

• Train a network to predict elbow torque as a function of the neural state. Useful FVAFs are: 0 < FVAF <= 1

• Produce a figure that shows both the true torque and the predicted torque for the test fold.

Part 2: Multiple Runs

• Use your code base to execute a batch of experiments on OSCER

• The batch is a 2-dimensional grid: rotation (0...19) and number of training folds (1,2,3,5,10,18). Although single experiments will have a short running time (2-4 minutes), each one needs to be executed as a separate job on OSCER (we are preparing ourselves for more extensive experiments)

• Implement a second program that executes on your local machine and:
  • Loads all of the stored results

  • Computes average training/validation/testing FVAF for each training fold size (average across the rotations)

  • Generates a plot with three curves (one for each data set type): FVAF as a function of training set size

Hints

• There are lots of hints embedded in the provided code. Take the time to understand what is going on there.

• List comprehension is your friend

Reading Results Files

import os

def read_all_rotations(dirname, filebase):
    '''Read results from dirname from files matching filebase'''

    # The set of files in the directory
    files = fnmatch.filter(os.listdir(dirname), filebase)
    files.sort()
    results = []

    # Loop over matching files
    for f in files:
        fp = open("%s/%s"%(dirname,f), "rb")
        r = pickle.load(fp)
        fp.close()
        results.append(r)
    return results

Example:

filebase = "bmi_torque_0_hidden_30_drop_0.50_ntrain_%02d_rot_*_results.pkl"
results = read_all_rotations("results", filebase)

will find all files that match this string (* is a wildcard here).

Expectations

Looking Forward

What to Hand-In

• Hand in python files and Jupyter notebooks (to Canvas) and a single PDF that contains all pieces, including code and the results figures (to Gradescope).

  Within Jupyter, you can generate a PDF by selecting File/Export As/PDF.

  Note: the PDF generator is not particularly smart about not cutting off code on the right-hand-side of the PDF page. Check your PDF and if this is happening, then add newlines to make your lines shorter.

• Do not submit MSWord files.

Grading

• 20 pts: clean code for executing a single experiment (including documentation)
• 20 pts: True and predicted torque as a function of time
• 20 pts: Executing on OSCER
• 20 pts: clean code for bringing individual results files together
• 20 pts: FVAF as a function of training set size

andrewhfagg -- gmail.com

Last modified: Sun Feb 2 00:11:33 2020