CS 5043: HW3

Objectives

Implement network architectures that use different forms of regularization
Execute experiments in which we vary multiple hyper-parameters
Practice Holistic Cross-Validation for evaluating models
Use SLURM to perform a large batch of experiments
Implement code that brings the results from a large batch of experiments into a single analysis

Deadline: Tuesday, March 9th @11:59pm.
Hand-in procedure: submit PDF to the HW3 dropbox on Gradescope.
This work is to be done on your own. While general discussion about Python, TensorFlow and Keras is okay, sharing solution-specific code is inappropriate. Likewise, you may not download code solutions to this problem from the Internet.

We will use the same data set as in HW2. However, we will focus on predicting shoulder orientation (theta[0]).

You should modify your code from HW2 to complete this assignment.

Implement a network with at least 5 hidden layers.
Add dropout to your Input layer and each of your Hidden layers. There should be no other regularization.
Execute a 3-dimensional experiment: rotation x training set size x dropout rate. Rotations and training folds are the same as in HW2. Attempt a reasonable range of dropout probabilities (at least 4), including zero. Note that for the zero probability case, you should not include Dropout layers in your network.
Write code that aggregates the results together, showing validation FVAF as a function of training set size. On the same plot, show one curve for each of your dropout probability choices.
Don't forget to label your axes and to provide a legend.

Perform the same form of experiment as part 1, except choose a range of values for either L1 or L2 regularization. Include no regularization, too (note that a regularization parameter of zero does not seem to mean zero regularization, instead just don't include regularization in your network).
There should be no Dropout here.
Create a similar plot as above.

For both sets of runs (part 1 and part 2), you have already computed mean FVAF across the rotations (the result for each model is a matrix indexed by hyper-parameter value and training set size). For models and each training set size, identify the hyper-parameter value that maximizes validation performance (so, each training set size will have one "best" hyper-parameter value and corresponding model).
For each best model, report the mean FVAF for the test sets (again, this is mean across rotations). Produce a figure that shows test set FVAF as a function of training set size. This figure will have one curve for each model.

Hand in a single PDF that contains all pieces, including code, results figures, and batch scripts to Gradescope.
Within Jupyter, you can generate a PDF by selecting File/Export As/PDF.
Note: the PDF generator is not particularly smart about not cutting off code on the right-hand-side of the PDF page. Check your PDF and if this is happening, then add newlines to make your lines shorter.
Do not submit MSWord files.

20 pts: Clean code for part 1 and part 2 (including documentation)
20 pts: Figure: Validation set performance with varying Dropout rate
20 pts: Figure: Validation set performance with varying Lx regularization parameter value
20 pts: Clean code for selecting the best model
20 pts: Figure: Test set performance for the best hyper-parameter choices for each model

Last modified: Tue Mar 2 23:00:06 2021