Homework 2 - More ANN Fundamentals
Due Tuesday, February 16, 2021
Due Thursday, February 18, 2021
1. Motivation
Artificial neurons (ANs) can be combined in many ways to compute more
complex functions than could be computed by a single AN. The most
fundamental way is by combining ANs into a layered, feedforward neural
network (FFNN). Moreover, FFNNs may be used for many tasks but the two
most fundamental are classification and function approximation, of which
classification is the easier to visualize. These ANN fundamentals
— FFNNs and classification — are the topics of this
homework.
2. Goal
The goal of this assignment is to give you experience with basic
FFNNs and classification.
3. Assignment
Complete the following exercises:
Part 1 — FFNN Representation
- Consider a two-layer FFNN — that is, one with two layers of
computational elements (ANs) — used for classification in a 2D
space with augmented vectors. The ANs in this FFNN are all SUs and
their activation functions are identical to fAN as
given in Homework 1, Subpart 1.1. There are three ANs in the hidden
layer and one in the output layer. Given the weights
v1,1=−0.3, v2,1=0.3,
v3,1=−0.9, v1,2=0.2,
v2,2=0.3, v3,2=−0.9,
v1,3=0.5, v2,3=0.9,
v3,3=0.4, w1=1.0,
w2=0.0, w3=−0.8, and
w4=0.0, draw this FFNN.(Note this FFNN uses terminology following Engelbrecht in which each vj,i refers to a weight between unit i in the input layer and unit j in the hidden layer, and each wk,j refers to a weight between unit j in the hidden layer and unit k in the output layer. Also, in the augmented vector, the last input at each layer is a bias value of −1.)
- Draw the decision region encoded by this FFNN. Be sure to
indicate the γ1 side of the region. Be sure to
indicate which portion of the decision region is due to each hidden
layer AN. (Hint: If you're having difficulty figuring out exactly how
the decision region works out given the weights above, try plotting
the points from Exercise 3 immediately below and calculate their
classes by plugging their point values into the equations defined by
the weights above in Exercise 1.)
- Add the following points on the graph you just drew and
label the class of each according to the AN.
- (−5.0, −2.0)
- (5.0, −3.0)
- (0.0, 0.0)
- (−2.0, −4.0)
- (2.0, 2.0)
- (1.0, 5.0)
- (−1.0, 3.0)
- Explain the significance of the sign of
w4.
- Explain the significance of the value of
w2.
- Explain the significance of the relative values of
w1 and w3.
- Explain how the decision region for this FFNN would change
if the value of w2 were changed to 0.3 rather
than 0.0 and explain which points, if any, from those above
would be classified differently and which would be classified the
same. Be sure to discuss the relative values of w1,
w2, w3, and
w4.
- Explain how the decision region for this FFNN would change
if γ2 were changed to −1 rather than 0 and
explain which points, if any, from those above would be
classified differently and which would be classified the same. (For
this hypothetical, use a value of 0.3 for w2.)
- Give an example weight vector w for which a
γ2 value of −1 would give a different
classification for one of the points from exercise 3 than would a
γ2 value of 0. Explain your answer.
- Explain how the decision region for this FFNN would change
if fAN for the hidden layer ANs were changed to be a
linear activation function, in particular, if
fAN(net)=net for the hidden layer
ANs. (For the output AN, fAN remains unchanged in
this hypothetical.) Explain which points, if any, from those
above would be classified differently and which would be classified
the same. (For this hypothetical, use a value of −0.4 for
w2.)
Part 2 — FFNN Learning
Consider the two-layer FFNN described above in Part 1, Exercise 1,
except using sigmoidal logistic activation functions with λ=1.
This FFNN uses the backpropagation algorithm we covered in class with
η = 1.0. The target values for γ1 and
γ2 are 0.9 and 0.1, respectively.
- The update equations for weights between hidden and output layer is
- wkjnew = wkjold + η(tk - ok)(1 - ok)okyj.
- The update equations for weights between input and hidden layer is
- vjinew = vjiold + η∑(tk - ok)(1 - ok)okwkj(1 - yj)yjzi
where the summation is over K. (Here, following Engelbrecht, the target output values are tk, the actual output values are ok, the outputs of the units in the hidden layer are yj, and the inputs are zi.)
Complete the following exercises:
- Explain how the weights of this FFNN would be updated if
presented with the data item (−1.0, −1.0)
γ1. Show your work. Keep track of four
significant digits.
- Calculate the output value of the FFNN above if, after
learning on (−1.0, −1.0) γ1, you were to
present this data item to the FFNN again. Show your work. Keep
track of four significant digits.
- Explain whether the error value for the input (−1.0,
−1.0) γ1 increased or decreased due to
learning.
4. What to Turn In
You may write and draw your responses to this assignment neatly by
hand or type your answers and use graphing software to complete the
exercises for this assignment. If drawn, the diagrams should be neatly
drawn on engineering or graph paper. In any case, you should turn in to
the appropriate Canvas dropbox an electronic (perhaps scanned) copy of
your assignment, so that I can grade it online.