CS 5073 Take Home Exam

1. Motivation

To test your understanding of artificial neural networks (ANNs), evolutionary computation (EC), and neuroevolution, this exam requires you to design a neuroevolutionary system and experimental setup that, if implemented, could be used to study an interesting set of research questions related to topics we have covered in this class. You will need to think carefully about the interrelationships of ANN and EC design, as well as experimental design in general, in designing this neuroevolutionary system.

2. Goal

The goal of this assignment is to test your understanding of course concepts by requiring you to apply them to an authentic neuroevolutionary problem.

3. Assignment

Design a neuroevolutionary system and associated experimental setup to use for studying the following. Be sure to specify and justify all aspects of this system and setup. Write up your design in the form (length, tone, content, etc.) of an experimental design section of a research paper.

Note that this design should be sufficiently detailed such that any person who is proficient in the relevant topics (neuroevolution and software development) could implement code to carry out the neuroevolutionary experiments you describe and get results that are highly likely to be in accordance with the results that would be obtained by any other person who is also proficient in the relevant topics who independently implemented code to carry out the experiments you describe. That is, you do not need to explain neuroevolution or software development in your description. However, all of the particulars of your design necessary to carry out the experiments and arrive at the results should be given.

Note also that you will not actually implement this design nor carry out the experiments. You will only be designing and writing up the system and the experiments which someone could carry out. You will not include a results section because you will have no results to report but you should explain the types of data that would be collected, if someone were to carry out these experiments, and how that data would be analyzed.

Problem Description

Prof H wants to study the evolution of deep reinforcement learning for continuous action spaces. In particular, he has good reasons to believe that existing noise sampling hyperparameter update rules (such as those given by Williams, 1992 as well as Shah and Hougen, 2017) are suboptimal; that the placement of noise for stochastic sampling in output layer neural units (as suggested by Williams, 1992) is suboptimal and, moreover, that synaptic noise (as proposed by Shah and Hougen, 2017) may be more appropriate than unit-based noise; that intuition is not an adequate basis for designing the architecture of deep artificial neural networks; that allowing these components to co-evolve is likely to produce much more interesting systems than evolving them in isolation; and that the environment can have a drastic effect on evolutionary outcomes.

In designing your neuroevolutionary system and associated experiments, here are some questions to consider:

• How will you allow for substantial variability in noise hyperparameter update rules to be encoded in the genome?
• How will you allow for substantial variability in noise sampling placement to be encoded in the genome?
• How will you allow for substantial variability in deep artificial neural network architectures to be encoded in the genome?
• How will you allow for the co-evolution of these system components?
• How will you allow for substantial variability in all of these system characteristics to be selected for during the evolutionary process?
• Which aspects of evolved deep reinforcement learning systems are likely to be generalizable?
• How will you design the environment to foster the evolution of generalized reinforcement learning to the extent practical?
• Given that your experiment will necessarily be computationally expensive to deal with the evolution of these systems, how will you attempt to minimize computational overhead, while still allowing for the variability just mentioned?

Note that you do not have to consider all or even many possible neuroevolutionary designs and/or experiments appropriate to explore these ideas. A single system and a single experiment is sufficient.

References:

• Williams, Ronald J. 1992. “Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning.” Machine Learning 8 (3–4): 229–256. http://link.springer.com/article/10.1007/BF00992696.
• Shah, S. N. H., and D. F. Hougen. 2017. “Stochastic Synapse Reinforcement Learning (SSRL).” In 2017 IEEE Symposium Series on Computational Intelligence (SSCI), 1–8. IEEE. https://doi.org/10.1109/SSCI.2017.8285425.

5. What to Turn In

You will turn in to the appropriate drop box in Canvas a machine readable electronic copy of your work that completes the exercise above.