To test your understanding of artificial neural networks (ANNs), evolutionary computation (EC), and neuroevolution, this exam requires you to design a neuroevolutionary system that, if implemented, could be used to study an interesting set of research questions related to topics we have covered in this class. You will need to think carefully about the interrelationships of ANN and EC design, as well as experimental design in general, in designing this neuroevolutionary system.
The goal of this assignment is to test your understanding of course concepts by requiring you to apply them to an authentic neuroevolutionary problem.
Design an experimental neuroevolutionary system to use for studying the following. Be sure to specify and justify all aspects of this system. Write up your design in the form (length, tone, content, etc.) of an experimental design section of a research paper.
Prof H wants to study the relationships between risk aversion, learning from reward feedback, information sharing, and resources sharing. In particular, he wants to know the optimal knowledge and resource sharing strategy to minimize the total risk premium paid out by a large group of homogeneous agents while maximizing the survival of the individual agents.
In the initial scenario to be studied, there will be 100 agents and two options: A and B. Option A is consistently rewarding with a moderate reward value while option B is highly variable with a 50% chance of yielding a high reward and a 50% chance of yielding a low reward. The expected value of option B is higher than the expected value of option A but the low reward from option B is lower than the reward from option A.
The rewards received by the agents correspond to resources that they need for survival. Each agent has a capacity to store resources from one time step to the next. Each agent starts the scenario with a small amount of resources and consumes resources on each time step. The reward from option A on a single time step is equal to the amount consumed by an agent each time step.
Prof H wants to use an ANN as the decision-making system for each agent. The only possible action for each agent on each time step is to choose option A or option B. The possible inputs to each ANN are any or all of the state variables describing the current or past states of the scenario (e.g., the agent’s current resource level, the reward the agent received from the environment last time step) or derivative values that could be computed from these values (e.g., the average reward received by all agents since the start of the scenario for choosing option B).
Q1. What is the level of detail we need to put into the description of the design?
A1. The design should be sufficient for any person who is proficient in the relevant topics (neuroevolution, risk, and software development) to implement code that will carry out the neuroevolutionary experiments you describe and get results that are highly likely to be in accordance with the results that would be obtained by any other person who is also proficient in the relevant topics who independently implemented code to carry out the experiments you describe. That is, you do not need to explain neuroevolution, risk, or software development in your description but all of the particulars of your design necessary to carry out the experiments and arrive at the results should be given.
Q2. Do we need to implement the system, carry out the experiments, and/or report results?
A2. No. You only need to complete the design and write it up.
Q3. From my understanding, there are different approaches to designing a neuroevolutionary system; there are also different approaches to applying a specific system to this particular problem. Which part should we focus on for this exam? I've been focusing on the latter but I want to confirm.
A3. You are certainly welcome to make use of existing neuroevolutionary systems in your design for this exam. However, you should note that there is unlikely to be an existing neuroevolutionary system that is ideally structured to take on the experimental question described, so you will probably need to adapt any existing system(s) you propose to apply here.
Q3.1. Oh okay. In that sense, the main task here is to use EC to find the optimal ANN architecture for this specific problem.
A3.1. Your design may include evolving the ANN architecture. It could include evolving other aspects of the ANN, such weights, initial weights, weight update rules, input accumulation functions, and/or activation functions. It could include the evolution of policies related to knowledge and/or resource sharing. It could include survival of agents and/or population success measures. These ideas may be combined in whatever way you think is appropriate to address the research question posed in the exam. You just need to decide what is appropriate, explain it so that someone could implement it, and justify it so that someone who wanted to answer the research question would want to implement it.
Q4. It is stated that optimal knowledge and resource sharing strategies in order to minimize total risk premium and maximize survival is desired. From that, should I take it that the goal then is to design some neuroevolutionary system in which parameters of both knowledge sharing and resource sharing mechanisms are evolved? If so, what might a resource sharing strategy look like?
A4. Your design may include evolving knowledge and/or resource sharing strategies. It could include evolving aspects of the ANN, such as weights, initial weights, weight update rules, topology, input accumulation functions, and/or activation functions. It could include survival of agents and/or population success measures. These ideas may be combined in whatever way you think is appropriate to address the research question posed in the exam. You just need to decide what is appropriate, explain it so that someone could implement it, and justify it so that someone who wanted to answer the research question would want to implement it.
There are many possible resource sharing strategies that could be used, so long as all of the agents in a given population follow the same strategy (because they are supposed to be homogeneous). Components of the strategy would include who contributes to others (e.g., everyone, no one, those with high stored resources, those receiving high rewards on a given time step, individuals chosen at random, individuals chosen through round robin, etc.), who receives shared resources (e.g., everyone, no one, etc.), how much is shared (everything, nothing, a fraction of the total stored resources, a fraction of reward received on a given time step, everything over a threshold amount, etc.), how it is shared (common pool, one-to-one, random, etc.), etc. Put these components together and you have a strategy (e.g., no one shares with anyone, everyone contributes all of their reward on a given time step to a common pool from which everyone draws equally, etc.).
Q5. Another concern/misunderstanding: You wish to minimize total risk premium paid out by the group, while maximizing survival. My understanding is that risk premium is the difference between the expected value of B and the risk-free option when B is chosen. Won't this lead the group to simply never choose the risky option because this means they never pay any risk premium while insuring that every member survives because they get exactly how much they need to survive by choosing A?
A5. Actually, you have the sign reversed on risk premium. In the example I gave in class (A: $120,000 sure thing vs. B: 50:50 chance of $0 or $300,000 for an expected value of $150,000), you pay out a risk premium of $30,000 to choose A because you are giving up $30,000 in expected value in order to get rid of your risk. To minimize the risk premium paid out when the risky option has a higher expected value than the safe option, everyone would need to choose the risky option, not the safe option.
You will turn in to the appropriate drop box in D2L a machine readable electronic copy of your work that completes the exercise above.