CS 5453: Empirical Methods for Computer Science: Projects

The semester-long project constitutes a significant percentage of your class grade and is intended as a venue in which you explore the techniques that we have been discussing. The topic of your project will be one of your own choosing (subject to approval). Ideally, the research topic will be one in which you are already engaged and have conducted experiments. The focus of your project should be on the process of formulating hypotheses and experimental questions, and on conducting and evaluating experiments. The focus should not be on the design and implementation of the algorithms of study (although some of this will probably be required).

Your project will be developed incrementally. Over the course of the semester, there will be a total of five checkpoints. Each checkpoint consists of an oral and a written component. The oral components are presented in class (with slides) and will focus on the new work that you have done since the last checkpoint. The written component is a "living document" that will evolve and be added to at each checkpoint. You must make progress at each checkpoint in order to receive full credit.

We will be using the AAAI paper template. The latex and M$word templates are both contained within the AuthorKit.zip file. The total length of your document (at any stage) should never exceed six pages.

Possible general topics include (these are not exclusive):

• Comparison of multiple algorithms
• Examining the architectural components to your model that are key to the results that you see
• Examining parameter settings

Project Checkpoint 1: Proposal

Grading rubric: postscript and pdf

Oral presentation: February 15th

Written document (incorporating feedback from presentation): February 22

You should:

• Define and justify the research domain and question
• Define the experimental questions
• Identify the experiments to be performed in your exploratory analysis
  • What are the expected results? Why?

Project Checkpoint 2: Exploratory Analysis

Grading rubric: postscript and pdf

At this checkpoint, you have completed an initial exploratory analysis. In particular, you have examined a number of candidate experimental questions.

Oral presentation: March 3rd

Report on the results of this first set of exploratory experiments:

• What are the experiments?
• What are the results?
• Why do you think that you see these results?
• Which experimental directions appear most promising for further analysis?
• What is (are) your experimental hypothesis(es)?
• How much data do you think that you will need?

Written document: March 10th

Notes:

• Abstract: this is a one paragraph advertisement of your paper. You may not have a complete view of what you will do by the end of the semester, but you will have some guesses.

  Contents: What is your domain/problem? Why is it important (from the domain perspective, as well as computationally/mathematically)? What are the key insights and your approach (in brief)? What are (some of) the experiments? And what are the (anticipated) results?

• Introduction: use this to talk in more detail about the problem and solution domain.

• Prior work: here, you will begin to explain what others have done before you. In part, this is motivation for your approach.

• Exploratory analysis: explain your initial experimental questions and the results of your analysis. Also, talk about how this will lead to the next steps.

• Proper references (use the AAAI reference format)

• Endeavor to use active sentence construction (think active verbs). Avoid passive sentences (e.g., "the experiment was performed" and overuse of the third person (it is ok to say "I" or "we" sparingly).

• Do not use informal language (e.g., "run an algorithm")

• Check your spelling

Project Checkpoint 3: Experimental Evaluation

Grading rubric: postscript and pdf

Oral presentation: March 31st

Written document: April 7th

For this checkpoint, you have completed a first set of detailed experiments, focusing largely on the promising directions as identified in the 2nd checkpoint. You have also made a preliminary attempt at making use of the hypothesis testing tools that we have been learning in class. Because we have not covered the full spectrum of approaches, you many need to read ahead a bit.

For this checkpoint:

• Describe the experiments
• What are the detailed results?
• What are the results of your first hypothesis tests?
• Do the results support your research question?
• Does your research question change in any way?
• What are the new experimental questions to be pursued for the next checkpoint?

Project Checkpoint 4: Hypothesis Testing

Grading rubric: postscript and pdf

Oral presentation: April 19th

Written document: April 26th

For this checkpoint, you have completed a new set of experiments and have used hypothesis testing to analyze the results in detail.

• Describe the addition to your experiments.
• What are the detailed results?
• Describe and justify the hypothesis tests that you have chosen.
• What are the results of the hypothesis tests?
• What are the implications of the hypothesis test results?
• What are the next experimental questions to be pursued?

Project Checkpoint 5: Final

Grading rubric: postscript and pdf

Written document: May 5th

Oral presentation: May 11 (Final Exam Period)

At this point, you have a complete story to tell, both in your paper and in your presentation. Both should be comprehensive in their presentation of the full story. However, this does not mean that you need to present in either venue an exhaustive list of what you have done over the semester. Instead, select the hypotheses, experiments and analyses that are central to your main argument. This does not mean that you should gloss over results that do not support this main argument. But, it does mean that if you have deviated from your initial hypotheses, then you do not need to focus on them.

Back to Empirical Methods Home

fagg cs ou edu

Last modified: Thu Apr 7 17:02:00 2011