Systematic introduction to neural networks, biological foundations; important network classes and learning algorithms; supervised models (perceptrons, adalines, multi-layer perceptrons), support-vector machines, echo-state networks, non-supervised networks (competitive, Kohonen, Hebb), recurrent networks (Hopfield, CTRNNs - continuous-time recurrent neural networks), spiking neural networks, spike-time dependent plasticity, applications. Special consideration will be given to neural networks embedded in adaptive systems having to interact with the real world, such as embodied systems (in particular robots). Cooperation of neural control, morphology, materials, and environment. Evolutionary approaches to designing autonomous systems; interaction of learning and evolution. Network theory applied to brain networks; motifs.

Additional case studies will be discussed to deepen the understanding of neural networks, e.g. Neural interfacing - coupling neural systems with technology (in particular robotic devices), neural imaging studies, Adaptive Resonance Theory, models of categorization (ALCOVE, Edelman), Distributed Adaptive Control (DAC), neural gas and DRNNs - Dynamically Rearranging Neural Networks (neuro-modulator-based networks), neural network models of memory. Selected neural network and brain modeling projects.

This is an elementary, interdisciplinary introduction to neural networks, suited not only for computer scientists, but also for economists, biologists, psychologists, etc.

The lectures will be given in English.

This course is worth of 6 credit points.

If you wish, you can do the exercises in groups of two - please hand in only one task sheet. You can also do them on your own if you prefer.

Important

The page will be subjected to changes over time. Please stay up-to-date by checking it periodically. 

Final Exam

The final exam will be Friday, 15 June,2012, from 8.00 to 9.45am in the lecture room 2.A.01.

In order to attend the final exam, you must achieve 50% out of the total possible points of all the exercises (Task sheets 1-4) together (not each).

It will not be an open-book exam, so you are not allowed to use your books and notes. However, you don't have to learn any formulas by heart, as we will provide you with a sheet containing all formulas (but you do have to know which one applies to which type of Neural Net). No laptops or cell phones.

Please bring along:

• simple pocket calculator (no cell phones or other advanced programmable devices)
• your student ID card (required)

We wish you a lot of success.

News

• First lecture will start at 8:30 am on 24th of Feb 2012.
• 16.02.2012: The new version of the script is now online (see Materials section)
• 15.03.2012: even newer version of the script is now online! changes include pages 21-28 and 30-31
• 15.03.2012: Task sheet 1 is now online (see below).
• 21.03.2012: The subject of the lecture on 23.03.2012 will be ''Support Vector Machines'', instead of ''Lab Tour;Reading script and papers'' (check the updated schedule).
• 22.03.2012: Two links for video tutorials on Support Vector Machines added (check Links section).
• 23.03.2012: The lecture slides for Cascade Correlation can be found in the Materials section.
• 27.03.2012: There has been a mistake in the Matlab-script which is required to solve Question 10 of Task Sheet 1. The corrected version is now online. Thanks to Tobias Klauser for pointing us to this error. (Don't worry if you already solved this task using the old script. The quality of the results will not change, only the scale of the error function.
• 06.04.2012: Task sheet 2 is now online.
• 16.04.2012: The solution of Task sheet 1 is now online.
• 20.04.2012: Task sheet 3 is now online.
• 22.04.2012: The solution of Task sheet 2 is now online.
• 27.04.2012: If you have issues with Matlab-licenses, try this workaround: 
  https://files.ifi.uzh.ch/ailab/teaching/NeuralNetworks/Matlab_license_issue.txt
  We work on getting more licenses for the University, but this will take a while. If you still have problems, you can work on our lab-computers to solve the tasksheet. Just contact us by email and we can make an appointment.

• 30.04.2012: The evaluation of our Nerual Networks lecture is currently running as an online-survey. Each student should have received a link to the online-evaluation via email. Please participate and help us improving our lecture!!!
• 04.05.2012: The PCA-demo script is now online in the materials section.
• 07.05.2012: The solution of Task sheet 3 is now online.
• 11.05.2012: Task sheet 4 is now online.

Tentative Schedule

Materials

Recommended Literature:

• J. Hertz, A. Krogh, R. Palmer, "Introduction to the theory of neural computation", Addison-Wesley Publishing Company.
  A "classic"; a bit mathematical, but sound, written by physicists. Recommended as a complement to the lecture script. It covers most but not all topics of the class (e.g. Support Vector Machines, spiking neurons, etc.).
   

• S. Haykin, "Neural Networks: A comprehensive foundation", Prentice Hall.
  Very comprehensive, covers most of the topics of the class. Can also be used as an introductory textbook and as a complement to the class. It also introduces quite a few topics that go beyond the class.
   

• N. Cristianini, J. Shawe-Taylor, "An Introduction to Support Vector Machines and other kernel-based learning methods", Cambridge University Press.
  Nice introduction to kernel-based learning machines. Mainly for the mathematically minded student. Support Vector Machines will be covered in class and are included in the book by Haykin.
  
   
• R. Rojas, "Neural Networks - A Systematic Introduction", Springer-Verlag, 1996.
  A nicely and comprehensively written overview of the field with robotics application examples. The book can be downloaded for free here:
  

  http://www.inf.fu-berlin.de/inst/ag-ki/rojas_home/pmwiki/pmwiki.php?n=Books.NeuralNetworksBook

   

Materials that are useful for the understanding of the course:

• The Neural Network script
   
• Lecture slides from Qian Zhao about Cascade Correlation (PDF).
  Paper about Cascade Correlation (PDF)
   
• Tutorials, Simulators, Applets:

NeuralNetworks Matlab demo

GreekAlphabet (PDF)

Linear Algebra Collection of formulas (PDF)

Java NN-Simulator and Cascade Correlation applet (ZIP)

NN-Simulator Manual (PDF) 

ART (adaptive resonance theory) article by Gail A. Carpenter and Stephen Grossberg

Support Vector Machines: Nice and short introduction to SVM. (PDF)

What is a supprt vector machine? (PDF)

SOM2D Demo - Python implementation (ZIP) - Simple code and visualization of 2D SOM.

Tutorial on training recurrent neural networks (echo state network) (PDF)

MLP demo: Recognition of handwritten digits (ZIP-File) (using Matlab Neural Network Toolbox)

PCA demo: use hebbian learning to find the principle components (Matlab script)

Links

Some links on the internet that are useful for the understanding of the course

Cascade-Correlation Tutorial

Cascade-Correlation Wikipedia

pattern recognition applet in japanese

multilayer perceptron applet

Support Vector Machines video tutorial 1 by Colin Campbell

Support Vector Machines video tutorial 2 by Chih-Jen Lin

Support Vector Machine Java Applet 1

Support Vector Machine Java Applet 2

Pattern completion using a hopfield net

Pattern recognition applet using a hopfield net

Travelling Salesman Problem: using a hopfield network in order to find possible solutions

Self-Organizing Maps: Kohonen Network

Travelling Salesman Problem: With a Kohonen network you can get quite satisfying results

Self-Organizing Maps: Kohonen Network in 3D

NERO: Neuro Evolving Robotics Operatives

SVM Toy, from National Taiwan University