"Ever since computers were invented, we have wondered whether they might be made to learn. If we could understand how to program them to learn - to improve automatically from experience - the impact would be dramatic. Imagine computers learning from medical records which treatments are most effective for new diseases, houses learning from experience to optimize energy costs based on the particular usage patterns of their occupants, or personal software assistants learning the evolving interests of their users in order to highlight especially relevant stories from the online morning newspaper." Tom M. Mitchell, Machine Learning, Mc Graw-Hill 1999. !!! Objectives The course's objective is to transmit the theoretical basis of machine learning, to introduce various types of learning systems, including neural networks, support vector machines, decision trees and some unsupervised learning systems, and to give laboratorial practice on the use of several of these systems. The course is open to foreign students. All essential course materials are supplied in English. The classes will be in English if there is at least one student who requires it. !!! Faculty * Alexandre Jose Malheiro Bernardino. Email: alex(at)isr(dot)ist(dot)utl(dot)pt * Margarida Silveira. Email: msilveira(at)isr(dot)ist(dot)utl(dot)pt (Note - replace (at) by @ and (dot) by . ) !!! Schedule * Lectures ** Tuesdays and Thursdays, 15:30h-17.00h, room EA 3. * Labs ** Tuesdays and Thusdays, 17:00h-18:30h, room LSDC 1 (5th floor). * Questions (please book by email with 24h prior notice) ** Prof. Margarida Silveira: Tuesdays, 14:00, Room 5.15. (5th floor) ** Prof. Alexandre Bernardino: Fridays, 11:00, Room 5.15 (5th floor) !!! Lab Grades !!! * [[ http://www.isr.ist.utl.pt/~alex/aauto0910/Notas_lab_09_10.pdf | Download File ]] !!! Exams !!! * [[ http://www.isr.ist.utl.pt/~alex/aauto0910/docs/EXAMS.zip |Collection of exams from previous years and some solutions (requires password).]] * First Exam: 25/01, 9:00-12:00, rooms E3, EA 1. [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/exame10-01-25.pdf | Download (requires password)]] [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/exame10-01-25-sol1.pdf | Solutions Part 1]] [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/exame10-01-25-sol2.pdf | Solutions Part 2]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/PautaAAut2-2010.pdf | Classifications after first exam (reviewed)]] * Exam review: 2 February 2010, 14:00h, Room 5.15. * Extra schedule for questions: ** 21/01, 15:00h, room 5.15. ** 22/01, 15:00h, room 5.15. * Second Exam: 4/02, 9:00-12:00, rooms Q4.2 Q4.4 Q4.5. * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/PautaAAut4-2010.pdf | Classifications after second exam (reviewed)]] * Exam Review: 22 February 2010, 16:30h, Room 5.15. * Extra schedule for questions: ** 28/01, 15:00h, room 5.15. ** 29/01, 15:00h, room 5.15. ** 01/02, 15:00h, room 5.15. ** 02/02, 15:00h, room 5.15. ** 03/02, 15:00h, room 5.15. !!! Lecture Material Lecture 1 - Introduction * [[http://www.isr.ist.utl.pt/~alex/aauto0910/lecture1Intro.pdf | Lecture Slides]] * [[http://www.cs.cmu.edu/~tom/pubs/MachineLearning.pdf | (Additional Material) Whitepaper: The Discipline of Machine Learning, Tom Mitchell, July 2006]] * [[http://www.cs.cmu.edu/~tom/pubs/HumanMachineLearning_11_2006_web.pdf | (Additional Material) Presentation Slides: Human and Machine Learning, Tom Mitchell, November 2006]] Lecture 2 - Formulation of Machine Learning Problems * [[http://www.isr.ist.utl.pt/~alex/aauto0910/lecture2Fundamentals.pdf | Lecture Slides]] Lecture 3 - Basics of Function Approximation * [[http://www.isr.ist.utl.pt/~alex/aauto0910/lecture3Optimization.pdf | Lecture Slides]] Lecture 4 - Introduction to Neural Nets * [[http://www.isr.ist.utl.pt/~alex/aauto0910/lecture4nnets.pdf | Lecture Slides]] Lecture 5 - The ADALINE * [[http://www.isr.ist.utl.pt/~alex/aauto0910/lecture5Adaline.pdf | Lecture Slides]] Lecture 6 - Introduction to Multi-Layer Perceptrons * [[http://www.isr.ist.utl.pt/~alex/aauto0910/lecture6IntroMLP.pdf | Lecture Slides]] Lecture 7 - Backpropagation * [[http://www.isr.ist.utl.pt/~alex/aauto0910/lecture7Backprop.pdf | Lecture Slides]] * Also consult Multilayer Perceptrons, Luís Borges de Almeida (see Other Documents) Lecture 8 - Evaluation and Validation * [[http://www.isr.ist.utl.pt/~alex/aauto0910/lecture8Validation.pdf | Lecture Slides]] Lecture 9 - Dynamic Networks * [[http://www.isr.ist.utl.pt/~alex/aauto0910/lecture9DynamicNets.pdf | Lecture Slides]] Lecture 10 - Statistical Decision Theory * [[http://www.isr.ist.utl.pt/~alex/aauto0910/lecture10Bayes.pdf | Lecture Slides]] Lecture 11 - Statistical Decision Theory II * [[http://www.isr.ist.utl.pt/~alex/aauto0910/lecture11Bayes2.pdf | Lecture Slides]] Lecture 12 - Statistical Properties of Neural Networks * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/conditional-means.pdf | Slides in Portuguese from Prof. Fernando Silva about estimation of conditional means (requires password)]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/percentiles.pdf | Slides in Portuguese from Prof. Fernando Silva about estimation of medians and percentiles (requires password)]] Lecture 13 - Support Vector Machines * [[http://www.isr.ist.utl.pt/~alex/aauto0910/lecture13SVM.pdf | Lecture Slides]] * Also consult (see Other Documents) ** "A Tutorial on Support Vector Machines for Pattern Recognition", Christopher Burges. ** "Duality and Geometry in SVM Classifiers", Kristin Bennett and Erin Bredensteiner. Lecture 14 - Support Vector Machines II * [[http://www.isr.ist.utl.pt/~alex/aauto0910/lecture14SVM2.pdf | Lecture Slides]] Lecture 15 - Support Vector Machines III * [[http://www.isr.ist.utl.pt/~alex/aauto0910/lecture15SVM3.pdf | Lecture Slides]] * Also consult (see Other Documents) ** "A Tutorial on Support Vector Regression", Alex Smola and Bernhard Scholkopf. Lecture 16 - Unsupervised Learning I * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/density_estimation1.pdf | Slides in Portuguese from Prof. Fernando Silva about estimation of density functions with the Parzen Window method (requires password)]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/density_estimation2.pdf | Slides in Portuguese from Prof. Fernando Silva about estimation of density functions with Neural Networks (requires password)]] Lecture 17 - Unsupervised Learning II * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/max_likelihood.pdf | Slides in Portuguese from Prof. Fernando Silva about likelihood function and maximum likelihood estimation (requires password)]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/mixture-models.pdf | Slides in Portuguese from Prof. Fernando Silva about estimation of mixture models (requires password)]] Lecture 18 - Unsupervised Learning III * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/em.pdf | Slides in Portuguese from Prof. Fernando Silva about the Expectation Maximization Algorithm (requires password)]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/kmeans.pdf | Slides in Portuguese from Prof. Fernando Silva about the particularization of EM to K-means (requires password)]] * Also consult (see Other Documents) ** Ripley notes on EM. Lecture 19 - Seminar Monte Carlo Methods (by Dr. Ruben Martinez-Cantin) Lecture 20 - Data Normalization * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/datanormalization.pdf | Slides in Portuguese from Prof. Fernando Silva about Data Normalization (requires password)]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/adaptivenormalization.pdf | Slides in Portuguese from Prof. Fernando Silva about Adaptive Decorrelation (requires password)]] Lecture 21 - Principal Component Analysis * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/PCA.pdf | Slides in Portuguese from Prof. Fernando Silva about PCA (requires password)]] Lecture 22 - Conclusions on Principal Component Analysis * Consult the Tutorial on Principal Component Analysis by Javier Movellan (see Other Documents). Lecture 23 - Decision Trees * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/decisiontrees1.pdf | Slides in Portuguese from Prof. Fernando Silva about decision trees (requires password)]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/decisiontrees2.pdf | Slides in Portuguese from Prof. Fernando Silva about entropy (requires password)]] * Consult also the Tom Mitchel's book chapter on CART - Classification and regression trees (see Other Documents). Lecture 24 - Conclusion on Decision Trees * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/decisiontrees3.pdf | Slides in Portuguese from Prof. Fernando Silva about the decision trees (requires password)]] Lecture 25 - Seminar by Prof. Margarida Silveira. !!! Laboratory Sessions [[https://fenix.ist.utl.pt/disciplinas/aaut2/2009-2010/1-semestre/lab | In the official fenix page]] !!! Other Documents (require password) * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/ADALINELEARNING.zip | Matlab Examples for ADALINE learning]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/MLP.pdf | Multilayer Perceptrons, Luís Borges de Almeida]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/support-vector-machines4.pdf | "A Tutorial on Support Vector Machines for Pattern Recognition", Christopher Burges. ]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/duality-and-geometry-in-svms.pdf | "Duality and Geometry in SVM Classifiers", Kristin Bennett and Erin Bredensteiner.]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/karush-khun-tucker.pdf | "The Karush-Kuhn-Tucker Theorem", Moritz Kuhn. ]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/support-vector-regression.pdf | "A Tutorial on Support Vector Regression", Alex Smola and Bernhard Scholkopf. ]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/Ripley-EM.pdf | Ripley notes on the EM algorithm. ]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/Movellan-PCA.pdf | "A Tutorial on Principal Component Analysis", Javier Movellan. ]] * [[http://www.isr.ist.utl.pt/~alex/aauto0910/docs/Mitchell-CART.pdf | "Chapter on CART - Classification and REgression Trees", Tom Mitchel. ]] !!! Official Site * https://fenix.ist.utl.pt/disciplinas/aaut2/2009-2010/1-semestre !!! Want to learn more ? * International Conference on Machine Learning (ICML). * Conference on Neural Information Processing Systems (NIPS). * Annual Conference on Learning Theory (COLT). * Journal of Machine Learning Research (JMLR). ** This top journal is freely available online at http://www.jmlr.org. * Machine Learning. Published by Springer. ** http://www.springer.com/computer/artificial/journal/10994 !!! Video Links *[[http://videolectures.net/mlas06_mitchell_itm | Interview with Tom Mitchell]] *[[http://www.youtube.com/watch?v=_JS8z6RLuVM | The Machine Learning Rap]] !!! Acknowledgements The materials provided in this course have been inspired in the materials of previous courses by Prof: Fernando Silva, Prof. Luís Almeida and Prof. Jorge Marques.