Fall 2023

The purpose of this course is to provide first year PhD students in engineering and computing with a solid mathematical background for two of the pillars of modern machine learning, data science, and artificial intelligence: linear algebra and applied probability.

Instructor: Justin Romberg

Download the syllabus

Go to Piazza

Course Notes

Primer: A concise review of linear algebra
Primer: basic matrix-vector manipulations
Advanced primer on analysis (this discusses concepts that we will refer to repeatedly, but are not necessary to appreciate the vast majority of content in this course)

I. Linear Representations
Notes 1, introduction to function approximation and basis expansions
Notes 2, linear vector spaces
Notes 3, norms and inner products
Notes 4, linear approximation (Jupyter notebook for example at end)
Notes 5, orthobasis expansions
Notes 6, non-orthogonal bases

II. Regression using Least-Squares
Notes 7, regression as a linear inverse problem (see also regression_examples.ipynb)
Notes 8, least-squares in Hilbert space
Notes 9, (continuous) linear functionals in Hilbert space
Notes 10, RKHS, kernel regression, Mercer’s theorem

III. Solving and analyzing least-squares problems
Notes 11, symmetric systems of equations
Notes 12, the SVD and least-squares
Notes 13, stable least-squares
Notes 14, iterative methods: gradient descent and conjugate gradients

IV. Statistical Estimation and Prediction
Notes 15, probability review, MMSE estimation
Notes 16, Gaussian estimation, Gaussian random processes
Notes 16a, Gaussian graphical models
Notes 17, maximum likelihood estimation
Notes 18, bias, variance, consistency, efficiency, Stein’s paradox
Notes 19, Bayesian estimation
Notes 20, classification (Bayesian and nearest neighbor)
Notes 21, logistic regression
Notes 22, empirical risk minimization

V. Learning Structure from Data
Notes 23, principal components analysis
Notes 24, Gaussian mixture models

Further topics
Notes 25, stochastic gradient descent for least-squares problems