Machine Learning for Engineering¶

Instructor: Daning Huang¶

Introduction¶

This website hosts the slides for several Machine Learning-related courses that are taught at Penn State Aerospace.

  • The goal is to help you to
    • Understand fundamentals of machine learning
    • Learn technical details of ML algorithms
    • Learn how to implement some important algorithms
    • Use machine learning algorithms for your research and applications
  • Prerequisites: multivariate calculus, linear algebra, some familiarity with probability

Use arrows to navigate; esc to see outline view of slides.

Disclaimer¶

Keep deriving equations until you give up; handle the rest by ML

What we focus on:

  • Regression problems
  • Fusion of physics and data
  • Dynamical systems

What we don't focus on:

  • Classification problems
  • Image processing
  • Large language model, AIGC, etc.
    • But LLMs are highly recommended to assist your learning...
    • But we do cover some recent generative models, e.g., diffusion model
  • Reinforcement learning (but might come soon)

Specifically, the course consists of following topics:

  • Mathematical background
  • Linear regression
  • Gaussian process regression
  • Neural networks
  • High-Dimensional systems
  • Dynamical and differential systems

Next:

  • Move to right to access the links to the specific topics, or
  • See here for a more technical description of the scope.

Mathematical background

  • Linear Algebra
  • Probability Theory, Supp. notes
  • Optimization
  • Differential Equations

Linear regression

  • Basics
  • Regularization
  • Probability Formulation
  • Model Selection
  • Kernel Method
  • More Variants draft

Gaussian process regression

  • Basics
  • Formulation
  • Markov Chain Monte Carlo
  • Sampling
  • Bayesian Optimization (PDF) slides (draft)
  • Variants / PDF (older version)

Neural networks

  • Basics
  • Training
  • Automatic Differentiation / Harder Cases
  • Variational Inference draft
  • Common Architectures / Physics-Informed NN

High-Dimensional systems

  • Basics
  • Spatial Decomposition
  • Autoencoders
  • Clustering
  • Temporal Decomposition

Dynamical and differential systems

  • Basics
  • Linear System
  • Nonnormality, DMD Revisited
  • Tackling Nonlinearity
  • Sparse Regression
  • Koopman Operator Theory
  • Neural ODEs / Extra: Differentially-constrained problems
  • Diffusion Models

References¶

  1. [PRML] Pattern Recognition and Machine Learning, Christopher Bishop, 2006.
  2. [MLaPP] Machine Learning: A Probabilistic Perspective, Kevin P. Murphy, 2012.
  3. [GPML] Gaussian Processes for Machine Learning, C. E. Rasmussen and C. K. I. Williams, 2006.
  4. [DMSC] Data-driven Modeling and Scientific Computation, Nathan Kutz, 2013.

Relevant Courses¶

  1. More basic ones: Berkeley CS 189, Stanford CS 229
  2. CNN-specific: Stanford CS 231n
  3. ML with a Linear Algebra flavor: MIT 18.065

Credits:¶

  1. UMich EECS 545 inspired me a lot when I took the course in 2016 and when I initially developed the course myself in 2020.
  2. @evislomo for generating the initial set of homework solutions