CMU 10-414/714: Deep Learning Systems

Author

Chao Ma

Published

June 12, 2026

Notes from CMU 10-414/714 Deep Learning Systems, focused on the systems and mathematical foundations needed to build deep learning workloads.

Lecture 5: Auto Differentiation Implementation The general architecture of an AD library, followed by Needle’s graph representation, forward execution, gradient control, and reverse-mode implementation.

Lecture 4: Automatic Differentiation Numerical gradient checking, computation graphs, forward- and reverse-mode AD, gradient accumulation across multiple paths, and tensor reverse passes.

Lecture 3 Part II: Manual Neural Networks Manual backpropagation for two-layer networks, matrix gradients, upstream error signals, vector-Jacobian products, and forward/backward passes.

Lecture 3: Manual Neural Networks Linear hypothesis limitations, feature maps, nonlinear activations, two-layer networks, ReLU, fully connected networks, and why depth helps in practice.

Lecture 2: ML Refresher / Softmax Regression Hypothesis classes, loss functions, optimization, softmax regression, cross-entropy, gradient descent, SGD, and the softmax gradient.