CMU 10-414/714: Deep Learning Systems
Notes from CMU 10-414/714 Deep Learning Systems, focused on the systems and mathematical foundations needed to build deep learning workloads.
An LSTM Batch Is a Schedule, Not Just a Shape Lecture 16: LSTM gate packing, tensor layout, variable lengths, and truncated gradients.
An RNN Carries State; Training Carries a Graph Lecture 15: recurrent state, BPTT, LSTM gradient paths, and permitted context.
Lecture 14: Convolutional Network Implementation Loops, patch views and im2col: account for the memory behind a matrix multiplication.
Lecture 13: Hardware Acceleration Implementation Views, compaction, backend dispatch and host/device transfers: preserve the contract behind each tensor operation.
Lecture 12: GPU Acceleration Thread ownership, cooperative loading, and two levels of tiling: count each memory boundary separately.
Lecture 11: Convolutional Networks Shared weights, two matrix views, and the backward pass: input gradients, kernel gradients, and im2col workspace.
Lecture 10: Hardware Acceleration for Linear Algebra Strided views, SIMD, register tiles, and cache panels: count data reuse as well as arithmetic.
Lecture 9: Neural Network Library Implementation Parameter discovery, gradient boundaries, stable softmax, and multiple-output operators in Needle.
Lecture 8: Normalization, Dropout, and Model State Why unchanged weights can produce different predictions: normalization axes, BatchNorm running state, and dropout masks.
Lecture 7: Neural Network Library Abstractions Why residual blocks need a forward definition, not a new backward engine: modules, tensor graphs, and optimizer state.
Lecture 6: Optimization and Initialization Broadcast gradients, momentum, Adam, stochastic updates, and why initialization scale compounds with depth. Includes two explanatory animations.
Lecture 5: Auto Differentiation Implementation The general architecture of an AD library, followed by Needle’s graph representation, forward execution, gradient control, and reverse-mode implementation.
Lecture 4: Automatic Differentiation Numerical gradient checking, computation graphs, forward- and reverse-mode AD, gradient accumulation across multiple paths, and tensor reverse passes.
Lecture 3 Part II: Manual Neural Networks Manual backpropagation for two-layer networks, matrix gradients, upstream error signals, vector-Jacobian products, and forward/backward passes.
Lecture 3: Manual Neural Networks Linear hypothesis limitations, feature maps, nonlinear activations, two-layer networks, ReLU, fully connected networks, and why depth helps in practice.
Lecture 2: ML Refresher / Softmax Regression Hypothesis classes, loss functions, optimization, softmax regression, cross-entropy, gradient descent, SGD, and the softmax gradient.