ickma.dev
Welcome to ickma.dev
My learning notes and thoughts on math and machine learning.
Currently reading the Deep Learning book.
Deep Learning Book
Chapter 6.1: XOR Problem & ReLU Networks
How ReLU solves problems that linear models cannot handle.
Chapter 6.2: Likelihood-Based Loss Functions
The mathematical connection between probabilistic models and loss functions.
Chapter 6.4: Architecture Design - Depth vs Width
How depth enables hierarchical feature reuse and exponential expressiveness with fewer parameters.
Chapter 6.5: Back-Propagation and Other Differentiation Algorithms
The algorithm that makes training deep networks computationally feasible through efficient gradient computation.
Chapter 7 Prerequisites: Hessian Matrix, Definiteness, and Curvature
Essential second-order calculus concepts needed before Chapter 7 on optimization algorithms.
Chapter 7.1.1: L2 Regularization (Parameter Norm Penalty)
How L2 regularization shrinks weights based on Hessian eigenvalues, preserving important directions while penalizing less sensitive ones.
Chapter 7.1.2: L1 Regularization
L1 regularization uses soft thresholding to push small weights to exactly zero, creating sparse solutions that perform feature selection.
Chapter 7.2: Constrained Optimization View of Regularization
Regularization as constrained optimization: penalty form vs Lagrangian with KKT conditions and min-max dual training.
Chapter 7.3: Regularization and Under-Constrained Problems
Why regularization is mathematically necessary when solving under-constrained linear systems, and how it ensures invertibility.
Chapter 7.4: Dataset Augmentation
How transforming existing data can improve generalization and combat overfitting when training data is limited.
Chapter 7.5: Noise Robustness
Mathematical derivation showing how adding Gaussian noise to weights is equivalent to penalizing large gradients.
Mathematics
Reflections & Synthesis
- Deep Connections: Invertibility, Null Space, Independence, Rank, and Pivots Exploring how these fundamental concepts are different perspectives on information preservation in linear transformations.
MIT 18.06SC Linear Algebra
- Lecture 1: The Geometry of Linear Equations
Two powerful perspectives that reveal the hidden beauty of linear systems: row picture vs column picture. - Lecture 2: Elimination with Matrices
The systematic algorithm that transforms linear systems into upper triangular form for easy solution. - Lecture 3: Matrix Multiplication and Inverse
Five different perspectives on matrix multiplication, from element-wise computation to rank-1 decomposition, plus understanding when matrices can’t be inverted. - Lecture 4: LU Decomposition
Factoring matrices into Lower × Upper triangular form: the foundation of efficient numerical linear algebra and solving multiple systems with the same matrix. - Lecture 5.1: Permutation Matrices
Permutation matrices reorder rows and columns using a simple structure of 0s and 1s. - Lecture 5.2: Transpose
The transpose operation switches rows to columns, creating symmetric matrices. - Lecture 5.3: Vector Spaces
Vector spaces and subspaces: closed under addition and scalar multiplication. - Lecture 6: Column Space and Null Space
Column space determines which \(b\) make \(Ax = b\) solvable. Null space contains all solutions to \(Ax = 0\). - Lecture 7: Solving Ax=0 - Pivot Variables and Special Solutions
Systematic algorithm to find null space using pivot/free variables and RREF. Dimension of null space is n-r. - Lecture 8: Solving Ax=b - Complete Solution to Linear Systems
Complete solution is particular solution plus null space. Four cases based on rank: exactly determined (unique), overdetermined (0 or 1), underdetermined (infinite), and rank deficient (0 or infinite). - Lecture 9: Independence, Basis, and Dimension Linear independence prevents redundancy, basis is minimal spanning set, dimension measures space size. Rank-nullity theorem: dim(C(A)) + dim(N(A)) = n.
- Lecture 10: Four Fundamental Subspaces The four fundamental subspaces completely characterize any matrix: column space, null space, row space, and left null space.
- Lecture 11: Matrix Spaces, Rank-1, and Graphs Matrix spaces as vector spaces, rank-1 matrices, dimension formulas for subspace intersections and sums, and differential equations as vector spaces.