Information Theory

Author

Chao Ma

Published

April 19, 2026

Notes on entropy, information, coding, and probabilistic structure, starting from the Oxford Math information theory lectures.

Oxford Information Theory Lecture 3: Introducing Codes Entropy bounds, chain rules, Fano’s inequality, symbol codes, uniquely decodable codes, prefix codes, and the Kraft-McMillan theorem.

Oxford Information Theory Lecture 2: Basic Properties of Information Conditional divergence, Gibbs inequality, chain rules for divergence and mutual information, data processing, and the main entropy inequalities.

Oxford Information Theory Lecture 1: Defining Entropy and Information Surprise as negative log-probability, entropy in bits, KL divergence as mismatch cost, mutual information as dependence, and conditional entropy through the chain rule.

---
title: "Information Theory"
author: "Chao Ma"
date: "2026-04-19"
---

Notes on entropy, information, coding, and probabilistic structure, starting from the Oxford Math information theory lectures.

---

::: {.content-grid}

::: {.content-card}
**[Oxford Information Theory Lecture 3: Introducing Codes](oxford-lecture-3-introducing-codes.qmd)**
Entropy bounds, chain rules, Fano's inequality, symbol codes, uniquely decodable codes, prefix codes, and the Kraft-McMillan theorem.
:::

::: {.content-card}
**[Oxford Information Theory Lecture 2: Basic Properties of Information](oxford-lecture-2-basic-properties-information.qmd)**
Conditional divergence, Gibbs inequality, chain rules for divergence and mutual information, data processing, and the main entropy inequalities.
:::

::: {.content-card}
**[Oxford Information Theory Lecture 1: Defining Entropy and Information](oxford-lecture-1-defining-entropy-information.qmd)**
Surprise as negative log-probability, entropy in bits, KL divergence as mismatch cost, mutual information as dependence, and conditional entropy through the chain rule.
:::

:::