Markov Chains: State Diagrams and Steady State

What is a Markov chain?

A Markov chain is a random process that moves between a finite set of states step by step, where the next state depends only on the current state — not the full history. That "memoryless" property is called the Markov property.

The chain is fully described by a transition matrix $P$, where $P_{ij} = P(\text{next state} = j \mid \text{current state} = i)$. Each row sums to 1 (some next-state must happen).

The weather model

States: S (sunny), R (rainy).

| From \ To | S | R | |-----------|-----|-----| | S | 0.8 | 0.2 | | R | 0.4 | 0.6 |

So $P = \begin{pmatrix} 0.8 & 0.2 \\ 0.4 & 0.6 \end{pmatrix}$.

(Since tables do not render here, read it as: $P_{SS} = 0.8$, $P_{SR} = 0.2$, $P_{RS} = 0.4$, $P_{RR} = 0.6$.)

Step-by-step — steady-state probabilities

The steady-state (or stationary) distribution $\pi = (\pi_S, \pi_R)$ satisfies $\pi P = \pi$ and $\pi_S + \pi_R = 1$. It's the long-run fraction of days spent in each state.

Step 1 — Write the balance equations. For each state, the flow in equals the flow out: $\pi_S = 0.8 \, \pi_S + 0.4 \, \pi_R$ $\pi_R = 0.2 \, \pi_S + 0.6 \, \pi_R$

Step 2 — Simplify. Rearranging the first equation: $0.2 \, \pi_S = 0.4 \, \pi_R \implies \pi_S = 2\pi_R$

Step 3 — Apply $\pi_S + \pi_R = 1$: $2\pi_R + \pi_R = 1 \implies \pi_R = \tfrac{1}{3}, \quad \pi_S = \tfrac{2}{3}$

So in the long run, $\boxed{\pi_S = 2/3, \; \pi_R = 1/3}$ — about 67% of days are sunny.

Two-step transition

$P(\text{sunny in 2 days} \mid \text{sunny today}) = P_{SS}^2 + P_{SR} \, P_{RS} = 0.8^2 + 0.2 \cdot 0.4 = 0.64 + 0.08 = 0.72$. Equivalently, compute the $(S, S)$ entry of $P^2$.

Verification

Plug $\pi$ back into $\pi P$:

• $(2/3)(0.8) + (1/3)(0.4) = 8/15 + 2/15 = 10/15 = 2/3 \checkmark$
• $(2/3)(0.2) + (1/3)(0.6) = 4/30 + 6/30 = 10/30 = 1/3 \checkmark$

Common mistakes

• Forgetting rows sum to 1. Every row is a probability distribution over next states.
• Confusing "state" with "observation." A hidden-Markov model has hidden states and observations; a plain Markov chain's state is directly observable.
• Assuming a unique steady state. You need the chain to be irreducible (every state reachable from every other) and aperiodic. Our weather chain is both.

Try it in the visualization

Adjust the two transition probabilities and watch the state diagram, the $P$ matrix, and the steady-state bar chart all update together. Step the chain one day at a time to watch it converge toward $\pi$.