Projection onto a Subspace

Projection — the idea

Given a vector $\mathbf{b}$ and a subspace $V$, the orthogonal projection of $\mathbf{b}$ onto $V$, written $\operatorname{proj}_V(\mathbf{b})$, is the unique vector $\mathbf{p} \in V$ closest to $\mathbf{b}$. The residual $\mathbf{r} = \mathbf{b} - \mathbf{p}$ is orthogonal to every vector in $V$.

Decomposition: $\mathbf{b} = \underbrace{\mathbf{p}}_{\in V} + \underbrace{\mathbf{r}}_{\in V^\perp}$

This split is unique.

Projection formula via a matrix $A$

If $V = \operatorname{Col}(A)$ where $A$'s columns are linearly independent, then $\operatorname{proj}_V(\mathbf{b}) = A (A^T A)^{-1} A^T \, \mathbf{b}$

The matrix $P = A (A^T A)^{-1} A^T$ is the projection matrix.

Properties of any projection matrix:

• $P^2 = P$ (applying twice is the same as once — idempotent).
• $P^T = P$ (symmetric — for orthogonal projection).
• Eigenvalues are 0 (in $V^\perp$) or 1 (in $V$).

Step-by-step

$A = \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 0 & 0 \end{pmatrix}, \quad \mathbf{b} = \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix}$

$V = \operatorname{Col}(A)$ is the $xy$-plane in $\mathbb{R}^3$ — the set of all vectors with last coordinate zero.

Step 1 — $A^T A$. $A^T A = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{pmatrix} \begin{pmatrix} 1 & 0 \\ 0 & 1 \\ 0 & 0 \end{pmatrix} = \begin{pmatrix} 1 & 0 \\ 0 & 1 \end{pmatrix} = I_2$

Since $A^T A = I$, $(A^T A)^{-1} = I$.

Step 2 — $A^T \mathbf{b}$. $A^T \mathbf{b} = \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \end{pmatrix} \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix} = \begin{pmatrix} 1 \\ 2 \end{pmatrix}$

Step 3 — Projection. $\mathbf{p} = A \cdot \begin{pmatrix} 1 \\ 2 \end{pmatrix} = \begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix}$

Step 4 — Residual. $\mathbf{r} = \mathbf{b} - \mathbf{p} = \begin{pmatrix} 1 \\ 2 \\ 3 \end{pmatrix} - \begin{pmatrix} 1 \\ 2 \\ 0 \end{pmatrix} = \begin{pmatrix} 0 \\ 0 \\ 3 \end{pmatrix}$

Check $\mathbf{r} \perp V$: the plane $V$ is spanned by $\mathbf{e}_1, \mathbf{e}_2$. $\mathbf{r} \cdot \mathbf{e}_1 = 0$, $\mathbf{r} \cdot \mathbf{e}_2 = 0$ ✓

Visual interpretation

$\mathbf{b} = (1, 2, 3)$ is a vector in $\mathbb{R}^3$. The plane $V$ is the floor ($xy$-plane). The projection $\mathbf{p}$ is the shadow of $\mathbf{b}$ directly below it — $(1, 2, 0)$ on the floor. The residual $(0, 0, 3)$ is the straight-up vertical piece.

Projection onto a line

For $V$ spanned by a single non-zero vector $\mathbf{a}$, the formula simplifies dramatically: $\operatorname{proj}_\mathbf{a}(\mathbf{b}) = \dfrac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{a} \cdot \mathbf{a}} \, \mathbf{a}$

This is the scalar "coefficient of $\mathbf{a}$ in $\mathbf{b}$" times $\mathbf{a}$ itself. It's the 1-column case of the general formula above.

Where projections appear

• Least squares regression: the fitted values are the projection of $\mathbf{y}$ onto the column space of the design matrix.
• PCA: project data onto principal components.
• Orthogonal decompositions: separating a signal into parts (low-frequency + high-frequency).
• Image compression: project onto a few top basis vectors and discard the rest.
• Fourier series: coefficients are projections of $f(x)$ onto basis functions.

Best approximation property

Among all vectors $\mathbf{v} \in V$, $\mathbf{p} = \operatorname{proj}_V(\mathbf{b})$ minimizes $\|\mathbf{b} - \mathbf{v}\|$. That's why projection is the foundation of least-squares approximation — it provides the closest point in a subspace.

Common mistakes

• Using the wrong columns. Only use columns that are linearly independent; otherwise $A^T A$ is singular.
• Forgetting that projection is idempotent. Applying $P$ twice gives the same result as once; $P(P \mathbf{b}) = P \mathbf{b}$.
• Confusing orthogonal projection with oblique projection. Oblique projections exist ($P^2 = P$ but not symmetric); our formula produces orthogonal ones.

Try it in the visualization

$\mathbf{b}$ and the subspace $V$ are drawn in 3D. A vertical dashed line from $\mathbf{b}$ to the subspace shows the projection path. The projected vector $\mathbf{p}$ sits on the plane; the residual $\mathbf{r}$ points perpendicular.