Singular Value Decomposition (SVD)

The SVD statement

Every $m \times n$ real matrix $A$ has a decomposition $A = U \Sigma V^T$

where

• $U$ is $m \times m$ orthogonal (columns = left singular vectors),
• $\Sigma$ is $m \times n$ diagonal with non-negative entries (singular values $\sigma_1 \ge \sigma_2 \ge \cdots \ge 0$),
• $V$ is $n \times n$ orthogonal (columns = right singular vectors).

SVD exists for any matrix — no invertibility, no squareness, no symmetry required.

Step-by-step

$A = \begin{pmatrix} 3 & 2 \\ 2 & 3 \end{pmatrix}$ (symmetric — SVD simplifies).

Step 1 — $A^T A$. $A^T A = \begin{pmatrix} 3 & 2 \\ 2 & 3 \end{pmatrix} \begin{pmatrix} 3 & 2 \\ 2 & 3 \end{pmatrix} = \begin{pmatrix} 13 & 12 \\ 12 & 13 \end{pmatrix}$

Step 2 — Eigenvalues of $A^T A$.

Characteristic polynomial: $(13 - \lambda)^2 - 144 = \lambda^2 - 26 \lambda + 169 - 144 = \lambda^2 - 26 \lambda + 25$.

$\lambda = \dfrac{26 \pm \sqrt{676 - 100}}{2} = \dfrac{26 \pm 24}{2} = 25, 1$.

Step 3 — Singular values.

$\sigma_i = \sqrt{\lambda_i}$: $\sigma_1 = 5, \quad \sigma_2 = 1$

$\Sigma = \begin{pmatrix} 5 & 0 \\ 0 & 1 \end{pmatrix}$

Step 4 — Right singular vectors (eigenvectors of $A^T A$).

$\lambda = 25$: $(A^T A - 25 I) \mathbf{v} = \mathbf{0}$: $\begin{pmatrix} -12 & 12 \\ 12 & -12 \end{pmatrix} \mathbf{v} = \mathbf{0} \implies \mathbf{v} = (1, 1)^T / \sqrt{2}$

$\lambda = 1$: $(A^T A - I) \mathbf{v} = \mathbf{0}$: $\begin{pmatrix} 12 & 12 \\ 12 & 12 \end{pmatrix} \mathbf{v} = \mathbf{0} \implies \mathbf{v} = (1, -1)^T / \sqrt{2}$

$V = \dfrac{1}{\sqrt{2}} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}$

Step 5 — Left singular vectors: $\mathbf{u}_i = A \mathbf{v}_i / \sigma_i$.

$\mathbf{u}_1 = A \mathbf{v}_1 / 5 = \dfrac{1}{5} \cdot \dfrac{1}{\sqrt{2}} \begin{pmatrix} 5 \\ 5 \end{pmatrix} = \dfrac{1}{\sqrt{2}} \begin{pmatrix} 1 \\ 1 \end{pmatrix}$

$\mathbf{u}_2 = A \mathbf{v}_2 / 1 = \dfrac{1}{\sqrt{2}} \begin{pmatrix} 1 \\ -1 \end{pmatrix}$

$U = \dfrac{1}{\sqrt{2}} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix}$

(Same as $V$ here because $A$ is symmetric positive-definite.)

Verification: compute $U \Sigma V^T$ — will reproduce $A$ exactly.

$U \Sigma = \dfrac{1}{\sqrt{2}} \begin{pmatrix} 5 & 1 \\ 5 & -1 \end{pmatrix}, \quad U \Sigma V^T = \dfrac{1}{2} \begin{pmatrix} 5 & 1 \\ 5 & -1 \end{pmatrix} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} = \dfrac{1}{2} \begin{pmatrix} 6 & 4 \\ 4 & 6 \end{pmatrix} = \begin{pmatrix} 3 & 2 \\ 2 & 3 \end{pmatrix} ✓$

Geometric interpretation

Any linear map decomposes as:

1. Rotate input via $V^T$ (align to principal axes).
2. Stretch axis-by-axis by $\sigma_i$ (the singular values).
3. Rotate output via $U$ (align to output directions).

For our matrix: the unit circle, passed through $A$, becomes an ellipse. The semi-axes are $\sigma_1 = 5$ (long) and $\sigma_2 = 1$ (short), aligned with $\mathbf{u}_1$ and $\mathbf{u}_2$ respectively.

Why SVD is the most important decomposition in linear algebra

• Exists for every matrix — no structural assumptions needed.
• Reveals rank: $\operatorname{rank}(A)$ = number of non-zero singular values.
• Best low-rank approximation (Eckart-Young): truncating the SVD at $k$ terms gives the optimal rank-$k$ approximation in both Frobenius and spectral norm.
• Pseudoinverse: $A^+ = V \Sigma^+ U^T$ with $\Sigma^+$ inverting non-zero singular values. Solves least-squares even when $A^T A$ is singular.
• Condition number: $\sigma_1 / \sigma_n$ measures sensitivity of linear systems to perturbations.
• Underlies PCA, latent-semantic analysis, recommender systems, denoising, and compression.

Compact/thin SVD

For an $m \times n$ matrix of rank $r$, the thin SVD keeps only the first $r$ singular values: $A = U_r \Sigma_r V_r^T$

where $U_r$ is $m \times r$, $\Sigma_r$ is $r \times r$, $V_r$ is $n \times r$. Stores and computes with far fewer numbers when $r \ll \min(m, n)$.

Common mistakes

• Taking square roots of negatives. Singular values are always non-negative (unlike eigenvalues). They come from eigenvalues of $A^T A$, which is positive semi-definite.
• Mixing up $U$ and $V$. $U$'s columns span the column space; $V$'s span the row space.
• Confusing SVD with eigendecomposition. SVD is for arbitrary matrices; eigendecomposition requires diagonalizability. For symmetric matrices they're closely related.

Try it in the visualization

Apply $A$ to a unit circle, producing an ellipse. Display $U$, $\Sigma$, $V^T$ as three sequential transformations: the circle rotates, stretches, and rotates again. Semi-axis lengths are the singular values; the major axis direction is $\mathbf{u}_1$.