for some eigenvector $\bm{v}$. A general real square matrix may have complex eigenvalues and eigenvectors, so we will limit ourselves to real symmetric matrices (that is, those satisfying $\bm{A}=\bm{A}^T$), which are guaranteed to have real eigenvalues and eigenvectors (note the converse is not true: a matrix does not necessarily have to be symmetric for all of its eigenvalues to be real). We denote the set of real symmetric $n\times n$ matrices as $\mathbb{S}^n$.

The eigenvector $\bm{v}$ in $\eqref{1}$ is free to take any non-zero magnitude, so we will assume $\|\bm{v}\|_2=1$. We can then left-multiply both sides of $\eqref{1}$ by $\bm{v}^T$ to obtain the relationship $\bm{v}^T\bm{A}\bm{v}=\lambda$. Thus the maximum eigenvalue $\lambda_{\max{}}(\bm{A})$ can be expressed as the optimization problem

It turns out that the maximum eigenvalue function $\lambda_{\max{}}:\mathbb{S}^n\to\R$ is convex, even though $\eqref{2}$ looks like a non-convex problem. Indeed, $\eqref{2}$ is a quadratically-constrained quadratic program (QCQP), but neither the objective function nor the constraint are convex (since we have made no definiteness assumptions about $\bm{A}$ and the constraint is a quadratic equality—we will come back to this).

We will explore the convexity of $\lambda_{\max{}}(\bm{A})$ in the rest of the post, which will allow us to have some fun with convex optimization and Lagrangian duality. Similar arguments also reveal that the minimum eigenvalue is a concave function, but we leave that as an exercise to the reader. If you’ve read Convex Optimization by Boyd and Vandenberghe (it’s freely available at the link provided) there shouldn’t be much here that’s totally new to you, but I think we can still enjoy ourselves.

Of course, if your goal is just to find the maximum eigenvalue of a (symmetric) matrix, there are much faster algorithms than the convex optimization problems we’ll explore in this post, but reasoning about eigenvalues is quite fundamental to convex programs arising in control theory, structural analysis, and portfolio optimization, among others. An interactive Python notebook that accompanies this post can be accessed directly in the browser here.

Convexity

Recall that a function is convex if a line segment drawn between any two points on the function lies on or above the function itself. In our case, we have a function of the form $g:\mathbb{S}^n\to\mathbb{R}$; by definition, it is convex if and only if

for any $\bm{A},\bm{B}\in\mathbb{S}^n$ and $t\in[0,1]$. Using $\eqref{2}$, we have

where we have used the fact that the sum of maxima is at least as large as the maximum of a sum, revealing that $\lambda_{\max{}}$ is indeed convex.

In general, the pointwise maximum over a family of convex functions is itself convex (see Figure 1 below for a visual example). That is, if a function $f(\bm{A},\bm{v})$ is convex in $\bm{A}$ for every $\bm{v}\in\mathcal{V}$, then the function

is also convex (see Section 3.2.3 of Convex Optimization). In our case, $f(\bm{A},\bm{v})\coloneqq\bm{v}^T\bm{A}\bm{v}$ is linear in $\bm{A}$, which means it is both convex and concave in $\bm{A}$, for every $\bm{v}$.

Figure 1: The function $g(x)\coloneqq\max_{i=1,2}\ f_i(x)$ (highlighted in red) is the pointwise maximum of two convex functions $f_1$ and $f_2$, and is therefore itself convex.

(Strong) Duality

Earlier we stated that $\eqref{2}$ is a QCQP, but that it is not a convex problem. For it to be convex, the constraint would need to define a convex set (for example, $\|\bm{v}\|_2\leq1$), and, since $\eqref{2}$ is a maximization problem, the objective function would need to be concave (with respect to the optimization variable $\bm{v}$). The function $\bm{v}^T\bm{A}\bm{v}$ is only concave if $\bm{A}$ is negative semidefinite, which means $\bm{v}^T\bm{A}\bm{v}\leq0$ for all $\bm{v}\in\mathbb{R}^n$, and which we denote as $\bm{A}\preccurlyeq\bm{0}$ (analogously, $\bm{A}$ is positive semidefinite if $\bm{v}^T\bm{A}\bm{v}\geq0$ for all $\bm{v}\in\mathbb{R}^n$, and is denoted $\bm{A}\succcurlyeq\bm{0}$). The eigenvalues of a negative semidefinite matrix are all non-positive (and those of a positive semidefinite matrix are all non-negative).

Luckily, despite its non-convexity, $\eqref{2}$ enjoys strong duality. This means that its (convex) dual problem has the same optimal value. To see this, let’s derive the dual problem. The Lagrangian of $\eqref{2}$ is

where $\bm{1}_n$ is the $n\times n$ identity matrix and $\lambda\in\mathbb{R}$ is the dual variable. The dual function is

where the $\sup$ is the supremum, or least upper bound (that is, it is the smallest value that is greater than every value of $\mathcal{L}(\bm{v},\lambda)$). When $\bm{A}-\lambda\bm{1}_n$ is negative semidefinite, the value $\bm{v}^T(\bm{A}-\lambda\bm{1}_n)\bm{v}$ is at most zero for any $\bm{v}\in\mathbb{R}^n$, and so $\mathcal{L}(\bm{v},\lambda)$ is at most $\lambda$; in all other cases it is unbounded above. Since we don’t care about the unbounded case, we restrict the domain to keep the dual function bounded. This yields the dual problem

which is a semidefinite program (SDP), because it includes a semidefinite constraint (that is, a linear matrix inequality). The dual problem is always convex, and SDPs in particular are a class of convex problems that can be solved efficiently with off-the-shelf software, as long as the number of optimization variables is not too large.

Diagonalization Interpretation

One way to interpret $\eqref{3}$ is to consider the eigendecomposition $\bm{A}=\bm{V}\bm{\Lambda}\bm{V}^T$, where $\bm{V}$ is the orthonormal matrix of eigenvectors and $\bm{\Lambda}$ is the diagonal matrix of eigenvalues, which we can use to rearrange the constraint $\bm{A}\preccurlyeq\lambda\bm{1}_n$ to the diagonalized form $\bm{\Lambda}\preccurlyeq\lambda\bm{1}_n$. Since both $\bm{\Lambda}$ and $\lambda\bm{1}_n$ are diagonal, the constraint simply requires $\lambda$ to be no smaller than the largest diagonal element (eigenvalue) of $\bm{\Lambda}$. Thus the minimum possible value of $\lambda$ is the maximum eigenvalue of $\bm{A}$, revealing that $\eqref{2}$ and $\eqref{3}$ have the same optimal values.

Extremal Interpretation

Another (related) way to interpret $\eqref{3}$ is as follows. We can rewrite the semidefinite constraint $\bm{A}\preccurlyeq\lambda\bm{1}_n$ as

Without loss of generality, take $\|\bm{v}\|_2=1$. Then the constraint becomes

which is equivalent to

Obviously, the minimum value of $\lambda$ is obtained at equality:

which is the same problem as $\eqref{2}$.

Geometric Interpretation

Finally, in the special case when $\bm{A}$ is positive definite (that is, all of its eigenvalues are strictly positive), it can be interpreted as defining an ellipsoid $\mathcal{E}$ in $n$-dimensional space centered at the origin:

where the eigenvectors of $\bm{A}$ define the direction of the principal semi-axes of $\mathcal{E}$ and the eigenvalues of $\bm{A}$ are the squares of the lengths of the semi-axes. The problem $\eqref{3}$ can then be interpreted as finding the (square of the) radius of the minimum-volume bounding sphere of $\mathcal{E}$. A two-dimensional example is shown in Figure 2.

Figure 2: The ellipse $\mathcal{E}$ can be defined by a $2\times2$ matrix $\bm{A}$ with eigenvalues $\lambda_1\geq\lambda_2>0$. The radius of the minimum-volume bounding sphere of $\mathcal{E}$ is $\sqrt{\lambda_1}$.

If you know of a good geometric interpretation of $\eqref{3}$ when $\bm{A}$ is a general real symmetric matrix with no definiteness assumptions, I’d like to hear about it!

General Semidefinite Constraints

Any semidefinite constraint is really just an eigenvalue constraint. For example, the constraint $\bm{Y}(\bm{x})\preccurlyeq\bm{0}$, where $\bm{x}$ is our optimization variable, just constrains the maximum eigenvalue of $\bm{Y}(\bm{x})$ to be at most zero. As long as $\bm{Y}$ is affine in $\bm{x}$, then the constraint $\bm{Y}(\bm{x})\preccurlyeq\bm{0}$ is convex. We can easily write the constraint from $\eqref{3}$ in this form by defining $\bm{Y}(\lambda)\coloneqq\bm{A}-\lambda\bm{1}_n$, which is affine in $\lambda$.

Trust Region Subproblem

In general, the problem of optimizing a quadratic objective function subject to a single quadratic constraint (equality or inequality) enjoys strong duality regardless of whether the objective and constraint are convex (see this paper as well as these slides for more information). This class of problem is often known as the trust region subproblem, because it arises as a step in trust region optimization methods. This result is the consequence of a theorem of alternatives known as the S-lemma.

Dual of the Dual

We can also take the dual of $\eqref{3}$. The Lagrangian of $\eqref{3}$ is

where $\mathrm{tr}(\cdot)$ denotes the matrix trace and $\bm{Z}\succcurlyeq\bm{0}$ is the dual variable. The dual function is

where $\inf$ is the infinimum, or greatest lower bound, so the dual problem is

which is also an SDP.

Relaxation Interpretation

It turns out that $\eqref{4}$ is just a semidefinite relaxation of $\eqref{2}$. To see this, observe that we can express $\eqref{2}$ in the equivalent form

which we obtained using the cyclic property of the trace. We can replace $\bm{v}\bm{v}^T$ with a matrix variable $\bm{Z}\succcurlyeq\bm{0}$ constrained to be rank one (because then $\bm{Z}$ can always be decomposed into the outer product $\bm{Z}=\bm{v}\bm{v}^T$), yielding

Finally, relaxing the problem by dropping the non-convex rank constraint gives us $\eqref{4}$. We say that this relaxation is tight because it retains the same optimal value as the original problem $\eqref{2}$.

Sum of the $k$ Largest Eigenvalues

Not only is the maximum eigenvalue of a real symmetric matrix a convex function, but so is the sum of the $k$ largest eigenvalues. If we arrange the eigenvalues of $\bm{A}$ in decreasing order $\lambda_1\geq\lambda_2\geq\dots\geq\lambda_n$, the sum of the $k$ largest is

where we have constrained each eigenvector to be orthonormal. We can also rewrite this problem in the nicer matrix form

where the columns of $\bm{V}\in\mathbb{R}^{n\times k}$ are the eigenvectors. This function is again the pointwise maximum over a family of convex functions, so it is itself convex.

Thanks to Connor Holmes for reviewing a draft of this post.

This post is aimed at those who have seen the inertial parameters before (for example, in the Newton-Euler equations), but who do not necessarily know all of their properties offhand. In particular, we focus on the set of inertial parameters that are realizable by a real physical object—that is, those that correspond to a valid (i.e., non-negative) mass density function. For a paper on this topic, I quite enjoy this one by Wensing, Kim, and Slotine, which informed a lot of my own understanding of rigid body inertial parameters.

Density

The density of a rigid body is described by a non-negative mass density function $\rho:\mathbb{R}^3\to\mathbb{R}_+$, where $\mathbb{R}_+$ denotes the set of non-negative real numbers. The density function can be thought of as an unnormalized probability distribution over three-dimensional space (we’ll come back to this analogy shortly), which assigns an infinitesimal mass value to each point in the body’s volume.

The inertial parameters are related to the density function by the integrals

where we are integrating over the position $\bm{r}\in\R^3$, and

forms a skew-symmetric matrix. The right-hand side of $\eqref{4}$ is also sometimes called the cross-product matrix because $\bm{a}^{\times}\bm{b}=\bm{a}\times\bm{b}$ for any $\bm{a},\bm{b}\in\mathbb{R}^3$. Since $\bm{r}^{\times}$ is skew-symmetric, it satisfies $(\bm{r}^{\times})^T=-\bm{r}^{\times}$ by definition, so it is also common to see $\eqref{3}$ written as

The inertia matrix is always taken with respect to a particular reference point; in $\eqref{3}$ we simply used the origin. Expressed about a general reference point $\bm{p}\in\mathbb{R}^3$, the inertia matrix is

It is often convenient to use the center of mass $\bm{c}$ as the reference point, in which case we have

where $\Delta\bm{r}=\bm{r}-\bm{c}$.

Probability Distribution Analogy

The quantity $m\bm{c}$ in $\eqref{2}$ is known as the first moment of mass. This is just the mean of the density function, so we can write

where $\mathbb{E}[\cdot]$ denotes the expected value under the distribution $\rho$. If we take $m=1$ such that $\rho$ is a proper (normalized) probability distribution, then $\mathbb{E}[\bm{r}]=\bm{c}$ and the covariance matrix is

which encodes the spread of the mass distribution about the center of mass.

More generally, we can define the quantity

which is known as the second moment matrix and does not require a normalized distribution. The second moment matrix has a one-to-one relationship with the inertia matrix, and is also taken about a particular reference point. In $\eqref{8}$, the reference point is the origin. If we use the center of mass as the reference point instead, we get

which corresponds to the covariance matrix from $\eqref{7}$.

Physical Consistency

Since the density $\rho$ is non-negative everywhere (as you cannot have a negative mass), it immediately follows from $\eqref{1}$ and $\eqref{6}$, respectively, that $m\geq0$ and $\bm{I}_c\succcurlyeq\bm{0}$; that is, the mass must be positive and the inertia matrix taken about the center of mass must be positive semidefinite (which means all of its eigenvalues are non-negative; we will see below that $\bm{I}_c\succcurlyeq\bm{0}$ implies $\bm{I}_p\succcurlyeq\bm{0}$ for any $\bm{p}\in\mathbb{R}^3$).

However, if $m=0$, then $\rho$ must be zero everywhere and therefore $\bm{I}_c=\bm{0}$ as well. To avoid this special case we will require that the mass is strictly positive, so our two conditions are

Some authors prefer to also restrict $\bm{I}_c$ to be strictly positive definite, which serves to exclude idealized zero-volume bodies like point masses, lines, or planes. We do not make this restriction.

Despite the fact that the two conditions in $\eqref{9}$ are together known in the literature as physical consistency, confusingly they are actually not quite sufficient to ensure that the inertial parameters correspond a non-negative mass density—we need one more condition on the inertia matrix.

Triangle Inequality

A valid inertia matrix (expressed about any reference point; here we use the origin for simplicity) must also satisfy the triangle inequality, which states that none of its eigenvalues is larger than the sum of the other two. To see this, we will make use of the identity

where $\bm{1}_3$ is the $3\times3$ identity matrix and $\mathrm{tr}(\cdot)$ denotes the matrix trace. Substituting into $\eqref{3}$, we get

Taking the trace of both sides of $\eqref{10}$, we get

Now let $\lambda_1\geq\lambda_2\geq\lambda_3\geq0$ be the eigenvalues of $\bm{I}$ and let $\bm{v}\in\mathbb{R}^3$ be the normalized (i.e., unit-length) eigenvector corresponding to $\lambda_1$, such that $\lambda_1=\bm{v}^T\bm{I}\bm{v}$. Substituting in $\eqref{10}$, we get

where the inequality follows because the second moment matrix is always positive semidefinite (this is easy to see from its definition $\eqref{8}$) and therefore $\bm{v}^T\bm{S}\bm{v}\geq0$ for any $\bm{v}$.

Finally, recalling that the trace of a matrix equals the sum of its eigenvalues, we can combine $\eqref{11}$ and $\eqref{12}$ to obtain

which we rearrange to obtain the triangle inequality:

Full Physical Consistency

A set of physically consistent inertial parameters where $\bm{I}$ also satisfies the triangle inequality are called fully physically consistent, which is a necessary and sufficient condition for the inertial parameters to be realizable by some non-negative mass density $\rho$.

Pseudo-Inertia Matrix

We can gather the inertial parameters into the $4\times4$ pseudo-inertia matrix

where $\tilde{\bm{r}} = [\bm{r}^T,1]^T$ is the homogeneous representation of the point $\bm{r}$.

It turns out that necessary and sufficient conditions for a set of inertial parameters to be fully physically consistent are

These conditions are convenient because they are convex, and can therefore be included as constraints in convex optimization problems for parameter identification or robust constraint verification. (For the purposes of numerical optimization, we can relax the constraint $m>0$ to $m\geq\epsilon$ for some small $\epsilon\geq0$, since strict inequalities don’t make sense in this case.)

Why do $m>0$ and $\bm{\Pi}\succcurlyeq\bm{0}$ imply full physical consistency? The Schur complement theorem tells us that if $m>0$, then

where $\bm{S}_c=\bm{S}-m\bm{c}\bm{c}^T$ (see $\eqref{14}$ below). We already saw that a positive semidefinite second moment matrix yields a fully physically consistent inertia matrix in the previous section on the triangle inequality. Here we need to prove the other direction; that is, if $\bm{S}$ is not positive semidefinite, then $\bm{I}$ must not be fully physically consistent (again, we will use the origin as the reference point here, but the same logic holds for any reference point, including the center of mass).

Let us assume that $\bm{I}\succcurlyeq\bm{0}$ but $\bm{S}\not\succcurlyeq\bm{0}$, which means that there exists some unit-length vector $\bm{v}\in\mathbb{R}^3$ such that $\bm{v}^T\bm{S}\bm{v}<0$. Using $\eqref{10}$, we have

We also know that $\bm{v}^T\bm{I}\bm{v}\leq\lambda_1$, and therefore $\lambda_1>\mathrm{tr}(\bm{S})$. Combined with $\eqref{11}$, we have

Rearranging the above equation reveals that $\lambda_1>\lambda_2+\lambda_3$, showing that the triangle inequality is not satisfied and therefore $\bm{I}$ is not fully physically consistent.

Changing the Reference Frame

Parallel Axis Theorem

We can manipulate the equation for the inertia matrix expressed about an arbitrary point $\bm{p}\in\mathbb{R}^3$ from $\eqref{5}$ to obtain

where $\Delta\bm{p}=\bm{p}-\bm{c}$ and we have used the fact that $\int_{\mathbb{R}^3}\rho(\bm{r})\Delta\bm{r}^{\times}d\bm{r}=\bm{0}$. This result is known as the parallel axis theorem, and is used to translate the inertia matrix to and from the center of mass.

Notice that $\eqref{13}$ implies that $\bm{I}_p\succcurlyeq\bm{I}_c$ for any reference point $\bm{p}\in\mathbb{R}^3$, with equality if and only if $\bm{p}=\bm{c}$. This means that it is easier (i.e., less energy is required) to rotate a rigid body about its center of mass than any other point.

To translate between two arbitrary points $\bm{p}$ and $\bm{q}$, we have

where $\Delta\bm{q}=\bm{q}-\bm{c}$. The analogous rule for the second moment matrix is

Full Spatial Transformations

The parallel axis theorem handles translations, but suppose we want to transform the inertial parameters by a general spatial transformation (i.e., translation and rotation) from frame $\{a\}$ to frame $\{b\}$. Let

be the homogeneous transformation matrix that maps points from $\{a\}$ to $\{b\}$, where $\bm{R}_{ba}\in SO(3)$ is the rotation and $\bm{p}^{ab}_b\in\mathbb{R}^3$ is the position of the origin of $\{a\}$ with respect to $\{b\}$ expressed in the coordinates of $\{b\}$. To map the inertial parameters from $\{a\}$ to $\{b\}$, we simply represent them as the pseudo-inertia matrix $\bm{\Pi}_a$ in $\{a\}$ and apply the “sandwich” rule

to obtain their representation $\bm{\Pi}_b$ in $\{b\}$.

We can use $\eqref{15}$ to obtain the parallel-axis theorem rule for $\bm{S}$ in $\eqref{14}$ by applying the pure translation

to the pseudo-inertia matrix expressed about the center of mass

which yields

where $-\Delta\bm{p}$ is the location of the center of mass with respect to $\bm{p}$.

More on the Inertia Matrix

We will conclude with a few more interesting properties of the inertia and second moment matrices.

$\bm{S}$ vs. $\bm{I}$

The second moment matrix $\bm{S}$ encodes the spread of the mass distribution while $\bm{I}$ encodes its resistance to rotation. To help understand this, consider the simple example shown below (borrowed from Chapter 2 of my PhD thesis).

Two point masses distributed along the $x$-axis. The $z$-axis points out of the page.

This system consists of two point masses, each with mass $0.5$, placed at $\pm1$ unit distance from the origin along the $x$-axis. The inertia and second moment matrices for this system

where $\bm{S}$ shows that the mass is spread along the $x$-axis and $\bm{I}$ shows that this spread resists rotation about the $y$- and $z$-axes.

From $\bm{I}$ to $\bm{S}$

Given $\bm{I}$, we can recover $\bm{S}$ by rearranging $\eqref{10}$ and substituting in $\eqref{11}$ to obtain

What happens when $\bm{I}$ does not satisfy the triangle inequality? Let

which is positive semidefinite but has eigenvalues $\lambda_1=1$ and $\lambda_2=\lambda_3=0$, so the triangle inequality is not satisfied. Using the above equation, the corresponding second moment matrix is

which is clearly not positive semidefinite and is therefore invalid.

Bigger $\bm{S}$, Bigger $\bm{I}$

Another interesting (and intuitive) property is that if $\bm{S}'\succcurlyeq\bm{S}$, then $\bm{I}'\succcurlyeq\bm{I}$. This means that when the mass distribution is more spread out, then the body’s resistance to rotation is increased. To prove this fact, consider the relationship

where $\Delta\bm{S}=\bm{S}'-\bm{S}\succcurlyeq\bm{0}$. Given any vector $\bm{u}\in\R^3$, we have

where $\lambda_{\max}(\Delta\bm{S})\geq0$ is the largest eigenvalue of $\Delta\bm{S}$, which shows that $\bm{I}'\succcurlyeq\bm{I}$.

Common Inertia Matrices

Expressions for the inertia matrix of common shapes with uniform density are available on Wikipedia. We have also derived the inertia matrix for ellipsoidal and cuboid shells in previous blog posts.

Thanks to Philippe Nadeau for reading a draft of this post.

from an optimization perspective. It is not too interesting when the system has a single unique solution; instead, we spent most of the post exploring the other two possibilities:

1. the system is inconsistent and has no solutions;
2. the system is under-determined and has infinite solutions.

In the first case, we chose the value $\bm{x}^\star$ that minimizes the least squares error $\|\bm{A}\bm{x}-\bm{b}\|^2_2$. In the second case, we choose the solution $\bm{x}^\star$ satisfying $\eqref{1}$ that is closest to some other desired point $\bm{y}$. Here we revisit both of these cases from a geometric perspective, which will reveal that both problems can be interpreted as projections onto a linear subspace. We also examine the addition of damping in both cases.

An interactive Python notebook that accompanies this post can be accessed directly in the browser here.

No Solutions

Let’s begin with a simple example. Consider a line in $\mathbb{R}^m$ that passes through the origin, defined as the set of points

where $\bm{a}\in\mathbb{R}^m$ is a non-zero vector representing the line’s direction. In other words, $\mathcal{L}$ is the linear subspace spanned by $\bm{a}$. Since a line is a one-dimensional subspace, we need only one variable $x\in\mathbb{R}$ to parameterize it.

Suppose we want to find the point $\bm{a}x^\star$ on the line that is closest to some other point $\bm{b}\in\mathbb{R}^m$ in terms of Euclidean distance, as shown below in Figure 1. This problem is called the vector projection of $\bm{b}$ onto $\bm{a}$.

Figure 1: A line (in red) passing through the origin $\mathcal{O}$ in the direction of the vector $\bm{a}$. We want to find the closest point on the line to an arbitrary point $\bm{b}$, which known as the vector projection of $\bm{b}$ onto $\bm{a}$.

Assuming that $\bm{b}$ is not actually on the line, the system $\bm{a}x=\bm{b}$ is inconsistent and has no solution. Instead, we want to find the point that solves the least-squares problem

where $\bm{e}(x)=\bm{a}x-\bm{b}$ is the error vector. However, we will turn to geometry rather than optimization theory to solve it.

Figure 2: The vector representing the closest point $\bm{a}x^\star$ (in blue) on the line to $\bm{b}$ is just the projection of $\bm{b}$ onto the line. This is equivalent to the condition $\bm{a}^T\bm{e}(x^\star)=0$; that is, the difference between $\bm{a}x^\star$ and $\bm{b}$ is orthogonal to the line.

By looking at Figure 2, we see that $\bm{e}(x)$ has minimum length when it is orthogonal to the line, which means

where $\bm{a}x^\star$ is the closest point. Rearranging $\eqref{3}$, we obtain the solution

This looks very similar to the solution to the least-squares problem we saw last time. Indeed, if we generalize $\eqref{2}$ from a one-dimensional subspace to an $n$-dimensional subspace, we get the column space

of the matrix $\bm{A}=[\bm{a}_1,\dots,\bm{a}_n]\in\mathbb{R}^{m\times n}$, where $\bm{x}=[x_1,\dots,x_n]^T$ such that $\bm{A}\bm{x}=\sum_{i=1}^n\bm{a}_ix_i$. The least-squares problem becomes

with $\bm{e}(\bm{x})=\bm{A}\bm{x}-\bm{b}$, the equation $\eqref{3}$ becomes $\bm{A}^T\bm{e}(\bm{x}^\star)=\bm{0}$, and the solution $\eqref{4}$ becomes

where $\bm{A}^+:=(\bm{A}^T\bm{A})^{-1}\bm{A}^T$ is the (left) Moore-Penrose pseudoinverse that we saw last time. The corresponding closest point in the subspace $\mathcal{C}(\bm{A})$ is

Projection

The matrix $\bm{A}\bm{A}^+$ is the projection matrix that maps arbitrary points to the closest point in $\mathcal{C}(\bm{A})$. A projection matrix on $\mathbb{R}^n$ is a matrix $\bm{U}\in\mathbb{R}^{n\times n}$ that satisfies $\bm{U}\bm{U}=\bm{U}$ (that is, it is idempotent), which means that multiplying a vector by $\bm{U}$ more than once has no additional effect. We can easily verify that $\bm{A}\bm{A}^+$ satisfies this property:

Since $\bm{A}\bm{A}^+$ is also symmetric (this is easy to check and left to the reader), it is an orthogonal projection matrix. Indeed, all of the projections we describe in this post are orthogonal.

This is the geometric interpretation of the least-squares problem $\eqref{6}$: it is just an (orthogonal) projection of $\bm{b}$ onto the column space $\mathcal{C}(\bm{A})$.

Damping

Recall that we introduced damping into the least-squares problem to bias the solution toward a smaller norm and avoid problems with ill-conditioning. Returning to our example with the line $\mathcal{L}$ defined in $\eqref{2}$, the damped least squares problem is

where $\alpha\geq0$ is the damping factor.

How do we think about this problem geometrically? Consider again the projected error $\eqref{3}$: this equation says that the projections of $\bm{b}$ and its closest point $\bm{a}x^\star$ onto $\bm{a}$ must be equal. In the damped case, we are willing to sacrifice this equality to make the magnitude of $x^\star$ smaller, as shown in Figure 3 below.

Figure 3: Damping pulls $x^\star$ closer to the origin by padding the value $\bm{a}^T\bm{a}x^\star$ with the term $\alpha x^\star$, which added together must equal $\bm{a}^T\bm{b}$.

Instead of $\bm{a}^T\bm{b}=\bm{a}^T\bm{a}x^\star$, we add an extra term $\alpha x^\star$ to the right hand side so that $x^\star$ becomes smaller, yielding

This solution generalizes to

in the matrix case, where $\bm{A}^+(\alpha):=(\bm{A}^T\bm{A}+\alpha\bm{1}_n)^{-1}\bm{A}^T$ is the damped Moore-Penrose pseudoinverse we saw last time. The matrix $\bm{A}\bm{A}^+(\alpha)$ is not a projection matrix when $\alpha>0$, because repeated applications to a vector continue to shrink that vector’s magnitude.

Infinite Solutions

Now let’s look at the case when there are infinite solutions to $\eqref{1}$, and we need to pick one.

In $\eqref{2}$ we defined a line in $\mathbb{R}^m$ with a single variable $x$ representing its single dimension. In $\eqref{5}$, we generalized to an $n$-dimensional subspace parameterized by $n<m$ variables. Instead, we could parameterize it with a full $m$ variables subject to $p=m-n$ constraints. Using this approach, we can define a linear subspace of $\mathbb{R}^m$ as

where each row of $\bm{A}\in\mathbb{R}^{p\times m}$ defines one of the $p$ constraints. Each row of $\bm{A}$ is orthogonal to the subspace itself, and we assume the rows of $\bm{A}$ are all linearly independent. This means that the columns of $\bm{A}^T$ span the $p$-dimensional subspace of $\mathbb{R}^m$ that is orthogonal to $\mathcal{S}$. Indeed, $\mathcal{S}$ is just the nullspace $\mathcal{N}(\bm{A})$, and the span of $\bm{A}^T$ is $\mathcal{C}(\bm{A}^T)$, which is called the range space of $\bm{A}$; in general, the range space and nullspace are orthogonal to each other. (For more information, refer to this unit on the four fundamental subspaces of linear algebra.)

Let’s generalize $\eqref{7}$ so that it does not necessarily pass through the origin by defining the affine subspace

which now satisfies the constraint $\eqref{1}$. We can interpret $\mathcal{S}'$ as the set of points in $\mathcal{S}$ offset by some vector $\bm{x}_0\in\mathbb{R}^m$, such that $\mathcal{S}'=\{\bm{x}+\bm{x}_0\mid\bm{x}\in\mathcal{S}\}$, where

Figure 4 below shows what $\mathcal{S}'$ looks like when it is just a line in $\mathbb{R}^2$, which requires a single constraint defined by $\bm{A}=[\bm{a}^T]\in\mathbb{R}^{1\times2}$ and $b\in\mathbb{R}$.

Figure 4: The line $\mathcal{S}'$ (in red) is defined by the constraint $\bm{a}^T\bm{x}=b$, where $b=\bm{a}^T\bm{x}_0$.

Suppose we want to find the point $\bm{x}^\star\in\mathcal{S}'$ that has the smallest Euclidean norm; that is, we want to solve the problem

which is equivalent to finding the point that is closest to the origin.

As can be seen in Figure 5, the optimal solution $\bm{x}^\star$ of $\eqref{9}$ is orthogonal to $\mathcal{S}'$, which means that it lives in $\mathcal{C}(\bm{A}^T)$; that is, there must exist a vector $\bm{\lambda}\in\mathbb{R}^m$ such that $\bm{x}^{\star}=\bm{A}^T\bm{\lambda}.$ We saw last time that $\bm{\lambda}$ can be interpreted as the vector of Lagrange multipliers for the constraint $\eqref{1}$; the optimal value of $\bm{\lambda}$ ensures that $\bm{x}^\star$ does indeed satisfy $\eqref{1}$.