Motivation: A minimization problem

The discussion here follows closely the formulation in [1, Chapter 3, pp. 113-115, pp. 144-152]. Suppose we expect $\theta(t)$ to be a linear function of $t$, so that we expect to write the measurements $y(t)$ of $\theta$ in the form,

$$y(t) = \theta(t) + n(t) = a + bt + n(t) ,$$ (1)

where $n(t)$ is ``measurement noise.'' Given $M$ measurements at times $t_1, t_2,\dots,t_M$, we write $M$ equations

$$\mathbf{E}\mathbf{x} + \mathbf{n} = \mathbf{y},$$ (2)

where

$$\mathbf{E} = \left[ \begin{array}{cc} 1 & t_1 1 & t_2 \vdot... ...eft[ \begin{array}{c} n(t_1) n(t_2) \vdots n(t_M) \end{array} \right] .$$ (3)

The ``best'' solution is obtained by minimizing, wrt $x_i$ (here, $x_1 = a$ and $x_2 = b$), the quantity

$$J := \mathbf{n}^T\mathbf{n} = (\mathbf{E}\mathbf{x}-\mathbf{y})^T(\mathbf{E}\mathbf{x}-\mathbf{y}) .$$ (4)

Differentiating wrt $x_i$ leads to the normal equations,

$$\mathbf{E}^T \mathbf{E} \mathbf{x} = \mathbf{E}^T \mathbf{y} ,$$ (5)

The solution is not $\mathbf{x} = \mathbf{E}^{-1}\mathbf{y}$ - the inverse $\mathbf{E}$ or $\mathbf{E}^T$ is not even defined for $M \ne 2$. It could be

$$\tilde{\mathbf{x}} = (\mathbf{E}^T\mathbf{E})^{-1}\mathbf{E}^T \mathbf{y} .$$ (6)

if $(\mathbf{E}^T\mathbf{E})^{-1}$ can be found under certain assumptions. An alternate approach, to be described next, is to use the singular value decomposition of the matrix $\mathbf{E}$. This latter approach can be beneficial when $M\ne N$, when $(\mathbf{E}^T\mathbf{E})^{-1}$ does not exist, or when the eigenvectors of $M\times M$ matrix $\mathbf{E}$ are not orthonormal. It is also helpful not only because singular values, unlike eigenvalues, exist for every matrix but also because the singular vectors tell us something about the ``dynamics'' under $\mathbf{E}$. (The last sentence couldn't have been more vague, but I will leave it at that...) See [1] or email (yes, that is a verb too) Juan for an extended discussion.