Completing the square in matrix form
I am having trouble understanding how to complete the square in matrix form. I can't find any source online for a clear, final equation for that. However, I would also like to grasp the intuition behind it. Thanks.
To give an example:
$$ x'Mx-2b'x=(x−M^{−1}b)′M(x−M^{−1}b)−b′M^{−1}b $$
How do I get that solution?
$\endgroup$ 73 Answers
$\begingroup$Based on your example, it is implicit that $M$ is symmetric (because you have $M'^{-1}M=I$). Hence $M=SS'$ for some $S$. Let $u=x-M^{-1}b$ and you get the expression\begin{align} uSS'u'-b(S')^{-1}Sb'. \end{align}Let's say you set it equal to a certain complex number $z$. Then\begin{align} uSS'u'=b(S')^{-1}Sb'+z. \end{align} Note that $S$ is invertible and both sides are complex numbers. The left hand side is the norm-squared of a complex vector. You can choose any vector $v$ with norm-squared equal to the right hand side and let $u=S^{-1}v=x-M^{-1}b$. Hence $x=S^{-1}v+M^{-1}b$.
$\endgroup$ $\begingroup$I used a trick from here:
$$ x'Mx-2bx = x'Mx-b'x-b'x $$ we can add $MM^{-1}$ in the middle terms:$$ x'Mx-b'M^{-1}Mx-b'M^{-1}Mx $$
now let $$ M^{-1}b=u $$
so rewrite
$$ x'Mx-u'Mx-u'Mx = $$
which becomes a simple square. Notice that we need to substract the third term
$$ (x-u)M(x-u)-u'Mu = (x-M^{-1}b)M(x-M^{-1}b)-(M^{-1}b)'MM^{-1}b = $$
finally simplify the last bit$$ (x-M^{-1}b)M(x-M^{-1}b)-b'M^{-1}MM^{-1}b = (x-M^{-1}b)M (x-M^{-1}b)-b'M^{-1}b $$
So what I learnt from this is that the trick is to try to get the expression in the form of $ x'Mx-2b'Mx $
$\endgroup$ $\begingroup$If it helps, the general form of matrix algebra is analogous to algebra with scalar variables except multiplication of matrices don't commute, so we write quadratic terms as a vector transpose times its transpose, $x^T x$, instead of with a square like $x^2$, and matrix inverses are always as $M^{-1}$ instead of say $1/M$.
So the matrix algebra expression $x^T M x - 2 b^T x $ is like the real-valued polynomial $M x^2 - 2bx$.
How can we complete the square for $M x^2 - 2bx$? Well we can rewrite by factoring out the quadratic coefficient as $M(x^2 - 2 (b/M) x)$ and we see how to complete the square as$$M (x^2 - 2 (b/M) x + (b/M)^2 - (b/M)^2) = M((x - b/M)^2 - (b/M)^2) = (x - b/M)^2 M - b^2/M$$ Actually this is almost exactly our matrix expression $(x - b M^{-1})^T M (x - bM^{-1}) - b^T M^{-1} b $ except written out in matrix form so all the dimensions and non-commutativity works out. You can now do the same complete the square steps in matrix form and see the equations look very similar.
$\endgroup$
