# Proof of Quadratic Approximation to $f(\mathbf{\vec{x}})$

Let $$\mathbf{\vec{x}} = \left[\begin{array}{c} x_1 \\ x_2 \\ x_3 \\ . \\ . \\ . \\ x_n \end{array}\right]$$ and $$f(\mathbf{\vec{x}}): \mathbb{R}^n \to \mathbb{R}$$, $$n \in \mathbb{N}$$ be a scalar-valued function, then its Quadratic approximation $$Q(\mathbf{\vec{x}})$$ about a point $$\mathbf{\vec{x}}_0$$ is given by:

$$Q(\mathbf{\vec{x}}) = f(\mathbf{\vec{x}}_0) + \mathbf{\nabla}f(\mathbf{\vec{x}}_0)^T (\mathbf{\vec{x} - \vec{x}}_0) + \dfrac{1}{2} (\mathbf{\vec{x} - \vec{x}}_0)^T \mathbf{H}_f (\mathbf{\vec{x}}_0) (\mathbf{\vec{x} - \vec{x}}_0)$$

where $$\mathbf{\nabla}f$$ is the gradient and $$\mathbf{H}_f$$ the Hessian matrix of $$f$$.

We know that the function $$Q(\mathbf{\vec{x}})$$ is suppose to have the same output, first and second partial derivatives as $$f(\mathbf{\vec{x}})$$ at the point $$\mathbf{\vec{x}}_0$$.

Clearly, $$Q(\mathbf{\vec{x}}_0) = f(\mathbf{\vec{x}}_0)$$. Now, how do I prove the following?:

$$\mathbf{\nabla}Q(\mathbf{\vec{x}}_0) = \mathbf{\nabla}f(\mathbf{\vec{x}}_0)$$

$$\mathbf{H}_Q(\mathbf{\vec{x}}_0) = \mathbf{H}_f(\mathbf{\vec{x}}_0)$$

• You might start by trying to differentiate $Q$. That shouldn’t be too difficult since most of the quantities in its definition are constants.
– amd
Mar 25, 2018 at 0:00
• But, how I am supposed to do $\dfrac{\partial ((\mathbf{\vec{x} − \vec{x}}_0)^T H_f(\mathbf{\vec{x}}_0) (\mathbf{\vec{x} − \vec{x}}_0))}{\partial x_1}$? Mar 25, 2018 at 13:39
• That’s just the dot product of a pair of vectors. Try a combination of the product and chain rules.
– amd
Mar 25, 2018 at 21:04

You can verify that the gradients and Hessian agree by direct calculation. Observe that both $\nabla f[\vec{\mathbf x}_0]$ and ${\mathbf H}_f[\vec{\mathbf x}_0]$ are constants, so the second term of $Q$ is a linear function of $\vec{\mathbf x}-\vec{\mathbf x}_0$ and the third term a quadratic form in $\vec{\mathbf x}-\vec{\mathbf x}_0$. (I use square brackets to indicate evaluation at a point in an attempt at disambiguation in the following.)
Analogous to the single-variable differentiation rule $(cx)'=c$ we have $$\nabla(\vec{\mathbf v}^T\vec{\mathbf x}) = \nabla\left(\sum_i v_ix_i\right) = \vec{\mathbf v},$$ therefore $$\nabla(\nabla f[\vec{\mathbf x}_0]^T(\vec{\mathbf x}-\vec{\mathbf x}_0)) = \nabla(\nabla f[\vec{\mathbf x}_0]^T\vec{\mathbf x})-\nabla(\nabla f[\vec{\mathbf x}_0]^T\vec{\mathbf x}_0) = \nabla f[\vec{\mathbf x}_0].$$ More generally, the differential of a linear function $A\mathbf x$ of $\mathbf x$ is $A$—the best linear approximation to a linear map is the map itself.
For the quadratic term, we can use the product rule $\nabla(u^Tv)=u^T\nabla v+(\nabla u)^Tv$: \begin{align} \nabla\left(\frac12(\vec{\mathbf x}-\vec{\mathbf x}_0)^T{\mathbf H}_f[\vec{\mathbf x}_0](\vec{\mathbf x}-\vec{\mathbf x}_0)\right) &= \frac12 (\vec{\mathbf x}-\vec{\mathbf x}_0)^T \nabla({\mathbf H}_f[\vec{\mathbf x}_0](\vec{\mathbf x}-\vec{\mathbf x}_0)) + \frac12(\nabla(\vec{\mathbf x}-\vec{\mathbf x}_0))^T{\mathbf H}_f[\vec{\mathbf x}_0](\vec{\mathbf x}-\vec{\mathbf x}_0) \\ &= \frac12 (\vec{\mathbf x}-\vec{\mathbf x}_0)^T {\mathbf H}_f[\vec{\mathbf x}_0] + \frac12 {\mathbf H}_f[\vec{\mathbf x}_0](\vec{\mathbf x}-\vec{\mathbf x}_0) \\ &= {\mathbf H}_f[\vec{\mathbf x}_0](\vec{\mathbf x}-\vec{\mathbf x}_0) \end{align} since ${\mathbf H}_f$ is symmetric. This vanishes at $\vec{\mathbf x}_0$, therefore $\nabla Q[\vec{\mathbf x}_0] = \nabla f[\vec{\mathbf x}_0]$. (We could’ve gotten to this without actually differentiating by noting that the result must be a linear combination of $x_i-x_{0i}$ terms, all of which vanish.)
For ${\mathbf H}_Q[\vec{\mathbf x}_0]$ we can proceed in a similar fashion by differentiating $\nabla Q = \nabla f[\vec{\mathbf x}_0]+{\mathbf H}_f[\vec{\mathbf x}_0](\vec{\mathbf x}-\vec{\mathbf x}_0)$ directly, producing ${\mathbf H}_f[\vec{\mathbf x}_0]$ per the earlier observation regarding the differential of a linear function, or note that the Hessian of a constant or linear function vanishes, while the Hessian of a quadratic form is 2 times the (constant) matrix of the form (analogously to the single-variable $(ax^2)''=2a$), so that the only term that survives when computing the Hessian directly from $Q$ is ${\mathbf H}_f[\vec{\mathbf x}_0]$.