Cost function simplifacitons

Question

Suppose I have a function $C(v_1,v_2)$. Which is a cost function.

I have that:

$$\Delta C \approx \frac{\partial C}{\partial v_1}\Delta v_1 + \frac{\partial C}{\partial v_2}\Delta v_2$$

Now $\triangledown C = \left[ \frac{\partial C}{\partial v_1} \ \ \ \frac{\partial C}{\partial v_2} \right]$ and define $\Delta v = [\Delta v_1 \ \ \ \Delta v_2]^{T}$

Then

$$\Delta C = \triangledown C \cdot \Delta v$$

Now if we choose $\Delta v = -\eta \triangledown C$, where $\eta > 0$ and is small.

Then we can rewrite the equation:

$$\Delta C = -\eta \ \triangledown C \triangledown C$$

Which, apparently, to this resource is:

$$\Delta C = -\eta || \triangledown C ||^2$$

I am not sure how that became a norm suddenly. And I cannot understand how this: [1x2]*[1x2] is a valid matrix/vector multiplication. That multiplication is undefined.

Answer 1

Maybe I'm misunderstanding, but if $\nabla C\in\mathbb{R}^{1\times 2}$ and $ \Delta v\in\mathbb{R}^{2\times 1} $, then the inner product is the matrix multiplication, i.e. $\nabla C\cdot \Delta v = \nabla C\Delta v $. Note that the differing shape means that $\Delta v = -\eta\nabla C^T$ is the proper assignment.

So then we get: \begin{align} \Delta C &= \nabla C\Delta v \\ &= -\eta\nabla C\nabla C^T \\ &= -\eta\sum_i [\nabla C]_i^2 \\ &= -\eta\,|| \nabla C ||_2^2 \end{align}

Side note: I think the convention is that the gradient (as a 1D Jacobian) is a row vector (also here), while all other "standard vectors" are column vectors.