# Cost function simplifacitons

Suppose I have a function $C(v_1,v_2)$. Which is a cost function.

I have that:

$$\Delta C \approx \frac{\partial C}{\partial v_1}\Delta v_1 + \frac{\partial C}{\partial v_2}\Delta v_2$$

Now $\triangledown C = \left[ \frac{\partial C}{\partial v_1} \ \ \ \frac{\partial C}{\partial v_2} \right]$ and define $\Delta v = [\Delta v_1 \ \ \ \Delta v_2]^{T}$

Then

$$\Delta C = \triangledown C \cdot \Delta v$$

Now if we choose $\Delta v = -\eta \triangledown C$, where $\eta > 0$ and is small.

Then we can rewrite the equation:

$$\Delta C = -\eta \ \triangledown C \triangledown C$$

Which, apparently, to this resource is:

$$\Delta C = -\eta || \triangledown C ||^2$$

I am not sure how that became a norm suddenly. And I cannot understand how this: [1x2]*[1x2] is a valid matrix/vector multiplication. That multiplication is undefined.

• I have dropped the transpose notation, because I think that the author has a typo.
– Naz
Nov 21, 2016 at 17:30
• I meant, that I have dropped the transpose notation in $\triangledown C$. This is because $\triangledown C \cdot \Delta v$ is an undefined matrix operation.
– Naz
Nov 21, 2016 at 17:45

Maybe I'm misunderstanding, but if $\nabla C\in\mathbb{R}^{1\times 2}$ and $\Delta v\in\mathbb{R}^{2\times 1}$, then the inner product is the matrix multiplication, i.e. $\nabla C\cdot \Delta v = \nabla C\Delta v$. Note that the differing shape means that $\Delta v = -\eta\nabla C^T$ is the proper assignment.