Confusion when using the product rule (differentiation) I'm quite confused by something related to the product rule that should be easy.
Let $\phi:\mathbb{R}^p \to \mathbb{R}$ a diferentiable function. Define
$$\varphi:\mathbb{R}^p \to \mathbb{R}, \quad \varphi(t) = e^{\phi(t)}, \quad t \in \mathbb{R}^p$$
The derivative of $\varphi$, using the chain rule, is given by:
$$\varphi'(t)=e^{\phi(t)} \phi'(t) = \varphi(t)\phi'(t)  \in \mathbb{R}^p$$
I am trying to use the product rule to find $\varphi''(t)$:
$$\varphi''(t)= \varphi'(t)\phi'(t) +\varphi(t)\phi''(t)$$
But note that the respective dimensions are not compatible:
$$\underbrace{\varphi''(t)}_{p\times p}= \underbrace{\varphi'(t)}_{p\times 1} \, \,\underbrace{\phi'(t)}_{p\times 1} +\underbrace{\varphi(t)}_{1\times 1} \, \,\underbrace{\phi''(t)}_{p\times p}$$
I'm committing some bullshit because it doesn't make sense.
help!
 A: I usually use the following rule: functions to be derivated have to be in column  vectors, i.e. $p\times 1$ in your case. Note that :
$$\varphi'(t) = \varphi(t) \phi'(t)\,\,\ \hbox{is }\, 1\times p.$$
So first you have to transpose...
$$\varphi'(t)^T =  \phi'(t)^T \varphi(t)\,\,\ \hbox{is }\, p\times 1 $$
and derive (remark that secod derivatives are symmetrics):
\begin{align}
\varphi''(t)&=\phi''(t) \varphi(t) + \phi'(t)^T \varphi'(t) \\
&=\varphi(t)\left[ \phi''(t)  + \phi'(t)^T  \phi'(t)\right].
\end{align}
Now the dimensions are ok!
A: If $\phi$ is a function mapping $p$ numbers to $1$ number, then the first derivative of $\phi$ has $p$ components, and the second derivative has $p^2$ components. This is true in general. But it is a matter of convention how to (and if) to write these components in matrix/vector form.

*

*Do you want to use a column- or a row- vector for the derivative? If you decide to "always use column vectors", the second derivative will we a "vector of vectors" which is impractical to write down.

*For the second derivative, the convention is to write one derivative as column, and the other as a row. This results in a nice $p\times p$ matrix called the "Hessian Matrix". (Thanks to Schwartz' theorem, the order of the two derivatives usually does not matter).

*But this inconsistency (sometimes using columns, sometimes rows) means you have to be careful about the chain rule and such. Just writing a formula like
\begin{align}
\varphi''(t)= \varphi'(t)\phi'(t) +\varphi(t)\phi''(t)
\end{align}
is kinda meaningless without specifying which of all these derivatives should be written as rows, and which as columns. Though in general formulas like these work out fine if you replace the implicit multiplication by either matrix multiplication, or inner product, or outer product. There is usually only one choice possible to make all the dimensions work out. For this reason, people tend to be somewhat sloppy in the notation.

Finally, you can of course avoid all of this confusion if you work with indices explicitly. This is somewhat more tedious, but can be helpful to clarify notations.
