How to find convex conjugate I want to derive the convex conjugate of
$$\varphi(\theta) = e^{x^\top \theta} - y x^\top \theta, ~y \in \mathbb R_+.$$
$$\varphi^*(z) = ?$$
So, I need to find $$\sup(z^\top \theta-e^{x^\top \theta} + y x^\top \theta)$$
I found gradient:
$$\nabla_\theta (\varphi_\theta)=z-e^{x^\top \theta}x + y x$$
So, I need to solve with respect $\theta$ $$\nabla_\theta (\varphi_\theta)=0$$
But I can't understand how to find the root of this equation. Please help me to find this optimal $\theta$.
 A: From scratch: According to the definition you have to find
$$\tag{1}
\varphi^*(\boldsymbol{z})=\sup_{\boldsymbol\theta}\big(\boldsymbol{z}^\top\boldsymbol\theta-\varphi(\boldsymbol\theta)\big)\,
$$
which is different from the supremum that you wrote. The gradient of the function in the supremum is
$$\tag{2}
\nabla_{\boldsymbol \theta}\big(\boldsymbol{z}^\top\boldsymbol\theta-\varphi(\boldsymbol\theta)\big)=\boldsymbol{z}-\nabla_{\boldsymbol \theta}\varphi(\boldsymbol\theta)=\boldsymbol{z}-\boldsymbol{x}\,e^{\boldsymbol{x}^\top\boldsymbol\theta}+y\,\boldsymbol{x}
=\boldsymbol{z}+\boldsymbol{x}\,\big(y-\,e^{\boldsymbol{x}^\top\boldsymbol\theta}\big)\,.
$$

*

*I do not think that when the dimension of the vectors $\boldsymbol{x,z,\theta}$ is larger than one that there exists always a closed form solution.


*Reason: The gradient in (2) is zero only when $\boldsymbol z$ and $\boldsymbol x$ are linearly dependent. If this is the case there are typically many solutions $\boldsymbol\theta$
that can be calculated from
$$\tag{3}
0=z_i+x_i\big(y-e^{\boldsymbol{x}^\top\boldsymbol\theta}\big)
$$
by picking any index $i$ for which $x_i$  is not zero. This $\boldsymbol\theta$ will automatically satisfy (3) for all other indices $i\,.$


*When $\boldsymbol z$ and $\boldsymbol x$ are linearly dependent then
$\boldsymbol{z^\top\theta}$ equals $\boldsymbol{x^\top\theta}$ up to a
factor. Further, for each solution $\boldsymbol\theta$ of (3) the scalar product
$\boldsymbol{x^\top\theta}$ is the same, therefore $\boldsymbol{z^\top\theta}$ is the same.
Writing the gradient (2) as
$$\tag{4}
\boldsymbol{z}-\boldsymbol{x}\big(\varphi(\boldsymbol\theta)+y\,\boldsymbol{x^\top\theta}\big)+y\,\boldsymbol{x}
$$
we get
$$
\boldsymbol{z^\top\theta}-\boldsymbol{x^\top\theta}\,\big(\varphi(\boldsymbol\theta)+y\,\boldsymbol{x^\top\theta}\big)+y\,\boldsymbol{x^\top\theta}\,.
$$
If $\boldsymbol\theta$ is a solution of (3) this is zero and therefore,
$$
\boldsymbol{z^\top\theta}-\varphi(\boldsymbol\theta)-\boldsymbol{x^\top\theta}\,\big(\varphi(\boldsymbol\theta)+y\,\boldsymbol{x^\top\theta}\big)+y\,\boldsymbol{x^\top\theta}=-\varphi(\boldsymbol\theta)\,.
$$
This leads to
$$
\varphi^*(\boldsymbol z)= 
\boldsymbol{x^\top\theta}\,\big(\varphi(\boldsymbol\theta)+y\,\boldsymbol{x^\top\theta}\big)+y\,\boldsymbol{x^\top\theta}-\varphi(\boldsymbol\theta)\,.
$$
Note that this is only a function of $\boldsymbol x^\top\theta$ and this
scalar product depends only on a single component of $\boldsymbol{z}$ through the way (3) was solved.
