Is the matrix of Lagrange multipliers associated to a symmetric matrix itself symmetric? Given the optimisation problem
\begin{equation}
\min_{V} f(V) \quad \text{s.t.} \quad C(V) = 0
\end{equation}
where $V \in \mathbb{R}^{p \times k}$ and $C(V) \in \mathbb{R}^{k \times k}$. The Lagrangian is
\begin{equation}
L(V,\Lambda) = f(V) - \langle C(V) , \Lambda \rangle
\end{equation}
where $\Lambda \in \mathbb{R}^{k \times k}$ is the associated matrix of Lagrange multipliers.
If $C(V)$ is symmetric, does this imply that $\Lambda$ is also symmetric ?
 A: The constraint $C(V)=0$ is really just a set of $n^{2}$ constraints
$C_{i,j}(V)=0$, $i=1, 2, \ldots, k$, $j=1, 2, \ldots, k$,
with lots of redundancy, since $C_{i,j}(V)=C_{j,i}(V)$.
The trace inner product of $\Lambda$ and $C(V)$ in the Lagrangian can be written as
$\langle C(V),\Lambda \rangle=\mbox{tr}(C(V)^{T}\Lambda)=\sum_{i=1}^{k} \sum_{j=1}^{k} \Lambda_{i,j}C_{i,j}(V)$
Since $C_{i,j}=C_{j,i}$, this simplifies to
$\langle C(V), \Lambda \rangle=\sum_{i=1}^{k} \Lambda_{i,i}C_{i,i}(V)+\sum_{i=1}^{k} \sum_{j=i+1}^{k} (\Lambda_{i,j}+\Lambda_{j,i}) C_{i,j}(V)$
The actual Lagrange multiplier for the $i,j$th constraint is now $\Lambda_{i,j}+\Lambda_{j,i}$ whenever $i \neq j$ and $\Lambda_{i,j}$ when $i=j$.
It should be clear that you can require $\Lambda$ to be symmetric if you want to, but that you could also let $\Lambda$ be nonsymmetric.  In the first case, $\Lambda_{i,j}=\Lambda_{j,i}$.  In the second case, you've introduced some non-uniqueness in that any pair of $\Lambda_{i,j}$ and $\Lambda_{j,i}$ with the same sum are equivalent.
