Mappings of Adjoint Linear Operators Let
$$
\begin{align*}
    \mathsf{A}:~ \mathbb{R}^n 
    & \rightarrow \mathbb{R}^m \\ 
    \boldsymbol{x}
    & \mapsto \boldsymbol{y} = \boldsymbol{A} \boldsymbol{x}.
\end{align*}
$$
The adjoint of the linear operator $\mathsf{A}$, denoted by $\mathsf{A}^*$, is defined via
$$
\begin{align*}
    \left\langle \mathsf{A}(\boldsymbol{x}),\boldsymbol{y} \right\rangle
    & = \left\langle \boldsymbol{x},\mathsf{A}^*(\boldsymbol{y}) \right\rangle
\end{align*}
$$
for all $\boldsymbol{x} \in \mathbb{R}^n$ and $\boldsymbol{y} \in \mathbb{R}^m$.
$\mathsf{A}^{*}(\boldsymbol{y}) = \boldsymbol{A}^{\mathsf{T}} \boldsymbol{y}$
Consider the linear map given by
$$
\begin{align*}
    & \boldsymbol{Y} = \mathsf{C}(\boldsymbol{X}) = \boldsymbol{A} \boldsymbol{X} \boldsymbol{B}
\end{align*}
$$

*

*What would the adjoint of $\mathsf{C}$ be interms of matrices $\boldsymbol{A}$ and $\boldsymbol{B}$.


*Let $\boldsymbol{Z} = \mathsf{C}^*( \mathsf{C} (\boldsymbol{X}) )$. How can I compute the closed form of Z interms A,B?

I have tried approaching this problem using the given definition for a vector and not a matrix but I am going in circles.
What would be a good start here?
 A: Take the matrices $A,B$ from the definition of $\mathsf C$ to be of sizes $p \times m, n \times q$ respectively. With that, $\mathsf C$ is a linear map $\mathsf C:\Bbb R^{m \times n} \to \Bbb R^{p \times q}$.
As you've obliquely noted, $\mathsf C$ takes matrices and produces matrices, whereas the map $\mathsf A$ from your definition of the adjoint takes vectors and produces vectors. If we are to apply your definition to $\mathsf C$, we'll need to extend your definition to this new context. To that end, we'll define the following inner product (known as the "Frobenius" or "Hilbert-Schmidt" inner product) over $\Bbb R^{m \times n}$ and $\Bbb R^{p \times q}$. For matrices $X,Y$ of the same shape, define
$$
\langle X,Y \rangle = \operatorname{tr}(X^TY) = \sum_{i,j} x_{ij} y_{ij}.
$$
To compute the adjoint of $\mathsf C$, note that for $m \times n$ matrices $X,Y$, we have
\begin{align}
\langle \mathsf C(X),Y \rangle &= 
\langle AXB,Y \rangle 
= \operatorname{tr}((AXB)^TY)
= \operatorname{tr}(B^TX^TA^TY)
\\ & = 
\operatorname{tr}(X^TA^TYB^T) = \langle X, A^TYB^T \rangle.
\end{align}
It follows that the adjoint of $\mathsf C$ is given by $\mathsf C^*(Y) = A^TYB^T$. From there, we can compute
$$
\mathsf C^*(\mathsf C(X)) = A^TAXB^TB.
$$
