Given the expression
$$
\small{ S \left[ 1-Q\left(\frac{\log \det \left(\mathbf{I}+\left( \mathbf{I} + \mathbf{U} \mathbf{X} \mathbf{X}^H \mathbf {U}^{H} \right)^{-1} \left(\mathbf {U} \mathbf {Y} \mathbf {Y}^H \mathbf {U}^{H}\right)\right)-c}{\sqrt{\left(1-\left( \det \left(\mathbf{I}+\left( \mathbf{I} + \mathbf{U} \mathbf{X} \mathbf{X}^H \mathbf {U}^{H} \right)^{-1} \left(\mathbf {U} \mathbf {Y} \mathbf {Y}^H \mathbf {U}^{H}\right)\right) \right)^{-2}\right) (\log_2 e)^2 /n}} \right) \right] }
$$
First, build the common matrix expression
$$\eqalign{
\def\LR#1{\left(#1\right)}
\def\qiq{\quad\implies\quad}
\def\A{A^{-1}}
A &= {I+UXX^HU^H} &\qiq dA=U\,dX\,X^HU^H \\
B &= {UYY^HU^H} &\qiq dB=0\quad\{{\rm constant}\} \\
Z &= {I+\A B} &\qiq dZ=-\A dA\,\A \\
}$$
Then build the common scalar expression
$$\eqalign{
\def\p{\partial}
\def\grad#1#2{\frac{\p #1}{\p #2}}
\def\dgrad#1#2{\frac{d #1}{d #2}}
\def\Z{Z^{-1}}
\def\ZZ{Z^{-T}}
\def\AA{A^{-T}}
\def\a{\alpha}
\def\b{\beta}
\def\g{\gamma}
\def\l{\lambda}
\def\e{e^{-2\a}}
\def\ee{\LR{1-\e}}
\a &= \log\det(Z) &\qiq d\a= \ZZ:dZ \\
\b &= {\sqrt n}\log(2)&\qiq d\b=0\qquad\{\b\:{\rm is\:constant}\} \\
\g &= \frac {\b\LR{\a-c}}{\ee^{1/2}} &\qiq
\dgrad{\g}{\a} = \frac{\b\ee+\a\b\LR{c-\a}\e} {\ee^{3/2}} \\
}$$
Using these variables, the objective function simplifies to
$$\l = S\Big[1-Q\LR{\g}\Big]$$
I have no idea what your Q-function is, but I'll assume that you know how to calculate its derivative
$$P(\g) = \dgrad{Q(\g)}{\g}\qiq dQ = P\,d\g $$
Now calculate the differential and gradient of $\l$
$$\eqalign{
d\l &= -S\,dQ \\
&= -SP\:d\g \\
&= -SP\LR{\dgrad{\g}{\a}\:d\a} \\
&= -\dgrad{\g}{\a}\:SP\:{\ZZ:dZ} \\
}$$
Let's collect everything on the LHS of $dZ$ into
a new matrix variable before continuing
$$\eqalign{
d\l &= -M:dZ \\
&= M:\LR{\A\,dA\,\A} \\
&= {\AA M\AA}:dA \\
&= \LR{\AA M\AA}:\LR{U\,dX\,X^HU^H} \\
&= \LR{U^T\AA M\AA U^*X^*}:dX \\
\grad{\l}{X} &= {U^T\AA M\AA U^*X^*} \;=\; G \\
}$$
So $G$ is the gradient wrt $X.\:$
The gradients wrt the conjugate variables are
$$\eqalign{
G^* = \grad{\l}{X^*} \qquad G^H = \grad{\l}{X^H} \\
\\
}$$
The above derivation utilizes this well-known gradient and calculates derivatives in the Wirtinger sense.
It also uses a colon to denote the Frobenius product
$$\eqalign{
A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij}
\;=\; {\rm Tr}\!\LR{A^TB} \\
A:A &= \|A\|^2_F \qquad \{{\rm Frobenius\:norm}\} \\
}$$
The properties of the underlying trace function allow the terms in such a product to be rearranged in many fruitful ways, e.g.
$$\eqalign{
A:B &= B:A \\
A:B &= A^T:B^T \\
C:\LR{AB} &= \LR{CB^T}:A &= \LR{A^TC}:B \\
}$$
