How can I prove $\det(\overline M)=\overline{\det(M)}$?

Of course $\overline M$ is the complex conjugate of an $n\times n$ matrix $M$.

Someone gave me advice to use the definition of determinant, then it means I have to use cofactor expasion here?

-
It is easier with the closed expression of $\det(A)$ as a sum over all permutation in $n$ : $\det(A)=\sum_{\sigma \in S_n}sgn(\sigma)\cdots$ –  Bebop Apr 28 '13 at 15:37
Antoine's answer is great and transparent. As an alternative, you can triangularize $M=PUP^{-1}$ and observe that this reduces to the upper triangular case where the determinant is just the product of the diagonal terms. The only trick is to observe that $\overline{P^{-1}}=\overline{P}^{-1}$. But I would still go with Antoine's answer. –  1015 Apr 28 '13 at 16:06

If you want to use the Laplace development (cofactors), then you can do it by induction.

The case for $1\times1$ matrices is obvious, so let $n>1$ and assume the result for $(n-1)\times(n-1)$ matrices.

If $A=[a_{ij}]$ is an $n\times n$ matrix, denote by $A_{ij}$ the matrix obtained by removing the $i$-th row and the $j$-th column. To not complicate notations, let $B=\bar{A}=[b_{ij}]$ with $b_{ij}=\overline{a_{ij}}$.

The expansion of $\det B$ along its first line is

$$\det B=(-1)^{1+1}b_{11}\det B_{11}+(-1)^{1+2}b_{12}\det B_{12}+\dots+ (-1)^{1+n}b_{1n}\det B_{1n}$$

By induction hypothesis, $\det B_{ij}=\overline{\det A_{ij}}$, so you can write

\begin{align} \det B&= (-1)^{1+1}\overline{a_{11}}\,\,\overline{\det A_{11}}+ (-1)^{1+2}\overline{a_{12}}\,\,\overline{\det A_{12}}+\dots+ (-1)^{1+n}\overline{a_{1n}}\,\,\overline{\det A_{1n}} \\[4pt] &= (-1)^{1+1}\overline{a_{11}\det A_{11}}+ (-1)^{1+2}\overline{a_{12}\det A_{12}}+\dots+ (-1)^{1+n}\overline{a_{1n}\det A_{1n}} \\[4pt] &=\overline{ (-1)^{1+1}a_{11}\det A_{12}+ (-1)^{1+2}a_{12}\det A_{12}+\dots+ (-1)^{1+n}a_{11}\det A_{1n} } \\[4pt] &=\overline{\det A} \end{align}

-

You have the following definition of the determinant:

$$\det(M) = \sum_{\sigma\in\mathfrak{S}_n}\varepsilon(\sigma)\prod_{i=1}^{n}m_{i,\sigma(i)}.$$

Then one has $$\det(\bar M) = \sum_{\sigma\in\mathfrak{S}_n}\varepsilon(\sigma)\prod_{i=1}^{n}\bar m_{i,\sigma(i)} = \sum_{\sigma\in\mathfrak{S}_n}\varepsilon(\sigma)\overline{\prod_{i=1}^{n}m_{i,\sigma(i)}} = \overline{\sum_{\sigma\in\mathfrak{S}_n}\varepsilon(\sigma)\prod_{i=1}^{n}m_{i,\sigma(i)}} = \overline{\det M},$$ all this equalities holding because conjugation is a $\mathbb R$-algebra homomorphism.

-
Sorry, I can't understand some of your notations. Can you explain it more simply just appropriate to linear algebra level? –  postman Apr 28 '13 at 15:45
@postman you can read wikipedia for the permutation definition of determinant. –  mez Apr 28 '13 at 16:02