A step in the equality $ \widehat{(T^{*}u)}=|\det T|^{-1}((T^{t})^{-1}))^{*}\hat{u} $ Suppose $ u\in S^{'}(\mathbb{R}^{n}) $, where $ S^{'}(\mathbb{R}^{n}) $ is the dual space of the Schwartz-class functions $ S $; $ T^{*}u $ is the pullback via 
$ T $, i.e. if $ T: \mathbb{R}^{n}\to\mathbb{R}^{n} $ is a linear bijection and $ 
u\in S^{'}(\mathbb{R}^{n})\cap C(\mathbb{R}^{n}) $, we can define its pullback under $ T $ by
$$
T^{*}u=u\circ T.
$$
Note that a change of variables gives
$$
\begin{align} (T^{*}u)(\phi)&=\int u(Tx)\phi(x)dx\\
&= \int u(y)|\det T^{-1}|\phi (T^{-1}y)dy\\
&= u(|\det T^{-1}|\phi(T^{-1})),
\end{align}$$
so, for general $ u\in S^{'}(\mathbb{R}^{n}) $, we define the pullback using the left and right sides of this equality.
With the above notation, prove that:

$$
\widehat{(T^{*}u)}=|\det T|^{-1}((T^{t})^{-1}))^{*}\hat{u}.
$$

The Fourier transform is defined to be $$ \hat{f}(\xi)=\int_{\mathbb{R}^{n}}e^{-i<x, \xi>}f(x)dx $$
for $ f(x)\in L^{1}(\mathbb{R}^{n}) $.

Here is what I have:
$$
\begin{align}
(\widehat{T^{*}u})(\phi(x)) &= (T^{*}u)(\hat{\phi})\\
&=\int_{\mathbb{R}^{n}}T^{*}(u(x))\hat{\phi}(x)dx\\
&= \int_{\mathbb{R}^{n}}\widehat{T^{*}[u(x)]}\phi(x)dx\\
&= \int_{\mathbb{R}^{n}}\widehat{u[T(x)]}\phi(x)dx\\
&= \int_{\mathbb{R}^{n}}\left[\int_{\mathbb{R}^{n}}e^{-i<s, \xi>}u[T(s)]ds\right]\phi(x)dx\\
\end{align}
$$
How to continue? I am wondering that where does the transpose come from?
 A: You were off to a good start. For any $\phi \in \cal S(\mathbb R^n)$ the definitions provide
$$\widehat{T^*u}(\phi) = \int_{\mathbb R^n} T^*u(x) \hat \phi(x) \, dx = \int_{\mathbb R^n} u(Tx) \hat \phi(x)\, dx.$$
Now change variables: with $y = Tx$ you get $dy = |\det T| dx$ so that
$$ \int_{\mathbb R^n} u(Tx) \hat \phi(x)\, dx = |\det T|^{-1} \int_{\mathbb R^n} u(y) \hat \phi (T^{-1}y) \, dy.$$
Time to investigate the Fourier transform. Observe
$$ \hat \phi (T^{-1} y) = \int_{\mathbb R^n} e^{-i \langle z,T^{-1}y \rangle} \phi(z) \, dz$$
and $\langle z,T^{-1}y \rangle = \langle (T^{-1})^tz,y \rangle = \langle (T^t)^{-1}z,y \rangle$, where the last equality uses the fact that the matrix representing $T$ is real.
Now combine the formulas. I will be a bit cavalier with the use of Fubini's theorem trusting that you can make it precise.
\begin{align*} \widehat{T^*u}(\phi) &= |\det T|^{-1} \int_{\mathbb R^n} \int_{\mathbb R^n} u(y)  e^{-i \langle (T^t)^{-1}z,y \rangle} \phi(z) \, dz dy \\ &= |\det T|^{-1} \int_{\mathbb R^n} \int_{\mathbb R^n} e^{-i \langle (T^t)^{-1}z,y \rangle}  u(y)\, dy   \phi(z) \, dz.
\end{align*}
As long as $u \in L^1(\mathbb R^n)$ (which doesn't seem to be implied by the hypotheses, but whatever) the inner integral equals
$ \hat u ((T^t)^{-1}z) $ so that
$$\widehat{T^*u}(\phi) =  |\det T|^{-1} \int_{\mathbb R^n} \hat u ((T^t)^{-1}z) \phi(z) \, dz =  |\det T|^{-1} \int_{\mathbb R^n} ((T^t)^{-1})^* \hat u(z) \phi(z) \, dz$$ where the last expression is
$|\det T|^{-1} ((T^t)^{-1})^* \hat u (\phi).$ Finally disregard $\phi$ to obtain
$$\widehat{T^*u} = |\det T|^{-1} ((T^t)^{-1})^* \hat u.$$
A: Let us start with showing that $\hat \phi \circ T^{-1} = |\det(T)| \, \widehat{\phi \circ T^t}$:
$$
(\hat \phi \circ T^{-1})(\xi) 
= \hat \phi (T^{-1}\xi)
= \int \phi(x) \, e^{-i\langle x, \, T^{-1} \xi \rangle} \, dx
= \int \phi(x) \, e^{-i\langle (T^{-1})^t x, \, \xi \rangle} \, dx \\
= \{ y = (T^{-1})^t x \}
= \int \phi(((T^{-1})^t)^{-1} x) \, e^{-i\langle y, \, \xi \rangle} \, |\det((T^{-1})^t)^{-1}| \, dy \\
= |\det(T)| \int \phi(T^t x) \, e^{-i\langle y, \, \xi \rangle} \,  \, dy
= |\det(T)| \, \widehat{\phi \circ T^t}(\xi)
$$
Now, we can show what you want to show. I use a different notation than you; instead of $u(\phi)$ I write $\langle u, \phi\rangle.$
$$
\langle \widehat{T^* u}, \phi \rangle
= \langle T^* u, \hat \phi \rangle
= \langle u, |\det T|^{-1} \hat \phi \circ T^{-1} \rangle
= |\det T|^{-1} \langle u, \hat \phi \circ T^{-1} \rangle \\
= \{ \text{ by what was shown above } \} \\
= \langle u, \widehat{\phi \circ T^t} \rangle
= \langle \hat u, \phi \circ T^t \rangle
= |\det(T^{-1})| \, \langle ((T^t)^{-1})^* \hat u, \phi \rangle
$$
