Which substitution do I use? For $x>0$, let $f(x)=\int_0^\infty e^{-t-x^2/t} t^{-1/2}dt$.
a) show that $f(x)=x\int_0^\infty e^{-t-x^2/t} t^{-3/2}dt$ via an adequate substitution.
b) Calculate $f'(x)$ and show that $f(x)=Ce^{-2x}$ for some constant $C$ (no need to evaluate $C$).
I tried some substitutions but none worked, so I hadn't even tried the second part.
 A: We define $f:\mathbb{R}^+\rightarrow\mathbb{R}$ by
$$f(x):=\int_{0}^{\infty}e^{-t-\frac{x^2}{t}} \frac{dt}{t^{1/2}}.$$
For part a), use the substitution $u=\frac{x^2}{t} \iff t=\frac{x^2}{u}$. Then 
$$t+\frac{x^2}{t}=\frac{x^2}{u}+u,\\
\sqrt{u}=\frac{x}{\sqrt{t}}\implies \frac{1}{\sqrt{t}}=\frac{\sqrt{u}}{x},\\
dt = -\frac{x^2}{u^2}du\implies \frac{dt}{\sqrt{t}}=-\frac{x}{u^{3/2}}du,$$
and so,
$$f(x)=\int_{0}^{\infty}e^{-t-\frac{x^2}{t}} \frac{dt}{t^{1/2}}\\
=-\int_{\infty}^{0}e^{-u-\frac{x^2}{u}} \frac{x}{u^{3/2}}du\\
=x\int_{0}^{\infty}e^{-u-\frac{x^2}{u}} \frac{du}{u^{3/2}}\\
=x\int_{0}^{\infty}e^{-t-\frac{x^2}{t}} \frac{dt}{t^{3/2}},$$
as was to be demonstrated.
For part b), differentiate under the integral sign:
$$f'(x)=\frac{d}{dx}\int_{0}^{\infty}e^{-t-\frac{x^2}{t}} \frac{dt}{t^{1/2}}\\
=\int_{0}^{\infty}\frac{\partial}{\partial x}e^{-t-\frac{x^2}{t}} \frac{dt}{t^{1/2}}\\
=\int_{0}^{\infty}\left(\frac{-2x}{t}\right)e^{-t-\frac{x^2}{t}} \frac{dt}{t^{1/2}}\\
=-2x\int_{0}^{\infty}e^{-t-\frac{x^2}{t}} \frac{dt}{t^{3/2}}\\
=-2f(x),$$
where in the last line we've made use of the identity proven in part a). So since $f(x)$ satisfies the 1st-order linear ODE $f^\prime+2f=0$, we must have
$$f(x)=Ce^{-2x},$$
for some constant $C$, as was to be demonstrated.

Optional: To find $C$, note that $f(x)=Ce^{-2x}\implies f(0)=C$. Then going back to the original definition of $f(x)$,
$$f(0)=\int_{0}^{\infty}\frac{e^{-t}}{\sqrt{t}}dt=\int_{0}^{\infty}\frac{e^{-x^2}}{|x|}\,2x\,dx=2\int_{0}^{\infty}e^{-x^2}\,dx=\sqrt{\pi}.$$
Hence,
$$f(x)=\sqrt{\pi}e^{-2x}.$$
