The expression
$f(x) = x / e^{x^2} \tag{1}$
should be interpreted in the same way we would interpret $x / exp(x^2)$;that is,
$f(x) = x e^{-x^2} = x \exp (-x^2). \tag{2}$
The tip-off here is that the exponent $2$ of $x$ in the expression $x^2$ is in fact written as a superscript of $x$; in item (a) of the question, $(e^x)^2 = (e^x)(e^x) = e^{2x}$; but
$e^{x^2}$ should be thought of as $e^{xx} = (e^x)^x$; as Stephen Montgomery points out in his answer, the evaluation of the exponents is right to left.
Having hopefully clarified the issue, we have
$f'(x) = (x e^{-x^2})' = e^{-x^2} + x e^{-x^2} (-2x) \tag{3}$
using the Leibniz product rule and the formula $d / dx (e^u) = e^u (du / dx)$. After the simple algebra,
$f'(x) = e^{-x^2}(1 - 2x^2). \tag{4}$
Hope this helps. Cheerio,
and as always,
Fiat Lux!!!