How does the fundamental theorem of calculus help to simplify difficult exponential functions. I am very confused by the value of the fundamental theorem of calculus to this initial value problem from Logan, D. (2017). A first course in differential equations. Springer.

Sure we have found the value of the arbitrary integration constant but now we have an indefinite integral with a free parameter.  I'm not clear on what this is meant to illustrate.  It has left me confused about why the FTC is so important since solving $x'=e^{-t^2}$ was what motivated discussing the FTC earlier in the chapter.
 A: The comment about "pass[ing] off a full undefined integral as a solution to a problem" seems to capture the substance of the issue.
The FTC is being used here an a guarantee of existence for a certain (particularly simple) class of initial value problems,
$$
x' = f,\qquad x'(t_{0}) = x_{0},
$$
with $f$ an arbitrary but specified continuous function.
Suppose $f$ is a bounded function on some real interval $I$ containing $t_{0}$. For each real $t$ in $I$, the integral $\int_{t_{0}}^{t} f(s)\, ds$ of $f$ may be defined using only lower and upper sums of $f$. If in addition $f$ is continuous, the integral may be proven to exist. (This is usually proven by showing $f$ is uniformly continuous, and then deducing the lower and upper sums may be made as close as we like with a sufficiently fine partition.)
Separately (by the FTC), if $f$ is continuous, then the real-valued function $F$ defined by
$$
F(t) = x_{0} + \int_{t_{0}}^{t} f(s)\, ds
$$
solves the initial value problem above. Again, one part of the non-trivial content is the existence of a solution, particularly that every continuous, real-valued function on $I$ has an antiderivative. It was not a priori obvious that our IVP has a solution at all, nor that every continuous function on an interval has an antiderivative. (If $f$ has removable and/or jump discontinuities, for example, this IVP does not have solutions in this "classical" sense.)
It's true that the integral representation $F$ is not of itself "explicit," but mathematically it still represents a substantial guarantee.
A: Roughly speaking, the Fundamental Theorem of Calculus allows one to transform an equation involving a derivative into an equation involving an integral. The point is that instead of differentiating the unknown function $x$ one can integrate the known function $e^{-s^2}$. In this case it just happens to be the case that integrating this known function is not straightforward; see How to integrate $\int e^{-t^{2}} \space \, \mathrm dt $ using introductory calculus methods.
The last sentence refers to this: https://www.desmos.com/calculator/aoxb6zxyyb , where the red curve is the graph of the unique solution $x(t)$ and the draggable blue line is its linear approximation.
A: The equation given is that $$x'(t)=e^{-t^2}$$ with $$x(0)=2.$$ With the former equation, you can integrate on both sides of the equation, obtaining $$\int_0^sx'(t)\,\mathrm{d}t=\int_0^se^{-t^2}\,\mathrm{d}t.$$ By the fundamental theorem of calculus, $$\int_0^sx'(t)\,\mathrm{d}t=x(s)-x(0).$$ Therefore, $$x(s)-x(0)=\int_0^se^{-t^2}\,\mathrm{d}t,$$ which is equivalent to $$x(s)=2+\int_0^se^{-t^2}\,\mathrm{d}t.$$
To address your comments: there are no free parameters in the expression $$2+\int_0^se^{-t^2}\,\mathrm{d}t.$$ I have no idea why you think there is one, but perhaps your concern is that $$\int_0^se^{-t^2}\,\mathrm{d}t$$ cannot be simplified in terms of elementary functions. However, the book never claimed you can simplify it in terms of elementary functions. The book states "Calculators and computer algebra systems plot such functions easily." This is because this integral is a well-known function, the error function. The error function is defined by $$\operatorname{erf}(s):=\frac2{\sqrt{\pi}}\int_0^se^{-t^2}\,\mathrm{d}t.$$ Therefore, we have that $$x(t)=\frac{\sqrt{\pi}}2\operatorname{erf}(t)+2.$$
