So because of the great help from Daniel I was able to solve the excercise. Because there are no other answers I will share my solution.
The claim that $F$ is continuous is actually not dependent on the FTC at all but rather a separate theorem. Also the potential discontinuities of $f$ play no role, because the theorem relies just on the fact that $f$ is Riemann integrable and therefore bounded. For a proof of this see here.
The second claim is then just the FTC applied to the points where $f$ is continuous. There are actually different versions of the FTC which differ in their preconditions. If the precondition is that $f$ is continuous on an interval then it is actually important that we have only finitely many points of discontinuity because otherwise we might not be able to split the interval into subintervals where $f$ is continuous (e.g. if $f$ would be discontinuous at every irrational number, we couldn't make an interval between two rational numbers where $f$ is continuous, because there would be another irrational number in it). If the preconditions is just so, that $f$ is continuous at some point $x_0$ then there can even be infinitely many discontinuities and it would still work.
Most definitions of the FTC I've seen actually require an interval where $f$ is continuous. I will include a proof of the stronger version which just requires pointwise continuity below.
Let $[a,b]$ be a compact interval and $f: [a,b] \to \mathbb R$ a Riemann-integrable function. Let $F: [a,b] \to \mathbb R$ with $F(x) = \int_a^x f(t) \, dt$ and $x_0 \in [a,b]$. If $f$ is continuous at $x_0$ then $F$ is differentiable at $x_0$ and $F'(x_0)=f(x_0)$.
We now prove that $F'(x_0) = \lim_{x \to x_0} \frac{F(x)-F(x_0)}{x-x_0} = f(x_0)$ which proves that $F$ is differentiable at $x_0$ with derivative $f(x_0)$. Recall from the definition of a limit, that
$\lim_{x \to x_0} g(x)=L$ iff for every $\varepsilon \gt 0$ there is $\delta \gt 0$ such that for all $x \in [a,b] \setminus \{x_0\}$ with $|x-x_0|\lt\delta$ we have $|g(x)-L|\lt\varepsilon$.
Let $\varepsilon \gt 0$. By continuity of $f$ at $x_0$ there exists $\delta \gt 0$, such that for all $x \in [a,b]$ with $|x-x_0|\lt\delta$ we have $|f(x)-f(x_0)|\lt\varepsilon$. Then it follow also that for all $x \in (x_0,x_0+\delta)$
$$\begin{aligned} \biggl\lvert \frac{F(x)-F(x_0)}{x-x_0}-f(x_0) \biggr\rvert &= \biggl\lvert \frac{1}{x-x_0} \Big( \int_a^x f(t) \,dt - \int_a^{x_0} f(t) \,dt \Big) - f(x_0) \biggr\rvert \\&= \biggl\lvert \frac{1}{x-x_0} \Big( \int_{x_0}^x f(t) \,dt \Big) - f(x_0) \biggr\rvert \\&= \biggl\lvert \frac{1}{x-x_0} \Big( \int_{x_0}^x f(t) \,dt \Big) - \frac{1}{x-x_0} \int_{x_0}^x f(x_0) \,dt \biggr\rvert \\&= \biggl\lvert \frac{1}{x-x_0} \int_{x_0}^x f(t)-f(x_0) \,dt \biggr\rvert \\&\le \frac{1}{x-x_0} \int_{x_0}^x \biggr\rvert f(t)-f(x_0) \biggr\rvert \,dt \\&\lt \frac{1}{x-x_0} \int_{x_0}^x \varepsilon \,dt \\&= \frac{1}{x-x_0} (x-x_0) \varepsilon = \varepsilon \end{aligned}$$
where we used the additivity and the triangle inequality property of Riemann integrals. Similarly for for all $x \in (x_0-\delta,x_0)$
$$\begin{aligned} \biggl\lvert \frac{F(x)-F(x_0)}{x_0-x}-f(x_0) \biggr\rvert &= \biggl\lvert \frac{F(x_0)-F(x)}{x_0-x}-f(x_0) \biggr\rvert \\&= \biggl\lvert \frac{1}{x_0-x} \Big( \int_a^{x_0} f(t) \,dt - \int_a^x f(t) \,dt \Big) - f(x_0) \biggr\rvert \\&= \biggl\lvert \frac{1}{x_0-x} \Big( \int_x^{x_0} f(t) \,dt \Big) - f(x_0) \biggr\rvert \\&= \biggl\lvert \frac{1}{x_0-x} \Big( \int_x^{x_0} f(t) \,dt \Big) - \frac{1}{x_0-x} \int_x^{x_0} f(x_0) \,dt \biggr\rvert \\&= \biggl\lvert \frac{1}{x_0-x} \int_x^{x_0} f(t)-f(x_0) \,dt \biggr\rvert \\&\le \frac{1}{x_0-x} \int_x^{x_0} \biggr\rvert f(t)-f(x_0) \biggr\rvert \,dt \\&\lt \frac{1}{x_0-x} \int_x^{x_0} \varepsilon \,dt \\&= \frac{1}{x_0-x} (x_0-x) \varepsilon = \varepsilon \end{aligned}$$
Thereby we proved our statement.
