Can we relax hypothesis of Fundamental theorem of calculus? Let $F$ is continuous  $[a,b]$ and differentiable on $[a,b]$ and $F'(x)=f(x)$ for $x\in [a,b]$. Assume that $f$ is Riemann integrable. Then Fundamental theorem of calculus say that
$$F(x)-F(a)=\int_{a}^x f(t) dt$$

My Question is: Can we say that $$F(x)-F(a)=\int_{a}^x f(t) dt$$ hold true if we  remove the assumption that $F$ is differentiable at  $a$ and $b$.

My thoughts: The proof uses the mean value theorem to prove the theorem but mean value requires only that $F$ is continuous $[a,b]$ and differentiable on $(a,b)$.
 A: There are other versions of the fundamental theorem of Calculus that are in the spirit of what you wrote. Some require extensions to the Riemann integration, which are beyond the scope of a first year college Calculus class. The most common extension in mathematics is Lebesgue integration, there is also a less common but also useful extension, called  gauge integration develop independently by  several people: Denjoy, Henstock-Kurzweil and others.

In  Lebesgue integration these are mainly two results:
Theorem 1L. If $F:[a,b]\rightarrow\mathbb{C}$ is absolutely continuous, then $F'$ exists $\lambda$--a.s., is integrable (in the sense of Lebesgue) over $[a,b]$, and
$$
F(x)-F(a)=\int^x_a F'(t)\,dt, \quad a\leq x\leq b.
$$
Theorem 2L. Let  $F:[a,b]\rightarrow\mathbb{C}$ be  continuous. If $F$ is  differentiable on $[a,b]$, with the exception of a countable set pf points,   and  $F'$ is integrable (in the sense of Lebesgue, never mind the set of exceptional points where $F'$ is not defined) then,
$$
F(x)-F(a)=\int^x_a F'(t)\,dt,\quad a\leq x\leq b.
$$

For the Henstock integral, there is a version similar the Theorem 2. above
Theorem 2H: Suppose
that the function $F$ is continuous  differentiable at all
but a countable collection of points in $[a,b]$. Then its derivative $F'$ is integrable (in the sense of Henstock-Kurzweil) on $[a,b]$, and
$$
F(x)-F(a)=\int^x_a F'(t)\,dt,\quad a\leq x\leq b.
$$

Theorems 1L, 2L   are studied in courses of Lebesgue integration.  A good source at the undergraduate level is Stein and Shakarchi's Real Analysis book.
A good place for the study of this type of integral is Bartle's book "Modern theory of integration". This type of integration can be done in a way that is similar to Riemann integration.

A: A stronger version of the Fundamental Theorem of Calculus is given in the following paper:
Michael W. Botsko and Richard A. Gosser, "Stronger Versions of the Fundamental Theorem of Calculus",
The American Mathematical Monthly, Vol. 93, No. 4 (Apr., 1986), pp. 294-296.
FTC: Let $f$ be Riemann integrable on $[a, b]$, and let $g$ be a continuous function
on $[a, b]$ such that $g'_{+}(x) = f(x)$ for all $x$ in $(a, b)$ where $g'_{+}(x)$ is the right derivative. Then
$\int_a^b f(x) \mathrm{d} x = g(b) - g(a)$.
Remark: By the way, I need it when I encountered the following problem.
Suppose $f: [0, 1]\to \mathbb{R}$ is non-decreasing and concave, with $f(0)=0$ and $f(1)=1$. Prove that
$$\frac{\int_0^1 f^2 \mathrm{d} x}{\int_0^1 f \mathrm{d} x} \ge \frac{2}{3}.$$
A: Apostol gives the theorem in following manner

FTC: Let $f:[a, b] \to\mathbb {R} $ be Riemann integrable on $[a, b] $ and let $g:(a, b) \to\mathbb {R} $ be such that $g'(x) =f(x) $ for all $x\in(a, b) $. Then the limits $$\lim_{x\to a^{+} } g(x), \lim_{x\to b^{-}} g(x) $$ exist and we have $$\int_{a} ^{b} f(x) \, dx=\lim_{x\to b^-} g(x) - \lim_{x\to a^+} g(x) $$

Thus essentially you don't need the $F$ in your question to be differentiable (or even continuous or defined) at end points $a, b$.

On request of user @sani via  comment I give a proof of the above mentioned theorem.
Let $$F(x) =\int_{a} ^{x} f(t) \, dt\tag{1}$$ Since $f$ is Riemann integrable on $[a, b] $ it is bounded on $[a, b] $ and let $M$ be an upper bound for $|f|$ on $[a, b] $. Then $$|F(x+h) - F(x) |=\left|\int_x^{x+h} f(t) \, dt\right|\leq M|h|$$ if both $x, x+h$ lie in $[a, b] $. This proves that $F$ is continuous on $[a, b] $.
Consider $g$ defined on $(a, b) $ such that $g'(x) =f(x) $ on $(a, b) $. Let $c\in(a, b) $. By the usual FTC we have $$g(x) =g(c) +\int_{c} ^{x} f(t) \, dt$$ for all $x\in(a, b) $ and using $(1)$ we can write above equation as $$g(x) =g(c) +F(x) - F(c) \tag{2}$$ Since $F$ is continuous on $[a, b] $ we can see the limits of RHS of $(2)$ as $x\to a^+$ and as $x\to b^{-} $ exist and we have $$\lim_{x\to a^+} g(x) =g(c) +F(a) - F(c) $$ and $$\lim_{x\to b^-} g(x) =g(c) +F(b) - F(c) $$ Subtracting these two equations we get $$F(b) - F(a) =\lim_{x\to b^-} g(x) - \lim_{x\to a^+} g(x) $$ Note that $F(a) =0$ and $F(b) =\int_a^b f(x) \, dx$ via definition $(1)$ and the proof for the above mentioned theorem is complete.
