Your attempt fails, for the reason that you mention : namely that $\lfloor x \rfloor$ isn't differentiable. However, the fact that we know some things about $\lfloor x \rfloor$, allows us to perform a more general procedure which yields the correct answer.
Now, what most people will tell you is that in an equation involving floors, you just need to simplify the floors by breaking the domain into subdomains where the floors simplify (because the argument of the floor has the same integral value). This makes sense.
However, we can use a generalized construction called the Riemann-Stieltjes integral, along with an integration by parts for Riemann-Stieltjes integrals , to make life easier.
I was quite surprised I couldn't direct you to an MSE resource which contained any of this material. My guess is that on most occasions we're just looking at the answer in a simple scenario, and in that case the technique of splitting into subdomains works the best and doesn't necessitate elaboration.
Let $f,g$ be real valued functions on $[a,b]$. We call a set of points $a \leq x_0<x_1<...<x_n = b$ as a partition $P$ of $[a,b]$. Corresponding to a partition $P$, we may associate lower and upper RS sums in the way we do so for the Riemann integral. For example, $$U(f,g,P) = \sum_{i=0}^{n-1} (g(x_{i+1})-g(x_i))\sup_{t \in[x_i,x_{i+1}]} f(t) \\ L(f,g,P) = \sum_{i=0}^{n-1} (g(x_{i+1})-g(x_i))\inf_{t \in[x_i,x_{i+1}]} f(t)$$
and then you define $f$ to be RS integrable with respect to $g$ on $[a,b]$ if for all $\epsilon>0$, you can find a $\delta>0$ such that for every partition $P$ with mesh (the mesh of a partition is the largest distance between two points in the partition) at most $\delta$, we have that $U(f,g,P) - L(f,g,P)< \epsilon$.
Now it's easy to prove that as the partition mesh goes to zero, these quantities $U(f,g,P)$ and $L(f,g,P)$ converge to some constant $L$, that you call the integral of $f$ with respect to $g$ on $[a,b]$ and denote $L = \int_a^b f \mathrm{d}g$.
We can prove a couple of things about the RS integral (and more, but these will suffice). Note that all statements and assumptions are on an interval $[a,b]$ :
If $g(x) = x$ then any function that's usually integrable (sufficient criteria include being piecewise continuous, which $\lfloor x\rfloor$ is , for example) is RS integrable and vice versa, and the values of the integral is the same.
If $f$ is a continuous function and $g$ is a function which is bounded and monotone increasing, then the RS integral of $f$ with respect to $g$ exists.
One of the nice statements is the RS integration-by-parts , picked up from here.
Suppose $f$ and $g$ are bounded functions with no common discontinuities on the interval $[a,b]$, and the Riemann-Stieltjes integral of $f$ with respect to $g$ exists. Then the Riemann-Stieltjes integral of $g$ with respect to $f$ exists, and $$ \int_a^b g \mathrm{d}f = f(b)g(b) - f(a)g(a) - \int_a^b f \mathrm{d}g$$
Which means that the statement you have made, in light of attempting to integrate $\lfloor x\rfloor$, is actually correct , but in a different sense. Applying the theorem to our situation :
$$
\bbox[yellow ,border:2px solid red]{
\int_a^b \lfloor x\rfloor dx = b \lfloor b \rfloor - a \lfloor a \rfloor - \int_a^b x d(\lfloor x\rfloor)}
$$
is a theoretically correct statement within the confines of the RS integral.
However, this is actually not the most useful statement, till we know what it means to evaluate $\int_a^b xd(\lfloor x\rfloor)$. For that, let's use the definition of the RS integral.
Namely, we know that the integral exists, so let's not worry about existence. We need a sequence of partitions whose mesh size goes to zero : the best is to just take the uniform mesh. Let $P_n : a =x_0 < ... < x_n = b$ where $x_j = \frac{j}{n} a + \frac{n-j}{n}b$. Now, we get :
$$
U(x,\lfloor x\rfloor,P) = \sum_{i=0}^{n-1} (\lfloor x_{i+1}\rfloor - \lfloor x_i \rfloor) \sup_{t \in [x_i,x_{i+1}]} t = \sum_{i=0}^{n-1}(\lfloor x_{i+1}\rfloor - \lfloor x_i \rfloor) x_{i+1}
$$
look at that difference though. It's going to be $1$ only when there's an integer less than or equal to $x_{i+1}$ and strictly bigger than $x_i$. Otherwise, it's going to be zero. I'm not going to be fully rigorous at this point, but you can now use the logic I mentioned to prove that the integral is actually equal to the sum of all integers between $a$ and $b$, with $b$ inclusive(if it's an integer) and $a$ exclusive (even if it is an integer).
That, using floors, can be translated to the formula :
$$
\int_a^b x \mathrm{d}(\lfloor x\rfloor) = \frac{(\lfloor b\rfloor)((\lfloor b\rfloor)+1)}{2} - \frac{(\lfloor a+1\rfloor)((\lfloor a+1\rfloor)+1)}{2}
$$
which should give you a full expression for $\int_a^b \lfloor x \rfloor dx$ following resubstitution into a highlighted formula (note : I sincerely hope I made no errors!)
We covered :
A generalization of integration by parts.
A rough example computation of the RS integral for a function with jumps.
An exact formula for the statement in question using the generalization.
Addendum : Note that $\lfloor x\rfloor$ is "differentiable" in what is called a distributional sense. Roughly speaking , for a function that "bends but doesn't break" like $|x|$, this is continuous but only piecewise differentiable. For such functions, it is possible to find an "integration-by-part companion" in the following sense.
Let $g$ be a function that is continuous but only piecewise differentiable, and zero at the boundaries $a$ and $b$ (this is not really a problem, but I just want to leave those boundary terms out). We call a "function"(note : it's not exactly a function, but more a "class of functions" : I'll be abusing notation because I don't want to cover the deeper details) $h$ as a "by-part companion" of $g$, if for all continuously differentiable $f$ we have $\int_a^b f'(x)g(x)dx = -\int_a^b f(x)h(x)$ dx. In some sense, even if $g$ doesn't work in a "traditional by-part sense", you can find a function $h$ which does the job.
You'd expect $h$ to be related to the derivative of $g$. For example, if $g(x) = |x|$, then $h(x) = \frac{|x|}{x}$ (the derivative of $g$ where it's not zero) with $h(0)$ being equal to any value (!) works.
Now, for functions which do jump (take the floor function, for example) we get so-called delta measures. I won't explain what these are in mathematical detail, but what happens is that jumps in functions are assigned a weight according to the level of the jump. For example, the floor function jumps by $1$ at integers, so we place a weight of $1$ at all integer points.
Now, it turns out that this theory is good enough to accommodate jumps as well : namely , if a function has jumps , you "integrate" with respect to the "weights" assigned at the jumps, by merely taking the function value at these jumps and multiplying with the weights.
For example, consider $\lfloor x \rfloor$. On $[a,b]$, this is discontinuous at all integers that are bigger than $a$ and less than or equal to $b$. Now, at each point, the function jumps by $1$ so you assign a weight "$1$" at this point. Now, you can prove that for any $f$ which isn't discontinuous at an integer, we have that $\int_a^b f d(\lfloor x\rfloor)$ is the sum of $f$ values at all integers between $a$ and $b$.
Such "weights" are called Delta measures , because they are weights at one point. These weights get incorporated into the concept of a "measure" in general, which are the most general objects that we can "integrate" functions against. So you have a long way to go!
