When faced with a problem in Calculus involving piecewise functions (such as the Heaviside), you can almost always make it easier on yourself but dividing into cases where the function takes different simpler non-piecewise forms. Then evaluate those individually.
Another way to approach it in this case: we are guaranteed that $0 < \tau < t$ at all times, just due to the integration bounds. This means that $t - \tau > 0$ for the whole integration, so $H(t - \tau) = 1$. This reduces it to
$$\int_0^t \frac{\tau}{\sqrt{\tau^2-\alpha^2}} H(\tau-\alpha) d\tau$$
At this point, as you stated -- if $\tau < a$, then the integrand vanishes. And for $\tau > \alpha$, the $H(\tau - \alpha)$ becomes 1. So we can break it up into these two parts. (As I said, case-splitting!)
$$ \int_0^t \frac{\tau}{\sqrt{\tau^2-\alpha^2}} H(\tau-\alpha) d\tau = $$
$$\int_0^\alpha \frac{\tau}{\sqrt{\tau^2-\alpha^2}} H(\tau-\alpha) d\tau + \int_\alpha^t \frac{\tau}{\sqrt{\tau^2-\alpha^2}} H(\tau-\alpha) d\tau =$$
$$\int_\alpha^t \frac{\tau}{\sqrt{\tau^2-\alpha^2}} d\tau$$
which should work out fine. (Keep in mind that this final form is only true if $\alpha>0$, though! )