In other words, how can we be sure we arrived at the contradiction because of the proposition $P$ we assumed, and not because of some other proposition $Q$ we are not even aware of?
Let's consider the possibilities for $Q.$
I will assume the Law of the Excluded Middle,
because I do not know enough about logic systems without it
to say whether proof by contradiction works in any of them.
I also assume that the proof is set in some axiomatic system $A,$
since I do not know how to do proofs outside of any axiomatic system.
So we either have that $Q$ is always true under $A,$ or $Q$ is not always true under $A.$
Case 1
If $Q$ is always true under $A,$ and $Q$ leads to a contradiction,
then $A$ is an inconsistent system.
It really is not suitable for any kind of proof, including
proof by contradiction.
Case 2
If $Q$ is not always true under $A,$ then either $Q$ is entailed by the axioms $A$ together with the assumption $P$ (but not by $A$ alone) or
$Q$ is not entailed by the axioms $A$ together with the assumption $P.$
Let's split this case into two sub-cases.
Case 2a
If $Q$ is not entailed by the axioms $A$ together with the assumption $P,$
then $Q$ represents an unwarranted assumption that we made.
This makes the proof invalid even if its conclusion is true.
Our human fallibility makes it possible for such things to happen.
It is possible to introduce such an assumption into a direct proof as well,
making it equally invalid.
Case 2b
If $Q$ is entailed by the axioms $A$ together with the assumption $P$
but not by $A$ alone,
then $P$ indeed is necessary in order to arrive at the contradiction,
and the theorem is provable.
But if $Q$ really is instrumental in making some of the steps in the
proof, and we were not aware of this fact,
then I think again we have an invalid proof of a true conclusion.
Again, this is a possible result of human fallibility
that can also happen in direct proofs.
Conclusion
Taking all the cases together, there really are only two ways for a proof by contradiction to go wrong. One is if we are working with an inconsistent set of axioms, which is very bad news altogether.
The other is if we have an unjustified step somewhere in the proof.
An example of an "unjustified step" occurred famously in Andrew Wiles's first announcement that he had proved Fermat's Last Theorem.
Someone (actually, I think multiple people)
found a mistake in the proof he presented.
After he made a considerable additional effort, he was finally able to present a proof without that mistake, and this proof was accepted.
Some comments under the original question raised the issue of how we check the intermediate steps of a proof by contradiction,
claiming that it is easier to check the steps in a direct proof
since their conclusions are all true.
Things that we want to prove typically have the form $S\implies T,$
for which a direct proof typically involves assuming $S$ and then
showing that $T$ follows.
In the intermediate steps of the proof, we have some facts that
depend on $S,$ which we cannot "check" by simply observing that they are true; we can check them by verifying the logic in every step leading up to that part of the proof, or we can check them by coming up with an alternative proof showing that they follow from $S.$
We may also introduce some known facts (which do not depend on $S$) in the course of the proof, which we can check simply by verifying that they are true facts.
A third possibility is that we derive something from $S$ that we could have known to be true without assuming $S.$
This is wasteful; we could improve the proof by simply introducing these facts as known without showing a logical derivation from $S.$
The same things happen in proof by contradiction.
We will have some steps that we can check only by checking every step in the logic leading up to them or by devising an alternative proof,
we may have known facts that we can check more easily,
and we may even have wasted effort by deriving something from our
(false) assumption that we could have simply brought in as a known fact.