The following specifically answers this part of OP's question: "I am not looking for a different solution. I need help to continue with my solution".
We have $$RHS=\left[\sqrt{\left[p^2+f^2+2pf\right]}\right]=\left[\sqrt{p^2+\left[f^2+2pf\right]}\right]$$
Now we have $$f^2+2pf \gt 0$$
Correct thus far.
But how to prove $$f^2+2pf \lt 1$$
That's not what you need to prove (and, in fact, doesn't hold true in general).
What has to be proved at this point is that:
$$
\begin{align}
\left\lfloor \sqrt{\lfloor x \rfloor} \right\rfloor = p \;\;&\iff\;\; \left\lfloor \sqrt{\left\lfloor (p+f)^2 \right\rfloor} \right\rfloor = p \\[5px]
&\iff\;\; p \le \sqrt{\left\lfloor (p+f)^2 \right\rfloor} \lt p+1 \\[5px]
&\iff\;\; \color{blue}{p^2 \le \left\lfloor (p+f)^2 \right\rfloor \lt (p+1)^2} \tag{1}
\end{align}
$$
But $0 \lt f \lt 1\,$, and therefore $p \lt p+f \lt p+1\,$, so:
$$
\begin{align}
p^2 \lt (p+f)^2 \lt (p+1)^2 \;\;&\implies\;\; \color{blue}{p^2} = \left\lfloor p^2 \right\rfloor \color{blue}{\le \left\lfloor (p+f)^2 \right\rfloor \lt} \left\lfloor (p+1)^2 \right\rfloor = \color{blue}{(p+1)^2} \tag{2}
\end{align}
$$
(The implication follows from the property of the greatest integer function that $\,a \lt b \implies \lfloor a \rfloor \le \lfloor b \rfloor\,$, with strict inequality $\,\lfloor a \rfloor \lt \lfloor b \rfloor\,$ if $\,b\,$ is an integer.)
The above proves $(2)\,$, which is identical with $(1)\,$, and therefore concludes the proof.