1. Solving $(5m+3)(3m+1)=n^2$
Here is an elementary approach that only uses modular arithmetic.
We do not assuming $m\geq 0$ initially, to highlight where it is used exactly in this approach.
It was shown above that since
We have $d=\gcd(5m+3,3m+1)$ divides $4$.
If $d=1$, then $|5m+3|$ is a square so either $5m+3$ or $-(5m+3)$ is a square. However since squares are $0,1$ or $4$ modulo $5$, this is impossible. Hence we must have $d=2$ or $4$. For either case $m$ is odd so $m=2k+1$.
Now the equation reduces to
We divide by $4$ into
(5k+4)(3k+2) = (n/2)^2
so now $d'=\gcd(5k+4,3k+2)=1$ or $2$.
Suppose first that $d'=2$, so $k$ must be even, say $k=2s$. The new equation after dividing by $4$ is
(5s+2)(3s+1) = (n/4)^2
Now $\gcd(5s+2,3s+1)=1$ so, like earlier, $5s+2$ or $-(5s+2)$ must be a square. However this is impossible as squares are $0,1$ or $4$ modulo $5$.
Therefore $d'=1$, hence we have $|3k+2|$ is a square so either $3k+2$ or $-(3k+2)$ is a square. Squares are $0$ or $1$ modulo $3$ so this is impossible for the former. For the latter case, $k\leq -1$ or else $-(3k+2)$ is negative and cannot be square.
We are reduced to a necessary condition that $m=2k+1$ and $k\leq -1$. In particular $k=-1$ is possible, giving $(m,n)=(-1,2)$. This is the part where requiring $m$ to be positive comes into play: As $m=2k+1$, positive $m$ forces $k\geq 0$ and so there are no solutions.
2. Quadratic diophantine in two variables
For two-variable quadratic diophantine equations, of the form
You can refer to posts like this and this.
The idea is you can multiply by integers and use integral linear transformations to convert into
X^2 - uY^2 = v
so original integer solutions $(x,y)$ transforms into integer solutions $(X,Y)$. By finding all integer solutions to the latter, we can check which ones satisfies the original. The only difficult case is when $u>0$ and not a square, which leads to a series of Pell type equations.
In a more algebraic setting, these questions are well answered by the theory of prime norms in quadratic number fields. Books like Cox's "Primes of form $x^2-ny^2$" addresses it.
For more than two variables indeed there is an algorithm, but unfortunately I do not know much about it.