$(a+b)^2+4ab$ and $a^2+b^2$ are both squares I cannot find a complete answer to the following problem (this is the source):

Q.  Find all positive integers $(a,b)$ for which $(a+b)^2+4ab$ and $a^2+b^2$ are both squares.


Just something: clearly if $(a,b)$ works then $(a/c,b/c)$ works as well, where $c$ is the greatest common divisor of $(a,b)$. Hence we can assume they are coprime. 
List of known primitive solutions: As remarked by Michael below, this is equivalent to solve the equation $(x^2-y^2)^2+12xy(x^2-y^2)+4x^2y^2=z^2$ with positive integers $x,y$. Adding the constraint $\mathrm{gcd}(x,y)=1$, by computer calculations we can see that all solutions $(x,y)$ with $x,y \le 30000$ are only the following ones: $(3,2)$, $(5,1)$, $(7,85)$, $(39,46)$, $(2717,1380)$, $(4097, 1337)$. This leads to the primitive solutions
$$
(a,b)=(5,12),
$$ 
$$
(a,b)=(5477689,7498920).
$$ 

Edit (05 Dec 2015): the ones above are the unique primitive solutions also with the (last new) constraint $x,y \le 80000$..
 A: Let $q^2 = (a+b)^2$, $p^2 = a^2 + b^2$ and $r^2 = (a+b)^2 + 4ab$. Note that $r^2 + p^2 = 2q^2$. Now let $X = r+p$, $Y = r - p$ and $Z = 2q$.
You can easily show that $r^2 + p^2 = 2q^2$ if and only if $X^2 + Y^2 = Z^2$.
So, each solution can be found as follows:


*

*Pick a Pythagorean triple $(X, Y, Z)$ in which $Z$ is even.

*Let $q = \frac{Z}{2}$, $r = \frac{X+Y}{2}$ and $p = \frac{X-Y}{2}$.

*Check if there is any Pythagorean triple $(a', b', p)$ for all $a', b'$ that $a'+b' = q$.


Hope this helps :)
A: In general, to make two quadratic polynomials as squares like,
$$a^2+b^2 = c^2\tag1$$
$$(a+b)^2+4ab = d^2\tag2$$
will yield one quartic to be made a square. Given the complete solution to Pythagorean triples as $a,b = m(x^2-y^2),\,2mxy$, then,
$$m^2(x^4 + 12 x^3 y + 2 x^2 y^2 - 12 x y^3 + y^4) = d^2$$
We can suppress $m,y$ without loss of generality,
$$u^4 + 12 u^3 + 2 u^2  - 12 u+1=v^2\tag3$$
Since you've found rational points to $(3)$ like $u =x/y = 3/2,\; 5/1,\; 7/85,\dots$, then it is birationally equivalent to an elliptic curve, so there is an infinite more (like $u = 616264191/46041814$). This yields,
$$a,b =377661704472473885,\;56747842513764948$$
Thus your system $(1)$, $(2)$ has an infinite number of primitive integer solutions, all of which are rational points on $(3)$.
P.S. For an elementary discussion on how to find more points on $(3)$, see this post.
A: HINT.- If $(a+b)^2+4ab=m^2$ then $4ab$ must be of the form $2h(a+b)+h^2$ (since $m=a+b+h$ in integers non negatives).
Besides $a^2+b^2=n^2\iff a=2rs$; $b=r^2-s^2$ (choosing $a$ even); $n=r^2+s^2$.
You deduce the diophantine equation $$h^2+2(r^2-s^2+2rs)h=8rs(r^2-s^2)$$ which you can try to solve.
Besides, if the discriminant of this equation is a square, you can get a set of solutions, functions of $(r,s)$ (not the complet set of solutions maybe). Hence another possible way to work would be try to solve the diophantine equation $$(r^2-s^2+2rs)^2+8rs(r^2-s^2)=t^2$$ in order to obtain as many solutions as this equation could have.
