Prove the given determinant Prove the given determinant:
$$\left|
\begin{matrix}
a&b&ax+by \\
b&c&bx+cy \\
ax+by&bx+cy&0 \\
\end{matrix}\right|=(b^2-ac)(ax^2+2bxy+cy^2)$$
I didn't get any idea. Please help me solve this.
 A: HINT:
Set $$C_3'=C_3-xC_1-yC_2$$  to find 
$$\left|
\begin{matrix}
a&b&0 \\
b&c&0 \\
ax+by&bx+cy&-x(ax+by)-y(bx+cy) \\
\end{matrix}\right|=?$$
A: Develop the determinant along the last line. You get
$$D_{a,b,c}(x,y)=(ax+by)\begin{vmatrix}b&ax+by\\c&bx+cy\end{vmatrix}-(bx+cy)\begin{vmatrix}a&ax+by\\b&bx+cy\end{vmatrix}$$
Now compute the $2\times 2$ déterminants to get
$$D_{a,b,c}(x,y)=(ax+by)\left(b(bx+cy)-c(ax+by)\right)-(bx+cy)\left(a(bx+cy)-b(ax+by)\right)$$
It simplifies quite well into
$$\begin{align}D_{a,b,c}(x,y)
=&(ax+by(b^2-ac)x+(bx+cy)(b^2-ac)y\\
=&(b^2-ac)(ax^2+2bxy+cy^2)
\end{align}$$
A: A bit late in the day, here is the symmetric matrix version of this, writing $P^TAP = B,$ where $B$ is also symmetric with obvious determinant. We say that $A$ and $B$ are "congruent"
$$
\left(
\begin{array}{ccc}
1&0&0 \\
0&1&0 \\
-x&-y&0 \\
\end{array}
\right)
\left(
\begin{array}{ccc}
a&b&ax+by \\
b&c&bx+cy \\
ax+by&bx+cy&0 \\
\end{array}
\right)
\left(
\begin{array}{ccc}
1&0&-x \\
0&1&-y \\
0&0&1 \\
\end{array}
\right) =
\left(
\begin{array}{ccc}
a&b&0 \\
b&c&0 \\
0&0& - \left(a x^2 + 2bxy+cy^2 \right) \\
\end{array}
\right)
$$
