Finding two missing vertexes of square given two points and equations of diagonal lines The diagonal line equations are $4x-5y+3=0$ and $5x+4y-27=0$.
Vertex $A$ was given $(-1,8)$, I figured it belongs to the second line equation. I have found the center of the square by solving diagonal line equation system $ O(3,3)$, and opposite of $A$ vertex $C(7,-2)$ from midpoint equation.
Now I'm not sure how to proceed with the points $B$ and $D$. I tried $|AB|=|BC|$ , but as I don't know the lengths of sides that didn't work out.
 A: You correctly got the intersection of the two lines as being at the point $O=(3,3)$.  Labeling the points clockwise, we know point $A=(-1,8)$ and the point $C=(7,-2)$ from the problem statement.  We also know that $\overline{AO}\perp \overline{OD}$ (they are perpendicular) and that $|\overline{AO}|=|\overline{OD}|$ (they are of the same length).
The linesegment $\overline{AO}$ can be described with the vector $\langle 4,-5\rangle$ (if you havn't seen vectors before, it is a way of saying it traveled 4 spaces to the right and 5 spaces down).  To find the vector describing $\overline{OD}$, reverse the order and apply a negative to one of the entries (in this case the first entry, had you changed the other entry it would have taken you to the point $B$) changing it from $\overline{AO}=\langle 4,-5\rangle$ to $\overline{OD}=\langle -5,-4\rangle$.
It is easy to verify that these representations have the same magnitude and are perpindicular, so travel from $(3,3)$ along the vector $\langle -5,-4\rangle$ to arrive at $D$.  Do so similarly for $B$.
Hover your mouse below for a picture of the situation if desired.

 

A: You know the center of the square is $(3,3)$.  You need a line segment along the other line of length $\sqrt{4^2+5^2}$ in each direction, so increase $x$ by $5$ and $y$ by $4$ to get one corner, and decrease them by the same amount for the other.
A: The slope of the first diagonal is $\frac{4}{5}$. The slope of the other (on which $A$ and $C$ lie) is $-\frac{5}{4}$. 
Let's find $B$, which will be to the right and above $A$, as well as to the right and above $C$. 
To get from $A$ to $O$, you follow diagonal $2$: increase the $x$-coordinate by $4$ (going right four units) and decrease the $y$-coordinate by $5$ (going down five units). To find $B$ from $O$, we should follow diagonal $1$: increase the $x$-coordinate by $5$ (going right five units) and increase the $y$-coordinate by $4$ (going up four units). 
This means $B$'s coordinates are $(8,7)$.
