OK, it seems you've understood why the positions of M and N are both needed--as someone noted above, technically they wouldn't have to be midpoints, but midpoints are good enough.
As to how to solve, you seem confused about two things: (A) Why $\overline{BM}$ and $\overline{AN}$ are perpendicular, and (B) how to find the area of $\triangle APQ$, knowing (A).
That said, I'm avoiding anything other than standard geometry and simple algebra. This problem is relatively hard (for high school geometry), but it can be done without resorting to trig, without graphing the lines and using the line equations, and in fact without even calculating the lengths of the various segments.
I'm skipping a step or two here and there, but nothing essential. I'm sure Miss Studenka, my HS geometry teacher, would disapprove, but alas.
Proof that $\overline{BM} \perp \overline{AN}$
Given:
- $ABCD$ is a square
- $N$ is the midpoint of $\overline{AD}$
- $M$ is the midpoint of $\overline{CD}$
We can show:
- $AB = AD$ from the definition of a square.
- $\angle BAD = \angle ADC = 90°$, also from the definition of a square.
- $AM = DN$, from (4) and the defined midpoints.
- $\triangle ABM \cong \triangle DAN$, by the side-angle-side theorem.
- $\angle MBA = \angle NAD$, by congruence.
- $\angle NAB = 90° - \angle NAD = 90° - \angle MBA$, from (5) and (8)
- $\angle NAB + \angle MBA = 90°$, rearranging (9)
- $\angle NAB + \angle MBA + \angle BQA = 180°$, from the definition of a triangle.
- $\angle BQA = 90°$ by subtraction. $\square$
OK, that's done. How do we find the area now? First, note that $\triangle APQ$ is a right triangle, so if we know any two of its sides we can find the area. Second, we want to note that $\triangle MAB \sim \triangle MQA \sim \triangle AQB$. Let's actually show that, using the triple-angle theorem:
- $\angle MAB = \angle MQA = \angle AQB = 90°$ as we've already shown they're right triangles.
- $\angle BMA = \angle AMQ = \angle BAQ$. The first equality is true because $M,Q,$ and $B$ are collinear; the second equality we showed in (8).
- $\angle ABM = \angle QBA = \angle QAM$. These are true by the same logic as in (14).
Therefore all three triangles are similar. Now, for simplicity, we're going to ignore everything but A, B, Q, M, and the segments between those four points for a moment, and use the following substitutions: $AM = a, AB = b, BM = c, BQ = d, QM = e$, and $AQ = h$. Note that $h$ here is the altitude of the right triangle. Also note that $c = d+e$. Now because our triangles are similar, we know:
$$\frac{e}{h} = \frac{h}{d} \implies h^2 = de$$
$$\frac{a}{b} = \frac{h}{d} \implies \frac{a^2}{b^2} = \frac{h^2}{d^2} \implies \frac{a^2}{b^2} = \frac{de}{d^2} = \frac{e}{d} \implies a^2d = b^2e$$
But because $c = d+e$, this means
$$a^2d = b^2(c-d) \\
a^2d = b^2c - b^2d \\
(a^2 + b^2)d = b^2c \\
c^2d = b^2c \\
d = \frac{b^2}{c}$$
Similar logic but substituting the other direction gets us $e = \frac{a^2}{c}$. Now, using the side length of the square $AB = s$:
- $AQ = h = \sqrt{de} = \sqrt{\frac{a^2b^2}{c^2}} = \frac{ab}{\sqrt{a^2+b^2}} = \frac{\frac{s}{2} \cdot s}{\sqrt{(\frac{s}{2})^2+s^2}} = s\sqrt{\frac15}$
Now we need another side. $AP$ is easier because once again we have two similar triangles: $\triangle PAM$ and $\triangle PCB$:
- $\angle PAM = \angle PCB$ as opposite interior angles of parallel lines
- $\angle PMA = \angle PBC$ for the same reason
- $\angle APM = \angle BPC$ as opposite angles
- $\triangle PAM \sim \triangle PCB$
- $BC = 2 \cdot AM$ because of the midpoint
- $CP = 2 \cdot AP$ because the triangles are similar
- $3 \cdot AP = AC$ and so $AP = \frac13 \cdot AC$ by algebra
- $AP = \frac13 \cdot AC = \frac13 \sqrt{2s^2} = s\sqrt{\frac29}$
We now have one leg ($AQ = s\sqrt{\frac15}$) and the hypotenuse ($AP = s\sqrt{\frac29}$) of $\triangle APQ$. That's enough to get the other leg and the area: $PQ = \sqrt{AP^2 - AQ^2} = s\sqrt{\frac{1}{45}}$ making the area
$$A = \frac12 ab = \frac12 \cdot s \sqrt{\frac15} \cdot s \sqrt{\frac{1}{45}} = \frac{s^2}{30}$$
And since the problem gave us $s^2 = 120 \text{ cm}^2$, the area we want is $4 \text{ cm}^2$.
The keys to solving this without any trig are showing that the intersection at $Q$ is perpendicular, then recognizing the sets of similar triangles. (I'm shocked no other answer pointed out $\triangle BPC \sim \triangle APM$. That's what lets us skip the trig.) Of note: the calculations I showed for finding $h$ apply to all right triangles. One interesting fact I couldn't shoehorn in is that you can find the length of that altitude with the "Inverse Pythagorean Theorem":
$$\frac{1}{h^2} = \frac{1}{a^2} + \frac{1}{b^2}$$
It's in there but kind of hidden, at the point where we found $h$ from $\frac{ab}{c}$.
If you have any questions or steps you can't follow, post a comment.