Let's label the rows 1 through 8 and the columns a through h, as is standard in chess notation.
To place eight rooks in the case with the NW quadrant (rows 1-4, columns a-d) missing:
- you must place four rooks in the NE quadrant (rows 1-4, columns e-h). There are 4! ways to do this.
- you must then place four rooks in rows 5-8 - but you can't place them in any of columns e-h, so you essentially must place four rooks in the the 4-by-4 square of rows 5-8, columns a-d. There are 4! ways to do this. This gives $4!^2$ as the answer.
If instead you have a 2-by-2 NW corner missing (say a1, a2, b1, b2):
- you must first place two rooks in rows 1 and 2. There are $6 \times 5$ ways to do this.
- now you must place six rooks in rows 3-8, but there are two columns struck out (the columns in which you places the rooks in the first two rows) so there are $6!$ ways to do this. This gives $6 \times 5 \times 6!$ as the answer.
As a sanity check, if you have a 1-by-1 NW corner missing (say a1) this approach gives $7 \times 7!$. So the probability that a random arrangement of rooks doesn't contain a rook in a1 is $(7 \times 7!)/8! = 7/8$, and the probability that a random arrangement of rooks does contain a rook in a1 is therefore $1 - 7/8 = 1/8$. This is what you'd expect by symmetry.