There is a very good question behind this post, but the true answer lies beyond linear algebra and in the realm of algebraic or symplectic geometry. Since you are only learning Sylvester's Law, I will only give a rather superficial answer.
The quotient space $X$ is non-Hausdorff, not even $T_1$ (for all $n\ge 1$), hence, it cannot be a manifold. In order to see this, take any nonzero matrix $A\in Y=M_n({\mathbb R})$, a sequence of matrices $B_i\in G=GL_n({\mathbb R})$ converging to the zero matrix and observe that
$$
\lim_{i\to\infty} B^T_i A B_i = {\mathbf 0}.
$$
From this, you conclude that the corresponding congruence classes $[A]\in X$ and $[{\mathbf 0}]\in X$ have the property that every neighborhood of $[{\mathbf 0}]$ contains $[A]$. One can verify that the space $X$ is contractible, hence, very boring from the viewpoint of algebraic topology. However, this is not the end of the story.
The above observation suggests that taking the naive quotient is simply a wrong thing to use in this setting. There are several areas of math where it is defined how to make the "right" quotient in this setting, one is GIT = Geometric Invariant Theory (pioneered by David Mumford, although many early ideas go back to David Hilbert), the other area is Symplectic Geometry. I will not attempt to describe what the "right" quotient means in this situation. Briefly, there are two open and dense subsets $Y_{sst}$ (semistable matrices) and $Y_{st}$ (stable matrices) in $Y$, which are both invariant under the action of the group $G$ and satisfy $Y_{st}\subset Y_{sst}$. In order to form the "right" quotient, one takes the naive quotient of $Y_{st}$ by the action of $G$, while for the quotient of $Y_{sst}$ one uses the extended orbit equivalence: $A\sim B$ if and only if the closures of their orbits in $Y_{sst}$ have nonempty intersection. Then the "right" quotient space (also called "Mumford quotient" or GIT quotient) is
$$
Z=Y//G= Y_{sst}/\sim.
$$
I did not try hard to compute $Y_{sst}$ and $Y_{st}$ in this example, but I am pretty sure that $Y_{st}$ consists of invertible matrices. It is also follows from the definition that zero matrix does not belong to $Y_{sst}$.
I did not try to analyze the geometry/topology of the quotient space $Z$: This usually requires very serious work.
- Lastly, if you are just interested in exhibiting representatives of congruence classes of matrices, you can find it in
Lee, Jeffrey M.; Weinberg, David A., A note on canonical forms for matrix congruence, Linear Algebra Appl. 249, 207-215 (1996). ZBL0886.15008.
as well as in
De Terán, Fernando, Canonical forms for congruence of matrices and (T)-palindromic matrix pencils: a tribute to H. W. Turnbull and A. C. Aitken, SeMA J. 73, No. 1, 7-16 (2016). ZBL1338.15030.
where more references are given.
In particular, you will see that the quotient space $X$ has the cardinality of continuum. However, the authors did not think in terms of algebraic geometry, so they did not address the question of geometry/topology of the "right" quotient space. I am not sure if algebraic geometers ever looked at the problem. When I have more time, I will post this question on Mathoverflow.
