Inverse metric on Riemannian manifold The metric is used ubiquitously in General Relativity. It is a multilinear symmetric tensor
$$
g: TM \times TM \to \mathbb{R}
$$
which physicists write as
$$
ds^2\equiv g=g_{\mu\nu}dx^\mu \ dx^\nu,
$$
with a symmetric tensor product implied between the differentials. 
The metric is, actually, a section of the appropriate bundle; but let's ignore this complication since I want to direct your attention on the so-called "inverse" metric.
With the extra assumption of $g$ being non-degenerate: given $g$, and a vector $\partial_\mu$, I can define a covector as 
$$
\left[g(\partial_\mu,\cdot)\right]=g_{\mu\rho} \ dx^\rho
$$
In physics they use the inverse metric too, with both indices "upstairs". Viewing it mathematically, I cannot understand the meaning of inverse. 

How can I attain something of the sort
  $$
g^{\mu\nu}
$$
  given $g_{\mu\nu}$ and demanding that $g^{\mu\nu}g_{\nu\alpha}=\delta^\mu_\alpha$. 

Also the word "inverse" seems to be a misnomer; since such thing as
$$
g^{-1}: \mathbb{R}\to TM \times TM
$$
is unattainable.
 A: It only makes sense to speak of the inverse of a $(1,1)$-tensor, i.e. a section of $T^*M\otimes TM$. The metric $g$ however is a $(2,0)$-tensor, i.e. a section of $T^*M\otimes T^*M$ and what you call inverse metric is in the differential geometry literatur usually called the dual metric. This is a $(0,2)$-tensor $\check g$, which is defined by $\check g(\alpha,\beta)=g(\alpha^\sharp, \beta^\sharp)$ (for covectors $\alpha, \beta$), where $\sharp$ denotes the musical isomorphism.
However, if you want to write things in coordinates you obtain two matrices define by $g_{ij}=g(\partial_{x_i},\partial_{x_j})$ and $\check g^{ij} = \check g (d x^i, d x^j)$. So far the choice of lower vs. upper indices was arbitrary, but a computation shows that $g_{ij} \check g^{jk}=\delta_i^k$, i.e. the matrices $(g_{ij})$ and $(\check g^{ij})$ are mutual inverses. In standard notation the $\check{}$ is usually left away in the coordinate representation.
Of course you can also argue the other way around: Given a local coordinate expression $(g_{ij})$ you can invert the matrix to arrive at $\check g^{ij}$ and then check that the local definition $\check g(d x^i,d x^j):=\check g^{ij}$ actually yields a global $(0,2)$-tensor.
In short: Given  the metric $g$, there is a natural way to obtain a $(0,2)$-tensor $\check g$. In local coordinates  this corresponds to inverting a matrix. Globally however, it does not make sense to invert $g$. 
A: The inverse metric is defined by the property that:
$$ g\circ g^{-1} = \delta $$
where the Kronecker delta is defined to be the section of $T^*M\otimes TM$ that is acts as the identity on $T^*M$:
$$ \delta := \sum_{i = 1}^{n} \mathrm{d}x^i \otimes \partial_{x^i} $$
In other words, if there were a general way to compute the inverse metric without computing the inverse matrix, then the method can be applied to the case of a constant metric on a linear space (that is, a simple inner product space) to compute the inverse matrix of a matrix without, computing the inverse matrix.
