Prove matrix inequality in inner product space

If $\{u_i\}_{i=1,\dots,d}$, $\{v_i\}_{i=1,\dots,d}$ are two sets of vectors in an inner product space such that $$\langle u_i, v_j\rangle= \delta_{ij}\;,$$ then prove that the two $d\times d$ matrices $A$ and $B$ with entries defined by $$A_{ij}=\langle u_i, u_j\rangle\;,\qquad B_{ij}=\langle v_i, v_j\rangle\;,$$ satisfy the matrix inequality $$A\geq B^{-1}\;,$$ i.e. $A-B^{-1}$ is a positive semidefinite matrix.

Let $\mathcal V$ be the span of all $v_i$s. Decompose each $u_i$ into the sum $x_i+w_i$, where $x_i\in V^\perp$ and $w_i\in\mathcal V$. Then \begin{cases} \langle w_i,v_j\rangle=\delta_{ij},\\ A_{ij}=\langle x_i,x_j\rangle+\langle w_i,w_j\rangle,\\ B_{ij}=\langle v_i,v_j\rangle. \end{cases} Let $\mathcal X$ be the span of all $x_i$s. Suppose $\dim\mathcal V=p$ and $\dim\mathcal X=q$. By picking any two orthonormal bases of $V$ and $X$, we may rewrite the above in matrix form: \begin{cases} V^\ast W=I,\\ A=X^\ast X+W^\ast W,\\ B=V^\ast V, \end{cases} where $V$ and $W$ are $p\times d$ and $X$ is $q\times d$. Since $p=\dim\mathcal V\le d$, the condition $V^\ast W=I$ implies that $p$ must be equal to $d$ and hence $V^\ast=W^{-1}$. Thus $B^{-1}=W^\ast W\le A$.