A fractional ideal is invertible iff it is a projective module. To find an example of a non-invertible one it is therefore enough to exhibit an example of an ideal which is not projective.
A simple example is the ideal $(x,y)$ in $R=k[x,y]$.
Let us check this ideal is not projective «by hand» showing that the natural map $\phi:(a,b)\in R\oplus R\mapsto ax+by\in (x,y)$, which is surjective, does not admit a section $\psi:(x,y)\to R\oplus R$. We suppose otherwise, that is, that there is such a $\psi$, and we let $\psi(x)=(f,g)$ and $\psi(y)=(r,s)$. We have $0=\psi(xy-yx)=x\psi(y)-y\psi(x)$, so that $yf=xr$ and $yg=xs$. It follows that $r$ and $s$ are multiples of $y$, and $f$ and $g$ are multiples of $x$. Now, as $\phi\circ\psi$ is the identity map of $(x,y)$, we have $x=\phi(\psi(x))=\phi(f,g)=xf+gy$, but now we know that $xf+gy$ is in the ideal $(x^2,y)$, which does not contain $x$: this is impossible.