Can there be non trivial self-dual 5-forms on a 10-dimensional compact orientable manifold without boundary? I am puzzled about the following.
Let $(M,g)$ be a compact, orientable Riemannian manifold without boundary. We define the usual inner product $(,)$ for two r-forms $\alpha,\beta\in\Omega^r(M)$ by the formula
$$
(\alpha,\beta) = \int_M \alpha\wedge \star\beta\, .
$$
This inner product is positive-definite, i.e. for any $\alpha\in\Omega^r(M)$, we have
$$
(\alpha.\alpha)\geq 0\, ,
$$
and $(\alpha,\alpha)=0$ if and only if $\alpha =0$.
For definiteness I focus now on a ten-dimensional manifold: dim $M$=10.
Let us consider $\gamma$ a self-dual 5-form, i.e. $-i \gamma = \star \gamma$ (I am using the convention where when acting on a r-form, $\star^2=(-1)^{r(m-r)}$ for dim $M$=m. Moreover, I could have considered anti-self-dual forms, by replacing $-i$ by $i$ in $-i \gamma = \star \gamma$). Then we have
$$
\gamma \wedge \star \gamma = - \gamma \wedge \star \gamma = 0\, ,
$$
where I used the self-duality condition as well as the fact that for a $r$-form $\omega$ and a $p$-form $\eta$, we have $\omega \wedge \eta = (-1)^{rp}\eta\wedge \omega$. But then we have $(\gamma,\gamma)=0$ and thus $\gamma=0$.
My question is: is there a flaw in the above argument?
My puzzle originally comes from the fact that in physics, we often encounter non-trivial 5-forms on 10-dimensional Riemannian manifold. The example that I have in mind is a case where the manifold is not compact ($AdS_5\times S^5$), so strictly speaking there is no contradiction with the above argument. But as far as I can see, the compactness is only necessary to define the integral of the inner product. But in the present case the product $(\gamma,\gamma)=0$, so there is no integration problem (there is no convergence problem).
I admit that the last argument is very hand-waving so I guess this is the place where a there's flaw but I'd like to know what you think.
Thank you for you answer(s).
Edit: Let me add one comment that might sharpen my puzzle.
In the example I have in mind, where $F\in\Omega^5(AdS_5 \times S^5)$ is such that $\star F = -iF$, we have moreover that
$$
\int_{S^5} F
$$
exist. Moreover, it is independent of the radial coordinate (if using the Poincaré coordinates of $AdS_5$). This suggests that there is really nothing wrong with $F$ at infinity, hence the above integrals for the inner product $(,)$ still make sense for this 5-form.
 A: You're mixing up the real and complex as well as Riemannian and pseudo-Riemannian.
Let $X$ be a $2n$-dimensional Riemannian manifold. Then a self-dual $n$-form is a $+1$-eigenvector of the Hodge star map
$$\star: \Omega^n(X; \Bbb R) \longrightarrow \Omega^n(X; \Bbb R).$$
We have the well-known formula
$$\star^2 = (-1)^{n(2n - n)} \mathrm{Id} = (-1)^{n^2} \mathrm{Id}.$$
Now if $n$ is odd, as it is in your case, Then $\star^2 = -\mathrm{Id}$, so it cannot have $\pm 1$ as eigenvalues, and hence there are no self-dual or anti-self-dual $n$-forms. $\star$ does, however, have $\pm i$ as eigenvalues in this case, so you can define (anti-)self-dual $n$-forms as $\pm i$-eigenvectors of $\star$, as you did in your original post. Notice, however, that when you do this you are now working with complex $n$-forms and you should not be using the real inner product any longer.
To complicate matters further, you're really interested in Lorentzian manifolds. One way to define the Hodge star is by
$$\alpha \wedge \star \beta = \langle\alpha, \beta\rangle\, \mathrm{vol},$$
where $\langle \cdot, \cdot \rangle$ is the inner product induced by the metric $g$. This is the same inner product you integrate to get the global inner product, and when $X$ is a Lorentzian manifold, $g$ is indefinite and hence the inner product on forms is indefinite as well. As you know, there is nothing wrong with having vectors of length zero in an indefinite inner product space (e.g. lightlike vectors in Minkowski space).
As a final remark, on a $2n$-dimensional Lorentzian manifold, we actually have
$$\star^2 = (-1)^{n(2n - n) + (2n - 1)}\mathrm{Id} = (-1)^{n^2 + 2n - 1}\mathrm{Id}$$
(the signature of the metric affects things now). When $n$ is odd (as in your $10$-dimensional case), we then see that $\star^2 = \mathrm{Id}$ on a Lorentzian manifold (of dimension $4k + 2$), so we actually use the usual definition of self-duality (without any $i$'s). When $n$ is even (e.g. in Minkowski space), then $\star^2 = -\mathrm{Id}$ and we have to use the alternate definition of self-duality and work with complex inner product spaces.
A: Quick summary: Nothing in your question seems to depend on dimension 10; looking at dimension 2 suggests that your problem formulation is suspect. 
Let's look at a much simpler situation: the torus, with angular coordinates $\theta$ and $\phi$. In this case, $d\theta \wedge d\phi$ is the area-form, and clearly has nonzero integral. It's been a long time since I studied this stuff, but I believe that in this case, we have something like $\star d\theta = i d\phi$ and $\star d\phi = i d\theta$, so that $\star^2 = -1$ as expected by your formula. 
Now every form can be written as a sum of a self-dual and an anti-self-dual form: you just compute $s = \frac{1}{2}(\omega - i \star \omega)$ and $a = \frac{1}{2}(\omega + i \star \omega)$, sort of like splitting a matrix into a sum of a symmetric and skew-symmetric matrix. Doing this for $\omega = d\theta$ (and dropping the 1/2), we get the self-dual form
$\sigma = d\theta + d\phi$
That's clearly a nontrivial self-dual 1-form on the torus. But if you evaluate $(\sigma, \sigma)$, you evidently get zero. So your claim about the positive-definiteness of the inner product seems suspect. 
Assuming that you can fix up that bit, I suspect that taking the 10-manifold M to be a cartesian product of 5 copiss of the torus, and taking $\omega = \sigma_1 + \ldots + \sigma_5$, where $\sigma_i$ is the form above, on the $i$th torus, will give you the thing you are looking for. 
A: I can't speak to the case of using complex numbers as the base field.  Using the reals only, I believe that in a space of dimensionality $2n$, there are self-dual $n$-vectors and $n$-forms only when $n$ is itself even.
This can be seen in your formula $\gamma \wedge \gamma = (-1)^{n^2} \gamma \wedge \gamma$.  If $n$ is even, then you just get $\gamma \wedge \gamma = \gamma \wedge \gamma$, a true statement.  This makes sense; in 4d spacetime, there are self-dual 2-vectors (some EM-fields, for instance).  I won't consider this a proof that such dual $n$-vectors and $n$-forms exist; merely, they are not ruled out using this argument.
If $n$ is odd, then your argument $\gamma \wedge \gamma = - \gamma \wedge \gamma$ holds, and thus in 10d we see that there are no self-dual 5-vectors.  For the same reason, there can be no self-dual 1-vectors in a 2d space (as John pointed out).
