Showing that plane curve of degree $5$ is not trigonal. I am trying to prove that if $X \subset \mathbb{P}^2$ is a nonsingular curve of degree $5$ (and hence of genus $6$), that $X$ admits no linear system of degree $3$ and dimension $1$. This was also asked here but the solution doesn't make sense to me. This is from Harthsorne's IV.5 exercises, so I was hoping to use the results of that section to solve this problem.
Let $H$ be a very ample effective divisor associated to the embedding $X \subset \mathbb{P}^2$.
First of all, why can we assume that $X$ is not hyperelliptic (ie. that $|K|$ gives an embedding into $\mathbb{P}^5$)?
Once we have this, we can apply Riemann-Roch to $|2H|$ to get $$h^0(2H) - h^0(K - 2H) = 10 + 1 - 6 = 5$$ on the other hand $K - 2H$ is a degree $0$ divisor, so it either has a global section and $K \sim 2H$, or it does not and $K \not \sim 2H$.

But now if $h^0(K - 2H) = 1$, then $\dim |2H| = 5 = 1/2 \deg(2H)$, which would contradict that $X$ is not hyperelliptic by Clifford's theorem (since $h^0(K - 2H) = 1$ so $2H$ is special). Hence the only possibility is that $K \not \sim 2H$. (See edit: This part is not quite right.)

This is even more confusing, since the linked answer states that 'the canonical embedding factors as $X \to \mathbb{P}^2 \stackrel{|\mathcal{O}(2)|}\to \mathbb{P}^5$'... which can only be the case when $K \sim 2H$.
Is this answer just incorrect, or am I missing something here? Also, can anyone offer any help on this problem? A degree 5 plane curve can be hyperelliptic right?
Thanks!
Edit: My conclusion in the quoted section is not right, since the conclusion of Clifford's theorem is that $2H \sim K$, so there's no contradiction, and this is a possibility. On the other hand, it seems like $2H \not \sim K$ is also a possibility, so I am still very confused about this answer.
 A: Let me first point out that you can compute the canonical divisor on a plane curve in a pretty straightforward fashion: by example II.8.20.3, for $X$ a plane curve of degree $d$ we have $\omega_X \cong \mathcal{O}_X(d-3)$, or $K_X\sim(d-3)H$. So in your case with $X$ a plane quintic we have $K\sim 2H$, and so $K$ is very ample. Then by proposition IV.5.2, this means $X$ cannot be hyperelliptic.
I'm not familiar enough with rational scrolls to comment on the rest of Sasha's answer, so let me present a different solution method.
First, reinterpret Riemann-Roch as follows: from $l(D)-l(K-D)=\deg D + 1 -g$, we can rearrange a little to get $l(D)-1=\deg D -1 -(g-1-l(K-D))$. Now we interpret $l(D)-1$ as $\dim |D|$ and $g-1-l(K-D)$ as the dimension of the linear span of $D$ under the canonical embedding: the canonical embedding $\varphi_K$ embeds $C$ in to $\Bbb P^{g-1}$, and $\Gamma(X,\mathcal{O}_X(K-D))$ is the space of hyperplanes vanishing on $\varphi_K(D)$. This gives $$\dim |D| = \deg D - 1 - \dim \overline{\varphi_K(D)},$$ or that the dimension of $|D|$ is the difference between the "expected" dimension of the linear span of $\deg D$ points and the actual dimension of the linear span of $D$ under the canonical embedding. (This is sometimes known as "Geometric Riemann-Roch", but I don't know how common this terminology is.)
Why does this help us solve this problem? If we have a divisor $D$ so that $\deg D = 3$ and $\dim |D|=1$, the formula above says that any effective divisor linearly equivalent to $D$ must lie on a line under the canonical embedding. Since the morphism determined by the $g_3^1$ has to be separable, it's generically unramified, so almost all fibers are three points and each of these fibers must lie on a line under the canonical embedding. We'll show that can't happen by finding a global section of the canonical bundle which vanishes on two of these points but not the third (such a global section would correspond to a hyperplane under the canonical embedding which contains two points but not the third, an impossibility if the three points were on a line).
In our case, $K\sim 2H$, so given three points on our curve we can pretty easily find the required global section: take a quadratic form on $\Bbb P^2$ which hits two points and misses the third, for instance by taking a product of two linear forms.
This generalizes: you can show that no plane curve of degree $d>2$ admits a divisor $D$ with $\dim |D|=1$ and $\deg D < d-1$, assuming $\deg D$ is a prime or coprime to the characteristic.
