Rather than thinking of the equation $y^2 = f(x)$ as a singular model of the curve $C/k$ we are interested in, think of it as a defining equation for its function field $F := k[x, y]]/(y^2 - f(x))$ (a quadratic extension of the rational function field $k(x)$). Up to isomorphism there is a unique smooth projective curve $C$ with function field $k(C) = F$, and we can take this as our definition for $C$ without needing to write down a smooth projective model for it (which will necessarily need to sit in $\mathbb{P}^3$ not $\mathbb{P}^2$).
Via Serre duality, one can formally relate the Hasse-Witt matrix, which is defined in terms of the $p$-power Frobenius acting on $H^1(C, \mathcal{O}_C)$, where $\mathcal{O}_C$ is the structure sheaf, with the action of the Cartier operator on $H^0(C, \Omega_F)$, where $\Omega_F$ is the space of differential forms on $C$ (Kähler differentials), which can be defined purely in terms of the function field $F = k(C)$. The action of the Cartier operator $C$ on $H^0(C, \Omega_F)$ is dual to the action of the $p$-power Frobenius acting on $H^1(\mathcal{O}_C)$ (so one takes $p$th-roots rather than $p$th powers). In general the matrices are not exactly the same, but when $k = \mathbb{F}_p$ (the case we are interested in), they are the same, since over $\mathbb{F}_p$ taking a $p$th root or a $p$th power is the same thing.
Depending on how much algebraic geometry you know some of this terminology may be unfamiliar, but the key point is that instead of thinking about the curve, one can translate everything into the function field setting, where working with the equation $y^2 = f(x)$ makes perfect sense.
One can restrict the Cartier operator to the space $\Omega_F(0)$ of regular (holomorphic) differentials, and it is enough to consider its action on a basis for $\Omega_F(0)$, which has dimension $g$ as an $\mathbb{F}_p$-vector space (one can even take this as a definition of the genus $g$). For hyperelliptic curves it is easy to write down a basis for $\Omega_F(0)$: take $\omega_i := x^{i - 1}dx/y$ for $0 \le i < g$.
One can explicitly compute the action of the Cartier operator on this basis (the Cartier operator can be defined purely algebraically), and it is not hard to show that$$C(\omega_j) = \sum_i c_{ip - j}^{1/p}\omega_i,$$where $c_k$ is the coefficient of $x^k$ in $f^{(p - 1)/2}$. When $k = \mathbb{F}_p$ we have $c_{ip - j}^{1/p} = c_{ip - j}$, and the matrix of the Cartier operator is exactly the Hasse-Witt matrix.
You might find the paper "A formula for the Cartier operator on plane algebraic curves" by Stöhr and Voloch helpful, see here.
- K.O. Stöhr, J.F. Voloch. A formula for the Cartier operator on plane algebraic curves. J. Reine Angew. Math. 377, 49–64 (1987).
It explains all this in more detail and in greater generality.
Some papers throw in the condition that $f(x)$ cannot have multiple roots (e.g. on page 52 of the paper you mentioned). Is there a particular reason why this condition has to be satisfied?
If $f(x)$ is not square-free (equivalently, has no multiple roots over the algebraic closure), then if we let $h(x)$ be the square-free part of $f(x)$ (i.e. keep just one copy of each repeated factor) then $k[x, y]/(y^2 - f(x))$ is isomorphic to $k[x, y]/(y^2 - h(x))$. This is completely analogous to the fact that $\mathbb{Q}(\sqrt{45})$ and $\mathbb{Q}(\sqrt{3})$ are the same field.
So strictly speaking there is no harm in allowing $f$ to be divisible by a square, but there is no reason to allow this and there are good reasons not to. For example, the genus would be smaller than we would expect given the degree of $f$ and all our formulas will be off — the Hasse-Witt matrix of $y^2 = x^{101} + x^{96}$ is a $2 \times 2$ matrix, not a $50 \times 50$ matrix.
Also, another reference you might find helpful is Stichtenoth's Algebraic Function Fields and Codes, see here. He covers elliptic and hyperelliptic function fields in Chapter VI.
- H. Stichtenoth. Algebraic function fields and codes. Second edition. Graduate Texts in Mathematics, 254. Springer-Verlag, Berlin, 2009. xiv+355 pp.