Understanding some details in calculating the Hilbert function and polynomial. I am practicing finding the Hilbert function and Hilbert polynomial for $S/I$ of the following:
$S = k[x_1, x_2, x_3, x_4]$ and the monomial ideal $I = (x_1 x_2 x_3, x_2x_3x_4, x_1x_4).$
Note that, I am using the notation used below (based on Eisenbud pg. 320 -321):
If $I = (m_1, \dots, m_l)$ min. set of monomial generator, $I' = (m_1, \dots, m_{l-1}) \subsetneq I,$ and $d = \operatorname{deg}m_l.$
$$S\mu \xrightarrow{\varphi} S/I' \rightarrow S/I \rightarrow 0$$
$$\mu \mapsto m_l + I^{'}$$
Then we have:
$$\operatorname{ker}{\varphi} = (I': m_l) = J = (m_1/\operatorname{gcd}(m_1, m_l), \dots , m_{l-1}/\operatorname{gcd}(m_{l - 1}, m_l))$$
Therefore, $\operatorname{im}(\varphi) = S/J . \mu$ and we have the s.e.s:
$$0 \rightarrow (S/J)\mu \xrightarrow{\tilde{\varphi}} S/I' \rightarrow S/I \rightarrow 0. $$
Regard $S\mu$ as a graded $S-$module, $\operatorname{deg}\mu = d.$ Then the s.e.s. is graded. So for each $i \in \mathbb Z,$ we get a s.e.s. of vector spaces
$$0 \rightarrow (S/J)_{i - d} \rightarrow (S/I')_{i} \rightarrow (S/I)_i \rightarrow 0 $$ which implies that $H_{S/I}(i) + H_{S/J}(i -d) = H_{S/I'}(i)$ and this gives us an algorithm to find $H_{S/I}(i).$
Now, here is my attempt to answer the question:
1- Put $m_l = x_1x_4, I' = (x_1x_2x_3, x_2x_3x_4)$ then
$J = (I': x_1x_4) = (x_1x_2x_3/\operatorname{gcd}(x_1x_2x_3, x_1x_4), x_2x_3x_4/\operatorname{gcd}(x_2x_3x_4, x_1x_4)) = (x_2x_3, x_2x_3)= (x_2x_3)$
And $S/J = k[x_1, x_4],$ so $H_{S/J}(d) = \binom{2 + d -1}{d} = \binom{1 + d}{d} = 1 + d$ where $d = \operatorname{deg} m_l $ and $H_{S/J}(i - 2) = i - 1, i \geq 2$
2- Put $m_{l}^{'}= x_2x_3x_4$, $I^{''} = (x_1x_2x_3)$ then $\operatorname{deg}m_{l}^{'} = 3,$ and $J' = (I'': x_2x_3x_4) = (x_1x_2x_3/\operatorname{gcd}(x_1x_2x_3, x_2x_3x_4)) = (x_1x_2)$
And $S/J' = k[x_3, x_4],$ so $H_{S/J'}(d) = \binom{2 + d -1}{d} = \binom{1 + d}{d} = 1 + d$ where $d = \operatorname{deg} m' $ and $H_{S/J'}(i - 3) = i - 2, i \geq 3$
So,
$$H_{S/I^{'}}(i) + H_{S/J^{'}}(i - 3) = H_{S/I^{''}}(i).$$
Now, since  $H_{S/J^{'}}(i - 3) = i - 2, i \geq 3,$ it remains to find $H_{S/I^{''}}(i)$.
Since we have $S/I^{''} = \frac{k[x_1, x_2, x_3]}{(x_1x_2x_3)}[x_4],$ a polynomial ring over the indeterminate $x_4,$ monomials: $x_1^a x_4^b, x_3^a x_4^b,$ then $$S/I^{''} = k[x_1, x_4] + k[x_3, x_4].$$
But then I am not sure how to calculate  $H_{S/I^{''}}(d)$? Also, I am noticing that I will count $x_4$ twice (am I correct?), then how can I get rid from this in the expression for $H_{S/I^{''}}(d)$?
Also, I do not understand what should be called Hilbert polynomial and in what determinants it is? Could anyone clarify this to me, please?
Any help will be greatly appreciated!
 A: First, in your setting, when $i$ large enough, Hilbert function will become a polynomial and it is called Hilbert polynomial (so you have a compact way to write down formula of Hilbert function at all large values).
Second, to calculate Hilbert function of $S/I^{''}$ or more generally, to calculate Hilbert function of $S/(f)$ (where $f$ is a monomial or a homogeneous polynomial of degree $d$), we can use essentially the same exact sequence that you have made use of, i.e:
$$0\to S\stackrel{\cdot f}{\to} S \to S/(f) \to 0$$
this will give short exact sequences in graded component
$$0\to (S)_{i-d}\stackrel{\cdot f}{\to} (S)_i \to (S/(f))_i \to 0$$
so you will have $H_{S/f}(i) + H_{S}(i -d) = H_{S}(i)$.
Applying to your case, for all $i$
$$H_{S/(x_1x_2x_3)}(i)=H_{S}(i)-H_{S}(i -3)={i+3 \choose 3}-{i\choose 3}$$
In particular, here, the Hibert function agrees with Hilbert polynomial for all $i\ge 3$.
A quick comment in your computation:
1/ $S/J \not= k[x_1, x_4]$ and 2/$J'=(x_1)$.

Edits: The following is another way to calculate $H_{S/I''}$ as mathlove has requested in the comments.
First, $S/I^{''} = k[x_1, x_4] + k[x_3, x_4]$ as you wrote is not correct way to start. The point here is the monomials of degree $d$ in $S/I^{''}$ are of either one of the form $x_1^ax_2^bx_4^c$, $x_2^ax_3^bx_4^c$ or $x_3^ax_1^bx_4^c$.
Now, using inclusion-exclusion principle as we have used here: The Hilbert function and polynomial of $S = k[x_1, x_2, x_3, x_4]$ and $I = (x_1x_3, x_1x_4, x_2x_4)$ step clarification.
1/ The number of monomials $x_1^a x_{2}^b x_4^c$ that has degree $d$ is $d+2 \choose 2$ which is $\frac{(d+1)(d+2)}{2}$. These monomials include monomials of the form $x_2^bx_4^c$ and $x_1^ax_4^c$ of degree $d$ (when $a=0$).
2/ The number of monomials $x_2^a x_3^b x_4^c$ that has degree $d$ is $d+2 \choose 2$ which is $\frac{(d+1)(d+2)}{2}$. These monomials include monomials of the form $x_2^ax_4^c$ and $x_3^b x_4^c$ of degree $d$ (when $a=0$).
3/ The number of monomials $x_3^a x_1^b x_4^c$ that has degree $d$ is $d+2 \choose 2$ which is $\frac{(d+1)(d+2)}{2}$. These monomials include monomials of the form $x_3^ax_4^c$ and $x_1^b x_4^c$ of degree $d$ (when $a=0$).
Now, what are monomials of degree $d$ that we have counted twice? They are precisely $x_2^ax_4^b$ (from 1/ and 2/); $x_3^ax_4^b$ (from 2/ and 3/) and $x_1^ax_4^b$ ((from 3/ and 1/), so we have to substract those. How many of those? That is $3{d+1 \choose 1}$ (there are ${d+1 \choose 1}$ monomials $x_2^ax_4^b$ and so on).
However, when subtracting, we subtract more than we should, we have subtracted the monomial $x_4^d$ from our counting, so we have to add this back.
Combine all of these, the number that we want is $$3{d+2 \choose 2}-3{d+1 \choose 1}+1.$$
