Difference between $P(Y \ge y)$ and $P(Y > y)$. In the book "Probability: Theory and Examples", there is this theorem:

Lemma 2.2.8 If $Y \ge 0$ and $p > 0$ then $\mathbb E(Y^p) = \int_0^\infty py^{p-1} P (Y > y) \,  \mathrm dy$.

The proof is by using Fubini's Theorem as follows
\begin{align*}
\int_0^\infty py^{p-1} \mathbb P (Y > y)  \mathrm dy
& = \int_0^\infty \int_\Omega py^{p-1} 1_{(Y > y)} \, \mathrm dP \, \mathrm dy \\
& = \int_\Omega \int_0^\infty py^{p-1} 1_{(Y > y)} \, \mathrm dy \, \mathrm dP \\
& = \int_\Omega \int_0^Y py^{p-1} \mathrm dy \mathrm dP  \\
& = \int_\Omega Y^p \mathrm dP = \mathbb E Y^p \\
\end{align*}
For me it seems totally fine to replace $Y > y$ with $Y \ge y$ in this proof, but the conclusion would be $\mathbb E(Y^p) = \int_0^\infty py^{p-1} P (Y \ge y) \,  \mathrm dy$. So if $Y$ can only take nonnegative integer values, we have $\mathbb E Y = \sum_{y \ge 0} P(Y \ge y)$, which is wrong.
So why we can not replace $Y > y$ with $Y \ge y$?
 A: Lemma 2.2.8 is equally valid when one replaces $P[Y\gt y]$ in the integral by $P[Y\geqslant y]$ because $P[Y\gt y]=P[Y\geqslant y]$, for almost every $y$ with respect to the Lebesgue measure, since the set of values of $y$ such that $P[Y\gt y]\ne P[Y\geqslant y]$ is at most countable. 
This holds for every nonnegative random variable $Y$, either continuous or discrete or neither continuous nor discrete.
A: There is nothing wrong with: $\mathbb E Y = \sum_{y \ge 0} P(Y \ge y)$ as long as $Y$ takes non-negative integer values. See the Wikipedia section on Discrete distribution taking only non-negative integer values.
A: As observed by $\textbf{Did}$, the lemma is equally valid when $P(Y > y)$ is replaced by $P(Y \ge y)$, since $P(Y > y)=P(Y \ge y)$ almost everywhere; this applies to every non-negative random variable $Y$.
In general, however, $P(Y \ge y)$ and $P(Y > y)$ are equivalent if and only if $P(Y = y)=0$, which is always true for continuous random variables, since a single point leaves the integral unaffected. For discrete random variables, this is not the case, since a point does affect the sum.
Finally, I am not sure where your formula for the expectation of a non-negative discrete random variable $Y$ comes from; as you noticed, it is easy to derive that
$$\operatorname{E}[Y] = \sum\limits_{y=1}^\infty P(Y \ge y).$$
