Assuming you've mis-written the definition of $S$, and it's just all ordered pairs of integers, then answer is 5. Consider just $x$-coordinates: once you have two of the same parity (both even, both odd), their average will be an integer. The same goes for the $y$-coordinates.
But you need the average to be an integer for both $x$ and $y$ at the same time. All you now need to do is write down all possibilities in the form
If any one of these appears twice in your set, you're done. If you have just the first three, then you're NOT done. But if you have all 4, then there are still no pairs whose average is in $S$. To be concrete: the four points $(0,0), (1, 0), (0, 1),$ and $(1,1)$ have the property that no average of two of them has integer coordinates.
But if you add one more pair, it'll have to match one of these four, so you've found a pair whose average is in $S$.
[Apologies for writing "4" before -- I mis-read your question as "what's the largest number of points you can have with NO averages being in $S$?"]