Let $$X$$ be a random variable. According to CLT, the distribution of means of samples of $$X$$ converges to normal as the sample size grows.

Suppose we sample $$X$$ $$n$$ times and observe data $$[x, x, x, ..., x, x+\epsilon]^T$$ (that is, $$n-1$$ times $$x$$ and $$x + \epsilon$$ once, where $$x \in \mathbb{R}$$ and $$\epsilon > 0$$). If we compute the one sample $$t$$ statistic as

$$t = \frac{\bar{X} - \mu}{s /\sqrt{n}}$$

setting $$\mu = x$$ ($$H_0$$ is that $$\mu = x$$) and compute the p-value, surprisingly enough, with increasing $$n$$, we observe decreasing $$p$$. As far as I know, the conclusion of this observation is that the larger the sample is (and therefore the more $$x$$'s we observe), the less likely $$x$$ is to be the true mean of $$X$$. But why? This feels counter-intuitive. Can anyone explain?

Given the sample, it is easy to show that $$\bar x = x + \frac{\epsilon}{n}, \quad s^2 = \frac{\epsilon^2}{n}.$$ Then the test statistic for the hypothesis $$H_0 : \mu = x$$ is $$T \mid H_0 = \frac{\bar x - x}{s/\sqrt{n}} = \frac{\epsilon/n}{\epsilon/n} = 1.$$ This is independent of the sample size $$n$$. Since $$T$$ is Student $$t$$-distributed with $$n-1$$ degrees of freedom, the limit as $$n \to \infty$$ is standard normal, thus $$\Pr[T > 1 \mid H_0] > \Pr[Z > 1 \mid H_0] \approx 0.158655.$$ This furnishes a strict lower bound on the $$p$$-value of this test for this kind of observed data, and you would never reject $$H_0$$ for any reasonable $$\alpha$$.
While it is true that $$p$$ decreases as an increasing function of $$n$$, despite the strength of evidence seemingly suggesting that the true mean is $$x$$, what the $$p$$-value is reflecting is the increased sensitivity of the test to deviation from the null mean as the sample size increases. In other words, there are two competing effects at work here:
1. With more observations equaling $$x$$, the sample mean is tending toward the null mean.
2. With larger sample size, the test is more likely to reject $$H_0$$ for the same value of the test statistic.