This thought came to me yesterday when I was doing a simple first moment approximation of a function of a discrete random variable, using the standard Taylor expansion. For a discrete random variable, strictly speaking we should be working with finite difference, so why would using the Taylor expansion which assumes a continuous variable work? Are there convergence results showing that this method works just as well for discrete vs. continuous r.v.? or is there actually a difference in how well the approximation works depending on whether the r.v. is discrete vs. continuous?

As requested, here is a specific example to illustrate the question.

Consider discrete random variable $X$ with known mean $\mu$ and known variance $\sigma^2$, and we'd like to approximate $\mathbb{E}[g(X)] = \mathbb{E}[\exp(-X)]$.

A standard approach is to use Taylor expansion for the function $g$ evaluated at $\mu$ to approximate $\exp(-X)$,

$$\mathbb{E}[g(X)] = \mathbb{E}[\exp(-X)] \approx g\left(\mu\right)+\frac{g^{\prime \prime}\left(\mu\right)}{2} \sigma^{2} = \exp(-\mu) + \frac{\exp(-\mu)}{2}\sigma^2$$

I can see how this is perfectly reasonable if $X$ is a continuous r.v., say defined on $[0, \infty)$. But it seems there is a technical issue with taking derivative if $X$ is discrete, say $X$ is defined only on non-negative integers, i.e. $X \in \{0, 1, 2, ...\}$, as the usual derivative can only be applied on continuous domain.

  • $\begingroup$ This is just a guess: I suspect that it's justified because of the function you were using, not the random variable itself -- but it's hard to tell since this post is light on details. Any chance you could update the question with a bit more specificity about the approximation you did? $\endgroup$ – Aaron Montgomery Apr 1 '20 at 17:51
  • 1
    $\begingroup$ Sure, I'll add in details. Sorry I kept the question general on purpose, as I don't see this distinction being brought up in any text I've studied from, regardless of the specific function or random variable, namely the distinction of discrete vs continuous in taylor expansion approximations $\endgroup$ – barnacleboy Apr 1 '20 at 18:17
  • $\begingroup$ No worries! I think the reason you haven't seen that distinction is likely because most reasonable functions you'd write down tend to be analytic on their domains (e.g. $e^x, \sqrt x$), and it's that, not anything about the random variable, that's the crux. But again, I'm kind of guessing here; hopefully I will be able to comment further on a longer post. $\endgroup$ – Aaron Montgomery Apr 1 '20 at 18:23

The appropriateness of the use of the Taylor series in this case (and in every case where it should be used) hinges on the function, not on the random variable.

To clarify the issue, consider the following statement:

$e^{0.5} \approx 1 + \frac{0.5}{1!} + \frac{0.5^2}{2!} + \frac{0.5^3}{3!}$

This statement holds because $e^x$ is a nice function, not because there's anything special about the number $0.5$. Note that no other numbers need to be in consideration at all for the statement to make sense.

Random variables work in the same way. For a given $\omega \in \Omega$, the quantity $X(\omega)$ is a fixed number, and the Taylor approximation holds just like it did for the example of $0.5$ above. You can make this claim simultaneously across many values of $\omega$ to get an approximate equality involving the random variable, and then you can apply the linearity of expectation to get the desired result.

  • $\begingroup$ Thank you, ok it makes sense that we are just replacing whatever function we have with an approximate version, so it should only depend on the property of the function. But just intuitively, it feels like for continuous variable, this moment approximation should somehow be "better" than for discrete variable, where each possible value is further apart. Am I making any sense? For Taylor approximation, we expect more accurate result if $\text{Var}(X)$ is small, more density is centered near the average, but for discrete variable, the density is not as bunched up. I hope I'm making some sense. $\endgroup$ – barnacleboy Apr 1 '20 at 18:50
  • 1
    $\begingroup$ I get your question, but: focus again on the example of approximating $e^{0.5}$ to get the right perspective here. The reason it's useful to have $var(X)$ be small is that we prefer the variable to not be far from $0$ (or more typically, its mean), because the standard Taylor approximation breaks down for inputs for from $0$. It again comes down to the essential properties of $e^x$ and the quality of the Taylor approximation, not to anything particular about the variable itself (other than that we want it to live where the Taylor approximations are good). $\endgroup$ – Aaron Montgomery Apr 1 '20 at 18:53
  • 1
    $\begingroup$ Got it, I think my question has been answered, so I'll accept your answer. I guess it just comes down to the moments of random variable in determining how well the approximation is, not so much the discrete vs continuous nature $\endgroup$ – barnacleboy Apr 1 '20 at 19:01
  • $\begingroup$ Exactly right. ${}$ $\endgroup$ – Aaron Montgomery Apr 1 '20 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.