# Difference between two Gaussian stochastic variables

Suppose I have a sequence of time steps $$t_1 < t_2 < t_3< \dots. To each time step corresponds a random Gaussian variable $$X_1,X_2,X_3,\dots,X_n$$. Since $$X = \{X_1,X_2,X_3,\dots,X_n\}$$ is normally distributed, I can say that

$$X \sim N \left( 0, \sigma^{2} \right)$$

where the distribution of $$X$$ has zero mean (assumption) and standard deviation $$\sigma$$. If I define some stochastic variable $$Y_j$$ such that

$$Y_j = X_{j} - X_{j-1}$$

with $$Y=\{Y_1,Y_2,\dots, Y_n\}$$ for $$2 \leq j \leq n$$, what can I say about the distribution of $$Y$$? Naively, I'd think that

$$Y \sim N \left( 0, \sigma^{2} \right)$$

which means that $$Y$$ is distributed the same way as $$X$$. My intuitive understanding is that since any $$X$$ is normally distributed, then any linear combination of $$X$$ is also normally distributed with the same mean and variance. Is this correct? If so, how can I show this rigorously?

• Use the definition of $Y_j$ and the definition of variance. Commented Oct 20, 2022 at 22:09
• Are the $X_i$ mutually independent? Otherwise it gets a lot more complicated fast. Commented Oct 20, 2022 at 23:43
• @Matija Yes. They are independent and identically distributed random variables Commented Oct 21, 2022 at 14:05
• @RodrigodeAzevedo Can you elaborate on what you mean? Commented Oct 21, 2022 at 14:06
• @kowalski Compute $\Bbb E \left( Y_k \right)$ and $\Bbb E \left( Y_k^2 \right)$. Commented Oct 21, 2022 at 14:40

Let $$X\sim\mathcal N(0,\sigma^2I_n)$$ be a vector of $$n$$ IID centered normal variables with variance $$\sigma^2$$ and let $$Y=AX$$, where $$I_n$$ is the identity matrix and $$A=\begin{pmatrix} 1 & 0 & 0 & \dots & \dots & 0 & 0 & 0\\ -1 & 1 & 0 & \dots & \dots & 0 & 0 & 0\\ 0 & -1 & 1 & \dots & \dots & 0 & 0 & 0\\ 0 & 0 & -1 & \ddots & \dots & 0 & 0 & 0\\ \vdots & \vdots & \vdots & \ddots & \ddots & \vdots & \vdots & \vdots\\ 0 & 0 & 0 & \dots & \ddots & 1 & 0 & 0\\ 0 & 0 & 0 & \dots & \dots & -1 & 1 & 0\\ 0 & 0 & 0 & \dots & \dots & 0 & -1 & 1 \end{pmatrix}.$$ This gives $$Y_1=X_1$$ and $$Y_j=X_j-X_{j-1}$$ otherwise. Using integration by substitution for $$X=A^{-1}Y$$ (notice that $$|A|=1$$) and $$x^{\mathrm t}$$ to denote the transpose, we have \begin{aligned} \mathbb P(Y\in\mathcal E)&= \int\unicode{120793}\{Ax\in\mathcal E\}\frac{1}{\sqrt{(2\pi)^n\sigma^2}}e^{-\frac{1}{2}x^{\mathrm{t}}(\sigma^2I_n)^{-1}x}\mathrm dx\\ &=\int\unicode{120793}\{x\in\mathcal E\}\frac{|A^{-1}|}{\sqrt{(2\pi)^n\sigma^2}}e^{-\frac{1}{2}x^{\mathrm{t}}\Sigma^{-1}x}\mathrm dx, \end{aligned} where $$\Sigma^{-1}=(A^{-1})^{\mathrm{t}}(\sigma^2I_n)^{-1}A^{-1}=(A(\sigma^2I_n)A^{\mathrm t})^{-1}=(\sigma^2AA^{\mathrm{t}})^{-1}$$. Recall the multiplicativity of the determinant, which gives $$|A^{-1}|=1$$, and thereby $$Y\sim\mathcal N(0,\sigma^2AA^{\mathrm{t}})$$. This shows that $$Y$$ is a multivariate normal vector. Clearly, we want to know the covariance exactly, so $$\Sigma=\sigma^2AA^{\mathrm{t}}=\sigma^2 \begin{pmatrix} 1 & -1 & 0 & \dots & \dots & 0 & 0 & 0\\ -1 & 2 & -1 & \dots & \dots & 0 & 0 & 0\\ 0 & -1 & 2 & \dots & \dots & 0 & 0 & 0\\ 0 & 0 & -1 & \ddots & \dots & 0 & 0 & 0\\ \vdots & \vdots & \vdots & \ddots & \ddots & \vdots & \vdots & \vdots\\ 0 & 0 & 0 & \dots & \ddots & 2 & -1 & 0\\ 0 & 0 & 0 & \dots & \dots & -1 & 2 & -1\\ 0 & 0 & 0 & \dots & \dots & 0 & -1 & 2 \end{pmatrix}.$$ Normal variables are independent iff they are uncorrelated. Thus, for example, $$Y_n$$ is independent of $$(Y_m)_{m\le n-2}$$, but clearly, the components of $$Y$$ are not mutually independent (since not all off-diagonal entries are $$0$$).

• Thank you very much for your answer. I think this is what I'm looking for but I'm having some difficulties understanding it. Firstly, I understand that you used integration by substitution. I know the expression for multivariate PDF, can you elaborate what happened in the first line of the integral going to the second? Commented Oct 31, 2022 at 18:51
• The substitution (the injective continuously differentiable map, or diffeomorphism) is $y\mapsto A^{-1}y$, with inverse $y=Ax$. The theorem tells us that when we replace $x$ by $A^{-1}y$ everywhere, we have to multiply the determinant of the Jacobian. But the Jacobian of the linear function $A^{-1}y$ is just $A^{-1}$, which explains the factor $|A^{-1}|$. When we replace $x$ in the exponent, we get $(A^{-1}x)^t(\sigma^2I)^{-1}A^{-1}x$. Then we have $(A^{-1}x)^t=x^t(A^{-1})^t$, which explains $\Sigma$. Commented Oct 31, 2022 at 19:09
• Thank you. I have had a more thorough understanding of your answer once I saw (probabilitycourse.com/chapter6/6_1_5_random_vectors.php)[this] article. One other question, how should I interpret the elements of the covariance matrix $\Sigma$? The diagonals are the variances between $X_{i},X_{i}$ or $Y_{i},Y_{i}$? Are there any $Y$ components in the $\Sigma$? Commented Oct 31, 2022 at 20:47
• The matrix $\Sigma$ is the covariance matrix of $Y$. The diagonal entry $\Sigma_{ii}=\mathbb E[Y_i^2]$ for $i=j$ is the variance of $Y_i$ (becuase the expectation is $0$). The entry $\Sigma_{ij}=\mathbb E[Y_iY_j]$ for $i\neq j$ is the covariance of $Y_i$ and $Y_j$. If this is $0$, then the two are uncorrelated, and because this is a Gaussian distribution, also independent. Commented Oct 31, 2022 at 20:58

Credits to @Matija for the answer, but here's my shorter take on it.

Suppose $$X\sim N(0,1)$$. If $$Y=AX+b$$, then $$Y\sim N(b,A^{T}A)$$

Proof:

First, denote the PDF of $$X$$ as $$f_{X}(x)$$ such that

$$f_{X}(x) = \frac{1}{\sqrt{(2\pi)^{n}}}\text{Exp}\left(-\frac{1}{2\sigma^{2}}X^{T}X\right)$$

One can ask, given $$f_{X}(x)$$, what is the PDF of $$Y$$? Namely $$f_{Y}(y)$$? I claim (without proof) that if $$Y=f(X)$$, then injectively, $$X=f^{-1}(Y)=G(Y)$$ where $$f^{-1}\equiv G$$. Then

$$f_{Y}(y) = f_{X}(G(y))|J|$$

where $$J$$ is the Jacobian, with matrix elements defined as

$$J_{ij}=\frac{\partial{G_{i}}}{\partial Y_{j}}$$

Since $$Y$$ is a simple linear combination of $$X$$, $$J$$ can be easily computed:

$$Y = Ax+b \\ X = A^{-1}(Y-b) \equiv G(Y)$$

Hence $$\partial G/\partial Y = A^{-1}$$ and $$|J| = |A^{-1}| = |A|^{-1}$$. Since we know the PDF for $$X$$, we can calculate $$f_{X}(G(y))$$:

$$f_{X}(G(y)) = \frac{1}{\sqrt{(2\pi)^{n}}}\text{Exp}\left(-\frac{1}{2\sigma^{2}}\left(A^{-1}(Y-b)\right)^{T}\left(A^{-1}(Y-b)\right)\right) \\ =\frac{1}{\sqrt{(2\pi)^{n}}}\text{Exp}\left(-\frac{1}{2\sigma^{2}}\left((Y-b)^{T}A^{-T}\right)\left(A^{-1}(Y-b)\right)\right)\\ =\frac{1}{\sqrt{(2\pi)^{n}}}\text{Exp}\left(-\frac{1}{2\sigma^{2}}(Y-b)^{T}(A^{T}A)^{-1}(Y-b)\right)\\ =\frac{1}{\sqrt{(2\pi)^{n}}}\text{Exp}\left(-\frac{1}{2\sigma^{2}}(Y-b)^{T}(C)^{-1}(Y-b)\right)$$

where $$C=A^{T}A$$. Note that

$$|C| = |A^{T}A| = |A^{T}||A|=|A|^{2}$$

hence

$$|A| = \sqrt{|C|} \longrightarrow |A|^{-1} = \frac{1}{\sqrt{|C|}}$$

Therefore, the final PDF of $$Y$$ gives

$$f_{Y}(y) = f_{X}(G(y))|J|\\ =\frac{1}{\sqrt{|C|}}f_{X}(G(y))\\ =\frac{1}{\sqrt{(2\pi)^{n}|C|}}\text{Exp}\left(-\frac{1}{2\sigma^{2}}(Y-b)^{T}(C)^{-1}(Y-b)\right)$$

But this is simply the PDF for a multivariate distribution with mean $$b$$ and covariance matrix $$C$$. Hence I can say that

$$Y\sim N(b,C)$$