# Uniform Law of Large numbers for independent but non-identical random matrices

Suppose $W_i=\frac{\delta_i}{\pi_i} S^\prime_i(y_i,\vec\theta)$, where $\delta_i$ is iid Bernoulli r.v, $y_i$ is fixed variable and $\vec \theta$ is a vector of parameters. $S^\prime_i(y_i,\vec\theta)=\frac{\partial}{\partial\vec \theta}S_i(y_i,\vec\theta)$. Suppose the vector $$S_i(y_i,x_i,\vec\theta) = \begin{bmatrix} y_i-\theta_1 &\\(y_i-\theta_1)^2-\theta^2_2 \end{bmatrix}$$ then

$$\frac{\partial}{\partial \vec \theta} S_{i}(y_i,\vec \theta) \text{ will be a matrix of order } 2 \times 2 \text{ of fixed quantities }.$$ Now $W_i$ is independent but non-identical random matrices because $E(W_i) = E(\delta_i/\pi_i) S^\prime_{i} (y_i,\vec \theta)=S^\prime_{i}(y_i,\vec \theta)$ since $E(\frac{\delta_i}{\pi_i}|\pi_i)=1$ and $y's$ are fixed. I need to show that $$S_N=\frac{1}{N}\sum_{i=1}^{N} W_i \text{ convergence in probability to } E(S_N) \text{uniformly}.$$ Any quick help would be greatly appreciated.

• This question cannot be answered because the descriptions of $\theta$ and $S_i(\theta)$ are unclear. [Why is a "matrix of fixed quanties" indexed by both $\theta$ and $i$?] [How can we verify the "Hence $W_i$ is independent but non-identical"?] – Michael Dec 5 '16 at 23:58
• Thanks Michael. I added the detail. Please now have a look. – ZAH Dec 6 '16 at 15:02
• Some details seem irrelevant (such as the $\theta$ vector). Yet, you do not specify anything about the $y_i$ sequence. There is no reason for this to converge to anything. It looks like you have a deterministic sequence of matrices $\{M_i\}_{i=1}^{\infty}$ and an i.i.d. Bernoulli sequence $\{\delta_i\}_{i=1}^{\infty}$, and you are studying $\frac{1}{n}\sum_{i=1}^n \delta_i M_i$. Consider the special case when $P[\delta_i=1]=1$ for all $i$. Well, why should the time average $\frac{1}{n}\sum_{i=1}^n M_i$ converge? – Michael Dec 6 '16 at 17:28
• Also, it does not make sense to say "convergence in probability to $E[S_N]$" since that is parameterized by $N$. We can converge to a constant or to a random variable, not to a sequence. I assume you really want to study $\frac{1}{n}\sum_{i=1}^n [\delta_i M_i - E[\delta_i M_i]]$. – Michael Dec 6 '16 at 17:34
• $E[S_N]=\sum_{i=1}^{N}S^\prime_i(y_i,\theta)$ is constant in my case because for finite population of size N, $E[S_N]$ is a constant. $\frac{1}{N}\sum_{i=1}^{N}[\delta_iM_i-E[[\delta_iM_i]]$ converge to zero in probability uniformly can also serve my purpose as you assumed in your above comment. – ZAH Dec 6 '16 at 20:44

If we have a sequence of (finite sized) matrices $(a_{ij}(n))$ such that $\lim_{n\rightarrow\infty} a_{ij}(n) = 0$ for all entries $(i,j)$, then the entries indeed converge uniformly to 0 (uniform over all finite entries). So you can reduce the problem to scalars.
If $\{X_i\}_{i=1}^{\infty}$ is a random process of mutually independent random variables, and if $\{a_i\}_{i=1}^{\infty}$ is a deterministic sequence of real numbers, then we can define $S_n = \frac{1}{n}\sum_{i=1}^n X_i a_i$. Then $S_n - E[S_n]$ will converge to 0 in probability (as $n\rightarrow \infty)$ under some mild assumptions on the random process $\{X_i\}$ and on the sequence $\{a_i\}$. Just compute the variance of $S_n - E[S_n]$ to derive the properties that you need.
In general, the quantity $S_n - E[S_n]$ will not converge to 0 in any sense. For example, if $\{X_i\}$ is i.i.d. Bernoulli with $P[X_i=0]=P[X_i=1]=1/2$, and if $a_i = 2^i$, you can see that $S_n$ will jump whenever $X_n=1$ and will never converge to anything.