Suppose you have $N_1$ numbers, $x_1,x_2,\cdots,x_{N_1}$. Their average is
$$\langle x\rangle=\frac{\sum_{i=1}^{N_1}x_i}{N_1}$$
Similarly, the average of $N_2$ numbers $y_1,y_2,\cdots,y_{N_2}$ is
$$\langle y\rangle=\frac{\sum_{i=1}^{N_2}y_i}{N_2}$$
On combining the observations, by definition, the average comes to be
$$\langle z\rangle=\frac{\sum_{i=1}^{N_1}x_i+\sum_{i=1}^{N_2}y_i}{N_1+N_2}$$
However, the average of the averages is
$$\langle z\rangle '=\frac{\frac{\sum_{i=1}^{N_1}x_i}{N_1}+\frac{\sum_{i=1}^{N_2}y_i}{N_2}}{2}$$
which is not the same as given by the definition. This discrepancy is because the number of observations that lead to each average is different. Note that if $N_1=N_2$, both become equivalent.
Instead, we need to take the number of observations into account. As the average is given by the sum of observations by the total number of observations, we need to know the sum of observations. Clearly, this can be obtained as
$$\text{Sum of observations}=N_1\langle x\rangle+N_2\langle y\rangle$$
$$\text{Total number of observations}=N_1+N_2$$
Thus,
$$\langle z\rangle=\frac{N_1\langle x\rangle+N_2\langle y\rangle}{N_1+N_2}$$
Note that average of $\langle x\rangle$ and $\langle y\rangle$ will be
$$\langle z\rangle '=\frac{\langle x\rangle+\langle y\rangle}{2}$$
Thus, the result is similar to taking the average, except that it appears as if $\langle x\rangle$ appears $N_1$ times and $\langle y\rangle$ appears $N_2$ times. This is known as the weighted average, i.e., the contribution of $\langle x\rangle$ and $\langle y\rangle$ to the result depends on their "weight", which is the number of elements used to obtain them.
In some sense, the average of all numbers is the average of averages, with the constraint that the "weight", or contribution of each average is taken care of.