# Intuitive explanantion for random vectors and estimation theory

Suppose $X$ is a random vector and $X \sim f_\theta(x)$. I have a question about random vectors.

Q1. Why do we consider each entry of a random vector to be $X_i \sim f_\theta(x_i)$? I think I can visualise it as being a vector in space, so its exact position is random. Each of its coordinates will be randomly chosen. Each entry has a probability distribution. Is this the correct intuition? I think I am getting confused by the use of $f_\theta(x_i)$, so can someone explain?

Q2. Further, is there some intuition behind why joint probability density becomes the product when each entry is independent?

Q3. Why is each observation considered to be an entry of a random vector during parameter estimation? In this case, why can't each entry be drawn simply from a distribution $X_i \sim f_\theta(x_i)$ which is exactly the same for all $i$? Why do we assume that the random variable is different for different measurements?

Lots of vague questions here. I'll try to interpret and give useful answers.

$1.$ You should begin by visualizing a population (or process) from which values $X_i, X_2, \dots, X_n$ are sampled sequentially and at random. The density or point mass function $f$ describes the distribution of the population. The argument of $f$ may be written as $x_i,$ but that is a sometimes useful, sometimes confusing, convention. For example, one rule is a density function has $\int_{-\infty}^\infty f(x)\,dx = 1.$ But writing $\int_{-\infty}^\infty f(t)\,dt = 1$ or $\int_{-\infty}^\infty f(\xi)\,d\xi = 1$ means exactly the same thing.

$2.$ If we have $X_1$ with $P(X_1 = 0) = f_{X_1}(0) = 1/2$ and $P(X_1 = 1) = f_{X_1}(1) = 1/2$ and similarly for independent random variable $X_2,$ then you may see something like $$P(\{X_1 = 1\}\cap\{X_2 = 0\}) = P(X_1 = 1, X_2 = 0) = P(X_1 = 1)P(X_2 = 0)\\ = f_{X_1}(1)f_{X_2}(0) = f_1(1)f_2(0) = f(1)f(0) = 1/4,$$ where the second term is a simplifying convention, the third is because of independence, the fourth uses the PMF notation, and the last is a lazy abbreviation used because the two PMFs are the same. Such an equation might be summarized as $f_{X_1}(x_1)f_{X_2}(x_2) = 1/4,$ for $x_1, x_2 = 0, 1.$ Or by $f_{1}(x_1)f_{2}(x_2) = 1/4$ or $f(x_1)f(x_2) = 1/4,$ depending on how much the author chooses to simplify the notation. Similar simplified notation is used for PDFs (density functions of continuous distributions), but then expressions such as $f_{X_1,X_2}(x_1,x_2) = f_{X_1}(x_1)f_{X_2}(x_2)$ or $f_{1,2}(x_1,x_2) = f_{1}(x_1)f_{2}(x_2)$ or $f(x_1,x_2) = f(x_1)f(x_2)$ need to be justified by explanations or proofs.

Different authors take different degrees of care when introducing simplified notation, and students need to be very attentive to what is going on as each simplification is explained or introduced. You might wonder why the simplifications are used, but once you get used to them, you will probably agree they make for easier reading. And just in typing this, I have been reminded that simplifications also help cut down on typographical errors.

$3.$ It is not always necessary or even helpful to view observations $X_1, X_2, \dots, X_n$ as a vector. (But, if you are using certain kinds of software, you may see some 'vector' notation from the start.) Here are a couple of fairly elementary examples in which viewing data as a vector may be useful:

(a) We may say that $(X_1, X_2, \dots, X_n)$ is a vector in $n$-space. Then the sample mean $\bar X = \frac{1}{n}\sum_i X_i,$ imposes one linear restriction on the data and may be considered to 'use' one dimension. Then one reason that the the sample variance $S_X^2 = \frac{1}{n-1}\sum_i (X_i = \bar X)^2$ has $n-1$ in the denominator is that estimating the variance 'uses' the remaining $n - 1$ dimensions. For normal data, this point of view helps explain why $\bar X$ and $S_X^2$ are (stochastically) independent random variables---even though they are obviously not functionally independent because $\bar X$ appears in the definition of $S_X^2.$

(b) Consider a balanced one-factor ANOVA design with $g$ treatment groups and $r$ replications in each group, for a total of $gr$ observations ($gr$ dimensions). The degrees of freedom appearing in a standard ANOVA table refer to the dimensionalities of orthogonal sub-spaces of dimentionalities $g-1$ (for factor) and $g(r-1)$ (for error). This amounts to $(g-1) + g(r-1) = gr -1$ dimensions. The remaining dimension is for the grand mean of all $gr$ observations. When you add SS(Factor) + SS(Error) = SS(Total), that is the Pythagorean Theorem in $gn-1$ dimensions. This viewpoint is useful in deriving the distribution of the $F$-statistic used for testing whether treatment groups have equal population means.