In the linear regression model $$y = \beta_1 X_1 + \cdots + \beta_p X_p + \varepsilon \, ,$$ can the regressors $\{X_i\}_{i \in \{1, \ldots, p\}}$ be considered as random variables?

I know that what we really have is $n$ observations of the relationship between the dependent variable and the regressors, i.e., $$y_i = \beta_{i1} x_{i1} + \cdots +\beta_{ip} x_{ip} + \varepsilon_i \, ,$$ but does it make sense to see things like $\Pr[X_1 = 0]$, where $X_1$ is the same $X_1$ as in the first equation? Would that be equal to $\frac{1}{n} \sum_{i = 1}^n [x_{i1} = 0]$ or does it refer to the probability according to a theoretical distribution?


Yes, it is meaningful to imagine the regressors are sampled from a theoretical distribution. The quantity $P(X_1=0)$ would then refer to the theoretical distribution for $X_1$, not to the observed values $x_{1,1},\ldots,x_{1,p}$ of $X_1$. Note that in elementary treatments of regression we assume the regressors are non-random, i.e., they are known constants (equivalently, we are conditioning on the the values of the regressors).

  • $\begingroup$ So, if the theoretical distribution of $X_1$ were known, it would be preferable to use it in order to calculate $\Pr[X_1 = 0]$ than just simply count how many times $x_{i1} = 0$ (for all $i \in \{1, \ldots, n\}$) and divide it by $n$, right? $\endgroup$ – Cromack Apr 14 '16 at 18:44
  • $\begingroup$ If the theoretical distribution of $X_1$ were known, then you should use it to calculate $P(X_1=0)$. The empirical frequency is only an estimate of the true probability. Then again, I don't know how complicated the theoretical calculation would be, or whether you're really being asked for the theoretical prob vs the empirical frequency. $\endgroup$ – grand_chat Apr 14 '16 at 19:22
  • $\begingroup$ I am asked about the probability that two given predictors $X_1$ and $X_2$ are collinear, which may be different if calculated by using the theoretical or the empirical distributions. $\endgroup$ – Cromack Apr 14 '16 at 19:32
  • $\begingroup$ That probability sounds like a theoretical calculation. If you've got one data set of $n$ observations, then the predictors $X_1$ and $X_2$ are either collinear or not. $\endgroup$ – grand_chat Apr 14 '16 at 19:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.