Suppose we are doing multiple linear regression:

$\boldsymbol{Y} = X\boldsymbol{\beta}+\epsilon$


$\epsilon \sim N(0,\bar{\Sigma})$

Now, suppose that one of the coefficients is defined using some function, for example

$\beta_1 = ax+bx^2$

where $x$ is some quantity of interest, for example a x-coordinate or something.

Now we can estimate $a,b$ and any other coefficients by maximising the log-likelihood of the normal distribution with respect to these parameters. Using these estimates for $a$ and $b$ we can get estimates for $\beta_1$.

We could also find the covariance matrix between $a,b$ and all other coefficients except $\beta_1$ using the Information matrix. Let this covariance matrix be denoted $\Sigma$.

Now my question is how to we obtain the covariance matrix between all the beta coefficients?

If the transformation of $\beta_1$ is linear in $a$ and $b$ (like above) then we can find the desired covariance matrix using

$A\Sigma A^T$

where A is a matrix describing the linear transformation. However what do we do if the transformation is non-linear in $a$ and $b$?

I know the delta method exists and I can obtain the variance in the $\beta_1$ but how can this be extended to find the covariance between say $\beta_1$ and $\beta_2$?


Suppose instead of estimating $(\beta_0 \beta_1 \beta_2...,\beta_p)$ we want to estimate

$(\beta_0 , \beta_1=exp(ax^2) , \beta_2...,\beta_p)$

where data has been generated using $\beta_1=exp(ax^2)$. Using the log-likelihood we can estimate the most likely values of

$(\beta_0 , a , \beta_2...,\beta_p)$

and we can use the Information matrix to find the covariance of these parameters. My question is how to find the covariance matrix of

$(\beta_0 , \beta_1 , \beta_2...,\beta_p)$.

If $\beta_1$ was linear in a, we could use some transformation of the normal random variable by multiplying by a matrix A to get the covariance matrix $A\Sigma A^T$, but what now when $\beta_1$ is not linear in a?


A general delta method should get you the whole variance-covariance of $\hat{\beta}$, or if you're using MLE you could use the standard score function approach. For example, you can look up how the standard errors of a logistic or a probit regression are calculated, which are both canonical examples of equations that are non-linear in $\beta$.

More specifically, I'm not sure I understand your restriction on $\beta_1$. Could you edit your question to give a more concrete or clear example?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.