Tag Info

New answers tagged

0

Let's try this. \begin{align} \sum_{i=1}^N \mathbf{X}_i^{\top}\mathbf{\Omega}^{-1}\mathbf{X}_i&=\mathbf{X}_1\mathbf{\Omega}^{-1}\mathbf{X}_1+\mathbf{X}_2\mathbf{\Omega}^{-1}\mathbf{X}_2+\cdots+\mathbf{X}_N\mathbf{\Omega}^{-1}\mathbf{X}_N \\ &=\begin{bmatrix} \mathbf{X}_1^{\top} & \mathbf{X}_2^{\top} & \cdots & \mathbf{X}_N^{\top} ...


0

I assume the residuals you are referring to come from a linear regression model of some sort. Then, in that case, the residual standard error is the MSE (mean square error - you can Google it and easily find the formula, although it is just the square root of the sum of the squared residuals divided by (n-k-1)). Note, it is a statistic computed from a given ...


0

If you are going into big data, you should also take a data mining/machine learning course - its essential for actually getting the samples to do stats on. Also, you need to include a forecasting course. Your list is covers the other main areas. You'll also want to become familiar with R if you are not already.


2

It is generally true that if $A$ is an $n\times n$ invertible and if $A^{-1}$ has a "square root" $C$, also $n\times n$, such that: $$ A^{-1} = C^2 $$ then $C^{-1} A^{-1} C^{-1} = I$ holds. The first fact we need is that since $A$ is invertible, $A^{-1}$ is invertible, and this implies $C$ is invertible. For if not, then there would exist a nonzero ...


1

We know \begin{equation} A^{-1}=CC \end{equation} Inverting both sides (Is this allowed? Depends on whether $C$ is invertible, which I do not know...), we would have \begin{equation} A=C^{-1}C^{-1} \end{equation} Multiply both sides on the left by $C$, and then on the right by $C$, so we get \begin{equation} CAC=I \end{equation} or, in the "$1/2$" notation, ...


0

\begin{align*} E(\hat\beta) &= E\left(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 + \sum^{n}_{i=1}x_i\epsilon_i}{\sum^{n}_{i=1}x^2_i + \lambda}\right) = E \left(\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda} \right)=\frac{\beta_{true}\sum^{n}_{i=1}x_i^2 }{\sum^{n}_{i=1}x^2_i + \lambda} \\ var(\hat\beta) &= ...


0

The scatterplot of the Miles ($X$-axis) vs Faults ($Y$-axis) indicates that the number of faults has in most cases decreased in year 2001, the two variables (# Faults and line length) are positively related, thus the regression coefficient is positive and probably close to $15/40=3/8$ (since in $x$ we move from $10$ to $50$ and in $y$ from $15$ to ...


1

First in statistical speak: Our model is $Y_i = \beta_0 + \beta_1 X_i + \varepsilon_i$. This means that $\beta_1$ is the amount that we expect $Y$ to increase by when $X$ increases by 1 [or decrease if $\beta_1 < 0$]. Now in terms of the problem: In this problem $X$ is weekly income and $Y$ is weekly consumption expenditure so $\hat{\beta_1}$ is our ...


1

Consider first $\beta>0$. First order condition with respect to $\beta$ would imply $$\frac{\partial L(\beta)}{\partial\beta}=0 \Rightarrow -\sum^{n}_{i=1} 2(y_i-\beta x_i)x_i +\lambda=0\Rightarrow \sum^{n}_{i=1}(y_i-\beta x_i)x_i=\lambda/2\Rightarrow \beta=\frac{-\lambda/2+\sum^{n}_{i=1}y_ix_i}{\sum^{n}_{i=1}x^2_i}$$ So your optimal estimate is ...


1

In linear regression usually exponential functions are logarithmized: $$ \log(1-f(x))=x/b$$ Don't know if it helps in your case. Google: Linear regression, transformations.


1

I study regression, and I used to wonder this very question myself. Now I've come to the conclusion it's because of the geometry and linear algebra behind regression. Suppose we collect data on $n$ observations and to run a regression. When we minimize the sum of squared residuals, the way we do this (using Ordinary Least suares) is via projection matrices. ...


3

Even when I'm not doing anything to do with differentiation, then I still like the squares. In fact, consider the $l_p$ norm, where you sum the $p$-powers, then raise to the power $1/p$: $$ || x ||_{l_p}^p = \sum_i |x|_i^p . $$ You can use this to "penalise" errors more aggressively. For example, when you square, it penalises more than just taking the ...


14

Actually there are some great reasons which have nothing to do with whether this is easy to calculate. The first form is called least squares, and in a probabilistic setting there are several good theoretical justifications to use it. For example, if you assume you are performing this regression on variables with normally distributed error (which is a ...


4

Have a look at a this simple example. Let our 'model' be $h(x):=\textrm{constant}$. Call this constant we are trying to estimate h. Squared case: $$\min_{h\in \mathbb{R}}\sum_{i=1}^m (h-y_i)^2$$ take the derivative with respect to $h$ and set it to $0$ $$\sum_{i=1}^m 2(h-y_i)=2h\sum_{i=1}^m 1-2\sum_{i=1}^my_i=2hm-2\sum_{i=1}^my_i=0$$ $$\Leftrightarrow$$ ...


2

One reason is the rest of calculations you need to do on the cost function. For example to minimize the cost function (e.g. in gradient descent), you need to take derivations. Taking derivations of a squared value is much more easier than an absolute value.


7

If $h(x)$ is linear with respect to the parameters, the derivatives of the sum of squares leads to simple, explicit and direct solutions (immediate if you use matrix calculations). This is not the case for the second objective function in your post. The problem becomes nonlinear with respect to the parameters and it is much more difficult to solve. But, it ...


0

Now that I understand your statement better, I think I can better answer this question. Is your model with one regressor a subset of your model with multiple regressors? E.g. are you using polynomials where the regressors are the coefficients? If so, then either your doing your fit wrong. Or, your fitting is not determining the best values. Let's think ...


0

By "Number of Regressors" do you mean number of datapoints? The Mean Square Error normalizes for the number of datapoints. In that sense, it's a comparable metric regardless of the number of datapoints that you're fitting. One issue with larger numbers of datapoints is that you may have a harder time fitting the model. For example, if your model is a ...


0

Since E(u) = 0, Cov(x,u) = E(xu')=E(E(xu'|x)). Now, E(xi uj|x)=xi E(uj|x)=0. Hence, the matrix E(xu'|x) = 0, so Cov(x,u)=0. Hence, each column of X is uncorrelated with u.


1

Model (2) has the merit that it takes into account any (linear) correlation of the result with the control variables, thus partly eliminating the confounding effect of nuisance variables. For example, if the only control variable is $X = 0$ for male and $X=1$ for female, and the experiment was performed on 7000 male subjects and only 3000 female subjects, ...


1

The concepts of "leverage" point and "influenctial" point are not equivalent. High leverage in regression analysis refers to observations that are outlying values of the independent variables. More technically, high leverage points can be defined as those having no neighbouring points in a $R^n$ space, where $n$ is the number of independent regression ...


1

$\newcommand{\var}{\operatorname{var}}\newcommand{\cov}{\operatorname{cov}}$ \begin{align} \var(y_{ij}-\bar{y}_i) & = \var(y_{ij}) -2\cov(y_{ij},\bar{y}_i) + \var(\bar y_i) \\[8pt] & = \sigma^2 - 2\frac{\sigma^2}{n_i} + \frac{\sigma^2}{n_i}. \tag{$*$} \end{align} The covariance in the second term is $$ \cov\left( y_{ij}, ...


0

Hint You have data points $(t_i,y_i)$. If you plot $y$ as a function of $t$ and suspect an exponential behavior, you then plot $\log(y)$ as a function of $t$ (this is a semi-log plot). If the data look now to be linear, then this confirm the model to be $$\log(y)=a +b t$$ To come back to the classical linear curve fit, define $z=\log(y)$ and use the ...


1

Let $$ X = \begin{bmatrix} 1 & x_1 & x_1^2 \\ \vdots & \vdots & \vdots \\ 1 & x_n & x_n^2 \end{bmatrix}. $$ Let $$ Y = \begin{bmatrix} y_1 \\ \vdots \\ y_n \end{bmatrix}. $$ Then the three entries in the $3\times 1$ matrix $(X^T X)^{-1}X^TY$ are the least-squares estimates of the coefficients in $y = \alpha+\beta x+\gamma x^2$. To ...


1

We don't know the expected value of the error term -and we do not argue that it equals the estimated value of the constant term, since the constant term may also estimate a "shift factor" anticipated by theory. Assume that no such shift factor is postulated, then the theory would tell you to specify $$y=\mathbf{x'\beta}+u$$ with no constant term. Hmmm, ...


2

$X$ isn't necessarily invertible (in fact, it is usually not square!) so you cannot distribute $\;^{-1}$ inside $(X'X)$.


3

The generalization is as follows: $$ L = \frac{1}{\sqrt{(2\pi)^n \left|\Sigma \right|}} \exp \left\{ -\frac{1}{2}(y-X\beta)^T \Sigma^{-1}(y-X\beta)\right\} $$ where $\Sigma$ is the covariance matrix of the multivariate Gaussian distribution, and $|\cdot|$ denotes its determinant. If the noises are independent, then it will be a diagonal matrix with noise ...


0

First of all, parameters of unweighted linear regression are very strongly influenced by the largest values of $y$, the dependent variable. When you linearize an exponential model and you have very small values, then the $\log(y)$ can be very large (negative) and strongly influence the results. I made an example fo your problem. I generated $20$ equally ...



Top 50 recent answers are included