# Tag Info

1

The Gauss-Markov theorem states that, under the usual assumptions, the OLS estimator $\beta_{OLS}$ is BLUE (Best Linear Unbiased Estimator). To prove this, take an arbitrary linear, unbiased estimator $\bar{\beta}$ of $\beta$. Since it is linear, we can write $\bar{\beta} = Cy$ in the model $y = \beta X + \varepsilon$. Furthermore, it is necessarily ...

1

1) The condition $\mathbb{E}[\tilde{\beta}]=\beta$ is just the condition "the estimator is unbiased" in mathematical form. Let's say you are considering the least squares estimator, then \begin{align} \mathbb{E}[\hat{\beta}] &= \mathbb{E}[(X^{\rm T}X)^{-1}X^{\rm T}Y]\\ &= \mathbb{E}[(X^{\rm T}X)^{-1}X^{\rm T}X\beta+\epsilon]\\ &= \beta, ... 0 Just make a linear regression by using matrices. Let the equation of the regression line be y_i=a\cdot x_i+b X=\left(\begin{array}{} 1& 2300 \\ 1& 2500 \\ 1& 2700 \\ 1& 2900 \\ 1& 3100 \\ 1& 3300 \\ 1& 3500 \\ 1& 3700\end{array} \right) Y=\left(\begin{array}{} 40 \\ 45 \\ 49 \\ 54 \\ 58 \\ 62 \\ 67 \\ 71 \end{array} ... 0 When you perform regression, you face two kinds of models Linear models are those where \frac{dy}{dp_k} are independent of all p_i's. The model could be off any apparent complexity such asy=a+bx^\pi+c\log(x)+d e^{-x\sqrt 2}$$Defining variables t_1=x^\pi, t_2=\log(x), t_3= e^{-x\sqrt 2} makes the model to be y=at_1+bt_2+ct_3 and then the ... 0 You can find the mathematics in this document, along with several other fitting techniques. The same web site has code to do the fitting. I have never used the fitting code, but I have used other code from this site, and found it be of very high quality. Mike Shaw's post says he used the algorithm decribed in this paper, which also looks very good, to me. 0 By \hat{Y}_i you mean the MMSE estimator (or the posterior mean) ? If so, remember a very important property of this estimator is that the MMSE estimator of Y is that the error \hat Y - Y is orthogonal to any function of Y. Thus E[(Y_i - \hat{Y}_i)(\hat{Y}_i - \bar{Y})] = 0. But in your case there are no expectations. Please verify that or give ... 2 There are many different ways to look at degrees of freedom. I wanted to provide a rigorous answer that starts from a concrete definition of degrees of freedom for a statistical estimator as this may be useful/satisfying to some readers: Definition: Given an observational model of the form$$y_i=r(x_i)+\xi_i,\ \ \ i=1,\dots,n$$where ... 0 What makes the model interesting is that the variance is proportional to the data x_i. This implies that x_i\geq 0. I am going to start by assuming that x_i>0. But once we've found the answer in that case I'll discuss the situation when that assumption does not hold; i.e., that x_i=0 for at least one i. Let us define the following quantities: ... 1 Since \hat{y_i} is determined from the linear regression, it has two degrees of freedom, corresponding to the fact that we specify a line by two points. When we consider the equation of a line in slope-intercept form, this becomes the slope value and the y-intercept value. When we subtract the mean response, \overline{y}, it cancels the y-intercept value ... 1 No and also independence of the X_i's (row vector, i-th row of design matrix) does not have anything to do with full rank either. For the design matrix you need additional assumptions. There are two ways in which people usually proceed: 1) Fixed (deterministic) design: Since it is deterministic, it either has full rank or it does not, so you just have to ... 1 No, none of these assumptions imply the full rank condition on X. It has to be assumed separately. The full rank condition on X does not mean independence of regressors from each other -- it just means that they are not related by a linear functional relationship. Independence of the error can be relaxed (to uncorrelatedness), and homoschedasticity and ... 0 The population parameter is equal to (equation (4) in your note) $$\beta_1 = \frac{cov(y,x)}{var(x)}-\frac{cov(e,x)}{var(x)}. \tag1$$ whereas the OLS estimate of it is given by $$\hat{\beta}_1 = \frac{\widehat{cov(y,x)}}{\widehat{var(x)}}. \tag2$$ As the sample size increases ... 0 With appropriate method names, you will be able to find a lot of references: you seem to be looking for Deming regression, orthogonal regression, or more often total least squares. It is illustrated below in the bivariate case: 0 For each who want to know the answer: You have to compute P_{L_H} analogous to P_L, just with the designmatrix X_0, that results under H_0. In my case that means if \beta_1 is null, the designmatrix for that is$$X_0 = X \beta_{H_0} = X \left(\begin{array}{cc} 1 \\ 0 \\ 1 \end{array}\right)$$where X is the designmatrix. Then compute$$P_{L_H} = ...

0

The classification problem data can be captured in one matrix and one vector, i.e. $\{X,y\}$. Then the relevant quantities are the vectors \eqalign{ f &= X^T\beta \cr p &= \sigma(f) \cr } and their differentials and logarithmic differentials \eqalign{ df &= X^Td\beta \cr dp &= p\circ(1-p)\circ df \cr\cr d\log(p) &= ... 0 what @mvw wrote is true, however, not using the information about the variances of the noise will, for sure, decrease the performance of your estimator. The problem given by @Jenny, is equivalent to $$Y = X \theta + n, n \sim {\mathcal{N}}(0,\Sigma),$$ where \Sigma \overset{def}{=} diag(\sigma_1^2, \sigma_2^2, ... 0 If you have any system of linear equations X \theta = Y $$one can show that$$ X^T X \theta^* = X^T Y $$has a solution \theta^*$$ \theta^* = (X^T X)^{-1} X^T Y $$which minimizes$$ e = X\theta - Y \ $$in the the Euclidean norm ("least squares"):$$ \lVert X\theta^* - Y \rVert_2 \le \lVert X\theta - Y \rVert_2 $$This is independent from the ... 0 I found the answer here: Weisstein, Eric W. "Least Squares Fitting--Logarithmic." 0 The question is a little confusing, because you mention fitting a logarithmic trendline, but then you suggest using a linear formula of the form y=cx-b. This would all make sense if your x values were already the logarithms of some original data points, say z_i, where x_i = \log z_i. Then fitting with a linear formula y=cx-b would be equivalent to ... 0 Assuming you have N values making up your functions, denote each point by f_i for i = 1,..,N and going from left to right. Then let another index be j. For j starting at 1 initially do a linear fit on the reduced data set \{f_j,..,f_N\} and let the y intercept for that value of j be C_j. From the data it is clear that as j increases C_j ... 1 Since \epsilon_i \sim N(0,9), it follows that (see the derivation bellow if you do not undestand why) $$\hat{b}-b \sim N\left(0,\frac{9}{\sum_i{(x_i-\bar{x})^2}}\right),$$ and, noticing that given the numbers above \sum_i{(x_i-\bar{x})^2}=16, this leads to$$ \sqrt{\frac{16}{9}} (\hat{b}-b) \sim N(0,1).$$Finally, let ... 0 your problem formulation is incorrect. note that W(y-X\beta)^T(y-X\beta) is a n \times n matrix since (y-X\beta)^T(y-X\beta) is a scalar. The weighted least squares problem is to minimize the scalar (y-X\beta)^TW(y-X\beta). If you now derive with respect to \beta you'll get the solution you're looking for. 0 Look up regression analysis, particularly the general linear model (in several variables). There is a huge literature on the subject, including estimating errors of the coefficients. 0 We can rewrite the regression equations slightly as$$ z_0 c_0 + z_1 c_1 + z_2 c_2 = y_t - a $$then arrange them as linear system$$ A x = b $$with$$ A = (z_0, z_1, z_2) \\ x = (c_0, c_1, c_2)^t \\ b = (y_t-a) $$where the z_i and y_t-a are column vectors each, having k components if your data list has k lines of such data. k \ge 3 would be ... 2$$ y = \alpha_0 + \alpha_1 x + (\beta_0 - \alpha_0)1_{x > x_0} + (\beta_1 - \alpha_1)x1_{x > x_0} + \epsilonLet \gamma_0 := \beta_0 - \alpha_0 and \gamma_1 := \beta_1 - \alpha_1. Now define X := \begin{bmatrix}1 && x && 1_{x > x_0} && x1_{x > x_0}\end{bmatrix} and \beta := \begin{bmatrix}\alpha_0 && ... 1 Here is the proof my teacher gaved me today, shorter than the very good one from Daniel. \begin{align*} R^2&=\frac{ESS}{TSS}=\frac{\sum(\hat y -\bar y)^2}{\sum (y_i-\bar y)^2}\\ &=\frac{\sum (\hat\beta_1+\hat \beta_2x_i-\bar y)^2}{\sum(y_i-\bar y)^2}\\ &=\frac{\sum (\bar y- \hat \beta_2\bar x+\hat \beta_2x_i-\bar y)^2}{\sum(y_i-\bar y)^2}\\ ... 0 The (Estimated) Variance of residuals in an OLS regression is simply: Var(e)=\frac{e'e}{n-(k+1)} $$where k+1 is the number of regressors (plus a constant). 0 To start you differentiate S=\sum_{i=1}^n \left(y_i-a_0-a_1x_i-a_1\overline x \right)^2 w.r.t. a_1. \frac{\partial S}{\partial a_1}=2\cdot \sum_{i=1}^n \left(y_i-a_0-a_1x_i-a_1\overline x \right)\cdot (x_i-\overline x)=0 Multipliying out the brackets and drop 2 \sum_{i=1}^n x_iy_i-\overline x \sum_{i=1}^n y_i-a_0\sum_{i=1}^n x_i+a_0\overline x ... 0 It's the usual: \begin{array}\\ \sum_{i=1}^n \frac{\alpha_1(X_i-\overline{X})}{n} &=\sum_{i=1}^n \frac{\alpha_1(X_i)}{n}-\sum_{i=1}^n \frac{\alpha_1(\overline{X})}{n}\\ &=\alpha_1\sum_{i=1}^n \frac{X_i}{n}-\alpha_1\sum_{i=1}^n \frac{\overline{X}}{n}\\ &=\alpha_1\overline{X}-\alpha_1\overline{X}\\ &=0 \end{array}  0 I think that stochasticboy321 made a very good comment to the question. When you minimize the some of squares for a function which is linear with respect to its parameters, the problem is simple and the so-called normal equations are particularly easy to solve (in particular using matrix calculations). Where the problem becomes more difficult is when the ... 0 I assume that y_i=\beta_0+\beta_1 x_i +\varepsilon_i Estimation of beta 1 why would$$\hat \beta_1=\frac{\sum(y_ix_i)}{\sum({x_i}^2)} - \frac{\beta_0*\sum(x_i)}{\sum(x_i^2)}?the intercept \beta_0 known, the least squares estimator of \beta_1 is still \begin{align*} \hat\beta_1 &=\frac{cov(x,y)}{var(x)}\\ &=\frac{\sum (x_i-\bar ... 1 If you standardize both your X and your Y, then you are predicting the values of standardized Y, in which case you can convert it back to its original scale after prediction. You can also only standardize your predictor X while leaving your Y unstandardized (this is more common) in order to predict Y based on the way X deviates from its mean. ... 1 E[b_1]=E[\frac{S_{xy}}{S_{xx}}]=E[\sum[\frac{[(x_i - \bar{X})*y_i]}{\sum(x_i-\bar{X})}]]=\frac{\sum[(x_i-\bar{x})*E(y_i)]}{\sum(x_i-\bar{X})} ...$$why the last move is legal? Because y_i are variables and (x_i - \bar{X}) are constants (remember, we want to estimate y_i, like the height of a tree, wich we don't know (assume we are too small), which ... 0 Part of the issue is whether you want your function to fit the data "as closely as possible", or if you want it to hit every data point exactly. For example, if you want to fit some data that appears linear, using linear least squares approximation to find the two coefficients which minimize the error is the right way to go. However, if you want an exact ... 1 To add to the answer from Ian Miller,$$S_{xx}=\sum x^2 -\frac{(\sum x)^2}{n}=\sum x^2 -n\bar{x}^2$$Intuitively, S_{xy} is the result when you replace one of the x's with a y.$$S_{xy}=\sum xy -\frac{\sum x \sum y}{n}=\sum xy -n\bar{x}\bar{y}$$Also, just for your information, the good thing about this notation is that it simplifies other parts ... 1 From matrix multiplication (H^2 = H), you can write$$h_{ii}=\sum_{j=1}^n h_{ji}^2$$for every i in 1,\ldots, n. Next, we have$$h_{ii}=h_{ii}^2 + \sum_{j=1, j\neq i}^n h_{ji}^2$$and then,$$h_{ii}-h_{ii}^2= \sum_{j=1, j\neq i}^n h_{ji}^2 Function $f(h_{ii}) = h_{ii}-h_{ii}^2$ has a local maximum in $h_{ii}=0.5$, and f(0.5) = 0.25. Thus, ...

1

$S_{xx}$ is the sum of the squares of the difference between each $x$ and the mean $x$ value. $S_{xy}$ is sum of the product of the difference between $x$ its means and the difference between $y$ and its mean. So $S_{xx}=\Sigma(x-\overline{x})(x-\overline{x})$ and $S_{xy}=\Sigma(x-\overline{x})(y-\overline{y})$. Both of these are often rearranged into ...

Top 50 recent answers are included