New answers tagged

1

Yes, you are right. What you are describing is what referred to in machine literature as non-robustness of LSE to outliers. The reason for this name is that, if you have a noisy point in your data, far away from the rest, the squared nature of the penalization tends to move the whole model towards that single noisy outlier. A more robust alternative to mean ...


2

This is one of those things that is hard to give a single answer on. At the end of the day there are modeling drawbacks indeed, but there are so many theoretical advantages that it is rather hard to resist. I think the best point I can make is the one I will make first: absolute differences (probably the most obvious idea to consider) have a serious problem....


0

Let $\mathrm A \in \mathbb R^{m \times n}$, where $m \gg n$, and $\mathrm y \in \mathbb R^m$. The linear system in $\mathrm x \in \mathbb R^n$ $$\mathrm A \mathrm x = \mathrm y$$ only has a solution when $\mathrm y$ is in the column space of $\mathrm A$, whereas the linear system $$\mathrm A^T \mathrm A \mathrm x = \mathrm A^T \mathrm y$$ always has at ...


0

Minimizing the $\ell_2$-norm of the residual is certainly not always the best thing to do, for the reason you said: it puts too much weight on outliers. For that reason people often minimize the $\ell_1$-norm of the residual. The $\ell_1$-norm is much more robust against outliers. (The $\ell_1$-norm does not consider it to be a disaster if a few components ...


0

It seems like you are looking for regression or fitting of a data set to a function. (If you don't care about knowing $f$ analytically, look up machine learning). Hopefully I am understanding you correctly. Usually, for numerical fitting, you pick a form for $f(X)$ and then computationally fit to it. For instance, in Python, you could fit a polynomial to it ...


3

You can find a least squares estimate of $x$ and $y$. You want to "solve" the overdetermined system $Az = c$, where \begin{equation} A = \begin{bmatrix} a_1 & b_1 \\ a_2 & b_2 \\ a_3 & b_3 \\ a_4 & b_4 \end{bmatrix} \end{equation} and \begin{equation} c = \begin{bmatrix} c_1 \\ c_2 \\ c_3 \\ c_4 \end{bmatrix}. \end{equation} In a least ...


0

The squared errors produce easier formulas. When implementing it in both cases you end up with a bunch of for loops and if statements so the difference then might not be so clear. However, if you intend to deal with formulas the squared error is easier. The expression involving squared errors is differentiable everywhere and the derivative is 'analytic': ...


-1

This offers an explanation. If this answer is better suited as a comment, please let me know and I will update it accordingly.


0

The formal arguments have already be given in Qiaochu Yuan's answer. So, let me give you a small example for a linear regression of the following data points $$\left( \begin{array}{cc} x & y \\ 1 & 5 \\ 2 & 8 \\ 3 & 11 \\ 4 & 14 \\ 5 & 16 \\ 6 & 19 \\ 7 & 22 \\ 8 & 24 \\ 9 & 27 \\ 10 & 30 \end{array} \...


5

From a Bayesian point of view, this is equivalent to assuming that your data is generated by a line plus Gaussian noise, and finding the maximum likelihood line based on that assumption. Using the absolute values means assuming that your noise has pdf proportional to $e^{-|x|}$ which is substantially less natural than assuming Gaussian noise (e.g. Gaussian ...


1

If the regression data are symmetric with respect to changing the sign of $y$, the least-squares approximation is the line $y=0$. The error is a sum of pairs $((y+a)^2 + (y-a)^2)$ all of which are minimized at $y=0$. If the data are samples from a symmetric distribution then $y=0$ is the expected regression line and the actual line will be a small random ...


2

As @hardmath mentioned in the comment, the results are perfectly logical. If $Y$ and $X$ are independent and each one is $\mathcal{N}(0,1)$, so clearly (from independence) $cov(X,Y)=0$ and the real intercept is $0$, because $(0,0)=(\mathbb{E}X, \mathbb{E}Y)$. Hence, the real regression line is simply $y=0+0 x+\epsilon=\epsilon$, where $\epsilon \sim \mathcal{...


0

Pearson's correlation coefficient between $X$ and $Y$ is given by $$ \rho = \frac{cov(X,Y)}{\sigma_X \sigma_Y}. $$ In the simple linear regression model $y=\beta_0 + \beta_1x + \epsilon$, the slope, $\beta_1$, has the following form/interpretation $$ \beta_1 = \frac{cov(X,Y)}{\sigma^2_X}, $$ hence, $$ \beta_1 = \frac{cov(X,Y)\sigma_Y}{\sigma_X \sigma_X \...


0

Proof for $f(\beta_0,\beta_1)$ being convex with respect to $(\beta_0,\beta_1)$: 1) $g(x)$ is convex $\Longleftrightarrow g(a^Tu)$ is convex with respect to $u$ for column matrices $a$ and $u$. 2) Sum of convex functions is convex. 3) $\ln(1+e^x)$ is convex. Let $g(x) = \ln(1+e^x)$ in 1). Let $a_i=-[y_i\ x_iy_i]^T$. $f(\beta_0,\beta_1)$ is then the sum ...


0

(1) follows from a case analysis, according to whether $x$ is positive or negative: thus $$\text{sign}(x) = {x \over |x|}$$ if $x \ne 0$. Applying this, we find $$p'(|x|) \cdot \text{sign}(x) \approx p'(|x|) \times {x \over |x|} \approx p'(|x_0|) \times {x \over |x_0|},$$ which justifies (1). (2) follows from a first-order Taylor series approximation for ...


0

Those four alternatives will return a function each, which fits a set of given data points. This is not a good mathematical model for your problem. What you might do is recording how often each number has been drawn, thus a historgram or frequency distribution. And then you might check, if in the case of many lottery drawings, if each possible lottery ...


1

As Jyrki Lahtonen commented, the problem reduces to finding the best value of $c$ for the model $$y=c(x-x_k)^2+y_k$$ based on $n$ data points $(x_i,y_i)$. If you define $z_i=(y_i-y_k)$ and $t_i=(x_i-x_k)^2$ then, the model reduces to $$z=c t$$ which corresponds to a linear regression without intercept. If you want to minimize the sum of squared errors, you ...


1

1) For the Alcohol retention, you don't even need a linear regression, as the relationship is perfectly linear (straight line), thus your model is: $$ y(t) = 0.19 -0.015t, $$ where $y$ denotes the Alcohol amount (g/dl) and $t$ time. 2) For the Caffeine, assuming that exponential decay model is reasonable fit, you should first perform a linearization, ...


0

You want to fit a second degree polynomial, i.e., find coefficients $(a,b,c)$ in: $$y=a+bx+cx^2.$$ The minimum is at $$0=b+2cx \iff x=-\frac{b}{2c}$$ The value at the minimum is $$f(x=-\frac{b}{2c})$$ So to fit your function, you could use ordinary least squares with constraints. The constraints are: $c<0$ (for minimum) $-\frac{b}{2c}=x_k$ $f(-\frac{b}{...


2

You have two degrees of freedom (the slope $m$ and $y$-intercept $b$ of the regression line, say), but also two constraints: The partial derivatives of the squared total error $E$ with respect to $m$ and with respect to $b$ must vanish. That means you expect only finitely many (local) minima. (As A.E. says, $E$ is strictly convex, so there is at most one ...


1

Basically, as you said, you are minimizing sum of squares, i.e., $$ \underset\beta{\arg\min} \sum_{i=1}^n(y_i-\beta_0 - \sum_{j=1}^p\beta_jx_j)^2 = \underset\beta{\arg\min}||y-X\beta|| $$ which has to be strictly convex function over the parametric space $\mathcal{B}$, otherwise the Hessian matrix of this quadratic form, $X'X$, won't be positive definite, ...


1

Assuming that you have $n$ data points $(x_i,y_i)$, you want to minimize $$SSQ=\sum_{i=1}^n\left( c_0+c_1x_i+\frac 12 c_2x_i^2+\frac 13 c_3x_i^3-y_i\right)^2$$ subject to constraints $a_i\leq c_i\leq b_i$. This is clearly an optimization problem with only bound constraints. Have a look here for doing it using Matlab. Edit By the way, if you have access to ...



Top 50 recent answers are included