Questions tagged [regression]

This tag is for questions on (linear or nonlinear) regression, which is a way of describing how one variable, the outcome, is numerically related to predictor variables. The dependent variable is also referred to as $~Y~$, dependent or response and is plotted on the vertical axis (ordinate) of a graph.

Filter by
Sorted by
Tagged with
130
votes
11answers
192k views

What is the difference between regression and classification?

What is the difference between regression and classification, when we try to generate output for a training data set $x$?
95
votes
6answers
66k views

derivative of cost function for Logistic Regression

I am going over the lectures on Machine Learning at Coursera. I am struggling with the following. How can the partial derivative of $$J(\theta)=-\frac{1}{m}\sum_{i=1}^{m}y^{i}\log(h_\theta(x^{i}))+...
58
votes
11answers
25k views

Why do we use a Least Squares fit?

I've been wondering for a while now if there's any deep mathematical or statistical significance to finding the line that minimizes the square of the errors between the line and the data points. If ...
40
votes
4answers
11k views

Why divide by $2m$

I'm taking a machine learning course. The professor has a model for linear regression. Where $h_\theta$ is the hypothesis (proposed model. linear regression, in this case), $J(\theta_1)$ is the cost ...
27
votes
5answers
13k views

Why get the sum of squares instead of the sum of absolute values?

I'm self-studying machine learning and getting into the basics of linear regression models. From what I understand so far, a good regression model minimizes the sum of the squared differences between ...
27
votes
2answers
26k views

Computational complexity of least square regression operation

In a least square regression algorithm, I have to do the following operations to compute regression coefficients: Matrix multiplication, complexity: $O(C^2N)$ Matrix inversion, complexity: $O(C^3)...
20
votes
3answers
14k views

Finding the intersection point of many lines in 3D (point closest to all lines)

I have many lines (let's say 4) which are supposed to be intersected. (Please consider lines are represented as a pair of points). I want to find the point in space which minimizes the sum of the ...
19
votes
1answer
435 views

On the integral $\int_{-\pi/2}^{\pi/2}\sin(x/\sin(x/\sin(x/\sin\cdots)))\,dx$

This question is the final one out of the set (see I and II), I promise! Consider $f_1(x)=\sin(x)$ and $f_2(x)=\sin\left(\frac x{f_1(x)}\right)$ such that $f_n$ satisfies the relation $$f_n(x)=\sin\...
16
votes
3answers
20k views

Derivation of the formula for Ordinary Least Squares Linear Regression

How was the formula for Ordinary Least Squares Linear Regression arrived at? Note I am not only looking for the proof, but also the derivation. Where did the formula come from?
16
votes
3answers
3k views

Best fit ellipsoid

Given a collection of points $P \subset \mathbb R^3$, a crude characterization of the "shape" of $P$ is sometimes given by the principal components. We construct a covariance matrix, e.g., if $P$ is ...
15
votes
4answers
7k views

Polynomial fitting where polynomial must be monotonically increasing

Given a set of monotonically increasing data points (in 2D), I want to fit a polynomial to the data which is monotonically increasing over the domain of the data. If the highest x value is 100, I don'...
13
votes
2answers
4k views

Why is polynomial regression considered a kind of linear regression?

Why is polynomial regression considered a kind of linear regression? This is what I mean by polynomial regression. For example, the hypothesis function is $$h(x; t_0, t_1, t_2) = t_0 + t_1 x + t_2 ...
13
votes
4answers
9k views

Deriving cost function using MLE :Why use log function?

I am learning machine learning from Andrew Ng's open-class notes and coursera.org. I am trying to understand how the cost function for the logistic regression is derived. I will start with the cost ...
13
votes
4answers
30k views

Correlation Coefficient and Determination Coefficient

I'm really new to linear regression and am trying to teach myself. In my textbook there's a problem that asks why $R^{2}$ in the regression of $Y$ on $X =$ the sample correlation between X and Y the ...
12
votes
1answer
11k views

easy to implement method to fit a power function (regression)

I want to fit to a dataset a power function ($y=Ax^B$). What is the best and easiest method to do this. I need the $A$ and $B$ parameters too. I'm using in general financial data in my project, which ...
12
votes
2answers
98k views

Linear regression: degrees of freedom of SST, SSR, and RSS

I'm trying to understand the concept of degrees of freedom in the specific case of the three quantities involved in a linear regression solution, i.e. $SST=SSR+SSE, $ i.e. Total sum of squares = ...
11
votes
5answers
38k views

Given a data set, how do you do a sinusoidal regression on paper? What are the equations, algorithms?

Most regressions are easy. Trivial once you know how to do it. Most of them involve substitutions which transform the data into a linear regression. But I have yet to figure out how to do a ...
11
votes
1answer
7k views

Lasso - constraint form equivalent to penalty form

We know that there are two definitions to describe lasso. Regression with constraint definition: $$\min\limits_{\beta} \|y-X\beta\|^2, \sum\limits_{p}|\beta_p|\leq t, \exists t $$ Regression with ...
10
votes
2answers
16k views

Linear regression for minimizing the maximum of the residuals

We know that simple linear regression will do the following thing: Suppose there are $n$ data points $\{y_i,x_i\}$, where $i=1,2,\dots,n$. The goal is to find the equation of the straight line $y=\...
10
votes
3answers
26k views

Prove $SST=SSE+SSR$

Prove $$SST=SSE+SSR$$ I start with $$SST= \Sigma (y_i-\bar{y})^2=...=SSE+SSR+ \Sigma 2( y_i-y_i^*)(y_i^*-\bar{y} )$$ and I don't know how to prove that $\Sigma 2( y_i-y_i^*)(y_i^*-\bar{y} )=0$ a ...
10
votes
1answer
72k views

How to find curve equation from data?

How do I find the formula when I only know some data points ? Usually I just use the Trendline option for diagrams in Excel, but this one eludes me. I expect it to be something like : ...
10
votes
1answer
9k views

What is the difference between Curve Fitting and Regression(Machine Learning)?

I know that Machine Learning regression algorithms try to find the function of the data. That is, if we have 1000 data points (x,y), to find a general continuous function that follows the trends of ...
9
votes
2answers
137k views

Sxx in linear regression

What is the meaning of 'Sxx' and 'Sxy' in simple linear regression? I know the formula but what is the meaning of those formulas?
9
votes
3answers
27k views

Fitting exponential curve to data

If I have a collection of data points that follow an exponential curve relationship, how can I manually construct the equation that defines the best-fit exponential curve for the data?
9
votes
3answers
567 views

Minimize $\|Ax-b\|$ but for $A$, not $x$

I have a machine learning regression problem. I need to minimize $$\sum_i \|Ax_i-b_i\|_2^2$$ However I am trying to find matrix $A$, not the usual $x$, and I have lots of example data for $x_i$ and $...
8
votes
2answers
8k views

connection between PCA and linear regression

Is there a formal link between linear regression and PCA? The goal of PCA is to decompose a matrix into a linear combination of variables that contain most of the information in the matrix. Suppose ...
8
votes
1answer
1k views

Linear regression is wrong?

I tried asking this in electronics as a question related to oscillators, but I wasn't able to get a satisfactory answer. I think the more math-y types here may shed some additional light on the ...
7
votes
9answers
1k views

Why work with squares of error in regression analysis?

In regression analysis one finds a line that fits best by minimizing the sum of squared errors. But why squared errors? Why not use the absolute value of the error? It seems to me that with squared ...
7
votes
1answer
2k views

Why do people fit polynomials?

Could someone explain the justification and limits of fitting polynomials to arbitrary data points? I mean what about square roots or fractional or inverse powers? Most of the time some wants to ...
7
votes
3answers
520 views

Are there variations on least-squares approximations?

In least-squares approximations the normal equations act to project a vector existing in N-dimensional space onto a lower dimensional space, where our problem actually lies, thus providing the "best" ...
7
votes
3answers
8k views

Log-likelihood gradient and Hessian

Considering a binary classification problem with data $D = \{(x_i,y_i)\}_{i=1}^n$, $x_i \in \mathbb{R}^d$ and $y_i \in \{0,1\}$. Given the following definitions: $f(x) = x^T \beta$ $p(x) = \sigma(f(...
7
votes
1answer
601 views

How to find line that has least distance to all points?

I need to find the line having minimal distance to all points. I found linear regression and linear interpolation algorithms. But their minimal distance is only in y-axis: $D = y - f(x)$. But I need ...
7
votes
2answers
429 views

Can I build a program that will tell me if a real world data set looks linear, logarithmic, exponential etc?

I have a bunch of real world data sets and from manually plotting some of the data in graphs, I've discovered some data sets look pretty much logarithmic and some look linear, or exponential (and some ...
7
votes
1answer
2k views

What is the Moore-Penrose pseudoinverse for scaled linear regression?

The matrix equation for linear regression is: $$ \vec{y} = X\vec{\beta}+\vec{\epsilon} $$ The Least Square Error solution of this forms the normal equations: $$ ({\bf{X}}^T \bf{X}) \vec{\beta}= {\bf{...
6
votes
4answers
2k views

Method of Least Squares-Why is it preferred? [duplicate]

Possible Duplicate: Why do we use a Least Squares fit? To find the normal equations for derivation of the regression line we use the method of least squares, We want to make the error smallest ...
6
votes
5answers
25k views

Assumption of a Random error term in a regression

In one of my recent statistics courses, our teacher introduced the linear regression model. The typical $y=\alpha + \beta X + \epsilon$, where $\epsilon$ is a "random" error term. The teacher then ...
6
votes
1answer
5k views

Analytic solution for matrix factorization using alternating least squares

The standard form for ridge regression aims to minimize the following cost function. $$ \min\ \ \sum_i(y_i-x_i^T\beta)^2 + \lambda\sum_j\beta^2_j $$ As described here, it's possible to differentiate ...
6
votes
2answers
592 views

Theoretical basis for overfitting

There are many examples in which making more "precise" predictions gives worse performance (e.g. Runge's phenomenon). My professor implied that there was a sound basis for choosing "simple" functions ...
6
votes
3answers
834 views

Understanding the difference between Ridge and LASSO

I'll start by presenting my current understanding of regularized loss minimization. In the context of ML, generally we are trying to minimize directly an empirical loss, something of the form $$\...
6
votes
2answers
18k views

How to fit a curve to a sinusoidal wave

I am wondering how to fit a sinusoidal wave (approximation). I would like to fit it in the form: $y = A\sin(Bx + C) + D$ where $A,\,B,\,C$ and $D$ are constants. The only constants I really care about ...
6
votes
2answers
435 views

How to find a closed curve approximation to contour lines that look like ellipses?

I am finding it very difficult to find resources giving a overview of techniques that can be used to approximate a closed curve. For example, the wikipedia article on curve fitting has further links ...
6
votes
1answer
2k views

Complexity of Gaussian Process algorithms is $\mathcal{O}(n^3)$

It is often quoted that the complexity of Gaussian Process algorithms is $\mathcal{O}(n^3)$ due to the need to invert an $n \times n$ matrix, where $n$ is the number of data points. But as far as I ...
6
votes
1answer
1k views

Expectation and orthogonal projection

Many books while introducing the regression problem, start with the assertion that any random variable $Y$ can be decomposed into two orthogonal terms $$ Y= E[Y|X]+\epsilon. $$ In the classical ...
6
votes
1answer
3k views

Prove the estimator $\hat{B}$ of ridge regression = mean of the posterior distribution under a Gaussian prior

I want to prove that the estimator of ridge regression is the mean of the posterior distribution under Gaussian prior. $$y \sim N(X\beta,\sigma^2I),\quad \text{prior }\beta \sim N(0,\gamma^2 I).$$ $...
6
votes
1answer
2k views

How to compare the similarity between functions?

I'm designing a web service that finds the regression function of a pattern within an image. I analyzed three images and found the following three regressions: 1) $f(x) = 74.7602 + 0.2005x - 0....
6
votes
1answer
161 views

Formula for straight part of a slightly bumpy line

Given a straight line that deviates from the horizontal by at most 15 degrees. On this straight line there are bumps on top at random places on the line. The combined width of the bumps is at most ...
6
votes
0answers
328 views

Perturbation theory for least squares for very different A, b

Consider the least squares problem $f(x;A,b) = \|Ax-b\|_2^2$ and define $x^*$ the minimizer of $f(x;\hat A,\hat b)$, and $\hat x$ the minimizer of $f(x; A_2, b_2)$. I want to put some bound on $\|...
5
votes
1answer
27k views

Can someone explain what plim is?

In my Introductory Econometrics class we discussed a concept of "plim" or "probability limit. I'm not sure what this means though and my professor doesn't explain it well at all. Can someone tell me ...
5
votes
2answers
1k views

why small L1 norm means sparsity?

I'm trying to understand regularization in machine learning. one way of regularization is adding a l1 norm to the error function. This is said to produce sparsity. But I can't understand. sparsity is ...
5
votes
2answers
398 views

Recommended Reading on Regression Analysis?

For a university project, I am implementing an automated regression analysis tool. However, I have very little background in statistics. So what books / articles / material would you suggest I could ...