Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

learn more… | top users | synonyms

42
votes
8answers
48k views

Regression vs Classification

This is more machine learning questions, but perhaps someone will be able to help. I would like to know what is the difference between regression and classification when we try to generate output for ...
38
votes
9answers
10k views

Why do we use a Least Squares fit?

I've been wondering for a while now if there's any deep mathematical or statistical significance to finding the line that minimizes the square of the errors between the line and the data points. If ...
24
votes
3answers
8k views

derivative of cost function for Logistic Regression

I am going over the lectures on Machine Learning at Coursera. I am struggling with the following. How can the partial derivative of ...
16
votes
6answers
3k views

Why get the sum of squares instead of the sum of absolute values?

I'm self-studying machine learning and getting into the basics of linear regression models. From what I understand so far, a good regression model minimizes the sum of the squared differences between ...
15
votes
3answers
6k views

Finding the intersection point of many lines in 3D (point closest to all lines)

I have many lines (let's say 4) which are supposed to be intersected. (Please consider lines are represented as a pair of points). I want to find the point in space which minimizes the sum of the ...
12
votes
3answers
2k views

Best fit ellipsoid

Given a collection of points $P \subset \mathbb R^3$, a crude characterization of the "shape" of $P$ is sometimes given by the principal components. We construct a covariance matrix, e.g., if $P$ is ...
11
votes
1answer
9k views

Computational complexity of least square regression operation

In a least square regression algorithm, I have to do the following operations to compute regression coefficients: Matrix multiplication, complexity: $O(C^2N)$ Matrix inversion, complexity: ...
11
votes
4answers
3k views

Polynomial fitting where polynomial must be monotonically increasing

Given a set of monotonically increasing data points (in 2D), I want to fit a polynomial to the data which is monotonically increasing over the domain of the data. If the highest x value is 100, I ...
10
votes
1answer
4k views

easy to implement method to fit a power function (regression)

I want to fit to a dataset a power function ($y=Ax^B$). What is the best and easiest method to do this. I need the $A$ and $B$ parameters too. I'm using in general financial data in my project, which ...
9
votes
2answers
1k views

Why is polynomial regression considered a kind of linear regression?

Why is polynomial regression considered a kind of linear regression? This is what I mean by polynomial regression. For example, the hypothesis function is $$h(x; t_0, t_1, t_2) = t_0 + t_1 x + t_2 ...
9
votes
3answers
313 views

Minimize $||Ax-b||$ but for $A$, not $x$

I have a machine learning regression problem. I need to minimize $$ \sum_i||Ax_i-b_i||_2^2 $$ However I am trying to find matrix $A$, not the usual $x$, and I have lots of example data for $x_i$ and ...
7
votes
3answers
2k views

Derivation of the formula for Ordinary Least Squares Linear Regression

How was the formula for Ordinary Least Squares Linear Regression arrived at? Note I am not only looking for the proof, but also the derivation. Where did the formula come from?
7
votes
5answers
15k views

Given a data set, how do you do a sinusoidal regression on paper? What are the equations, algorithms?

Most regressions are easy. Trivial once you know how to do it. Most of them involve substitutions which transform the data into a linear regression. But I have yet to figure out how to do a ...
7
votes
1answer
681 views

Linear regression is wrong?

I tried asking this in electronics as a question related to oscillators, but I wasn't able to get a satisfactory answer. I think the more math-y types here may shed some additional light on the ...
6
votes
2answers
735 views

Why do people fit polynomials?

Could someone explain the justification and limits of fitting polynomials to arbitrary data points? I mean what about square roots or fractional or inverse powers? Most of the time some wants to ...
6
votes
3answers
385 views

Are there variations on least-squares approximations?

In least-squares approximations the normal equations act to project a vector existing in N-dimensional space onto a lower dimensional space, where our problem actually lies, thus providing the "best" ...
6
votes
1answer
284 views

How to find line that has least distance to all points?

I need to find the line having minimal distance to all points. I found linear regression and linear interpolation algorithms. But their minimal distance is only in y-axis: $D = y - f(x)$. But I need ...
6
votes
2answers
11k views

How to fit a curve to a sinusoidal wave

I am wondering how to fit a sinusoidal wave (approximation). I would like to fit it in the form: $y = A\sin(Bx + C) + D$ where $A,\,B,\,C$ and $D$ are constants. The only constants I really care about ...
6
votes
2answers
368 views

Theoretical basis for overfitting

There are many examples in which making more "precise" predictions gives worse performance (e.g. Runge's phenomenon). My professor implied that there was a sound basis for choosing "simple" functions ...
6
votes
2answers
34k views

Linear regression: degrees of freedom of SST, SSR, and RSS

I'm trying to understand the concept of degrees of freedom in the specific case of the three quantities involved in a linear regression solution, i.e. $SST=SSR+SSE, $ i.e. Total sum of squares = ...
6
votes
3answers
5k views

Correlation Coefficient and Determination Coefficient

I'm really new to linear regression and am trying to teach myself. In my textbook there's a problem that asks why $R^{2}$ in the regression of $Y$ on $X =$ the sample correlation between X and Y the ...
6
votes
1answer
274 views

Can I build a program that will tell me if a real world data set looks linear, logarithmic, exponential etc?

I have a bunch of real world data sets and from manually plotting some of the data in graphs, I've discovered some data sets look pretty much logarithmic and some look linear, or exponential (and some ...
6
votes
1answer
106 views

Formula for straight part of a slightly bumpy line

Given a straight line that deviates from the horizontal by at most 15 degrees. On this straight line there are bumps on top at random places on the line. The combined width of the bumps is at most ...
5
votes
2answers
267 views

Recommended Reading on Regression Analysis?

For a university project, I am implementing an automated regression analysis tool. However, I have very little background in statistics. So what books / articles / material would you suggest I could ...
5
votes
2answers
5k views

Linear regression for minimizing the maximum of the residuals

We know that simple linear regression will do the following thing: Suppose there are $n$ data points $\{y_i,x_i\}$, where $i=1,2,\dots,n$. The goal is to find the equation of the straight line ...
5
votes
1answer
37k views

How to find curve equation from data?

How do I find the formula when I only know some data points ? Usually I just use the Trendline option for diagrams in Excel, but this one eludes me. I expect it to be something like : ...
5
votes
1answer
349 views

How to prove SSE and SSR are independent

Consider $Y=X\beta+\varepsilon$, where $X$ is n by p, $\beta$ is p by 1 and $\varepsilon$ is n by 1 with covariance matrix = var($\varepsilon$)=$\sigma^2 I$. Give expression for the regression and ...
5
votes
1answer
2k views

What is the difference between Curve Fitting and Regression(Machine Learning)?

I know that Machine Learning regression algorithms try to find the function of the data. That is, if we have 1000 data points (x,y), to find a general continuous function that follows the trends of ...
5
votes
2answers
140 views

Update a regression on the fly?

Say I have 100 people each with a height, weight, and age. I make a regression that predicts age based on height and weight. Now, I would like to update that model when I meet someone new. I don't ...
5
votes
1answer
60 views

Why do polynomial regressions have larger variance at the end?

In reading the book "An Introduction to Statistical Learning with Applications in R", I came across this graph: It shows that the point-wise variance is larger at the ends of the regression curve. ...
5
votes
1answer
93 views

Complexity of Gaussian Process algorithms is $\mathcal{O}(n^3)$

It is often quoted that the complexity of Gaussian Process algorithms is $\mathcal{O}(n^3)$ due to the need to invert an $n \times n$ matrix, where $n$ is the number of data points. But as far as I ...
5
votes
0answers
160 views

Prove the estimator $\hat{B}$ of ridge regression = mean of the posterior distribution under a Gaussian prior

I want to prove that the estimator of ridge regression is the mean of the posterior distribution under Gaussian prior. $$y \sim N(X\beta,\sigma^2I),\quad \text{prior }\beta \sim N(0,\gamma^2 I).$$ ...
5
votes
0answers
33 views

single variable is significant but overall test is not

I do a multiple regression with 3 independent variables $X_1$, $X_2$ and $X_3$. The correlation between $Y$ and $X_1$, $Y$ and $X_2$, and $Y$ and $X_3$, are each large and statistically significant. ...
5
votes
1answer
274 views

How to compare the similarity between functions?

I'm designing a web service that finds the regression function of a pattern within an image. I analyzed three images and found the following three regressions: 1) $f(x) = 74.7602 + 0.2005x - ...
4
votes
4answers
801 views

Method of Least Squares-Why is it preferred? [duplicate]

Possible Duplicate: Why do we use a Least Squares fit? To find the normal equations for derivation of the regression line we use the method of least squares, We want to make the error ...
4
votes
2answers
163 views

Rewriting the matrix equation $AX = YB$ as $Y = CX$?

Is it possible in general, if $A,B,C,X,Y$ are square and of the same dimensions? If so, does it generalize to non-square matrices (using a pseudoinverse)? I'm doing some curve fitting in which I have ...
4
votes
4answers
364 views

How to model prices?

This is my first question here. As I'm not a matematician I thought I'd ask here for advice how to approach something I'm working on as a hobby project. A bit of context Let's say there is a ...
4
votes
5answers
71 views

calculating least squares fit

I read this thread talking about 'why we use least squares' for curve fitting Why do we use a Least Squares fit? One answer by Chris Taylor begins with the assumption that we should look for $$ ...
4
votes
3answers
208 views

linear solution of curve fitting on multiple linear functions differing by a multiplier

I recently posted this question here but I thought this could be of interest also in mathematics, given I found a partially related question here I am facing the following problem. I know nonlinear ...
4
votes
3answers
712 views

Does the Least Squares Regression Method work for any line type?

I recently learned how to apply the least squares method to do linear regression. I also understand that it can be used for quadratic regression, by minimizing the error for three variables, two ...
4
votes
2answers
2k views

Fitting an exponential function to data

I have a noisy data set (the grey line in the graph below) that corresponds roughly to $y=m(1-2^{-x/k})$ where m and k are unknown constants. How can I determine the best-fit value of m and k? I ...
4
votes
1answer
187 views

When is Block-Partitioned Matrix Invertible?

Suppose I have a block partitioned matrix \begin{equation} \begin{bmatrix} \mathbf{X}_1^{\top}\mathbf{X}_1 & \mathbf{X}_1^{\top}\mathbf{X}_2 \\ \mathbf{X}_2^{\top}\mathbf{X}_1 & ...
4
votes
2answers
117 views

Why choosing Linear Regression?

First I would like to say that I am not a statistician nor am I good in the field. I have been collecting data for over a period of e.g 100 days and each day has a varying amount of data that I can ...
4
votes
2answers
145 views

Linear regression where undershooting isn't as bad as overshooting

Given a set of points $(x_i, y_i)$, least-squares linear regression finds the linear function $L$ such that $$\sum \varepsilon(y_i, L(x_i))$$ is minimized, where $\varepsilon(y, y') = (y-y')^2$ is the ...
4
votes
2answers
23k views

How to calculate a correction factor for two sets of numbers

Suppose one has a set of numbers. To help understand my question, suppose that these numbers are from two different temperature sensors. In this first example, both sensors are placed in the same ...
4
votes
2answers
1k views

How does one fit the curve $y = ae^{bx} + c$?

How does one fit the curve $y = ae^{bx} + c$? The method of taking logarithms of both sides does not simplify to allow linear regression. I can take the three equations derived from setting the ...
4
votes
1answer
414 views

Multicollinearity and SVD

I compute the Singular Value Decomposition of a $n \times n$ matrix. If the matrix is not full rank, and I have 2 collinear columns, I end up with one singular value equal to 0. Is it possible to find ...
4
votes
1answer
223 views

Straight line through data by eye - least squares? [closed]

I heard an interesting fact a while ago about how people draw a line through a cloud of points on a scatter plot. Usually, when calculating lines of best fit, we use the minimal the sum of squares of ...
4
votes
1answer
51 views

Using regression results to predict?

I run some Poisson regressions with the following results: (with number of associations an individual belongs to as the dependent variable) ...
4
votes
1answer
260 views

Who invented linearization of exponential datasets to find their approximating functions?

I just learned how to find the exponential function that approximates a dataset by taking the logarithm of the data points, doing a linear regression on that data, then working out the exponential ...