Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

learn more… | top users | synonyms

20
votes
6answers
3k views

Why do we use a Least Squares fit?

I've been wondering for a while now if there's any deep mathematical or statistical significance to finding the line that minimizes the square of the errors between the line and the data points. If ...
10
votes
2answers
2k views

Finding the intersection point of many lines in 3D (point closest to all lines)

I have many lines (let's say 4) which are supposed to be intersected. (Please consider lines are represented as a pair of points). I want to find the point in space which minimizes the sum of the ...
10
votes
3answers
1k views

Best fit ellipsoid

Given a collection of points $P \subset \mathbb R^3$, a crude characterization of the "shape" of $P$ is sometimes given by the principal components. We construct a covariance matrix, e.g., if $P$ is ...
9
votes
4answers
8k views

regression vs classification

THis is more machine learning questions, but perhaps someone will be able to help. I would like to know what is the diference between regression and classification when we try to generate output for a ...
9
votes
4answers
2k views

Polynomial fitting where polynomial must be monotonically increasing

Given a set of monotonically increasing data points (in 2D), I want to fit a polynomial to the data which is monotonically increasing over the domain of the data. If the highest x value is 100, I ...
8
votes
2answers
504 views

Why is polynomial regression considered a kind of linear regression?

Why is polynomial regression considered a kind of linear regression? This is what I mean by polynomial regression. For example, the hypothesis function is $$h(x; t_0, t_1, t_2) = t_0 + t_1 x + t_2 ...
8
votes
3answers
211 views

Minimize $||Ax-b||$ but for $A$, not $x$

I have a machine learning regression problem. I need to minimize $$ \sum_i||Ax_i-b_i||_2^2 $$ However I am trying to find matrix $A$, not the usual $x$, and I have lots of example data for $x_i$ and ...
7
votes
1answer
2k views

easy to implement method to fit a power function (regression)

I want to fit to a dataset a power function ($y=Ax^B$). What is the best and easiest method to do this. I need the $A$ and $B$ parameters too. I'm using in general financial data in my project, which ...
7
votes
1answer
452 views

Linear regression is wrong?

I tried asking this in electronics as a question related to oscillators, but I wasn't able to get a satisfactory answer. I think the more math-y types here may shed some additional light on the ...
6
votes
1answer
856 views

derivative of cost function for Logistic Regression

I am going over the lectures on Machine Learning at Coursera. I am struggling with the following. How can the partial derivative of ...
6
votes
3answers
332 views

Are there variations on least-squares approximations?

In least-squares approximations the normal equations act to project a vector existing in N-dimensional space onto a lower dimensional space, where our problem actually lies, thus providing the "best" ...
6
votes
1answer
186 views

How to find line that has least distance to all points?

I need to find the line having minimal distance to all points. I found linear regression and linear interpolation algorithms. But their minimal distance is only in y-axis: $D = y - f(x)$. But I need ...
6
votes
1answer
91 views

Formula for straight part of a slightly bumpy line

Given a straight line that deviates from the horizontal by at most 15 degrees. On this straight line there are bumps on top at random places on the line. The combined width of the bumps is at most ...
5
votes
3answers
1k views

Linear Regression?

How was the formula for Ordinary Least Squares Linear Regression arrived at? Note I am not only looking for the proof, but also the derivation. Where did the formula come from?
5
votes
2answers
373 views

Why do people fit polynomials?

Could someone explain the justification and limits of fitting polynomials to arbitrary data points? I mean what about square roots or fractional or inverse powers? Most of the time some wants to ...
5
votes
2answers
230 views

Recommended Reading on Regression Analysis?

For a university project, I am implementing an automated regression analysis tool. However, I have very little background in statistics. So what books / articles / material would you suggest I could ...
5
votes
2answers
2k views

Linear regression for minimizing the maximum of the residuals

We know that simple linear regression will do the following thing: Suppose there are $n$ data points $\{y_i,x_i\}$, where $i=1,2,\dots,n$. The goal is to find the equation of the straight line ...
5
votes
2answers
268 views

Theoretical basis for overfitting

There are many examples in which making more "precise" predictions gives worse performance (e.g. Runge's phenomenon). My professor implied that there was a sound basis for choosing "simple" functions ...
5
votes
2answers
5k views

How to fit a curve to a sinusoidal wave

I am wondering how to fit a sinusoidal wave (approximation). I would like to fit it in the form: $y = A\sin(Bx + C) + D$ where $A,\,B,\,C$ and $D$ are constants. The only constants I really care about ...
5
votes
1answer
232 views

Can I build a program that will tell me if a real world data set looks linear, logarithmic, exponential etc?

I have a bunch of real world data sets and from manually plotting some of the data in graphs, I've discovered some data sets look pretty much logarithmic and some look linear, or exponential (and some ...
5
votes
2answers
2k views

Correlation Coefficient and Determination Coefficient

I'm really new to linear regression and am trying to teach myself. In my textbook there's a problem that asks why R^2 in the regression of Y on X = the sample correlation between X and Y the whole ...
4
votes
1answer
3k views

Computational complexity of least square regression operation

In a least square regression algorithm, I have to do the following operations to compute regression coefficients: Matrix multiplication, complexity: $O(C^2N)$ Matrix inversion, complexity: ...
4
votes
4answers
290 views

How to model prices?

This is my first question here. As I'm not a matematician I thought I'd ask here for advice how to approach something I'm working on as a hobby project. A bit of context Let's say there is a ...
4
votes
3answers
320 views

Does the Least Squares Regression Method work for any line type?

I recently learned how to apply the least squares method to do linear regression. I also understand that it can be used for quadratic regression, by minimizing the error for three variables, two ...
4
votes
2answers
735 views

Fitting an exponential function to data

I have a noisy data set (the grey line in the graph below) that corresponds roughly to $y=m(1-2^{-x/k})$ where m and k are unknown constants. How can I determine the best-fit value of m and k? I ...
4
votes
2answers
87 views

Linear regression where undershooting isn't as bad as overshooting

Given a set of points $(x_i, y_i)$, least-squares linear regression finds the linear function $L$ such that $$\sum \varepsilon(y_i, L(x_i))$$ is minimized, where $\varepsilon(y, y') = (y-y')^2$ is the ...
4
votes
2answers
532 views

How does one fit the curve $y = ae^{bx} + c$?

How does one fit the curve $y = ae^{bx} + c$? The method of taking logarithms of both sides does not simplify to allow linear regression. I can take the three equations derived from setting the ...
4
votes
1answer
176 views

Who invented linearization of exponential datasets to find their approximating functions?

I just learned how to find the exponential function that approximates a dataset by taking the logarithm of the data points, doing a linear regression on that data, then working out the exponential ...
4
votes
0answers
129 views

How to compare the similarity between functions?

I'm designing a web service that finds the regression function of a pattern within an image. I analyzed three images and found the following three regressions: 1) $f(x) = 74.7602 + 0.2005x - ...
4
votes
1answer
40 views

Regression model for a shearing process

30 Widgets are randomly assigned to a shearing process. There are 3 such processes, each getting 10 widgets. The lengths of each widget are recorded before undergoing the shearing. The amount that ...
3
votes
5answers
3k views

Given a data set, how do you do a sinusoidal regression on paper? What are the equations, algorithms?

Most regressions are easy. Trivial once you know how to do it. Most of them involve substitutions which transform the data into a linear regression. But I have yet to figure out how to do a ...
3
votes
4answers
419 views

Method of Least Squares-Why is it preferred? [duplicate]

Possible Duplicate: Why do we use a Least Squares fit? To find the normal equations for derivation of the regression line we use the method of least squares, We want to make the error ...
3
votes
2answers
94 views

How do I use specific data points on a graph to determine an equation?

I need to find a function $f(x)$ such that the following data points would fit on it: $$f(1) = 0 \\ f(2) = 0.5 \\ f(4) = 1.0 \\ f(8) = 1.5 \\ \cdots $$ and so on. So the pattern is every time $x$ ...
3
votes
3answers
461 views

Correlation between variables

I asked this question on stats SE but did not find a suitable answer so far. Maybe someone can help. Given n random variables x1,...,xn (one-dimensional). The following is known (corr() = Pearson ...
3
votes
2answers
165 views

How many points to find a polynomial?

I would like to fit a formula $ax^b + cx^d+ e$ to a set of points. I have two questions. If my data were perfect, how many points do I need in the worst case to get $a,b,c,d,e$ exactly? If my data ...
3
votes
2answers
6k views

How to calculate a correction factor for two sets of numbers

Suppose one has a set of numbers. To help understand my question, suppose that these numbers are from two different temperature sensors. In this first example, both sensors are placed in the same ...
3
votes
1answer
143 views

Ridge Regression: $\hat{\beta} \rightarrow \beta$

I'm trying to find the probability limit of $$\hat{\beta} = \left( \sum x_i x'_i + \lambda I_k \right)^{-1} \left( \sum x_i y_i \right) $$ as $n \to \infty$, and $\lambda$ is some positive ...
3
votes
1answer
54 views

Minimum required data for cosine fit

With a minimum amount of (noisy) data-points, I need to find the amplitude of a simple cosine y=A*cos(x), where x is an angle from 0:2pi. I know how to fit data to the function, and I know how to ...
3
votes
1answer
72 views

Solution to linear system of equations

Notation. Let $y$, $a$, and $b$ be $n\times 1$, $p\times 1$, and $q\times1$ real vectors. Let also $X$ and $Z$ be $n\times p$ and $n \times q$ real matrices. Suppose that there is no solution, $a$, ...
3
votes
1answer
77 views

Constraining estimated linear regression coefficients over several regressions

I'm trying to run a series of simultaneous linear regressions, and I want to constrain the regression coefficients. For the standard ordinary least squares regression, the specification of the ...
3
votes
1answer
72 views

Why is $\sum x^2 _t \times \text{Var}(\beta)=\frac{\sum x^2 _t \times \sigma^2}{ \sum x^2 _t} = \sigma^2$?

I do not get this connection. Is is reliable to divide this equation by $\sum x^2 _t$ to get just $\sigma^2$ ? $$\sum x^2 _t \times E(\hat \beta - \beta)^2=\sum x^2 _t \times ...
3
votes
1answer
142 views

Straight line through data by eye - least squares? [closed]

I heard an interesting fact a while ago about how people draw a line through a cloud of points on a scatter plot. Usually, when calculating lines of best fit, we use the minimal the sum of squares of ...
3
votes
2answers
109 views

Can I consider shooting% as an independent variable

First time poster in the math section (a few posts in the stats section) and I am looking for clarification on a variable query that I have. Basically I enjoy sports and enjoy putting a mathematical ...
3
votes
2answers
448 views

Confidence interval of a random variable for an ordinary linear regression

I have a small problem. With my limited stats background I am not sure I am getting this one right. After fitting an ordinary linear regression model I get ...
3
votes
0answers
60 views

Can the sigmoid function approximate any function (or relation) where 0<y<1

I'm studying Machine Learning and Artificial Neural Networks. Some basic principles of Machine Learning are linear regression, multivariate linear regression, and nonlinear regression. The last of ...
3
votes
0answers
130 views

How to perform nonlinear regression with correlated errors?

I have a nonlinear least squares problem, but the errors are correlated. I could use R's nls function to do the regression if the errors were independent, but I don't know the right way to handle ...
3
votes
0answers
233 views

Least Square Method with Positive Parameters

this is my first post here in the Stack Exchange. A friend told me about this forum and I'm giving it a try. I searched a bit past threads, but couldn't find what I wanted, so I'm posting the problem ...
3
votes
0answers
776 views

Correlation and Regression Question

Two separate tests are designed to measure a students ability to solve problems. Several students are randomly selected to take both tests and the results are: $$ \begin{matrix} \text{Test A}(x) ...
3
votes
1answer
132 views

Bound of linear regression's object function

Randomly uniformly select $n$ numbers from a set $\{1,2,...,U\}$ with/without replacement, $y_i$ is the $i$th number selected, and $x_i$ is the rank of $y_i$ in the $n$ numbers. The rank is the order ...
2
votes
1answer
531 views

Polynomial fitting - how to fit and what is _polynomial fitting_

I don't understand what is polynomial fitting. Can anyone explain me how to fit a curve to given points?