Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.
15
votes
5answers
2k views
Why do we use a Least Squares fit?
I've been wondering for a while now if there's any deep mathematical or statistical significance to finding the line that minimizes the square of the errors between the line and the data points.
If ...
10
votes
3answers
960 views
Best fit ellipsoid
Given a collection of points $P \subset \mathbb R^3$, a crude characterization of the "shape" of $P$ is sometimes given by the principal components. We construct a covariance matrix, e.g., if $P$ is ...
9
votes
2answers
1k views
Finding the intersection point of many lines in 3D (point closest to all lines)
I have many lines (let's say 4) which are supposed to be intersected. (Please consider lines are represented as a pair of points). I want to find the point in space which minimizes the sum of the ...
8
votes
2answers
249 views
Why is polynomial regression considered a kind of linear regression?
Why is polynomial regression considered a kind of linear regression?
This is what I mean by polynomial regression. For example, the hypothesis function is
$$h(x; t_0, t_1, t_2) = t_0 + t_1 x + t_2 ...
8
votes
3answers
163 views
Minimize $||Ax-b||$ but for $A$, not $x$
I have a machine learning regression problem. I need to minimize
$$
\sum_i||Ax_i-b_i||_2^2
$$
However I am trying to find matrix $A$, not the usual $x$, and I have lots of example data for $x_i$ and ...
7
votes
1answer
361 views
Linear regression is wrong?
I tried asking this in electronics as a question related to oscillators, but I wasn't able to get a satisfactory answer. I think the more math-y types here may shed some additional light on the ...
6
votes
1answer
935 views
easy to implement method to fit a power function (regression)
I want to fit to a dataset a power function ($y=Ax^B$). What is the best and easiest method to do this. I need the $A$ and $B$ parameters too.
I'm using in general financial data in my project, which ...
6
votes
3answers
304 views
Are there variations on least-squares approximations?
In least-squares approximations the normal equations act to project a vector existing in N-dimensional space onto a lower dimensional space, where our problem actually lies, thus providing the "best" ...
6
votes
1answer
145 views
How to find line that has least distance to all points?
I need to find the line having minimal distance to all points. I found linear regression and linear interpolation algorithms. But their minimal distance is only in y-axis: $D = y - f(x)$.
But I need ...
6
votes
1answer
79 views
Formula for straight part of a slightly bumpy line
Given a straight line that deviates from the horizontal by at most 15 degrees.
On this straight line there are bumps on top at random places on the line. The combined width of the bumps is at most ...
5
votes
3answers
1k views
Linear Regression?
How was the formula for Ordinary Least Squares Linear Regression arrived at?
Note I am not only looking for the proof, but also the derivation. Where did the formula come from?
5
votes
2answers
269 views
Why do people fit polynomials?
Could someone explain the justification and limits of fitting polynomials to arbitrary data points? I mean what about square roots or fractional or inverse powers?
Most of the time some wants to ...
5
votes
2answers
886 views
Linear regression for minimizing the maximum of the residuals
We know that simple linear regression will do the following thing:
Suppose there are $n$ data points $\{y_i,x_i\}$, where $i=1,2,\dots,n$. The goal is to find the equation of the straight line
...
5
votes
2answers
231 views
Theoretical basis for overfitting
There are many examples in which making more "precise" predictions gives worse performance (e.g. Runge's phenomenon). My professor implied that there was a sound basis for choosing "simple" functions ...
5
votes
2answers
3k views
How to fit a curve to a sinusoidal wave
I am wondering how to fit a sinusoidal wave (approximation). I would like to fit it in the form: $y = A\sin(Bx + C) + D$ where $A,\,B,\,C$ and $D$ are constants. The only constants I really care about ...
5
votes
1answer
197 views
Can I build a program that will tell me if a real world data set looks linear, logarithmic, exponential etc?
I have a bunch of real world data sets and from manually plotting some of the data in graphs, I've discovered some data sets look pretty much logarithmic and some look linear, or exponential (and some ...
4
votes
2answers
200 views
Recommended Reading on Regression Analysis?
For a university project, I am implementing an automated regression analysis tool.
However, I have very little background in statistics.
So what books / articles / material would you suggest I could ...
4
votes
4answers
257 views
How to model prices?
This is my first question here.
As I'm not a matematician I thought I'd ask here for advice how to approach something I'm working on as a hobby project.
A bit of context
Let's say there is a ...
4
votes
2answers
325 views
Fitting an exponential function to data
I have a noisy data set (the grey line in the graph below) that corresponds roughly to $y=m(1-2^{-x/k})$ where m and k are unknown constants.
How can I determine the best-fit value of m and k?
I ...
4
votes
2answers
56 views
Linear regression where undershooting isn't as bad as overshooting
Given a set of points $(x_i, y_i)$, least-squares linear regression finds the linear function $L$ such that $$\sum \varepsilon(y_i, L(x_i))$$
is minimized, where $\varepsilon(y, y') = (y-y')^2$ is the ...
4
votes
1answer
142 views
Who invented linearization of exponential datasets to find their approximating functions?
I just learned how to find the exponential function that approximates a dataset by taking the logarithm of the data points, doing a linear regression on that data, then working out the exponential ...
3
votes
4answers
209 views
Method of Least Squares-Why is it preferred? [duplicate]
Possible Duplicate:
Why do we use a Least Squares fit?
To find the normal equations for derivation of the regression line we use the method of least squares, We want to make the error ...
3
votes
4answers
824 views
Polynomial fitting where polynomial must be monotonically increasing
Given a set of monotonically increasing data points (in 2D), I want to fit a polynomial to the data which is monotonically increasing over the domain of the data. If the highest x value is 100, I ...
3
votes
1answer
2k views
Computational complexity of least square regression operation
In a least square regression algorithm, I have to do the following operations to compute regression coefficients:
Matrix multiplication, complexity: $O(C^2N)$
Matrix inversion, complexity: ...
3
votes
3answers
278 views
Correlation between variables
I asked this question on stats SE but did not find a suitable answer so far. Maybe someone can help.
Given n random variables x1,...,xn (one-dimensional).
The following is known (corr() = Pearson ...
3
votes
2answers
135 views
How many points to find a polynomial?
I would like to fit a formula $ax^b + cx^d+ e$ to a set of points. I have two questions.
If my data were perfect, how many points do I need in the worst case to get $a,b,c,d,e$ exactly?
If my data ...
3
votes
3answers
138 views
Does the Least Squares Regression Method work for any line type?
I recently learned how to apply the least squares method to do linear regression. I also understand that it can be used for quadratic regression, by minimizing the error for three variables, two ...
3
votes
2answers
2k views
How to calculate a correction factor for two sets of numbers
Suppose one has a set of numbers. To help understand my question, suppose that these numbers are from two different temperature sensors. In this first example, both sensors are placed in the same ...
3
votes
1answer
219 views
How does one fit the curve $y = ae^{bx} + c$?
How does one fit the curve $y = ae^{bx} + c$?
The method of taking logarithms of both sides does not simplify to allow linear regression.
I can take the three equations derived from setting the ...
3
votes
1answer
94 views
Ridge Regression: $\hat{\beta} \rightarrow \beta$
I'm trying to find the probability limit of
$$\hat{\beta} = \left( \sum x_i x'_i + \lambda I_k \right)^{-1} \left( \sum x_i y_i \right) $$ as $n \to \infty$, and $\lambda$ is some positive ...
3
votes
1answer
111 views
Straight line through data by eye - least squares? [closed]
I heard an interesting fact a while ago about how people draw a line through a cloud of points on a scatter plot. Usually, when calculating lines of best fit, we use the minimal the sum of squares of ...
3
votes
2answers
94 views
Can I consider shooting% as an independent variable
First time poster in the math section (a few posts in the stats section) and I am looking for clarification on a variable query that I have. Basically I enjoy sports and enjoy putting a mathematical ...
3
votes
2answers
243 views
Confidence interval of a random variable for an ordinary linear regression
I have a small problem. With my limited stats background I am not sure I am getting this one right.
After fitting an ordinary linear regression model I get
...
3
votes
0answers
86 views
How to compare the similarity between functions?
I'm designing a web service that finds the regression function of a pattern within an image.
I analyzed three images and found the following three regressions:
1) $f(x) = 74.7602 + 0.2005x - ...
3
votes
0answers
26 views
Regression model for a shearing process
30 Widgets are randomly assigned to a shearing process.
There are 3 such processes, each getting 10 widgets.
The lengths of each widget are recorded before undergoing the shearing.
The amount that ...
3
votes
0answers
100 views
How to perform nonlinear regression with correlated errors?
I have a nonlinear least squares problem, but the errors are correlated. I could use R's nls function to do the regression if the errors were independent, but I don't know the right way to handle ...
3
votes
0answers
145 views
Least Square Method with Positive Parameters
this is my first post here in the Stack Exchange. A friend told me about this forum and I'm giving it a try.
I searched a bit past threads, but couldn't find what I wanted, so I'm posting the problem ...
3
votes
1answer
122 views
Bound of linear regression's object function
Randomly uniformly select $n$ numbers from a set $\{1,2,...,U\}$ with/without replacement, $y_i$ is the $i$th number selected, and $x_i$ is the rank of $y_i$ in the $n$ numbers. The rank is the order ...
3
votes
2answers
930 views
Correlation Coefficient and Determination Coefficient
I'm really new to linear regression and am trying to teach myself.
In my textbook there's a problem that asks why R^2 in the regression of Y on X = the sample correlation between X and Y the whole ...
2
votes
1answer
358 views
Polynomial fitting - how to fit and what is _polynomial fitting_
I don't understand what is polynomial fitting.
Can anyone explain me how to fit a curve to given points?
2
votes
2answers
2k views
regression vs classification
THis is more machine learning questions, but perhaps someone will be able to help. I would like to know what is the diference between regression and classification when we try to generate output for a ...
2
votes
4answers
267 views
Given a data set, how do you do a sinusoidal regression on paper? What are the equations, algorithms?
Most regressions are easy. Trivial once you know how to do it. Most of them involve substitutions which transform the data into a linear regression. But I have yet to figure out how to do a ...
2
votes
3answers
149 views
least squares regression in 3space
robjohn is giving me a hand with this, but in case anybody else knows...
I need to do a least-squares regression for linearity on a set of coordinates in 3space. If the dataset is linear, I need to ...
2
votes
2answers
62 views
How do I use specific data points on a graph to determine an equation?
I need to find a function $f(x)$ such that the following data points would fit on it:
$$f(1) = 0 \\
f(2) = 0.5 \\
f(4) = 1.0 \\
f(8) = 1.5 \\
\cdots $$
and so on. So the pattern is every time $x$ ...
2
votes
2answers
174 views
design matrices
Given a linear model $Y = X\beta + \epsilon$ with three treatments and six subjects where $X$ is the design matrix, suppose $X = \begin{matrix}1 & 1 & 0\\
1 & 1 & 0\\
1 & 0 ...
2
votes
1answer
319 views
Linear Regression with 3x3 Matrices
Here's my Homework Problem:
We can generalize the least squares method to other polynomial curves. To find the quadratic equation $y=a x^2+b x+c$ that best fits the points $(-1, −3)$, $(0, 0)$, ...
2
votes
1answer
227 views
How do I do a least squares fit of $a x + b y = 1$?
How do I do a least squares fit of the line equation $a x + b y = 1$, so that the points are as close to the line as possible? (Not just vertically close)
If I use the matrices
$$X = ...
2
votes
1answer
67 views
What is $Cov(\hat{Y},Y)$?
If $\hat{Y}$ is the OLS linear regression model for $Y$, what can I say about $Cov(\hat{Y},Y)$? Is this value 0?
2
votes
1answer
131 views
How to handle constant term in Least Squares Regression?
In the well known matrix form of a least squares regression
where I am trying to solve for B in Y = B1X1 + B2X2 + B3
I might be given X and Y sample data as something like
$X$ =
$\begin{bmatrix}
...
2
votes
3answers
60 views
Condition for $\det(A^{T}A)=0$
Is it always true that
$\det(A^{T}A)=0$, $\hspace{0.5mm}$ for $A=n \times m$ matrix with $n<m$?
From some notes I am reading on Regression analysis, and from some trials, it would appear this is ...
