Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

learn more… | top users | synonyms

21
votes
7answers
4k views

Why do we use a Least Squares fit?

I've been wondering for a while now if there's any deep mathematical or statistical significance to finding the line that minimizes the square of the errors between the line and the data points. If ...
10
votes
2answers
3k views

Finding the intersection point of many lines in 3D (point closest to all lines)

I have many lines (let's say 4) which are supposed to be intersected. (Please consider lines are represented as a pair of points). I want to find the point in space which minimizes the sum of the ...
10
votes
3answers
1k views

Best fit ellipsoid

Given a collection of points $P \subset \mathbb R^3$, a crude characterization of the "shape" of $P$ is sometimes given by the principal components. We construct a covariance matrix, e.g., if $P$ is ...
9
votes
4answers
11k views

regression vs classification

THis is more machine learning questions, but perhaps someone will be able to help. I would like to know what is the diference between regression and classification when we try to generate output for a ...
9
votes
4answers
2k views

Polynomial fitting where polynomial must be monotonically increasing

Given a set of monotonically increasing data points (in 2D), I want to fit a polynomial to the data which is monotonically increasing over the domain of the data. If the highest x value is 100, I ...
8
votes
2answers
571 views

Why is polynomial regression considered a kind of linear regression?

Why is polynomial regression considered a kind of linear regression? This is what I mean by polynomial regression. For example, the hypothesis function is $$h(x; t_0, t_1, t_2) = t_0 + t_1 x + t_2 ...
8
votes
1answer
4k views

Computational complexity of least square regression operation

In a least square regression algorithm, I have to do the following operations to compute regression coefficients: Matrix multiplication, complexity: $O(C^2N)$ Matrix inversion, complexity: ...
8
votes
3answers
222 views

Minimize $||Ax-b||$ but for $A$, not $x$

I have a machine learning regression problem. I need to minimize $$ \sum_i||Ax_i-b_i||_2^2 $$ However I am trying to find matrix $A$, not the usual $x$, and I have lots of example data for $x_i$ and ...
7
votes
2answers
1k views

derivative of cost function for Logistic Regression

I am going over the lectures on Machine Learning at Coursera. I am struggling with the following. How can the partial derivative of ...
7
votes
1answer
2k views

easy to implement method to fit a power function (regression)

I want to fit to a dataset a power function ($y=Ax^B$). What is the best and easiest method to do this. I need the $A$ and $B$ parameters too. I'm using in general financial data in my project, which ...
7
votes
1answer
472 views

Linear regression is wrong?

I tried asking this in electronics as a question related to oscillators, but I wasn't able to get a satisfactory answer. I think the more math-y types here may shed some additional light on the ...
6
votes
3answers
338 views

Are there variations on least-squares approximations?

In least-squares approximations the normal equations act to project a vector existing in N-dimensional space onto a lower dimensional space, where our problem actually lies, thus providing the "best" ...
6
votes
1answer
201 views

How to find line that has least distance to all points?

I need to find the line having minimal distance to all points. I found linear regression and linear interpolation algorithms. But their minimal distance is only in y-axis: $D = y - f(x)$. But I need ...
6
votes
1answer
95 views

Formula for straight part of a slightly bumpy line

Given a straight line that deviates from the horizontal by at most 15 degrees. On this straight line there are bumps on top at random places on the line. The combined width of the bumps is at most ...
5
votes
3answers
1k views

Linear Regression?

How was the formula for Ordinary Least Squares Linear Regression arrived at? Note I am not only looking for the proof, but also the derivation. Where did the formula come from?
5
votes
2answers
391 views

Why do people fit polynomials?

Could someone explain the justification and limits of fitting polynomials to arbitrary data points? I mean what about square roots or fractional or inverse powers? Most of the time some wants to ...
5
votes
2answers
235 views

Recommended Reading on Regression Analysis?

For a university project, I am implementing an automated regression analysis tool. However, I have very little background in statistics. So what books / articles / material would you suggest I could ...
5
votes
2answers
2k views

Linear regression for minimizing the maximum of the residuals

We know that simple linear regression will do the following thing: Suppose there are $n$ data points $\{y_i,x_i\}$, where $i=1,2,\dots,n$. The goal is to find the equation of the straight line ...
5
votes
2answers
6k views

How to fit a curve to a sinusoidal wave

I am wondering how to fit a sinusoidal wave (approximation). I would like to fit it in the form: $y = A\sin(Bx + C) + D$ where $A,\,B,\,C$ and $D$ are constants. The only constants I really care about ...
5
votes
2answers
279 views

Theoretical basis for overfitting

There are many examples in which making more "precise" predictions gives worse performance (e.g. Runge's phenomenon). My professor implied that there was a sound basis for choosing "simple" functions ...
5
votes
1answer
240 views

Can I build a program that will tell me if a real world data set looks linear, logarithmic, exponential etc?

I have a bunch of real world data sets and from manually plotting some of the data in graphs, I've discovered some data sets look pretty much logarithmic and some look linear, or exponential (and some ...
5
votes
2answers
55 views

Update a regression on the fly?

Say I have 100 people each with a height, weight, and age. I make a regression that predicts age based on height and weight. Now, I would like to update that model when I meet someone new. I don't ...
5
votes
2answers
2k views

Correlation Coefficient and Determination Coefficient

I'm really new to linear regression and am trying to teach myself. In my textbook there's a problem that asks why R^2 in the regression of Y on X = the sample correlation between X and Y the whole ...
4
votes
4answers
318 views

How to model prices?

This is my first question here. As I'm not a matematician I thought I'd ask here for advice how to approach something I'm working on as a hobby project. A bit of context Let's say there is a ...
4
votes
3answers
363 views

Does the Least Squares Regression Method work for any line type?

I recently learned how to apply the least squares method to do linear regression. I also understand that it can be used for quadratic regression, by minimizing the error for three variables, two ...
4
votes
2answers
819 views

Fitting an exponential function to data

I have a noisy data set (the grey line in the graph below) that corresponds roughly to $y=m(1-2^{-x/k})$ where m and k are unknown constants. How can I determine the best-fit value of m and k? I ...
4
votes
2answers
98 views

Linear regression where undershooting isn't as bad as overshooting

Given a set of points $(x_i, y_i)$, least-squares linear regression finds the linear function $L$ such that $$\sum \varepsilon(y_i, L(x_i))$$ is minimized, where $\varepsilon(y, y') = (y-y')^2$ is the ...
4
votes
2answers
638 views

How does one fit the curve $y = ae^{bx} + c$?

How does one fit the curve $y = ae^{bx} + c$? The method of taking logarithms of both sides does not simplify to allow linear regression. I can take the three equations derived from setting the ...
4
votes
1answer
188 views

Who invented linearization of exponential datasets to find their approximating functions?

I just learned how to find the exponential function that approximates a dataset by taking the logarithm of the data points, doing a linear regression on that data, then working out the exponential ...
4
votes
1answer
25 views

Using regression results to predict?

I run some Poisson regressions with the following results: (with number of associations an individual belongs to as the dependent variable) ...
4
votes
0answers
142 views

How to compare the similarity between functions?

I'm designing a web service that finds the regression function of a pattern within an image. I analyzed three images and found the following three regressions: 1) $f(x) = 74.7602 + 0.2005x - ...
4
votes
1answer
40 views

Regression model for a shearing process

30 Widgets are randomly assigned to a shearing process. There are 3 such processes, each getting 10 widgets. The lengths of each widget are recorded before undergoing the shearing. The amount that ...
3
votes
5answers
4k views

Given a data set, how do you do a sinusoidal regression on paper? What are the equations, algorithms?

Most regressions are easy. Trivial once you know how to do it. Most of them involve substitutions which transform the data into a linear regression. But I have yet to figure out how to do a ...
3
votes
4answers
467 views

Method of Least Squares-Why is it preferred? [duplicate]

Possible Duplicate: Why do we use a Least Squares fit? To find the normal equations for derivation of the regression line we use the method of least squares, We want to make the error ...
3
votes
2answers
100 views

How do I use specific data points on a graph to determine an equation?

I need to find a function $f(x)$ such that the following data points would fit on it: $$f(1) = 0 \\ f(2) = 0.5 \\ f(4) = 1.0 \\ f(8) = 1.5 \\ \cdots $$ and so on. So the pattern is every time $x$ ...
3
votes
3answers
500 views

Correlation between variables

I asked this question on stats SE but did not find a suitable answer so far. Maybe someone can help. Given n random variables x1,...,xn (one-dimensional). The following is known (corr() = Pearson ...
3
votes
2answers
63 views

Why Linear Regression

First i will like to say that i am not a statistician nor am i good in the field. I have been collecting data for over a period of e.g 100 days and each day has a varying amount of data that i can ...
3
votes
2answers
168 views

How many points to find a polynomial?

I would like to fit a formula $ax^b + cx^d+ e$ to a set of points. I have two questions. If my data were perfect, how many points do I need in the worst case to get $a,b,c,d,e$ exactly? If my data ...
3
votes
2answers
8k views

How to calculate a correction factor for two sets of numbers

Suppose one has a set of numbers. To help understand my question, suppose that these numbers are from two different temperature sensors. In this first example, both sensors are placed in the same ...
3
votes
1answer
16k views

How to find curve equation from data?

How do I find the formula when I only know some data points ? Usually I just use the Trendline option for diagrams in Excel, but this one eludes me. I expect it to be something like : ...
3
votes
1answer
152 views

Ridge Regression: $\hat{\beta} \rightarrow \beta$

I'm trying to find the probability limit of $$\hat{\beta} = \left( \sum x_i x'_i + \lambda I_k \right)^{-1} \left( \sum x_i y_i \right) $$ as $n \to \infty$, and $\lambda$ is some positive ...
3
votes
1answer
68 views

Minimum required data for cosine fit

With a minimum amount of (noisy) data-points, I need to find the amplitude of a simple cosine y=A*cos(x), where x is an angle from 0:2pi. I know how to fit data to the function, and I know how to ...
3
votes
1answer
80 views

Solution to linear system of equations

Notation. Let $y$, $a$, and $b$ be $n\times 1$, $p\times 1$, and $q\times1$ real vectors. Let also $X$ and $Z$ be $n\times p$ and $n \times q$ real matrices. Suppose that there is no solution, $a$, ...
3
votes
1answer
85 views

Constraining estimated linear regression coefficients over several regressions

I'm trying to run a series of simultaneous linear regressions, and I want to constrain the regression coefficients. For the standard ordinary least squares regression, the specification of the ...
3
votes
1answer
73 views

Why is $\sum x^2 _t \times \text{Var}(\beta)=\frac{\sum x^2 _t \times \sigma^2}{ \sum x^2 _t} = \sigma^2$?

I do not get this connection. Is is reliable to divide this equation by $\sum x^2 _t$ to get just $\sigma^2$ ? $$\sum x^2 _t \times E(\hat \beta - \beta)^2=\sum x^2 _t \times ...
3
votes
1answer
148 views

Straight line through data by eye - least squares? [closed]

I heard an interesting fact a while ago about how people draw a line through a cloud of points on a scatter plot. Usually, when calculating lines of best fit, we use the minimal the sum of squares of ...
3
votes
2answers
111 views

Can I consider shooting% as an independent variable

First time poster in the math section (a few posts in the stats section) and I am looking for clarification on a variable query that I have. Basically I enjoy sports and enjoy putting a mathematical ...
3
votes
2answers
501 views

Confidence interval of a random variable for an ordinary linear regression

I have a small problem. With my limited stats background I am not sure I am getting this one right. After fitting an ordinary linear regression model I get ...
3
votes
0answers
43 views

Determine whether ARMA(p,q) is stationary and/or invertible?

Determine whether an ARMA(p,q) process is stationary and invertible such that $y_t = \sum_{i=1}^{p} \phi_i y_{t-i} + \sum_{i=1}^{p} \theta_{i} \epsilon_{t-i}$ with the restriction that ...
3
votes
1answer
32 views

Logarithmic Functions algebra question

This is my first post and I honestly just want a second opinion on my answer to a question I got incorrect on an exam before I go arguing over it with my professor. Basically, is this mathematically ...