Questions tagged [regression]

This tag is for questions on (linear or nonlinear) regression, which is a way of describing how one variable, the outcome, is numerically related to predictor variables. The dependent variable is also referred to as $~Y~$, dependent or response and is plotted on the vertical axis (ordinate) of a graph.

Filter by
Sorted by
Tagged with
13
votes
2answers
4k views

Why is polynomial regression considered a kind of linear regression?

Why is polynomial regression considered a kind of linear regression? This is what I mean by polynomial regression. For example, the hypothesis function is $$h(x; t_0, t_1, t_2) = t_0 + t_1 x + t_2 ...
57
votes
11answers
24k views

Why do we use a Least Squares fit?

I've been wondering for a while now if there's any deep mathematical or statistical significance to finding the line that minimizes the square of the errors between the line and the data points. If ...
94
votes
6answers
65k views

derivative of cost function for Logistic Regression

I am going over the lectures on Machine Learning at Coursera. I am struggling with the following. How can the partial derivative of $$J(\theta)=-\frac{1}{m}\sum_{i=1}^{m}y^{i}\log(h_\theta(x^{i}))+...
16
votes
3answers
19k views

Derivation of the formula for Ordinary Least Squares Linear Regression

How was the formula for Ordinary Least Squares Linear Regression arrived at? Note I am not only looking for the proof, but also the derivation. Where did the formula come from?
12
votes
1answer
11k views

easy to implement method to fit a power function (regression)

I want to fit to a dataset a power function ($y=Ax^B$). What is the best and easiest method to do this. I need the $A$ and $B$ parameters too. I'm using in general financial data in my project, which ...
2
votes
3answers
195 views

Polynomial best fit line for very large values

not only are the x values large, the difference between them and the y values is huge. My data points: $22353120,720$ $24448725,671.427053270323$ $26544330,634.312274868634$ $28639935,566....
1
vote
2answers
1k views

Hat matrix with simple linear regression

In these lecture notes: However I am unable to work this out myself. When I multiply things out I get $\frac{1}{nS_{xx}}(\sum_{j=1}^n x_j^2 -2n\bar{x}x_i+nx_i^2)$. For things to be true, the terms ...
13
votes
4answers
9k views

Deriving cost function using MLE :Why use log function?

I am learning machine learning from Andrew Ng's open-class notes and coursera.org. I am trying to understand how the cost function for the logistic regression is derived. I will start with the cost ...
19
votes
1answer
426 views

On the integral $\int_{-\pi/2}^{\pi/2}\sin(x/\sin(x/\sin(x/\sin\cdots)))\,dx$

This question is the final one out of the set (see I and II), I promise! Consider $f_1(x)=\sin(x)$ and $f_2(x)=\sin\left(\frac x{f_1(x)}\right)$ such that $f_n$ satisfies the relation $$f_n(x)=\sin\...
20
votes
3answers
14k views

Finding the intersection point of many lines in 3D (point closest to all lines)

I have many lines (let's say 4) which are supposed to be intersected. (Please consider lines are represented as a pair of points). I want to find the point in space which minimizes the sum of the ...
3
votes
2answers
749 views

Why is minimizing least squares equivalent to finding the projection matrix $\hat{x}=A^Tb(A^TA)^{-1}$?

I understand the derivation for $\hat{x}=A^Tb(A^TA)^{-1}$, but I'm having trouble explicitly connecting it to least squares regression. So suppose we have a system of equations: $A=\begin{bmatrix}1 &...
3
votes
5answers
1k views

Why use the kernel trick in an SVM as opposed to just transforming the data?

Why use the kernel trick in a support vector machine as opposed to just transforming the data and then using a linear classifier? Certainly, we'll approximately double the amount of memory required ...
2
votes
2answers
294 views

Prove that $E(|(X+Y)(X-Y)|) \leq 2\sqrt{1-\rho²}$, where $\rho$ is correlation.

For two random variables $X,Y$ with mean $0$ and variance $1$, their correlation is $\rho$. We have to prove that $$E(|(X+Y)(X-Y)|) \leq 2\sqrt{1-\rho^2}.$$ But, I can't understand how the $\rho$ ...
0
votes
2answers
66 views

Linear Regression Computation as $y = ax$

I got a process which can be modelised as a Linear Regression matching an $y = ax$ equation. I can find on the internet computations to match an $y = ax +b$ equation like this $$ b = \frac{\sum y\sum ...
26
votes
2answers
26k views

Computational complexity of least square regression operation

In a least square regression algorithm, I have to do the following operations to compute regression coefficients: Matrix multiplication, complexity: $O(C^2N)$ Matrix inversion, complexity: $O(C^3)...
11
votes
5answers
37k views

Given a data set, how do you do a sinusoidal regression on paper? What are the equations, algorithms?

Most regressions are easy. Trivial once you know how to do it. Most of them involve substitutions which transform the data into a linear regression. But I have yet to figure out how to do a ...
15
votes
4answers
7k views

Polynomial fitting where polynomial must be monotonically increasing

Given a set of monotonically increasing data points (in 2D), I want to fit a polynomial to the data which is monotonically increasing over the domain of the data. If the highest x value is 100, I don'...
5
votes
2answers
1k views

why small L1 norm means sparsity?

I'm trying to understand regularization in machine learning. one way of regularization is adding a l1 norm to the error function. This is said to produce sparsity. But I can't understand. sparsity is ...
4
votes
1answer
5k views

What is $\operatorname{Cov}(\widehat{Y},Y)$?

If $\hat{Y}$ is the OLS linear regression model for $Y$, what can I say about $\operatorname{Cov}(\hat{Y},Y)$? Is this value $0$?
5
votes
2answers
9k views

Equations For Quadratic Regression

Does anyone know the specific equations for the three parameters in a least-squares quadratic regression? I'm looking for something like $\beta_1=,\beta_2=,\beta_3=$ for each of $y=\beta_1+\beta_2x+\...
3
votes
4answers
4k views

Fit exponential with constant

I have data whic would fit to an exponential function with a constant. $$y=a\cdot\exp(b\cdot t) + c.$$ Now I can solve an exponential without a constant using least square by taking log of y and ...
4
votes
1answer
1k views

Polynomial fitting - how to fit and what is _polynomial fitting_

I don't understand what is polynomial fitting. Can anyone explain to me how to fit a curve to given points?
3
votes
3answers
88 views

Determine which of $N$ points is not on $\sin(ax + b)$, where $a$ and $b$ are unknown.

Suppose $N$ points ($(x_1,y_1), (x_2,y_2), ... (x_N,y_N)$) are given from a curve $y=\sin(ax+b)$ where $a, b$ values are unknown. Before giving these $N$ points to you, $y$ coordinate of one point is ...
1
vote
2answers
1k views

Linear trend has to pass through a point

I need to interpolate a linear trend surface through a number of points but with the condition that the surface has to pass exactly through one of them. Can somebody give me any advice?
3
votes
2answers
438 views

Multivariate Quadratic Regression

I would like to make a polynomial regression, but for multivariate input data. In the univariate case, one can write polynomial regression as a multivariate linear regression problem and can come up ...
1
vote
1answer
85 views

Iterative Power Regression

If I have a set of data points that would fit inside a power equation of the form y = a*x^b, what is the best ITERATIVE method to find the values of 'a' and 'b'. I thought I could compute the error ...
-1
votes
2answers
531 views

Proving that the estimate of a mean is a least squares estimator?

I think this is a really simple question so please bear with me - I just had my first class in regression and I'm a little confused about nomenclature/labeling. Does anyone recommend some good ...
40
votes
4answers
11k views

Why divide by $2m$

I'm taking a machine learning course. The professor has a model for linear regression. Where $h_\theta$ is the hypothesis (proposed model. linear regression, in this case), $J(\theta_1)$ is the cost ...
11
votes
1answer
7k views

Lasso - constraint form equivalent to penalty form

We know that there are two definitions to describe lasso. Regression with constraint definition: $$\min\limits_{\beta} \|y-X\beta\|^2, \sum\limits_{p}|\beta_p|\leq t, \exists t $$ Regression with ...
13
votes
4answers
30k views

Correlation Coefficient and Determination Coefficient

I'm really new to linear regression and am trying to teach myself. In my textbook there's a problem that asks why $R^{2}$ in the regression of $Y$ on $X =$ the sample correlation between X and Y the ...
27
votes
5answers
13k views

Why get the sum of squares instead of the sum of absolute values?

I'm self-studying machine learning and getting into the basics of linear regression models. From what I understand so far, a good regression model minimizes the sum of the squared differences between ...
6
votes
1answer
5k views

Analytic solution for matrix factorization using alternating least squares

The standard form for ridge regression aims to minimize the following cost function. $$ \min\ \ \sum_i(y_i-x_i^T\beta)^2 + \lambda\sum_j\beta^2_j $$ As described here, it's possible to differentiate ...
3
votes
1answer
6k views

Construct / find the simplest function based on data

Let's say I have these 7 natural numbers (all between 0 and 255): 255, 23, 45, 32, 87, 52, 146 How can I find a function F(x) that, once computed, gives me back ...
3
votes
1answer
197 views

A problem on sinusoids

Given $N$ points $\{x_i\} \in (0,1)$ and $N$ real numbers $\{d_i\}$ such that $\sum\limits_{i=1}^{N}d_i = 0$. Can we find a function of the form $$f(x) = A_k\sin(\pi x + \theta_k), x_{k-1}\le x \le x_{...
1
vote
1answer
2k views

Show posterior probability takes the form of the logistic function

Suppose you have a D-dimensional data vector $x$ = ($x_1$, ..., $x_n$) and associated class variable $y \in \{0, 1\}$, which is Bernoulli with parameter $\alpha$. Assume the dimensions of $x$ are ...
16
votes
3answers
3k views

Best fit ellipsoid

Given a collection of points $P \subset \mathbb R^3$, a crude characterization of the "shape" of $P$ is sometimes given by the principal components. We construct a covariance matrix, e.g., if $P$ is ...
9
votes
3answers
27k views

Fitting exponential curve to data

If I have a collection of data points that follow an exponential curve relationship, how can I manually construct the equation that defines the best-fit exponential curve for the data?
7
votes
3answers
8k views

Log-likelihood gradient and Hessian

Considering a binary classification problem with data $D = \{(x_i,y_i)\}_{i=1}^n$, $x_i \in \mathbb{R}^d$ and $y_i \in \{0,1\}$. Given the following definitions: $f(x) = x^T \beta$ $p(x) = \sigma(f(...
6
votes
1answer
1k views

Expectation and orthogonal projection

Many books while introducing the regression problem, start with the assertion that any random variable $Y$ can be decomposed into two orthogonal terms $$ Y= E[Y|X]+\epsilon. $$ In the classical ...
5
votes
1answer
359 views

Two dimensional (discrete) orthogonal polynomials for regression

This question How to work out orthogonal polynomials for regression model and the answer https://math.stackexchange.com/a/354807/51020 explain how to build orthogonal polynomials for regression. ...
10
votes
2answers
16k views

Linear regression for minimizing the maximum of the residuals

We know that simple linear regression will do the following thing: Suppose there are $n$ data points $\{y_i,x_i\}$, where $i=1,2,\dots,n$. The goal is to find the equation of the straight line $y=\...
5
votes
2answers
6k views

Is the Square Root of an Inverse Matrix Equal to the Inverse of the Square Root Matrix?

I know in general that if a matrix $A$ is positive definite, then there exists a (unique?) square root matrix $B$, which is also positive definite, such that $BB=A$. Therefore, suppose $A$ is ...
4
votes
1answer
2k views

Variant of linear regression using perpendicular distance instead of vertical

Normally, linear regression asks for a pair of parameters m,b such that for a set of given points $\{x_i,y_i\}$ the variance of $y-m\cdot x-b$ is minimized (this minimizes the distance in y-direction ...
3
votes
1answer
549 views

Basic exponential regression

Background: I've been struggling with an exponential regression problem for about 8 months now (on and off): Vertically translated depreciation curve: Update the exponential regression coefficient ...
2
votes
1answer
5k views

Mean response in linear regression

What does mean response in linear regession mean? I don't understand the definition given in wikipedia. This is the definition: Mean response is an estimate of the mean of the $y$ population ...
2
votes
1answer
2k views

How to work out orthogonal polynomials for regression model

I put this question here as it has a pure maths element to it, even though it has a statistical twist. Basically, I have been given the table of data: $$\begin{matrix} i & \mathrm{Response} \, ...
2
votes
1answer
379 views

Machine Learning: Linear Regression models

I'm currently in a course learning about neural networks and machine learning, and I came across these two formulas in this textbook page on linear regression: 1) $y(x) = a + bx$ and 2) $y(x) = w^{...
2
votes
2answers
3k views

derivation of simple linear regression parameters

I know there are some proof in the internet, but I attempted to proove the formulas for the intercept and the slope in simple linear regression using Least squares, some algebra, and partial ...
1
vote
1answer
83 views

How the regression method affects the smoothness of the ’regression curve’ when we used them as a smoothing method?

The following Quizzes are the rough translation (with minor modification) of Quizzes No.08 of the exam of the "2019's semi-first grade of Japan Statistical Society Certificate (JSSC)" (See (ref 1) ). ...
0
votes
1answer
2k views

Why is $E(u)=0$ when an intercept is included in OLS Estimation?

I am reading Wooldridge's graduate econometrics text. There he states that when estimating the equation $y=\mathbf{x\beta}+u$ by OLS, if an intercept (constant term) is included in your $\mathbf{x}$ ...