Questions tagged [regression]

This tag is for questions on (linear or nonlinear) regression, which is a way of describing how one variable, the outcome, is numerically related to predictor variables. The dependent variable is also referred to as $~Y~$, dependent or response and is plotted on the vertical axis (ordinate) of a graph.

815 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
6
votes
0answers
311 views

Perturbation theory for least squares for very different A, b

Consider the least squares problem $f(x;A,b) = \|Ax-b\|_2^2$ and define $x^*$ the minimizer of $f(x;\hat A,\hat b)$, and $\hat x$ the minimizer of $f(x; A_2, b_2)$. I want to put some bound on $\|...
5
votes
0answers
58 views

single variable is significant but overall test is not

I do a multiple regression with 3 independent variables $X_1$, $X_2$ and $X_3$. The correlation between $Y$ and $X_1$, $Y$ and $X_2$, and $Y$ and $X_3$, are each large and statistically significant. ...
5
votes
0answers
378 views

What is ${\rm cov}(e_i, \hat y_i)$ in simple linear regression?

The model is $y_i = \beta_0 + \beta_1x_i + \epsilon_i$ What is ${\rm cov}(e_i, \hat y_i)$? What is ${\rm cov}(\epsilon_i, \hat \beta_1)$? What is ${\rm cov}(e_i, \epsilon_i)$? For 1, I am writing $...
5
votes
0answers
477 views

Multilinear or Tensor Regression?

Given input data $x_t\in \mathbb{R}^n$ and output data $y_t\in\mathbb{R}^m$, the closed form solution to $\min_A \sum_t \|y_t - Ax_t\|^2_2$ is given by $A = (XX^T)^{-1}XY^T$ where $x_t$ form the ...
4
votes
1answer
78 views

Estimating Spline curve by OLS. Is a good idea to fix the knots at Chebyshev sites?

I am writing my master's degree thesis on a novel method for fixing knots in an adaptive way and while reading the literature I've found many references to the so-called Chebyshev sites. This sites or ...
4
votes
0answers
90 views

Why do perfect square values to $ax^2 +ax +1$ form an exponential function?

While playing around with numbers using Python, I found that the set of values of x which fulfilled $$ax^2 + ax +1 = p^2$$ Where p is an integer form an exponential function. For example, $$3x^2 + ...
4
votes
0answers
39 views

Write the likelihood in terms of the unknown parameter β, $f_x$ and g

Consider $n$ i.i.d. pairs of random variables $(X_i, Y_i), i = 1, . . . , n,$ where $X_i ∈ R_p (p ≥ 1)$ and $Y_i ∈ R_p$. For each $i$, write $Y_i = X′_i\beta + \varepsilon_i$, where $E[\...
4
votes
0answers
684 views

Maximum Likelihood Estimator (MLE ) for the Gaussian-noise simple linear regression model

I am reviewing the method of maximum likelihood and I was looking at the Gaussian-noise simple linear regression model at this link. I understand up to equation (3) on page 2. I assumed it followed ...
4
votes
0answers
77 views

How to perform nonlinear regression with regressors affected by gaussian error?

I am trying to calibrate a sensor and I have a data set consisting of several observations of a 3-dimensional vector $X_i$, with $X_i=w_i + \epsilon_i$ where $w_i$ is the value that the sensor ...
4
votes
1answer
133 views

Weighted least squares with angular data

Suppose I have a system whose state is $\Theta=(\theta_1,\theta_2,\ldots,\theta_n)$, where $\theta_i\in[-\pi,\pi)$ (i.e., they are angles). I'd like to determine the most likely estimate of $\Theta$ ...
4
votes
0answers
314 views

Bayesian linear regression cost function

I am studying classification using linear regression . Now, I want to map it in Bayesian regression. Let talk about binary classification using linear regression again. Assume that I have a set $X=${...
4
votes
1answer
49 views

Regression model for a shearing process

30 Widgets are randomly assigned to a shearing process. There are 3 such processes, each getting 10 widgets. The lengths of each widget are recorded before undergoing the shearing. The amount that ...
3
votes
2answers
122 views

Ridge regression to minimize RMSE instead of MSE

Given a metrix $X$ and a vector $\vec{y}$, ordinary least squares (OLS) regression tries to find $\vec{c}$ such that $\left\| X \vec{c} - \vec{y} \right\|_2^2$ is minimal. (If we assume that $\left\| ...
3
votes
0answers
145 views

Condition number of $AA^T$ when $A$ is polynomial Vandermonde

Suppose I'm doing polynomial regression of degree $m$ $$p(x, \mathbf{w}) = w_0 + w_1x + \dotsb + w_mx^m$$ given training data $(x_1, t_1), \dotsc, (x_N, t_N)$. Suppose I'm using the loss function $...
3
votes
0answers
53 views

How far apart can L1 and L2 lines fit to the same data be?

Given $n$ points $(x_i, y_i)$ in the unit square with $x_i = {{i - 1} \over {n - 1}}$ uniformly spaced and $0 \leq y_i \leq 1$, consider the best-fit L1 line and the best-fit L2 line: $ \qquad$ L1 ...
3
votes
0answers
68 views

Asymptotic distribution of non-linear least squares

Assume an i.i.d sample of a scalar dependent variable $y_i$ and a $k$-dimensional regressor $x_i$. Assume that $\mathbb{E}[y_i \mid x_i] = \exp\left(x_i'\beta_0\right)$ and $Var(y_i \mid x_i) = \exp\...
3
votes
1answer
1k views

What are the steps to convert weighted sum of squares to matrix form?

I'm new to converting formulas to matrix form. But this is required for efficient machine learning code. So I want to understand the "right" way, not the cowboy stuff I do. Alright here we go, I'm ...
3
votes
2answers
918 views

Minimize the number of points in a piecewise linear approximation

I have $m$ data points $(x_i,y_i)$ in a given interval. I would like to find a piecewise linear function $f(x)$ that approximate these $m$ points with a minimum number of points $n$ so that my ...
3
votes
0answers
64 views

Derive the Hat Matrix to map actual response to estimated resposne

In order to measure the quality of a regression we can calculate the Hat Matrix. Using it we can estimate the response variable as if we used the predictor variables to regress them. For linear ...
3
votes
1answer
146 views

Detrending sine waves accurately

I am doing some data analysis where I look at electricity demand over the course of a day, but need to separate the intra-day (constant and periodic) components from daily changes (assumed linear). At ...
3
votes
0answers
57 views

Regression with many discrete and continuous predictors and few rows

I want to do regression on a dataset. It has one continuous dependent variable that I want to predict. It has many categorical and some continuous predictors. It only has a few rows. A simplified ...
3
votes
0answers
177 views

Uni-variate Moving Average Theta coefficients

Consider the Uni-variate Moving Average Models (MA models) MA(1) $$x_t = \mu + w_t +\theta_1w_{t-1}$$ or the second order moving average MA(2) $$x_t = \mu + w_t +\theta_1w_{t-1}+\theta_2w_{t-2}$$ ...
3
votes
1answer
40 views

Logarithmic Functions algebra question

This is my first post and I honestly just want a second opinion on my answer to a question I got incorrect on an exam before I go arguing over it with my professor. Basically, is this mathematically ...
3
votes
0answers
901 views

Can the sigmoid function approximate any function (or relation) where 0<y<1

I'm studying Machine Learning and Artificial Neural Networks. Some basic principles of Machine Learning are linear regression, multivariate linear regression, and nonlinear regression. The last of ...
3
votes
1answer
165 views

Proving $\text{Var}{(\hat{y}_h)} = \sigma^2 \left(\frac{1}{n} + \frac{(x_h-\bar{x})^2}{S_{xx}}\right)$

I have asked in another question how $\text{Var}{(\hat{y}_h)} = \sigma^2 \left(\frac{1}{n} + \frac{(x_h-\bar{x})^2}{S_{xx}}\right)$. Note that $\hat{y}_h$ = $b_0 + b_1X_h$ which is a regression line ...
3
votes
0answers
493 views

Minimizing L4/ L6/ L2N norm for linear regression

OLS regression minimizes the sum of the squared errors. The normal equation for an OLS for $L_2$ minimization is as follows: $$b= (A'A)^{-1}A'y$$ What would be the equation to minimize the $L_4$ norm ...
3
votes
0answers
99 views

Regressing $Y$ back on the residuals

Suppose I have the linear regression model $ \hat{y_i} = a + b x_i $ for $a,b$ obtained via OLS. How does one regress $y$ back on the residuals $\hat{e}_i = y_i - \hat{y}_i$? If we write $ \hat{\hat{...
3
votes
0answers
256 views

How to perform nonlinear regression with correlated errors?

I have a nonlinear least squares problem, but the errors are correlated. I could use R's nls function to do the regression if the errors were independent, but I don't know the right way to handle ...
3
votes
0answers
416 views

Least Square Method with Positive Parameters

this is my first post here in the Stack Exchange. A friend told me about this forum and I'm giving it a try. I searched a bit past threads, but couldn't find what I wanted, so I'm posting the problem ...
2
votes
0answers
11 views

Multiple Linear Regression using expected values instead of observations

Normally when doing multiple linear regression we use multiple observations of the features to estimate the coefficients, in my case I want to minimize the square error. This formula normally is: $\...
2
votes
0answers
20 views

Computing the variance of $Y$ given $Y_i=a+bX_i+u_i$, where the distribution of $u_i$ is known

I'm working on the following problem, which I quote verbatim from the assignment. Suppose that in the regression equation $$Y_i=a+bX_i+u_i$$ the coefficients are $a=1$ and $b=2$. Suppose also ...
2
votes
0answers
39 views

Vapnik-Chervonenkis dimension for (polynomial) interpolation/regression

I may have some misconceptions here, but by my understanding, the Vapnik-Chervonenkis dimension of a class of functions describes the number of points it can be guaranteed to classify correctly. I'm ...
2
votes
1answer
30 views

Time-series modeling

I'm wondering what methods can be used to predict a future value using past values. I looked into linear regression modeling, but this doesn't allow for a time value. As an example, say I have an ...
2
votes
0answers
27 views

Is it appropriate to use clustering to partition the dependent variable into separate datasets for a home price prediction model?

I'm struggling to decide how to deal with a heteroskedasticity problem in a home price prediction model I'm developing. The training set residuals are normally distributed around zero, but they have ...
2
votes
1answer
76 views

Rank of sub-matrix of projection matrix

Consider the projection matrix in Linear Regression $P=X(X^TX)^{-1}X^T$. If we have $n$ points, $P$ is an $n$ x $n$ matrix. We also know it satisfies a number of properties, including that it's ...
2
votes
0answers
41 views

Equation for the Human Spine

I have data that I believe might fit some kind of semi-sinusoidal trend line - I'm trying to derive an equation for the length of individual vertebrae in various species (I am not a mathematician). I'...
2
votes
0answers
54 views

How to solve linear regression with an uncommon error function?

For common linear regression problems, the error terms are $l2$ norm. In other words, the error between measurement (independent values) $y$, and estimate $\hat{y}=X\beta$ is measured as $||y-\hat{y}||...
2
votes
0answers
36 views

Fitting a spline: find coefficients using Fourier Transform?

I came up with a idea to estimate the coefficients of a B-spline fit by using the Fourier Transform but I don't know if it makes any sense to estimate them in this way. Given that $$s(x)=\sum_kc(k)\...
2
votes
1answer
95 views

Deriving the odds ratio of a 3-way interaction logistic regression model

Suppose a logistic regression model has three binary explanatory variables $x_1$, $x_2$ and $x_3$ used to estimate the probability of success. This model includes all three main effects, the three $2$-...
2
votes
0answers
43 views

Elastic Net as LASSO

Good evening everybody, I need help with an excercise on Regularised Regression. What I need to do is turn an Elastic-net problem: \begin{equation} argmin_\omega \Vert y-\Phi(x)^T \omega \Vert_2^2 + ...
2
votes
0answers
39 views

From Ng video: Using feature normalization with polynomial regression

In this video on machine learning by Andrew Ng, called "Features and Polynomial Regression", at time 4:34, he mentions the possibility of feature normalization in polynomial regression. By which he ...
2
votes
0answers
168 views

Linear regression with feature representation confusion - is design matrix column space the feature space?

I am trying to visualise the geometry of linear regression with feature representation. I have a regression problem with $n$ data pairs $\mathcal{D}:=\{(\mathbf{x},y)_{i}\}_{i=1}^{n}$, independent ...
2
votes
0answers
104 views

Integration by parts with empirical measure

I'm currently reading through the paper Asymptotic normality of nearest neighbor regression function estimates and am struggling to understand the asymptotic equalities that were shown in the proof of ...
2
votes
1answer
196 views

Minimization problem with latent function and splines

I have a dataset consisting of pairs $(x_i, y_i)$. I want to determine the function $f$, so $$ f(x)f(y) = 1 $$ with the constraint that $f(x) \leq x$, $f'(x) \geq 0$ and $f''(x) \geq 0$. I was ...
2
votes
0answers
53 views

Finding an interval by curve fitting

Suppose I have a function like the one in this graph: I have samples $x_i,y_i$ from this function, with some Gaussian noise (like the black points). The function behaves, in some interval $[a,b]$, ...
2
votes
1answer
52 views

Statistical limitations of conducting nonlinear regression of $x$ vs.$ f(y)$ as opposed to $y$ vs. $f(x)$

I derived a solution to a physics problem using a method of analysis that gives me an inversed relation, the independent variable $t$ expressed explicitly in terms of a nontrivial algebraic set of ...
2
votes
0answers
21 views

What are tolerance intervals for linear regression?

What are tolerance intervals for linear regression. I am trying to break down this paper into general terms, not a specific problem. What are they used for? What is the basic principles behind this ...
2
votes
0answers
53 views

Linear regression where the error is modified

I have a set of coordinates $\{(x_1,y_1),(x_2,y_2),\ldots,(x_n,y_n)\}$ , where for every $i<n$, $$ x_i \ll x_{i\space+\space1}\space\space\text{and}\space\space y_i\ll y_{i\space+\space1} $$ I know ...
2
votes
1answer
592 views

$x^TAx$ and the eigenvalues of $A$

Let A denote a symmetric matrix in $\mathbb R^{N\times N}$, and let $f$ denote the function defined for all $x\in \mathbb R^{N}$\{0} by $$f(x) = \frac{x^TAx}{x^Tx}$$ Let $\lambda_1,....,\lambda_N$ ...
2
votes
1answer
487 views

Linear regression with integer function

I have a dataset, that only takes integer values ($x$ and $y$ coordiantes). E.g. my data is the following: $x = (1,2,2,3,3), y = (1,2,3,3,4)$. I want to make a linear regression through the data, i.e....