Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

learn more… | top users | synonyms

3
votes
1answer
140 views

Bound of linear regression's object function

Randomly uniformly select $n$ numbers from a set $\{1,2,...,U\}$ with/without replacement, $y_i$ is the $i$th number selected, and $x_i$ is the rank of $y_i$ in the $n$ numbers. The rank is the order ...
2
votes
1answer
754 views

Polynomial fitting - how to fit and what is _polynomial fitting_

I don't understand what is polynomial fitting. Can anyone explain me how to fit a curve to given points?
2
votes
1answer
64 views

When is $\mathbf{X}^{T}\mathbf{X}+\lambda\mathbf{I}$ invertible?

The question is quite simple: for a $N \times p$ matrix $\mathbf{X}$ with real entries, when is $\mathbf{X}^{T}\mathbf{X}+\lambda\mathbf{I}$ invertible (where $\mathbf{I}$ is the $p \times p$ identity ...
2
votes
3answers
198 views

least squares regression in 3space

robjohn is giving me a hand with this, but in case anybody else knows... I need to do a least-squares regression for linearity on a set of coordinates in 3space. If the dataset is linear, I need to ...
2
votes
3answers
140 views

Proof of Gauss-Markov theorem

Theorem: Let $Y=X\beta+\varepsilon$ where $$Y\in\mathcal M_{n\times 1}(\mathbb R),$$ $$X\in \mathcal M_{n\times p}(\mathbb R),$$ $$\beta\in\mathcal M_{n\times 1}(\mathbb R ),$$ and $$\varepsilon\in\...
2
votes
2answers
603 views

Closed form for coefficients in Multiple Regression model

I want to find $\hat{\beta}$ in ordinary least squares s.t. $\hat{Y} = \hat{\beta}_0 + \hat{\beta}_1 X_1 + \cdots + \hat{\beta}_n X_n $. I know the way to do this is through the normal equation using ...
2
votes
2answers
44 views

Interpreting OLS Regression Coefficients with High Multicolinearity

I am having trouble understanding the interpretation of OLS coefficients when predictors are highly correlated. My understanding of OLS coefficients is that they estimate a change in the expected ...
2
votes
1answer
1k views

Intuition and the math behind normalization

What exactly is the purpose of normalization. From what I read, it is to adjust two different sets of values so you can compare them, but I don't understand why, nor the math behind it. Could anyone ...
2
votes
1answer
1k views

What is $Cov(\hat{Y},Y)$?

If $\hat{Y}$ is the OLS linear regression model for $Y$, what can I say about $\operatorname{Cov}(\hat{Y},Y)$? Is this value $0$?
2
votes
2answers
306 views

design matrices

Given a linear model $Y = X\beta + \epsilon$ with three treatments and six subjects where $X$ is the design matrix, suppose $X = \begin{matrix}1 & 1 & 0\\ 1 & 1 & 0\\ 1 & 0 & 1\...
2
votes
1answer
931 views

Linear Regression with 3x3 Matrices

Here's my Homework Problem: We can generalize the least squares method to other polynomial curves. To find the quadratic equation $y=a x^2+b x+c$ that best fits the points $(-1, −3)$, $(0, 0)$, $(...
2
votes
1answer
274 views

How do I do a least squares fit of $a x + b y = 1$?

How do I do a least squares fit of the line equation $a x + b y = 1$, so that the points are as close to the line as possible? (Not just vertically close) If I use the matrices $$X = \left[\begin{...
2
votes
2answers
863 views

connection between PCA and linear regression

Is there a formal link between linear regression and PCA? The goal of PCA is to decompose a matrix into a linear combination of variables that contain most of the information in the matrix. Suppose ...
2
votes
2answers
147 views

Linear Models - Regression Analysis

As a student learning Applied Regression Analysis, I come from a background with very little information about this topic. I understand that given $y = \beta_0 + \beta_1x_1 + \epsilon$ $E(y\mid x) = ...
2
votes
1answer
2k views

Hat Matrix Identities in Regression

I need to show that $\bar h= \sum{h_{ii}/n} = \operatorname{Tr}[H]/n = (p+1)/n$ Using the fact that $\operatorname{Tr}[AB]=\operatorname{Tr}[BA]$ and $H=X(X^TX)^{-1}X^T$. But I have no idea how to ...
2
votes
3answers
393 views

Residuals of regression model

Let's suppose I do a regression between earnings and age (and suppose I do not know the distribution of earnings). Would it be possible for the residuals to be normally distributed? I am thinking it ...
2
votes
2answers
467 views

Calculate Line Of Best Fit Using Exponential Weighting?

I know how to calculate a line of best fit with a set of data. I want to be able to exponentially weight the data that is more recent so that the more recent data has a greater effect on the line. ...
2
votes
1answer
268 views

How to handle constant term in Least Squares Regression?

In the well known matrix form of a least squares regression where I am trying to solve for B in Y = B1X1 + B2X2 + B3 I might be given X and Y sample data as something like $X$ = $\begin{bmatrix} ...
2
votes
1answer
22 views

Deriving the identity: $\hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}$

For some reason I am having an extremely hard time finding out how the following expression is derived $$ \hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} $$ Is ...
2
votes
1answer
46 views

How many data points are “enough” for linear regression?

I have data points $(x_t,y_t)$ generated from $y_t = a + b x_t + \epsilon$ where $\epsilon$ is gaussian error term with zero mean and unknown variance. I want to estimate coefficients $a$ and $b$ but ...
2
votes
1answer
151 views

Solving for a 3D point in a 5D graph given 3 pairs of 2D points.

I am attempting to solve the values $C$, $D$, and $S$, given three pairs of $[M,R]$. $$R = \frac {M}{C - MDC + DC\left(MS\right)^2}$$ I have been able to solve for a related equation (or rather, ...
2
votes
1answer
38 views

Question on regression

So I've been given this formula For regression $R^2=1 - \sum \frac{{(y_i - \hat{y}_i)}^2}{(y_1-\bar{y})^2}$ Now an obvious question that has come to me is why $R^2$ stays the same in certain ...
2
votes
1answer
38 views

How to take the derivative of Matrices

I was browsing the derivation of the Least Squares estimates and stumbled about this problem. It said that: $$E = (Y + XB)^2$$ $$\frac{dE}{dB} = -X^TY + X^TXB$$ It is to my understanding that the ...
2
votes
1answer
51 views

How can I plot this?

Given a bunch of data $x_i$ , $y_i$, how do I plot $$f(\theta_2,\theta_2)= \frac{1}{2M} \sum_{i=1}^{M} (\theta_1\cdot x_i -\theta_2 y_i)^2$$ in matlab? I know it should be parabolic, but my code (...
2
votes
1answer
944 views

Understanding Regularization parameters in Machine Learning/Statistics

Suppose I have the following $k$ degree polynomial regression model with a data set of size $n$ which includes a $k$-dimensional feature vector $x$ and an outcome denoted $t_i$ for each vector in the ...
2
votes
2answers
194 views

Polynomial regression - correctness and accuracy

I have just finished a code that performs polynomial regression, doing $(X'X)^{-1}X'y$ (where $X'$ is the transpose) to estimate the vector of coefficients. Now I'd like to add some check procedures ...
2
votes
1answer
2k views

Convert nonlinear regression equation to a linear regression equation

The question is: "Show how the nonlinear regression equation y=aX^B can be converted to a linear regression equation solvable by the method of least ...
2
votes
1answer
68 views

Choosing the right regression

I'm trying to analyze my sleep using regression analysis. Each night is rated (dependent variable). I'm trying to explain this rating with, for example, my sleep duration and each night's bed time's ...
2
votes
1answer
31 views

Determine which parameter has correlation with result and which is not

sorry for probably silly question, it's the first time when I need to do such work. I have large data set with regarding clicks on some element on web page. It contains some characteristics of such ...
2
votes
4answers
102 views

Condition for $\det(A^{T}A)=0$

Is it always true that $\det(A^{T}A)=0$, $\hspace{0.5mm}$ for $A=n \times m$ matrix with $n<m$? From some notes I am reading on Regression analysis, and from some trials, it would appear this is ...
2
votes
1answer
514 views

Arriving at the Logistic function from a Binomial Distribution and Maximum Likelihood

I've been trying to understand the origin of the Logistic function in Logistic regression: $$\Pr(Y=1|x;\theta)=\frac{1}{1+e^{-\theta x}}$$ I was lead to beilive that one could somehow arrive at this ...
2
votes
1answer
214 views

Find data to perform regression analysis

I'm trying to find some data (two continuous variables that I believe are correlated) online for which I can perform a regression anaylsis, my assignment sheet says: The data may be found anywhere ...
2
votes
1answer
6k views

Equations For Quadratic Regression

Does anyone know the specific equations for the three parameters in a least-squares quadratic regression? I'm looking for something like $\beta_1=,\beta_2=,\beta_3=$ for each of $y=\beta_1+\beta_2x+\...
2
votes
1answer
111 views

Linear relationship of a company's profit

Assume a linear relationship for a company that has several shops is not known. Let $Y_i$ be the profit the shop number $i$ makes in the coming year. Let $x_i$ be the size of the shop number $i$. ...
2
votes
2answers
1k views

How to do a regression with only integer values and a fixed intercept?

I need to write some code for an application that takes in a series of 2D points whose values are integers, and determines a polynomial regression that passes through the origin. I know how to do this ...
2
votes
4answers
89 views

Seeking a function based on its level set

I'm trying to create a function for a research project, but I fear my math knowledge is insufficient to derive it from the attached diagram I've created showing its desired behavior. I'm hoping ...
2
votes
3answers
553 views

How to deal with Linear Regression model with some data aggregated

Lets say I am trying to find a linear regression between Weight and Height of a person. $W=b_0+b_1 H+e$ The data I have gathered from 8 people is like this: ...
2
votes
1answer
24 views

Variance of Least Squares Estimator

Suppose a fit a line using the method of least squares to $n$ points, all the standard statistical assumptions hold, and I want to estimate that line at a new point, $x_0$. Denoting that value by $\...
2
votes
1answer
24 views

Can a prediction interval be interpreted as a probability?

Suppose I find a 90% prediction interval for some data distribution. This implies that if I sample large enough data from this distribution, then 90% of such data will lie inside the prediction ...
2
votes
1answer
212 views

Simple Regression ~ House Price Prediction

I am stuck with this question. "You have a data set consisting of the sales prices of houses in your neighborhood, with each sale time-stamped by the month and year in which the house sold. You ...
2
votes
1answer
10 views

Inverse function for non-linear regression purpouses

The Setting: I want to perform a regression onto data of that follows this shape: \begin{equation} U(x):=\sum_{i=1}^N\, a_ix^ie^{-b_ix} \end{equation} where the $a_i\in \mathbb{R}$ and the $b_i \in (...
2
votes
1answer
59 views

Estimating landing position for a slowly falling object using latitude, longitude, and altitude.

I have a weather balloon project, in which I intend to use GPS to locate the payload when it finally comes down again. I will make the computer send coordinates to a server every minute or so, as ...
2
votes
1answer
73 views

Relationship between $L^1$ norm and sparsity

I'm doing some research in the field of sparse representation and sparse modeling. I have two variables and their $L^1$ norm is calculated to make comparisons. As I take it the smaller the value of $...
2
votes
1answer
46 views

Predicting trends of timeseries data with ARIMA

I'm looking for an algorithm that can help identify abnormal trends in time-series metrics. The best I've been able to find so far is ARIMA (a completely new concept for me). We offer several ...
2
votes
1answer
28 views

Linear regression and standardization

I am trying to use a linear regression to model an expected value Y for an input X. X and Y have a large difference between them, so I was converting to standard (z) score, doing my calculation (...
2
votes
1answer
44 views

Discrete version of continuous SIR model

I'm working with a SIR infection model, which is $$\begin{array}{rcl} \frac{dS}{dt} & = & -\beta IS\\ \frac{dI}{dt} & = & \beta IS-\gamma I\\ \frac{dR}{dt} & = & \gamma I \end{...
2
votes
2answers
60 views

Fitting two parallel lines to a set of points

In two dimension I have a set of points X = $\{x_1,..., x_N\}$. I want to fit two parallel lines to these points like $l_1$ and $l_2$ $$l_1 = p_1 + \lambda n^\perp$$ $$l_2 = p_2 + \lambda n^\perp$$ $...
2
votes
1answer
678 views

Linear Regression Model, linearity in parameters/ variables

I am confusing with the wording here. I was reading a book on linear regression. "The primary concern for linear models is that they display linearity in the parameters. Therefore, when we refer to a ...
2
votes
1answer
97 views

Least-squares solution to a matrix equation?

Suppose I have $n$ observations of $m$ dependent variables $y_1,\dots,y_m$, and I believe they follow some model wherein they can all be written as linear combinations of some underlying variables $...
2
votes
2answers
86 views

Outlier detection with robust multiple regression model

I have a set of features (eg, location, income, budget, education) that I use to predict a continuous variable (say, amount spent per day on the internet). I am interested in detecting outliers. I ...