Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

learn more… | top users | synonyms

2
votes
1answer
1k views

What is $Cov(\hat{Y},Y)$?

If $\hat{Y}$ is the OLS linear regression model for $Y$, what can I say about $\operatorname{Cov}(\hat{Y},Y)$? Is this value $0$?
2
votes
2answers
303 views

design matrices

Given a linear model $Y = X\beta + \epsilon$ with three treatments and six subjects where $X$ is the design matrix, suppose $X = \begin{matrix}1 & 1 & 0\\ 1 & 1 & 0\\ 1 & 0 ...
2
votes
1answer
827 views

Linear Regression with 3x3 Matrices

Here's my Homework Problem: We can generalize the least squares method to other polynomial curves. To find the quadratic equation $y=a x^2+b x+c$ that best fits the points $(-1, −3)$, $(0, 0)$, ...
2
votes
1answer
272 views

How do I do a least squares fit of $a x + b y = 1$?

How do I do a least squares fit of the line equation $a x + b y = 1$, so that the points are as close to the line as possible? (Not just vertically close) If I use the matrices $$X = ...
2
votes
2answers
4k views

Prove $SST=SSE+SSR$

Prove $$SST=SSE+SSR$$ I start with $$SST= \Sigma (y_i-\bar{y})^2=...=SSE+SSR+ \Sigma 2( y_i-y_i^*)(y_i^*-\bar{y} )$$ and I don't know how to prove that $\Sigma 2( y_i-y_i^*)(y_i^*-\bar{y} )=0$ a ...
2
votes
1answer
562 views

QR factorization for ridge regression

I am solving an overdetermined system of equations: $$Ax= b$$ Using QR factorization, we can solve this system easily by posing it as: $$Rx= Q'b$$ I would like to regularize my estimate of $x$. I ...
2
votes
2answers
141 views

Linear Models - Regression Analysis

As a student learning Applied Regression Analysis, I come from a background with very little information about this topic. I understand that given $y = \beta_0 + \beta_1x_1 + \epsilon$ $E(y\mid x) = ...
2
votes
1answer
2k views

Hat Matrix Identities in Regression

I need to show that $\bar h= \sum{h_{ii}/n} = \operatorname{Tr}[H]/n = (p+1)/n$ Using the fact that $\operatorname{Tr}[AB]=\operatorname{Tr}[BA]$ and $H=X(X^TX)^{-1}X^T$. But I have no idea how to ...
2
votes
3answers
390 views

Residuals of regression model

Let's suppose I do a regression between earnings and age (and suppose I do not know the distribution of earnings). Would it be possible for the residuals to be normally distributed? I am thinking it ...
2
votes
2answers
454 views

Calculate Line Of Best Fit Using Exponential Weighting?

I know how to calculate a line of best fit with a set of data. I want to be able to exponentially weight the data that is more recent so that the more recent data has a greater effect on the line. ...
2
votes
1answer
239 views

How to handle constant term in Least Squares Regression?

In the well known matrix form of a least squares regression where I am trying to solve for B in Y = B1X1 + B2X2 + B3 I might be given X and Y sample data as something like $X$ = $\begin{bmatrix} ...
2
votes
1answer
147 views

Solving for a 3D point in a 5D graph given 3 pairs of 2D points.

I am attempting to solve the values $C$, $D$, and $S$, given three pairs of $[M,R]$. $$R = \frac {M}{C - MDC + DC\left(MS\right)^2}$$ I have been able to solve for a related equation (or rather, ...
2
votes
1answer
38 views

Question on regression

So I've been given this formula For regression $R^2=1 - \sum \frac{{(y_i - \hat{y}_i)}^2}{(y_1-\bar{y})^2}$ Now an obvious question that has come to me is why $R^2$ stays the same in certain ...
2
votes
1answer
38 views

How to take the derivative of Matrices

I was browsing the derivation of the Least Squares estimates and stumbled about this problem. It said that: $$E = (Y + XB)^2$$ $$\frac{dE}{dB} = -X^TY + X^TXB$$ It is to my understanding that the ...
2
votes
1answer
48 views

How can I plot this?

Given a bunch of data $x_i$ , $y_i$, how do I plot $$f(\theta_2,\theta_2)= \frac{1}{2M} \sum_{i=1}^{M} (\theta_1\cdot x_i -\theta_2 y_i)^2$$ in matlab? I know it should be parabolic, but my code ...
2
votes
1answer
192 views

Fitting a simple linear regression

A professor in the School of Business in a university polled a dozen colleagues about the number of professional meetings they attended in the past five years $x$ and the number of papers they ...
2
votes
2answers
428 views

connection between PCA and linear regression

Is there a formal link between linear regression and PCA? The goal of PCA is to decompose a matrix into a linear combination of variables that contain most of the information in the matrix. Suppose ...
2
votes
1answer
2k views

Convert nonlinear regression equation to a linear regression equation

The question is: "Show how the nonlinear regression equation y=aX^B can be converted to a linear regression equation solvable by the method of least ...
2
votes
1answer
64 views

Choosing the right regression

I'm trying to analyze my sleep using regression analysis. Each night is rated (dependent variable). I'm trying to explain this rating with, for example, my sleep duration and each night's bed time's ...
2
votes
1answer
30 views

Determine which parameter has correlation with result and which is not

sorry for probably silly question, it's the first time when I need to do such work. I have large data set with regarding clicks on some element on web page. It contains some characteristics of such ...
2
votes
1answer
1k views

Lasso - constraint form equivalent to penalty form

We know that there are two definitions to describe lasso. Regression with constraint definition: $$\min\limits_{\beta} \|y-X\beta\|^2, \sum\limits_{p}|\beta_p|\leq t, \exists t $$ Regression with ...
2
votes
4answers
100 views

Condition for $\det(A^{T}A)=0$

Is it always true that $\det(A^{T}A)=0$, $\hspace{0.5mm}$ for $A=n \times m$ matrix with $n<m$? From some notes I am reading on Regression analysis, and from some trials, it would appear this is ...
2
votes
1answer
488 views

Arriving at the Logistic function from a Binomial Distribution and Maximum Likelihood

I've been trying to understand the origin of the Logistic function in Logistic regression: $$\Pr(Y=1|x;\theta)=\frac{1}{1+e^{-\theta x}}$$ I was lead to beilive that one could somehow arrive at this ...
2
votes
1answer
173 views

Find data to perform regression analysis

I'm trying to find some data (two continuous variables that I believe are correlated) online for which I can perform a regression anaylsis, my assignment sheet says: The data may be found anywhere ...
2
votes
1answer
5k views

Equations For Quadratic Regression

Does anyone know the specific equations for the three parameters in a least-squares quadratic regression? I'm looking for something like $\beta_1=,\beta_2=,\beta_3=$ for each of ...
2
votes
1answer
110 views

Linear relationship of a company's profit

Assume a linear relationship for a company that has several shops is not known. Let $Y_i$ be the profit the shop number $i$ makes in the coming year. Let $x_i$ be the size of the shop number ...
2
votes
1answer
394 views

An intuitive explanation for neural networks as function approximators ?

We use normal linear regression for modelling functions on datasets . But Can someone explain how neural networks help in approximating more complex ,especially non-linear functions ? intuitively , ...
2
votes
4answers
89 views

Seeking a function based on its level set

I'm trying to create a function for a research project, but I fear my math knowledge is insufficient to derive it from the attached diagram I've created showing its desired behavior. I'm hoping ...
2
votes
3answers
522 views

How to deal with Linear Regression model with some data aggregated

Lets say I am trying to find a linear regression between Weight and Height of a person. $W=b_0+b_1 H+e$ The data I have gathered from 8 people is like this: ...
2
votes
1answer
5k views

How to calculate hyperbola from data points?

I have 4 data points, from which I want to calculate a hyperbola. It seems that the Excel trendline feature can't do it for me, so how do I find the relationship? The points are: (x,y) (3, 0.008) ...
2
votes
1answer
47 views

Estimating landing position for a slowly falling object using latitude, longitude, and altitude.

I have a weather balloon project, in which I intend to use GPS to locate the payload when it finally comes down again. I will make the computer send coordinates to a server every minute or so, as ...
2
votes
1answer
68 views

Relationship between $L^1$ norm and sparsity

I'm doing some research in the field of sparse representation and sparse modeling. I have two variables and their $L^1$ norm is calculated to make comparisons. As I take it the smaller the value of ...
2
votes
1answer
40 views

Predicting trends of timeseries data with ARIMA

I'm looking for an algorithm that can help identify abnormal trends in time-series metrics. The best I've been able to find so far is ARIMA (a completely new concept for me). We offer several ...
2
votes
1answer
25 views

Linear regression and standardization

I am trying to use a linear regression to model an expected value Y for an input X. X and Y have a large difference between them, so I was converting to standard (z) score, doing my calculation ...
2
votes
1answer
29 views

Discrete version of continuous SIR model

I'm working with a SIR infection model, which is $$\begin{array}{rcl} \frac{dS}{dt} & = & -\beta IS\\ \frac{dI}{dt} & = & \beta IS-\gamma I\\ \frac{dR}{dt} & = & \gamma I ...
2
votes
2answers
49 views

Fitting two parallel lines to a set of points

In two dimension I have a set of points X = $\{x_1,..., x_N\}$. I want to fit two parallel lines to these points like $l_1$ and $l_2$ $$l_1 = p_1 + \lambda n^\perp$$ $$l_2 = p_2 + \lambda n^\perp$$ ...
2
votes
1answer
92 views

Least-squares solution to a matrix equation?

Suppose I have $n$ observations of $m$ dependent variables $y_1,\dots,y_m$, and I believe they follow some model wherein they can all be written as linear combinations of some underlying variables ...
2
votes
1answer
71 views

Outlier detection with robust multiple regression model

I have a set of features (eg, location, income, budget, education) that I use to predict a continuous variable (say, amount spent per day on the internet). I am interested in detecting outliers. I ...
2
votes
1answer
36 views

Prove a result in multiple linear regression

This arises in multiple linear regression. Given $m, n \in \mathbb{N}$ and matrices $X \in \mathbb{R}^{m \times (n+1)} (m > n + 1), H = X(X'X)^{-1}X' \in \mathbb{R}^{m\times m}, I = I_m$ and $J ...
2
votes
1answer
86 views

Is it possible to have two lines of best fit?

Could you rig a data set to have two lines of equally good (and best) fit? Or is it impossible?
2
votes
1answer
23 views

Three-Perpendicular Theorem for linear regressions

For a random vector $X=(X_1,\ldots,X_p)'$, we define $$ \mathcal{L}(X)=\{b_0+b_1X_1+\cdots+b_pX_p,b_0,\ldots,b_p\in\mathbb{R}\}. $$ The linear regression of the $q$-dimensional random vector ...
2
votes
1answer
57 views

Standard Error in OLS Regression

Assuming I have the following linear regression set-up: $y_i = \alpha + x_i * \beta + \epsilon_i$ for $i = 1,2,..., n$. When I run the regession, I get a $\beta$ and $\alpha$ estimates, along ...
2
votes
1answer
105 views

Prove that $E(\mathbf{u}|\mathbf{X})=\mathbf{0}$ implies $Cov(\mathbf{x},\mathbf{u})=\mathbf{0}$

Let \begin{equation} \mathbf{y}=\mathbf{X}\mathbf{\beta}+\mathbf{u} \end{equation} where $\mathbf{y}=\begin{bmatrix}y_1 \\ \vdots \\ y_n\end{bmatrix}$, $\mathbf{X}=\begin{bmatrix}X_{11} & ...
2
votes
1answer
400 views

Least Squares Regression Matrix for Rational Functions

So first off no, this isn't a homework problem. Second, I'm trying to understand how this works, NOT find a program that will do it for me. Okay so I've known for a while how to use Gaussian-Jordan ...
2
votes
2answers
99 views

orthogonal matrices vs. orthogonal columns

I'm just reading a book on econometrics and now I'm stuck with a problem: There is a Theorem on "Orthogonal Partitioned Regression" which says: "In the multiple linear least squares regression of ...
2
votes
1answer
37 views

Gaussian prior favors values closest to zero?

I am reading an article on Bayesian Logistic Regression, where they're using Logistic Regression, imposing a Gaussian prior (with mean = 0) on its parameters. They state that a Gaussian prior favors ...
2
votes
2answers
134 views

derivation of simple linear regression parameters

I know there are some proof in the internet, but I attempted to proove the formulas for the intercept and the slope in simple linear regression using Least squares, some algebra, and partial ...
2
votes
2answers
75 views

Find parameters for curve fitting (simple linear regression involved?)

I would like to fit data in g~t scatterplot, where ...
2
votes
2answers
51 views

Merging Linear Regression

If I have built two linear regression models over sets $A$ and $B$, and now want a linear regression over set $A\cup{}B$. Is there a way to reuse what I already have?
2
votes
2answers
40 views

Regression model when under-estimations costs us more than over-estimations

We have a factory and we are planning how many items produce in 2014. During the learning process we minimize the mean squared error. But, under-estimations costs us more than over-estimations. Let's ...