Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

learn more… | top users | synonyms

2
votes
2answers
574 views

Closed form for coefficients in Multiple Regression model

I want to find $\hat{\beta}$ in ordinary least squares s.t. $\hat{Y} = \hat{\beta}_0 + \hat{\beta}_1 X_1 + \cdots + \hat{\beta}_n X_n $. I know the way to do this is through the normal equation using ...
2
votes
2answers
38 views

Interpreting OLS Regression Coefficients with High Multicolinearity

I am having trouble understanding the interpretation of OLS coefficients when predictors are highly correlated. My understanding of OLS coefficients is that they estimate a change in the expected ...
2
votes
1answer
1k views

Intuition and the math behind normalization

What exactly is the purpose of normalization. From what I read, it is to adjust two different sets of values so you can compare them, but I don't understand why, nor the math behind it. Could anyone ...
2
votes
1answer
1k views

What is $Cov(\hat{Y},Y)$?

If $\hat{Y}$ is the OLS linear regression model for $Y$, what can I say about $\operatorname{Cov}(\hat{Y},Y)$? Is this value $0$?
2
votes
2answers
306 views

design matrices

Given a linear model $Y = X\beta + \epsilon$ with three treatments and six subjects where $X$ is the design matrix, suppose $X = \begin{matrix}1 & 1 & 0\\ 1 & 1 & 0\\ 1 & 0 ...
2
votes
1answer
893 views

Linear Regression with 3x3 Matrices

Here's my Homework Problem: We can generalize the least squares method to other polynomial curves. To find the quadratic equation $y=a x^2+b x+c$ that best fits the points $(-1, −3)$, $(0, 0)$, ...
2
votes
1answer
273 views

How do I do a least squares fit of $a x + b y = 1$?

How do I do a least squares fit of the line equation $a x + b y = 1$, so that the points are as close to the line as possible? (Not just vertically close) If I use the matrices $$X = ...
2
votes
2answers
5k views

Prove $SST=SSE+SSR$

Prove $$SST=SSE+SSR$$ I start with $$SST= \Sigma (y_i-\bar{y})^2=...=SSE+SSR+ \Sigma 2( y_i-y_i^*)(y_i^*-\bar{y} )$$ and I don't know how to prove that $\Sigma 2( y_i-y_i^*)(y_i^*-\bar{y} )=0$ a ...
2
votes
2answers
143 views

Linear Models - Regression Analysis

As a student learning Applied Regression Analysis, I come from a background with very little information about this topic. I understand that given $y = \beta_0 + \beta_1x_1 + \epsilon$ $E(y\mid x) = ...
2
votes
1answer
2k views

Hat Matrix Identities in Regression

I need to show that $\bar h= \sum{h_{ii}/n} = \operatorname{Tr}[H]/n = (p+1)/n$ Using the fact that $\operatorname{Tr}[AB]=\operatorname{Tr}[BA]$ and $H=X(X^TX)^{-1}X^T$. But I have no idea how to ...
2
votes
3answers
391 views

Residuals of regression model

Let's suppose I do a regression between earnings and age (and suppose I do not know the distribution of earnings). Would it be possible for the residuals to be normally distributed? I am thinking it ...
2
votes
2answers
462 views

Calculate Line Of Best Fit Using Exponential Weighting?

I know how to calculate a line of best fit with a set of data. I want to be able to exponentially weight the data that is more recent so that the more recent data has a greater effect on the line. ...
2
votes
1answer
263 views

How to handle constant term in Least Squares Regression?

In the well known matrix form of a least squares regression where I am trying to solve for B in Y = B1X1 + B2X2 + B3 I might be given X and Y sample data as something like $X$ = $\begin{bmatrix} ...
2
votes
1answer
19 views

Deriving the identity: $\hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2}$

For some reason I am having an extremely hard time finding out how the following expression is derived $$ \hat{\beta}_1 = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sum (x_i - \bar{x})^2} $$ Is ...
2
votes
1answer
42 views

How many data points are “enough” for linear regression?

I have data points $(x_t,y_t)$ generated from $y_t = a + b x_t + \epsilon$ where $\epsilon$ is gaussian error term with zero mean and unknown variance. I want to estimate coefficients $a$ and $b$ but ...
2
votes
1answer
149 views

Solving for a 3D point in a 5D graph given 3 pairs of 2D points.

I am attempting to solve the values $C$, $D$, and $S$, given three pairs of $[M,R]$. $$R = \frac {M}{C - MDC + DC\left(MS\right)^2}$$ I have been able to solve for a related equation (or rather, ...
2
votes
1answer
38 views

Question on regression

So I've been given this formula For regression $R^2=1 - \sum \frac{{(y_i - \hat{y}_i)}^2}{(y_1-\bar{y})^2}$ Now an obvious question that has come to me is why $R^2$ stays the same in certain ...
2
votes
1answer
38 views

How to take the derivative of Matrices

I was browsing the derivation of the Least Squares estimates and stumbled about this problem. It said that: $$E = (Y + XB)^2$$ $$\frac{dE}{dB} = -X^TY + X^TXB$$ It is to my understanding that the ...
2
votes
1answer
50 views

How can I plot this?

Given a bunch of data $x_i$ , $y_i$, how do I plot $$f(\theta_2,\theta_2)= \frac{1}{2M} \sum_{i=1}^{M} (\theta_1\cdot x_i -\theta_2 y_i)^2$$ in matlab? I know it should be parabolic, but my code ...
2
votes
1answer
891 views

Understanding Regularization parameters in Machine Learning/Statistics

Suppose I have the following $k$ degree polynomial regression model with a data set of size $n$ which includes a $k$-dimensional feature vector $x$ and an outcome denoted $t_i$ for each vector in the ...
2
votes
1answer
229 views

Fitting a simple linear regression

A professor in the School of Business in a university polled a dozen colleagues about the number of professional meetings they attended in the past five years $x$ and the number of papers they ...
2
votes
2answers
672 views

connection between PCA and linear regression

Is there a formal link between linear regression and PCA? The goal of PCA is to decompose a matrix into a linear combination of variables that contain most of the information in the matrix. Suppose ...
2
votes
1answer
2k views

Convert nonlinear regression equation to a linear regression equation

The question is: "Show how the nonlinear regression equation y=aX^B can be converted to a linear regression equation solvable by the method of least ...
2
votes
1answer
65 views

Choosing the right regression

I'm trying to analyze my sleep using regression analysis. Each night is rated (dependent variable). I'm trying to explain this rating with, for example, my sleep duration and each night's bed time's ...
2
votes
1answer
31 views

Determine which parameter has correlation with result and which is not

sorry for probably silly question, it's the first time when I need to do such work. I have large data set with regarding clicks on some element on web page. It contains some characteristics of such ...
2
votes
4answers
101 views

Condition for $\det(A^{T}A)=0$

Is it always true that $\det(A^{T}A)=0$, $\hspace{0.5mm}$ for $A=n \times m$ matrix with $n<m$? From some notes I am reading on Regression analysis, and from some trials, it would appear this is ...
2
votes
1answer
505 views

Arriving at the Logistic function from a Binomial Distribution and Maximum Likelihood

I've been trying to understand the origin of the Logistic function in Logistic regression: $$\Pr(Y=1|x;\theta)=\frac{1}{1+e^{-\theta x}}$$ I was lead to beilive that one could somehow arrive at this ...
2
votes
1answer
202 views

Find data to perform regression analysis

I'm trying to find some data (two continuous variables that I believe are correlated) online for which I can perform a regression anaylsis, my assignment sheet says: The data may be found anywhere ...
2
votes
1answer
6k views

Equations For Quadratic Regression

Does anyone know the specific equations for the three parameters in a least-squares quadratic regression? I'm looking for something like $\beta_1=,\beta_2=,\beta_3=$ for each of ...
2
votes
1answer
110 views

Linear relationship of a company's profit

Assume a linear relationship for a company that has several shops is not known. Let $Y_i$ be the profit the shop number $i$ makes in the coming year. Let $x_i$ be the size of the shop number ...
2
votes
1answer
419 views

An intuitive explanation for neural networks as function approximators ?

We use normal linear regression for modelling functions on datasets . But Can someone explain how neural networks help in approximating more complex ,especially non-linear functions ? intuitively , ...
2
votes
4answers
89 views

Seeking a function based on its level set

I'm trying to create a function for a research project, but I fear my math knowledge is insufficient to derive it from the attached diagram I've created showing its desired behavior. I'm hoping ...
2
votes
3answers
544 views

How to deal with Linear Regression model with some data aggregated

Lets say I am trying to find a linear regression between Weight and Height of a person. $W=b_0+b_1 H+e$ The data I have gathered from 8 people is like this: ...
2
votes
1answer
5k views

How to calculate hyperbola from data points?

I have 4 data points, from which I want to calculate a hyperbola. It seems that the Excel trendline feature can't do it for me, so how do I find the relationship? The points are: (x,y) (3, 0.008) ...
2
votes
1answer
23 views

Can a prediction interval be interpreted as a probability?

Suppose I find a 90% prediction interval for some data distribution. This implies that if I sample large enough data from this distribution, then 90% of such data will lie inside the prediction ...
2
votes
1answer
114 views

Simple Regression ~ House Price Prediction

I am stuck with this question. "You have a data set consisting of the sales prices of houses in your neighborhood, with each sale time-stamped by the month and year in which the house sold. You ...
2
votes
1answer
8 views

Inverse function for non-linear regression purpouses

The Setting: I want to perform a regression onto data of that follows this shape: \begin{equation} U(x):=\sum_{i=1}^N\, a_ix^ie^{-b_ix} \end{equation} where the $a_i\in \mathbb{R}$ and the $b_i \in ...
2
votes
1answer
53 views

Estimating landing position for a slowly falling object using latitude, longitude, and altitude.

I have a weather balloon project, in which I intend to use GPS to locate the payload when it finally comes down again. I will make the computer send coordinates to a server every minute or so, as ...
2
votes
1answer
71 views

Relationship between $L^1$ norm and sparsity

I'm doing some research in the field of sparse representation and sparse modeling. I have two variables and their $L^1$ norm is calculated to make comparisons. As I take it the smaller the value of ...
2
votes
1answer
45 views

Predicting trends of timeseries data with ARIMA

I'm looking for an algorithm that can help identify abnormal trends in time-series metrics. The best I've been able to find so far is ARIMA (a completely new concept for me). We offer several ...
2
votes
1answer
28 views

Linear regression and standardization

I am trying to use a linear regression to model an expected value Y for an input X. X and Y have a large difference between them, so I was converting to standard (z) score, doing my calculation ...
2
votes
1answer
39 views

Discrete version of continuous SIR model

I'm working with a SIR infection model, which is $$\begin{array}{rcl} \frac{dS}{dt} & = & -\beta IS\\ \frac{dI}{dt} & = & \beta IS-\gamma I\\ \frac{dR}{dt} & = & \gamma I ...
2
votes
2answers
55 views

Fitting two parallel lines to a set of points

In two dimension I have a set of points X = $\{x_1,..., x_N\}$. I want to fit two parallel lines to these points like $l_1$ and $l_2$ $$l_1 = p_1 + \lambda n^\perp$$ $$l_2 = p_2 + \lambda n^\perp$$ ...
2
votes
1answer
96 views

Least-squares solution to a matrix equation?

Suppose I have $n$ observations of $m$ dependent variables $y_1,\dots,y_m$, and I believe they follow some model wherein they can all be written as linear combinations of some underlying variables ...
2
votes
2answers
81 views

Outlier detection with robust multiple regression model

I have a set of features (eg, location, income, budget, education) that I use to predict a continuous variable (say, amount spent per day on the internet). I am interested in detecting outliers. I ...
2
votes
1answer
41 views

Prove a result in multiple linear regression

This arises in multiple linear regression. Given $m, n \in \mathbb{N}$ and matrices $X \in \mathbb{R}^{m \times (n+1)} (m > n + 1), H = X(X'X)^{-1}X' \in \mathbb{R}^{m\times m}, I = I_m$ and $J ...
2
votes
1answer
104 views

Is it possible to have two lines of best fit?

Could you rig a data set to have two lines of equally good (and best) fit? Or is it impossible?
2
votes
1answer
23 views

Three-Perpendicular Theorem for linear regressions

For a random vector $X=(X_1,\ldots,X_p)'$, we define $$ \mathcal{L}(X)=\{b_0+b_1X_1+\cdots+b_pX_p,b_0,\ldots,b_p\in\mathbb{R}\}. $$ The linear regression of the $q$-dimensional random vector ...
2
votes
1answer
116 views

Solution of overdetermined polynomial system

Some of you will find this question pretty straightforward to answer, but I desperately need some help in solving a problem involving several equations and 2 unknowns, for an engineering application. ...
2
votes
1answer
60 views

Standard Error in OLS Regression

Assuming I have the following linear regression set-up: $y_i = \alpha + x_i * \beta + \epsilon_i$ for $i = 1,2,..., n$. When I run the regession, I get a $\beta$ and $\alpha$ estimates, along ...