Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

learn more… | top users | synonyms

1
vote
0answers
18 views

Matrix Decomposition: Difference between Cholesky Decomposition, Eigendecomposition and Jordan Normal Form Decomposition

I recently created a related topic about the square root matrix, in case you'd like to refer to that one. Here's what we want: Consider the matrix $\Omega=E(\mathbf{u}^{\top}\mathbf{u})$, where ...
0
votes
0answers
11 views

Matrix problem in Mixed Regression

the background is $y= X \beta +e$ y=n*1 X=n*p $\beta=p*1$ e=n*1 take singular value decomposition of X $X=P \Delta Q$ $\beta=QKP'y$ K is a digaonal matrix and depending on its form can represent ...
0
votes
0answers
8 views

Multivariate Regression

Suppose there are $n$ variables that map through a function to a single output variable $r$. Given a set of 50-100 data sets with accepted input and output values that satisfy this relation, is it ...
1
vote
2answers
70 views

Is the Square Root of an Inverse Matrix Equal to the Inverse of the Square Root Matrix?

I know in general that if a matrix $A$ is positive definite, then there exists a (unique?) square root matrix $B$, which is also positive definite, such that $BB=A$. Therefore, suppose $A$ is ...
0
votes
0answers
2 views

2d spatio-time regression

I have this question which intuitively appears to be simple, but I couldn't find a solution to it. Imagine you have a ball moving in one direction (x) and that the measurements of the movement are ...
0
votes
1answer
14 views

linear regression, expectation and mean squared error

Let us assume that data is generated according to a true model $$y_i = \beta_{true}x_i + \epsilon_i$$ for $i = 1, ..., n$ Assume that $x_i$ are fixed, and $\epsilon_i$~ N(0, $\sigma^2$) ...
0
votes
0answers
9 views

Multiple regression model

I have a multiple regression equation which as four quarters (maybe called them as parameters) ...
3
votes
0answers
35 views

Least square regression of a vector onto a space [on hold]

Suppose the basis vectors for a space are [1 0 0] and [0 1 2]. Now, I would like to find the least square projection of the vector [a b b] onto the mentioned space. How do I approach this?
2
votes
0answers
24 views

Predicting profit with price variation

I am currently working on a high school project that aims to predict profit from X amount of items to Y amount of profit based off a deviated sale price. For instance: I sale 10 cookies for 10 ...
1
vote
0answers
22 views

Round robin logistic regression

I have a poll with four answers (A,B,C,D) and response information about people who have taken that poll. I have created four models (one for each of the answers) in a one vs all. i.e. the model for ...
1
vote
0answers
22 views

Uni-variate Moving Average Theta coefficients

Consider the Uni-variate Moving Average Models (MA models) MA(1) $$x_t = \mu + w_t +\theta_1w_{t-1}$$ or the second order moving average MA(2) $$x_t = \mu + w_t +\theta_1w_{t-1}+\theta_2w_{t-2}$$ ...
-1
votes
1answer
15 views

Linear Regression question 1

I would really be grateful if someone could let me know how to answer Part (a) of Question 1. I believe i should scatter plot both x and y values separately for year 2000 and year 2001 on same graph ...
1
vote
0answers
9 views

Proving $Corr(\hat{e}_{ij}, \hat{e}_{jk}) = \frac{-1}{n_i-1}$ for $ j \neq k$

For the model of a single factor experiment: $y_{ij}= \mu + \alpha_i + e_{ij}$, $(1 \leq i \leq a, 1 \leq j \leq n_i)$, where a = the number of treatments, $n_i$ = the number of experimental units ...
1
vote
0answers
11 views

Principal Components vs Principal Directions

I'm trying to do statistical downscaling of some climate data and there is a module of principal component analysis by regression method required. I am confused with the different terms here. What is ...
0
votes
0answers
6 views

Inverting the inner product of a matrix with highly correlated columns

If matrix $X$ has highly correlated (but not exactly linearly dependent) columns, $X'X$ is still invertible but $(X'X)^{-1}$ will be large. Why is that the case? The motivation for this question ...
0
votes
1answer
14 views

Interpreting confidence interval of regression coefficient.

In a Simple Linear Regression analysis, independent variable is weekly income and dependent variable is weekly consumption expenditure. Here $95$% confidence interval of regression coefficient, ...
0
votes
1answer
23 views

linear regression model beta estimate

Suppose we want to estimate $\beta$ by minimizing $L(\beta)=\sum_{i=1}^n(y_i-\beta x_i)^2+\lambda|\beta|$, where $\lambda$ is a fixed positive constant. Calculate the estimate. How would I ...
0
votes
1answer
22 views

Linearize non linear function

Is it possible to linearize the function $f(x) = 1-exp(\frac{x}{b})$ so that one could use it in a linear regression?
0
votes
0answers
11 views

Question regarding Balanced Incomplete Block Design

Question: Consider a BIB design with a treatments, b blocks and c < a number of plots in each block where a,b,c ≥ 2. Let $n_{ij} = 1$ if an observation is made on the ith treatment in the jth ...
10
votes
6answers
854 views

Why get the sum of squares instead of the sum of absolute values?

I'm self-studying machine learning and am getting into the basics of linear regression models. From what I understand so far, a good regression model minimizes the sum of the squared differences ...
0
votes
2answers
15 views

When does Mean Square Error increase?

As far i know, we want the model to include as few regressors as possible because the variance of the prediction $\hat y$ increases as the number of regressor increases. But from the hald cement ...
1
vote
0answers
77 views

Normal equations for minimization of Frobenius norm least squares error

I'm having a hard time understanding the most efficient sequence of steps for deriving the normal equations for Frobenius norm least squares minimization. Here I want to minimize the norm of a matrix ...
1
vote
0answers
17 views

Variance-covariance matrix of a linear regression model

In finding the covariance matrix of a linear regression model I don't understand this step: $$ E[(b-\beta)(b-\beta)']=E[(X'X)^{-1}X'\epsilon\epsilon'X(X'X)^{-1}] $$ where we've been given that $$ ...
1
vote
1answer
50 views

Prove that $E(\mathbf{u}|\mathbf{X})=\mathbf{0}$ implies $Cov(\mathbf{x},\mathbf{u})=0$

Let \begin{equation} \mathbf{y}=\mathbf{X}\mathbf{\beta}+\mathbf{u} \end{equation} where $\mathbf{y}=\begin{bmatrix}y_1 \\ \vdots \\ y_n\end{bmatrix}$, $\mathbf{X}=\begin{bmatrix}X_{11} & ...
0
votes
1answer
17 views

Prros and cons of including controls in a regression?

Assume we have conducted a random experiment for the benefits of a drug. Let $Y_i$ be the outcome of interest , $X_i$ be some control variables (e.g. age, sex etc.) and $$D_i= ...
1
vote
1answer
17 views

Low Leverage in Residuals, Logistic Regression

I am doing an interpretation of logistic regression and I have an observation withh high residuals but low leverage. I thought that means that it is an outlier(bad prediction) but not ...
1
vote
1answer
50 views

Prove $\operatorname{Var}(\hat{e}_{ij}) = \sigma^2 \left(\frac{n_i-1}{n_i}\right)$

$\newcommand{\Var}{\operatorname{Var}}$ Let $y_{ij}$ denote the observed response of the $j$th experimental unit in the $i$th treatment group, and the $e_{ij}$ are i.i.d. $N(0,\sigma^2)$ experimental ...
1
vote
0answers
11 views

GMM estimation of linear regression with intercept restriction

Say I have a time series regression as follows: $$y_t = a_i + \beta_i x_t + \varepsilon_t^i \ \ ; \ \ t = 1, 2, \cdots, T \ \ \text{for each } i$$ Now say I impose the following restriction on the ...
0
votes
0answers
12 views

How does an increase in the order count in a conditional mean model affect the model?

How does an increase in the order count in a conditional mean model affect the model ,especially the effect on its filtering capacity, smoothness, lag ; while also its forecasting ability. By ...
0
votes
1answer
27 views

Semi-log re-expression to find an exponential model.

I'm unsure of how to approach this problem because of the 'semi-log' part, would I find the line of best fit, then log both sides on that equation until it is in exponential form? Thanks in advance. ...
2
votes
0answers
31 views

Best line fit for correlated points

Given in $\mathbb{R}^3$ are $n$ points $\mathbf{y}_i\sim N(\mathbf{y}_i-\mathbf{\hat{y}}_i, \mathbf{C}_i)$, which are normally distributed. I want to determine a best fit line $\mathbf{u}(\lambda) = ...
1
vote
0answers
24 views

Least square / Linear regression over a simplex

I have to solve the following least square problem: $$\hat{x} = \arg \min_{x \in S} \|Ax - b\|^2$$ If $S = \mathbb{R}^n$, then the solution is given by $$\hat{x} = (A^TA)^{-1}A^Tb$$ having posed ...
1
vote
1answer
29 views

Understanding polynomial regression

I'm looking for a good tutorial on how to calculate a "line of best fit" for non-linear data. I found this site: http://easycalculation.com/statistics/learn-regression.php which gives a very good ...
0
votes
1answer
38 views

Why is $E(u)=0$ when an intercept is included in OLS Estimation?

I am reading Wooldridge's graduate econometrics text. There he states that when estimating the equation $y=\mathbf{x\beta}+u$ by OLS, if an intercept (constant term) is included in your $\mathbf{x}$ ...
0
votes
0answers
28 views

OLS standard error that corrects for autocorrelation but not heteroskedasticity

Question: By mapping the OLS regression into the GMM framework, write the formula for the standard error of the OLS regression coefficients that corrects for autocorrelation but not ...
0
votes
0answers
15 views

How to find Expected Values of errors in OLS?

I am learning OLS and we are asked to show/explain whether the following statements are true or false. We are given that $\hat{U}= Y-\beta X$ and $E(U|X)=0$. I am able to do this for the first two ...
0
votes
0answers
19 views

Why use the expected value of y, E(y), in simple linear regression

I am learning about linear regression and I have ran into a bit of confusion. I'm trying to relate what I've learned in my probability and mathematical statistics course (in particular, expected ...
2
votes
1answer
33 views

Isn't the Hat Matrix just an identity matrix?

In Linear regression $y = X\beta + \epsilon$ The Hat matrix is defined to be $H = X(X^TX)^{-1} X^T$ . However. If I compute the equation for Hat matrix, I just get an identity matrix. My calculation ...
0
votes
0answers
10 views

Predicted value of polynomial regression models

Suppose that we have a polynomial linear regression as following $$ Y_i = \beta_0 + \beta_1 X_{i} + \beta_2X_{i}^2 + \epsilon_i, \quad i=1,\ldots, n $$ with $\epsilon_i \sim N(0,\sigma^2)$ and ...
0
votes
0answers
15 views

Analysis of variance question ($\sum(\hat{y}-\bar{y})\hat{e}_t=0$)

this is from an econometrics textbook, looking at how to arrive at the analysis of variance table for a linear regression: $\sum (y_{t}-\bar{y})^{2}$ =$\sum [(\hat{y_{t}}-\bar{y})+\hat{e_{t}}]^{2}$ ...
1
vote
1answer
34 views

Linear Regression with independent but non-identical noise

If I have this linear regression equation: $$y=X\beta+\epsilon $$ ($x$ and $\beta$ are vectors) The likelihood function can be written as $$L= \prod_{n=1}^N N(y_n ;x_n ,\beta ,\sigma^2)=(2\pi ...
0
votes
1answer
29 views

Exponential least squares biased to small y?

I'm working on fitting an exponential decay curve to a data set. While searching for techniques I found Wolfram's page describing how to easily accomplish it by taking the natural log of the ...
0
votes
0answers
67 views

weights go to infinity in logistic regression with linearly separable data

I have the loss function of logistic regression $L(W)$ = - $\sum_{i=1}^n {y_i}.log[\sigma(w^Tx)] + {(1-y_i)}.log[1- \sigma(w^Tx)]$ I have derived the Hessian and proven it's positive semi-definite ...
0
votes
0answers
19 views

Likelihood Functions of Nonparametric Simple Regression

I'm trying to find the likelihood function of a nonparametric simple regression model. Nonparametric statistics is new to me however, so I'm having some trouble wrapping my head around some of the ...
0
votes
1answer
32 views

More variables = better fit?

When fitting (let's say) a linear regression model, it is always true, that the more variables we include in our model, the better fit is (in R^2 sense)? I don't want to discuss here overfitting, ...
0
votes
0answers
13 views

Is there any easy way to QR-decompose two related matrices?

Matrix A and B are related in that: $A = W_1X$ and $B = W_2X$ Where $W_1$ and $W_2$ are diagonal matrices with all the same entries (but not in the same order). I am asking for the purpose of ...
1
vote
1answer
41 views

Prove that the the variance estimator $\widehat{\sigma}^2=MSE/(n-2)$ is biased is the simple linear regression model

This is in scope of the simple linear model. Im trying to prove that $\mathbb{E}\left(\widehat{\sigma}^2\right) = \sigma^2$ for $$\widehat{\sigma}^2 = \frac{1}{n-2}\sum^n_{i=1} ...
0
votes
0answers
19 views

How to solve this linear algebra equation / regression?

$Y = X D$ and $Y$ is $n\times 1$ known matrix. $X$ is $n\times k$ unknown matrix. $D$ is $k\times 1$ known matrix. How to solve for $X$?
3
votes
3answers
47 views

linear solution of curve fitting on multiple linear functions differing by a multiplier

I recently posted this question here but I thought this could be of interest also in mathematics, given I found a partially related question here I am facing the following problem. I know nonlinear ...
0
votes
1answer
48 views

Examining the effect of a quantitative factor on response.

To examine the effect of a quantitative factor temperature on yield,the researcher has a plan to use the following model for the analysis: $$y_{ix}=\beta_0+\beta_1 x+\epsilon_{ix}$$ where $y_{ix}$ ...