Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

learn more… | top users | synonyms

5
votes
0answers
175 views

Prove the estimator $\hat{B}$ of ridge regression = mean of the posterior distribution under a Gaussian prior

I want to prove that the estimator of ridge regression is the mean of the posterior distribution under Gaussian prior. $$y \sim N(X\beta,\sigma^2I),\quad \text{prior }\beta \sim N(0,\gamma^2 I).$$ $...
5
votes
0answers
38 views

single variable is significant but overall test is not

I do a multiple regression with 3 independent variables $X_1$, $X_2$ and $X_3$. The correlation between $Y$ and $X_1$, $Y$ and $X_2$, and $Y$ and $X_3$, are each large and statistically significant. ...
4
votes
0answers
76 views

Bayesian linear regression cost function

I am studying classification using linear regression . Now, I want to map it in Bayesian regression. Let talk about binary classification using linear regression again. Assume that I have a set $X=${...
4
votes
0answers
302 views

Multilinear or Tensor Regression?

Given input data $x_t\in \mathbb{R}^n$ and output data $y_t\in\mathbb{R}^m$, the closed form solution to $\min_A \sum_t \|y_t - Ax_t\|^2_2$ is given by $A = (XX^T)^{-1}XY^T$ where $x_t$ form the ...
3
votes
0answers
77 views

Two dimensional (discrete) orthogonal polynomials for regression

This question How to work out orthogonal polynomials for regression model and the answer http://math.stackexchange.com/a/354807/51020 explain how to build orthogonal polynomials for regression. ...
3
votes
0answers
34 views

Regression with many discrete and continuous predictors and few rows

I want to do regression on a dataset. It has one continuous dependent variable that I want to predict. It has many categorical and some continuous predictors. It only has a few rows. A simplified ...
3
votes
0answers
44 views

How to perform nonlinear regression with regressors affected by gaussian error?

I am trying to calibrate a sensor and I have a data set consisting of several observations of a 3-dimensional vector $X_i$, with $X_i=w_i + \epsilon_i$ where $w_i$ is the value that the sensor ...
3
votes
0answers
397 views

Can the sigmoid function approximate any function (or relation) where 0<y<1

I'm studying Machine Learning and Artificial Neural Networks. Some basic principles of Machine Learning are linear regression, multivariate linear regression, and nonlinear regression. The last of ...
3
votes
0answers
922 views

What is the Moore-Penrose pseudoinverse for scaled linear regression?

The matrix equation for linear regression is: $$ \vec{y} = X\vec{\beta}+\vec{\epsilon} $$ The Least Square Error solution of this forms the normal equations: $$ ({\bf{X}}^T \bf{X}) \vec{\beta}= {\bf{...
3
votes
0answers
230 views

Minimizing L4/ L6/ L2N norm for linear regression

OLS regression minimizes the sum of the squared errors. The normal equation for an OLS for $L_2$ minimization is as follows: $$b= (A'A)^{-1}A'y$$ What would be the equation to minimize the $L_4$ norm ...
3
votes
0answers
185 views

How to perform nonlinear regression with correlated errors?

I have a nonlinear least squares problem, but the errors are correlated. I could use R's nls function to do the regression if the errors were independent, but I don't know the right way to handle ...
3
votes
0answers
345 views

Least Square Method with Positive Parameters

this is my first post here in the Stack Exchange. A friend told me about this forum and I'm giving it a try. I searched a bit past threads, but couldn't find what I wanted, so I'm posting the problem ...
2
votes
0answers
24 views

What is the relationship between the function $\mathbb{E}(Y \mid X = x)$ and linear regression?

Consider the function $$ r(x) = \mathbb{E}(Y \mid X = x) $$ This has been called the regression function in a textbook I'm using. I'm trying to figure out the relationship between this function ...
2
votes
0answers
13 views

Derive the Hat Matrix to map actual response to estimated resposne

In order to measure the quality of a regression we can calculate the Hat Matrix. Using it we can estimate the response variable as if we used the predictor variables to regress them. For linear ...
2
votes
0answers
33 views

What does it mean to regress out current features?

First of all, I'd like to say that this is the intro to a homework problem. Please do not post any answers, I am only looking for clarification on some terminology in the setup. I am trying to ...
2
votes
0answers
28 views

Ridge Regression Centering Proof

This is a ridge regression problem. The following two problems are equivalent: $(w_t, b_\lambda ) = argmin_{w,b}\{\sum_{i=1}^m (y_i-b-w^Tx_i)^2+\lambda w^Tw\} $ $(w_t, b_\lambda )= argmin_{w,b}\{\...
2
votes
0answers
46 views

Does linear regression form a subspace?

The author writes Given a vector of inputs $X^T = (X_1, \dots ,X_p)$, we can predict an output $Y$ via $$ \hat{Y} = \beta_0 + \sum_{j = 1}^p X_j \beta_j$$ He goes on to note that if we include a 1 in ...
2
votes
0answers
23 views

standard deviation for regression

The first slide is the denifition of simple linear regression model, the second slides is an example I have two questions: 1.Did I get the right calculation of the standard deviation? 2.I still ...
2
votes
0answers
27 views

Non linear regression calculator

Are there any really good non linear regression calculators around the web? Or is something like matlab the best solution? I tried using excel and its solver tool, but it's complete garbage lol. ...
2
votes
0answers
28 views

Proof that correlation coefficient squared equals the coefficient of determination

Hi I as the title says I'm looking at the proof that $r^2$ = $R^2$ in the case of simple linear regression, but I don't understand one part. There are different versions of the proof, but in most of ...
2
votes
0answers
18 views

Problems with Exchange Procedure in Remez Algorithm

So first off: *** This code is not being used in production software. It is a personal project of mine, trying to understand approximation theory and advanced curve fitting. In other words, I'm ...
2
votes
0answers
26 views

combining multiple regression outputs

Suppose I have multiple regressions, along with their r-squares, standard-errors, etc.: $y(t) = \alpha_1 + \beta_1 x(t) + e_1$, where $t \in (\tau_0, \tau_1)$ $y(t) = \alpha_2 + \beta_2 x(t) + e_2$,...
2
votes
0answers
63 views

GLM for Poisson Regression for Soccer Ratings Not Converging

I have been trying to formulate a model of soccer matches to help me predict the outcomes. The model I'm trying to formulate involves using Poisson regression to assign attack and defence ratings to ...
2
votes
0answers
35 views

Can I adjust linear growth of a a subpopulation to a linear decay of the general population?

I need to estimate the amount of CF patients in Poland in the next four years. I have: estimations of the Polish population for the future years a CF patients' register for the last couple of years ...
2
votes
0answers
89 views

Show $\hat{\beta}$ and $s^2$ are independent?

I have the model: $y=X{\beta}+{\epsilon}$ I know $\hat{\beta}=(X'X)^{-1}X'y$ and that it is an unbiased estimator of ${\beta}$ and that $s^2=\hat{\epsilon}'\hat{\epsilon}/(n-k)$ and is an unbiased ...
2
votes
0answers
147 views

How to reach Moore-Penrose pseudoinverse solution to minimize error function

Edit I'm trying to figure the derivation of the Moore-Penrose pseudoinverse for linear regression. The starting expression is the standard error function. I'm not quite sure how to expand on this ...
2
votes
0answers
75 views

Predicting profit with price variation

I am currently working on a high school project that aims to predict profit from X amount of items to Y amount of profit based off a deviated sale price. For instance: I sale 10 cookies for 10 ...
2
votes
0answers
108 views

Uni-variate Moving Average Theta coefficients

Consider the Uni-variate Moving Average Models (MA models) MA(1) $$x_t = \mu + w_t +\theta_1w_{t-1}$$ or the second order moving average MA(2) $$x_t = \mu + w_t +\theta_1w_{t-1}+\theta_2w_{t-2}$$ ...
2
votes
0answers
293 views

Normal equations for minimization of Frobenius norm least squares error

I'm having a hard time understanding the most efficient sequence of steps for deriving the normal equations for Frobenius norm least squares minimization. Here I want to minimize the norm of a matrix ...
2
votes
0answers
37 views

Best line fit for correlated points

Given in $\mathbb{R}^3$ are $n$ points $\mathbf{y}_i\sim N(\mathbf{y}_i-\mathbf{\hat{y}}_i, \mathbf{C}_i)$, which are normally distributed. I want to determine a best fit line $\mathbf{u}(\lambda) = \...
2
votes
0answers
48 views

OLS standard error that corrects for autocorrelation but not heteroskedasticity

Question: By mapping the OLS regression into the GMM framework, write the formula for the standard error of the OLS regression coefficients that corrects for autocorrelation but not heteroskedasticity....
2
votes
0answers
34 views

Showing Hat matrix equal specific values

Consider a one way layout model $y_{ij}$ = $\mu_i + e_{ij}$ (1 $\leq$ i $\leq$ a, 1 $\leq$ j $\leq$ $n_i$) where a = 3 and $n_1$ = 2, $n_2$ = 3, $n_3$ = 4. Show that the hat matrix for this design ...
2
votes
0answers
106 views

Is it compulsory to make transformation to the econometric model in order to have only diagonal elements on variance-covariance matrix of errors?

I need some sharped and advanced advices for the following issue ... Model and its assumptions I'm working on the methodology of a two-way error component model. Here is the model: $y_{jis} = x_{jis}...
2
votes
0answers
131 views

What is ${\rm cov}(e_i, \hat y_i)$ in simple linear regression?

The model is $y_i = \beta_0 + \beta_1x_i + \epsilon_i$ What is ${\rm cov}(e_i, \hat y_i)$? What is ${\rm cov}(\epsilon_i, \hat \beta_1)$? What is ${\rm cov}(e_i, \epsilon_i)$? For 1, I am writing ...
2
votes
0answers
36 views

Smallest set of Liner equations, which exactly fit a set of points

I have a set of 2-d points,(it can be of any arbitrary dimension n). I want to find the minimum set of straight lines(linear equations) which exactly passes through the given 2-d points (unlike ...
2
votes
0answers
41 views

Sampling data prior to nonlinear regression

As my question shows it, I am not a statistician. My problem is that I have too many data points to be used in a nonlinear fit (I have millions of them, automatically acquired). Is there a methodology ...
2
votes
0answers
1k views

Least Squares “analytic expression” for fitting a 2D quadratic function to measurements

I have n scattered elevation measurements: $ \{x_i,y_i,z_i\}_{i=1..n} $ that I want to fit a quadratic function to: $ z = ax^2 + by^2 + cxy + dx + ey + f$. The problem can be written as a vector ...
2
votes
0answers
58 views

Computing evidence for least-squares fit

I'm at a loss trying to implement Bayesian model selection for standard least-squares polynomials fits. I have three polynomials of order $1$, $2$, and $3$, and a sequence of $(x,y)$ data points. ...
2
votes
0answers
53 views

Coefficient of determination

$$ \displaystyle \sum^n_{i = 1} (y_i - \bar{y})^2 = ( \displaystyle \sum^n_{i = 1} (y_i - \bar{y})^2 - \displaystyle \sum^n_{i = 1} (y_i - \hat{y}_i)^2 ) + \displaystyle \sum^n_{i = 1} (y_i - \hat{y}...
2
votes
0answers
172 views

Orthonormal Matrix weighted regression

$Q$ is a rectangular matrix with orthonormal columns. A linear system composed of $$Qx= b$$ is really easy to solve as: $$Q'Q=I$$ hence: $$x=Q'b$$ Given that $Q$ is orthonormal can this be used to ...
2
votes
0answers
91 views

Regressing $Y$ back on the residuals

Suppose I have the linear regression model $ \hat{y_i} = a + b x_i $ for $a,b$ obtained via OLS. How does one regress $y$ back on the residuals $\hat{e}_i = y_i - \hat{y}_i$? If we write $ \hat{\hat{...
2
votes
0answers
199 views

Effective model for calculating momentum or growth rate for a time series

I have a series of numbers tracking the performance of an entity on any given day. It's nothing but a simple integer for each date. For example, here's a series for Entity "X" ...
2
votes
0answers
505 views

Bare minimum of points in multiple polynomial regression

I have a question on multiple polynomial regression and the absolute minimum amount of points in the different terms. The minimum amount of points required for a second order polynomial would (in one ...
2
votes
0answers
100 views

Find $w$ as the minimizer of regularized logistic regression

Could someone point me to a reference on how to find $w$ as the minimizer of: $$ \frac{1}{2}\sum_{i=1}^{d}q_i(w_i-m_i)^2+\sum_{j=1}^{n}log(1+\exp(-y_jw^Tx_j)) $$ where $q_i$ (initialized with $\...
2
votes
0answers
537 views

Logistic regression algorithm in Casio and Texas Instruments calculators

When using logistic regression on a Casio or Texas Instruments calculator, the output is of the form $$f(x) = \frac{c}{1+ae^{-bx}} $$ The problem I have (when teaching in a class where both types of ...
2
votes
0answers
365 views

Finding a model for multiple non-linear regression

I want to implement a regression analysis, but I have problems with finding a model for the given data. There are $149$ $(x,y,z)$-values. $y$ values are all positive, $x$ is between $[-10, 10]$ and $z$...
2
votes
0answers
3k views

Derivation of standard error of beta in simple linear regression

Countless web pages show the equation for the standard error of the slope in a simple linear regression. For example: ...
2
votes
0answers
1k views

Correlation and Regression Question

Two separate tests are designed to measure a students ability to solve problems. Several students are randomly selected to take both tests and the results are: $$ \begin{matrix} \text{Test A}(x) &...
2
votes
0answers
3k views

Help with problem: Estimated Standard Deviation of Regression Equation (Simple Linear)

This is a practice problem. I've solved part (a). I have provided verified answers (from the published key) to all parts (a), (b) and & (c). I need help solving (b) and (c). Consider a simple ...
2
votes
0answers
313 views

Surface Function Fitting to Spherical Data

I have a set of geographic (longitude,latitude,value) data to which I would like to fit surface functions, specifically, the set of quadratic surfaces: $f(x,y)=Ax^2+Bx^2+Cxy+Dx+Ey+F$ At the moment, ...