Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

learn more… | top users | synonyms

4
votes
0answers
171 views

How to compare the similarity between functions?

I'm designing a web service that finds the regression function of a pattern within an image. I analyzed three images and found the following three regressions: 1) $f(x) = 74.7602 + 0.2005x - ...
3
votes
0answers
35 views

Bayesian linear regression cost function

I am studying classification using linear regression . Now, I want to map it in Bayesian regression. Let talk about binary classification using linear regression again. Assume that I have a set ...
3
votes
0answers
86 views

Determine whether ARMA(p,q) is stationary and/or invertible?

Determine whether an ARMA(p,q) process is stationary and invertible such that $y_t = \sum_{i=1}^{p} \phi_i y_{t-i} + \sum_{i=1}^{p} \theta_{i} \epsilon_{t-i}$ with the restriction that ...
3
votes
0answers
132 views

Can the sigmoid function approximate any function (or relation) where 0<y<1

I'm studying Machine Learning and Artificial Neural Networks. Some basic principles of Machine Learning are linear regression, multivariate linear regression, and nonlinear regression. The last of ...
3
votes
0answers
134 views

Multilinear or Tensor Regression?

Given input data $x_t\in \mathbb{R}^n$ and output data $y_t\in\mathbb{R}^m$, the closed form solution to $\min_A \sum_t \|y_t - Ax_t\|^2_2$ is given by $A = (XX^T)^{-1}XY^T$ where $x_t$ form the ...
3
votes
0answers
144 views

How to perform nonlinear regression with correlated errors?

I have a nonlinear least squares problem, but the errors are correlated. I could use R's nls function to do the regression if the errors were independent, but I don't know the right way to handle ...
3
votes
0answers
273 views

Least Square Method with Positive Parameters

this is my first post here in the Stack Exchange. A friend told me about this forum and I'm giving it a try. I searched a bit past threads, but couldn't find what I wanted, so I'm posting the problem ...
3
votes
0answers
868 views

Correlation and Regression Question

Two separate tests are designed to measure a students ability to solve problems. Several students are randomly selected to take both tests and the results are: $$ \begin{matrix} \text{Test A}(x) ...
2
votes
0answers
34 views

Predicting profit with price variation

I am currently working on a high school project that aims to predict profit from X amount of items to Y amount of profit based off a deviated sale price. For instance: I sale 10 cookies for 10 ...
2
votes
0answers
33 views

Best line fit for correlated points

Given in $\mathbb{R}^3$ are $n$ points $\mathbf{y}_i\sim N(\mathbf{y}_i-\mathbf{\hat{y}}_i, \mathbf{C}_i)$, which are normally distributed. I want to determine a best fit line $\mathbf{u}(\lambda) = ...
2
votes
0answers
14 views

Showing Hat matrix equal specific values

Consider a one way layout model $y_{ij}$ = $\mu_i + e_{ij}$ (1 $\leq$ i $\leq$ a, 1 $\leq$ j $\leq$ $n_i$) where a = 3 and $n_1$ = 2, $n_2$ = 3, $n_3$ = 4. Show that the hat matrix for this design ...
2
votes
0answers
99 views

Is it compulsory to make transformation to the econometric model in order to have only diagonal elements on variance-covariance matrix of errors?

I need some sharped and advanced advices for the following issue ... Model and its assumptions I'm working on the methodology of a two-way error component model. Here is the model: $y_{jis} = ...
2
votes
0answers
74 views

What is ${\rm cov}(e_i, \hat y_i)$ in simple linear regression?

The model is $y_i = \beta_0 + \beta_1x_i + \epsilon_i$ What is ${\rm cov}(e_i, \hat y_i)$? What is ${\rm cov}(\epsilon_i, \hat \beta_1)$? What is ${\rm cov}(e_i, \epsilon_i)$? For 1, I am writing ...
2
votes
0answers
33 views

Smallest set of Liner equations, which exactly fit a set of points

I have a set of 2-d points,(it can be of any arbitrary dimension n). I want to find the minimum set of straight lines(linear equations) which exactly passes through the given 2-d points (unlike ...
2
votes
0answers
45 views

Computing evidence for least-squares fit

I'm at a loss trying to implement Bayesian model selection for standard least-squares polynomials fits. I have three polynomials of order $1$, $2$, and $3$, and a sequence of $(x,y)$ data points. ...
2
votes
0answers
468 views

What is the Moore-Penrose pseudoinverse for scaled linear regression?

The matrix equation for linear regression is: $$ \vec{y} = X\vec{\beta}+\vec{\epsilon} $$ The Least Square Error solution of this forms the normal equations: $$ ({\bf{X}}^T \bf{X}) \vec{\beta}= ...
2
votes
0answers
39 views

Coefficient of determination

$$ \displaystyle \sum^n_{i = 1} (y_i - \bar{y})^2 = ( \displaystyle \sum^n_{i = 1} (y_i - \bar{y})^2 - \displaystyle \sum^n_{i = 1} (y_i - \hat{y}_i)^2 ) + \displaystyle \sum^n_{i = 1} (y_i - ...
2
votes
0answers
132 views

Minimizing L4/ L6/ L2N norm for linear regression

OLS regression minimizes the sum of the squared errors. The normal equation for an OLS for $L_2$ minimization is as follows: $$b= (A'A)^{-1}A'y$$ What would be the equation to minimize the $L_4$ norm ...
2
votes
0answers
152 views

Orthonormal Matrix weighted regression

$Q$ is a rectangular matrix with orthonormal columns. A linear system composed of $$Qx= b$$ is really easy to solve as: $$Q'Q=I$$ hence: $$x=Q'b$$ Given that $Q$ is orthonormal can this be used to ...
2
votes
0answers
83 views

Regressing $Y$ back on the residuals

Suppose I have the linear regression model $ \hat{y_i} = a + b x_i $ for $a,b$ obtained via OLS. How does one regress $y$ back on the residuals $\hat{e}_i = y_i - \hat{y}_i$? If we write $ ...
2
votes
0answers
140 views

Effective model for calculating momentum or growth rate for a time series

I have a series of numbers tracking the performance of an entity on any given day. It's nothing but a simple integer for each date. For example, here's a series for Entity "X" ...
2
votes
0answers
277 views

Bare minimum of points in multiple polynomial regression

I have a question on multiple polynomial regression and the absolute minimum amount of points in the different terms. The minimum amount of points required for a second order polynomial would (in one ...
2
votes
0answers
87 views

Find $w$ as the minimizer of regularized logistic regression

Could someone point me to a reference on how to find $w$ as the minimizer of: $$ \frac{1}{2}\sum_{i=1}^{d}q_i(w_i-m_i)^2+\sum_{j=1}^{n}log(1+\exp(-y_jw^Tx_j)) $$ where $q_i$ (initialized with ...
2
votes
0answers
374 views

Logistic regression algorithm in Casio and Texas Instruments calculators

When using logistic regression on a Casio or Texas Instruments calculator, the output is of the form $$f(x) = \frac{c}{1+ae^{-bx}} $$ The problem I have (when teaching in a class where both types of ...
2
votes
0answers
305 views

Finding a model for multiple non-linear regression

I want to implement a regression analysis, but I have problems with finding a model for the given data. There are $149$ $(x,y,z)$-values. $y$ values are all positive, $x$ is between $[-10, 10]$ and ...
2
votes
0answers
2k views

Derivation of standard error of beta in simple linear regression

Countless web pages show the equation for the standard error of the slope in a simple linear regression. For example: ...
2
votes
0answers
2k views

Help with problem: Estimated Standard Deviation of Regression Equation (Simple Linear)

This is a practice problem. I've solved part (a). I have provided verified answers (from the published key) to all parts (a), (b) and & (c). I need help solving (b) and (c). Consider a simple ...
2
votes
0answers
262 views

Surface Function Fitting to Spherical Data

I have a set of geographic (longitude,latitude,value) data to which I would like to fit surface functions, specifically, the set of quadratic surfaces: $f(x,y)=Ax^2+Bx^2+Cxy+Dx+Ey+F$ At the moment, ...
2
votes
0answers
65 views

Accurate computation for Linear Regression case

I am writing a program that inputs a sequence of points $(x_i,y_i)$ based on the user clicking on certain pixels in an image shown. The program should then find the "best -fitting" line in the least ...
1
vote
0answers
9 views

Regression question (details inside). Measuring the incremental impact on the dependent variable of one category over other categories

Formulate a regression equation you would use to test for the differences in ROE between firms that used tier 1 investment banks as their advisors and those that used tier 2 or tier 3 banks (note: ...
1
vote
0answers
13 views

Statistics linear modelling in R

Suppose I have a date set of the form: Test Subject/Sex/No. of mistakes made in the morning/ No. of mistakes made in the afternoon A / M / 2 / 5 B / F / 1 / 4 C / M / 3 / 5 D / F / 1 / 5 Suppose ...
1
vote
0answers
12 views

Cannot figure out autocavariance

The moving average model of order q has the form $$Y_t =β_0 +e_t +b_1e_{t−1} +b_2e_{t−2} +...+b_qe_{t−q}$$ where $e_t$ is a serially uncorrelated random variable with mean $0$ and variance $σ^2_e$. ...
1
vote
0answers
15 views

Time series regression

What if I have a stationary independent variable and 2 non-stationary dependent variables, and I want to run a regression on them, what model is the most appropriate?
1
vote
0answers
18 views

Derive the F-statistic for this model

Problem: Consider the model $y=\beta+\varepsilon$, where $\varepsilon\sim N_4(0,\sigma^2I)$, and $\sum_{i=1}^4\beta_i=0$. Derive the F-statistic for testing $H_0:\beta_1=\beta_2$. I tried to write the ...
1
vote
0answers
19 views

Outlier Contained in Prediction Interval (Tme series Forecasting Problem)

In my stats class today, the professor was showing us some output from MINITAB on a prediction interval that was calculated (from time series data using standard linear regression). For one of the ...
1
vote
0answers
11 views

derive the distribution(a multiple regression problem)

(Multiple regression model with p's predictor variables.) Derive the distribution of $$\frac{(b-\beta)X'X(b-\beta)}{MSE\cdot p}$$ As far as I know, $b\sim N(\beta,\sigma^2 (X'X)^{-1})$ $b-\beta ...
1
vote
0answers
18 views

Regression on even function?

Is there any test for whether or not the regression function is even? Suppose we have a model: $Y=g(X, \epsilon)$, where $Y, X$ are both one dimensional. My questions is how do we test for $g$ is an ...
1
vote
0answers
31 views

The Hessian Matrix I calculate is twice as much as it should be. Why?

I have a function "fkt." In this example, let it be as simple as $y=a \cdot x+b$. I have a real dataset with values obeying to the model. After regression of the points to the model, I find the ...
1
vote
0answers
38 views

Which projection, in $L_\infty$ norm or $L_2$ norm, is non-expansion?

I am just wondering which projection is non-expansion? Basically, I am wondering if $F$ is a projection operator then which norm would satisfy the following non-expansion property, where for a given ...
1
vote
0answers
49 views

Matrix Decompositions: Difference between Cholesky Decomposition, Eigendecomposition and Jordan Normal Form Decomposition

I recently created a related topic about the square root matrix, in case you'd like to refer to that one. Here's what we want: Consider the matrix $\Omega=E(\mathbf{u}^{\top}\mathbf{u})$, where ...
1
vote
0answers
13 views

Multiple regression model

I have a multiple regression equation which as four quarters (maybe called them as parameters) ...
1
vote
0answers
27 views

Round robin logistic regression

I have a poll with four answers (A,B,C,D) and response information about people who have taken that poll. I have created four models (one for each of the answers) in a one vs all. i.e. the model for ...
1
vote
0answers
30 views

Uni-variate Moving Average Theta coefficients

Consider the Uni-variate Moving Average Models (MA models) MA(1) $$x_t = \mu + w_t +\theta_1w_{t-1}$$ or the second order moving average MA(2) $$x_t = \mu + w_t +\theta_1w_{t-1}+\theta_2w_{t-2}$$ ...
1
vote
0answers
9 views

Proving $Corr(\hat{e}_{ij}, \hat{e}_{jk}) = \frac{-1}{n_i-1}$ for $ j \neq k$

For the model of a single factor experiment: $y_{ij}= \mu + \alpha_i + e_{ij}$, $(1 \leq i \leq a, 1 \leq j \leq n_i)$, where a = the number of treatments, $n_i$ = the number of experimental units ...
1
vote
0answers
13 views

Principal Components vs Principal Directions

I'm trying to do statistical downscaling of some climate data and there is a module of principal component analysis by regression method required. I am confused with the different terms here. What is ...
1
vote
0answers
113 views

Normal equations for minimization of Frobenius norm least squares error

I'm having a hard time understanding the most efficient sequence of steps for deriving the normal equations for Frobenius norm least squares minimization. Here I want to minimize the norm of a matrix ...
1
vote
0answers
22 views

Variance-covariance matrix of a linear regression model

In finding the covariance matrix of a linear regression model I don't understand this step: $$ E[(b-\beta)(b-\beta)']=E[(X'X)^{-1}X'\epsilon\epsilon'X(X'X)^{-1}] $$ where we've been given that $$ ...
1
vote
0answers
14 views

GMM estimation of linear regression with intercept restriction

Say I have a time series regression as follows: $$y_t = a_i + \beta_i x_t + \varepsilon_t^i \ \ ; \ \ t = 1, 2, \cdots, T \ \ \text{for each } i$$ Now say I impose the following restriction on the ...
1
vote
0answers
32 views

Least square / Linear regression over a simplex

I have to solve the following least square problem: $$\hat{x} = \arg \min_{x \in S} \|Ax - b\|^2$$ If $S = \mathbb{R}^n$, then the solution is given by $$\hat{x} = (A^TA)^{-1}A^Tb$$ having posed ...
1
vote
0answers
22 views

Orthogonalization of Variables

Let's assume we have three collinear variables/factors, $X_1$, $X_2$, $X_3$. Would there be a method to orthogonalize these variables in a simultaneous way: in other word, orthogonalizing them in such ...