Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

learn more… | top users | synonyms

4
votes
0answers
53 views

Bayesian linear regression cost function

I am studying classification using linear regression . Now, I want to map it in Bayesian regression. Let talk about binary classification using linear regression again. Assume that I have a set ...
3
votes
0answers
19 views

Regression with many discrete and continuous predictors and few rows

I want to do regression on a dataset. It has one continuous dependent variable that I want to predict. It has many categorical and some continuous predictors. It only has a few rows. A simplified ...
3
votes
0answers
30 views

How to perform nonlinear regression with regressors affected by gaussian error?

I am trying to calibrate a sensor and I have a data set consisting of several observations of a 3-dimensional vector $X_i$, with $X_i=w_i + \epsilon_i$ where $w_i$ is the value that the sensor ...
3
votes
0answers
136 views

Expected in-sample error of linear regression with respect to a dataset D

In my textbook, there is a statement mentioned on the topic of linear regression/machine learning, and a question, which is simply quoted as, Consider a noisy target, $ y = (w^{*})^T \textbf{x} + ...
3
votes
0answers
227 views

Can the sigmoid function approximate any function (or relation) where 0<y<1

I'm studying Machine Learning and Artificial Neural Networks. Some basic principles of Machine Learning are linear regression, multivariate linear regression, and nonlinear regression. The last of ...
3
votes
0answers
19k views

Linear regression: degrees of freedom of SST, SSR, and RSS

I'm trying to understand the concept of degrees of freedom in the specific case of the three quantities involved in a linear regression solution, i.e. $SST=SSR+SSE, $ i.e. Total sum of squares = ...
3
votes
0answers
184 views

Multilinear or Tensor Regression?

Given input data $x_t\in \mathbb{R}^n$ and output data $y_t\in\mathbb{R}^m$, the closed form solution to $\min_A \sum_t \|y_t - Ax_t\|^2_2$ is given by $A = (XX^T)^{-1}XY^T$ where $x_t$ form the ...
3
votes
0answers
160 views

Minimizing L4/ L6/ L2N norm for linear regression

OLS regression minimizes the sum of the squared errors. The normal equation for an OLS for $L_2$ minimization is as follows: $$b= (A'A)^{-1}A'y$$ What would be the equation to minimize the $L_4$ norm ...
3
votes
0answers
163 views

How to perform nonlinear regression with correlated errors?

I have a nonlinear least squares problem, but the errors are correlated. I could use R's nls function to do the regression if the errors were independent, but I don't know the right way to handle ...
3
votes
0answers
313 views

Least Square Method with Positive Parameters

this is my first post here in the Stack Exchange. A friend told me about this forum and I'm giving it a try. I searched a bit past threads, but couldn't find what I wanted, so I'm posting the problem ...
3
votes
0answers
1k views

Correlation and Regression Question

Two separate tests are designed to measure a students ability to solve problems. Several students are randomly selected to take both tests and the results are: $$ \begin{matrix} \text{Test A}(x) ...
2
votes
0answers
10 views

Problems with Exchange Procedure in Remez Algorithm

So first off: *** This code is not being used in production software. It is a personal project of mine, trying to understand approximation theory and advanced curve fitting. In other words, I'm ...
2
votes
0answers
21 views

combining multiple regression outputs

Suppose I have multiple regressions, along with their r-squares, standard-errors, etc.: $y(t) = \alpha_1 + \beta_1 x(t) + e_1$, where $t \in (\tau_0, \tau_1)$ $y(t) = \alpha_2 + \beta_2 x(t) + ...
2
votes
0answers
23 views

GLM for Poisson Regression for Soccer Ratings Not Converging

I have been trying to formulate a model of soccer matches to help me predict the outcomes. The model I'm trying to formulate involves using Poisson regression to assign attack and defence ratings to ...
2
votes
0answers
25 views

Can I adjust linear growth of a a subpopulation to a linear decay of the general population?

I need to estimate the amount of CF patients in Poland in the next four years. I have: estimations of the Polish population for the future years a CF patients' register for the last couple of years ...
2
votes
0answers
34 views

Show $\hat{\beta}$ and $s^2$ are independent?

I have the model: $y=X{\beta}+{\epsilon}$ I know $\hat{\beta}=(X'X)^{-1}X'y$ and that it is an unbiased estimator of ${\beta}$ and that $s^2=\hat{\epsilon}'\hat{\epsilon}/(n-k)$ and is an unbiased ...
2
votes
0answers
100 views

How to reach Moore-Penrose pseudoinverse solution to minimize error function

Edit I'm trying to figure the derivation of the Moore-Penrose pseudoinverse for linear regression. The starting expression is the standard error function. I'm not quite sure how to expand on this ...
2
votes
0answers
63 views

Predicting profit with price variation

I am currently working on a high school project that aims to predict profit from X amount of items to Y amount of profit based off a deviated sale price. For instance: I sale 10 cookies for 10 ...
2
votes
0answers
68 views

Uni-variate Moving Average Theta coefficients

Consider the Uni-variate Moving Average Models (MA models) MA(1) $$x_t = \mu + w_t +\theta_1w_{t-1}$$ or the second order moving average MA(2) $$x_t = \mu + w_t +\theta_1w_{t-1}+\theta_2w_{t-2}$$ ...
2
votes
0answers
200 views

Normal equations for minimization of Frobenius norm least squares error

I'm having a hard time understanding the most efficient sequence of steps for deriving the normal equations for Frobenius norm least squares minimization. Here I want to minimize the norm of a matrix ...
2
votes
0answers
35 views

Best line fit for correlated points

Given in $\mathbb{R}^3$ are $n$ points $\mathbf{y}_i\sim N(\mathbf{y}_i-\mathbf{\hat{y}}_i, \mathbf{C}_i)$, which are normally distributed. I want to determine a best fit line $\mathbf{u}(\lambda) = ...
2
votes
0answers
20 views

Showing Hat matrix equal specific values

Consider a one way layout model $y_{ij}$ = $\mu_i + e_{ij}$ (1 $\leq$ i $\leq$ a, 1 $\leq$ j $\leq$ $n_i$) where a = 3 and $n_1$ = 2, $n_2$ = 3, $n_3$ = 4. Show that the hat matrix for this design ...
2
votes
0answers
101 views

Is it compulsory to make transformation to the econometric model in order to have only diagonal elements on variance-covariance matrix of errors?

I need some sharped and advanced advices for the following issue ... Model and its assumptions I'm working on the methodology of a two-way error component model. Here is the model: $y_{jis} = ...
2
votes
0answers
93 views

What is ${\rm cov}(e_i, \hat y_i)$ in simple linear regression?

The model is $y_i = \beta_0 + \beta_1x_i + \epsilon_i$ What is ${\rm cov}(e_i, \hat y_i)$? What is ${\rm cov}(\epsilon_i, \hat \beta_1)$? What is ${\rm cov}(e_i, \epsilon_i)$? For 1, I am writing ...
2
votes
0answers
35 views

Smallest set of Liner equations, which exactly fit a set of points

I have a set of 2-d points,(it can be of any arbitrary dimension n). I want to find the minimum set of straight lines(linear equations) which exactly passes through the given 2-d points (unlike ...
2
votes
0answers
881 views

Least Squares “analytic expression” for fitting a 2D quadratic function to measurements

I have n scattered elevation measurements: $ \{x_i,y_i,z_i\}_{i=1..n} $ that I want to fit a quadratic function to: $ z = ax^2 + by^2 + cxy + dx + ey + f$. The problem can be written as a vector ...
2
votes
0answers
53 views

Computing evidence for least-squares fit

I'm at a loss trying to implement Bayesian model selection for standard least-squares polynomials fits. I have three polynomials of order $1$, $2$, and $3$, and a sequence of $(x,y)$ data points. ...
2
votes
0answers
650 views

What is the Moore-Penrose pseudoinverse for scaled linear regression?

The matrix equation for linear regression is: $$ \vec{y} = X\vec{\beta}+\vec{\epsilon} $$ The Least Square Error solution of this forms the normal equations: $$ ({\bf{X}}^T \bf{X}) \vec{\beta}= ...
2
votes
0answers
45 views

Coefficient of determination

$$ \displaystyle \sum^n_{i = 1} (y_i - \bar{y})^2 = ( \displaystyle \sum^n_{i = 1} (y_i - \bar{y})^2 - \displaystyle \sum^n_{i = 1} (y_i - \hat{y}_i)^2 ) + \displaystyle \sum^n_{i = 1} (y_i - ...
2
votes
0answers
162 views

Orthonormal Matrix weighted regression

$Q$ is a rectangular matrix with orthonormal columns. A linear system composed of $$Qx= b$$ is really easy to solve as: $$Q'Q=I$$ hence: $$x=Q'b$$ Given that $Q$ is orthonormal can this be used to ...
2
votes
0answers
88 views

Regressing $Y$ back on the residuals

Suppose I have the linear regression model $ \hat{y_i} = a + b x_i $ for $a,b$ obtained via OLS. How does one regress $y$ back on the residuals $\hat{e}_i = y_i - \hat{y}_i$? If we write $ ...
2
votes
0answers
172 views

Effective model for calculating momentum or growth rate for a time series

I have a series of numbers tracking the performance of an entity on any given day. It's nothing but a simple integer for each date. For example, here's a series for Entity "X" ...
2
votes
0answers
351 views

Bare minimum of points in multiple polynomial regression

I have a question on multiple polynomial regression and the absolute minimum amount of points in the different terms. The minimum amount of points required for a second order polynomial would (in one ...
2
votes
0answers
95 views

Find $w$ as the minimizer of regularized logistic regression

Could someone point me to a reference on how to find $w$ as the minimizer of: $$ \frac{1}{2}\sum_{i=1}^{d}q_i(w_i-m_i)^2+\sum_{j=1}^{n}log(1+\exp(-y_jw^Tx_j)) $$ where $q_i$ (initialized with ...
2
votes
0answers
446 views

Logistic regression algorithm in Casio and Texas Instruments calculators

When using logistic regression on a Casio or Texas Instruments calculator, the output is of the form $$f(x) = \frac{c}{1+ae^{-bx}} $$ The problem I have (when teaching in a class where both types of ...
2
votes
0answers
331 views

Finding a model for multiple non-linear regression

I want to implement a regression analysis, but I have problems with finding a model for the given data. There are $149$ $(x,y,z)$-values. $y$ values are all positive, $x$ is between $[-10, 10]$ and ...
2
votes
0answers
3k views

Derivation of standard error of beta in simple linear regression

Countless web pages show the equation for the standard error of the slope in a simple linear regression. For example: ...
2
votes
0answers
3k views

Help with problem: Estimated Standard Deviation of Regression Equation (Simple Linear)

This is a practice problem. I've solved part (a). I have provided verified answers (from the published key) to all parts (a), (b) and & (c). I need help solving (b) and (c). Consider a simple ...
2
votes
0answers
291 views

Surface Function Fitting to Spherical Data

I have a set of geographic (longitude,latitude,value) data to which I would like to fit surface functions, specifically, the set of quadratic surfaces: $f(x,y)=Ax^2+Bx^2+Cxy+Dx+Ey+F$ At the moment, ...
2
votes
0answers
67 views

Accurate computation for Linear Regression case

I am writing a program that inputs a sequence of points $(x_i,y_i)$ based on the user clicking on certain pixels in an image shown. The program should then find the "best -fitting" line in the least ...
1
vote
0answers
10 views

LMS Update Rule

I'm starting to study machine learning using Andrew Ng's class notes. I understand conceptually how linear regression works, but am having trouble with this equation: $$ \theta_j := \theta_j + ...
1
vote
0answers
24 views

Optimal place to measure for simple linear regression/fitting

Suppose I have a linear model $y_i=a\cdot x_i+b+\epsilon_i$, where $y_i,\epsilon_i,a,b\in\mathbb{R}$, $x_i\in[-1,1]$. I can take n measurements of $y_i$ at $x_i$, where $n\in\mathbb{N}$. $\epsilon_i$ ...
1
vote
0answers
18 views

How to compare experimental data with teorethical prediction

I would like to know, what is the method to approximate experimental data to teorethical one. I have heard about polynomial regression. After calculating particular matrices and solving set of ...
1
vote
0answers
18 views

LRT and Wald Test in Multivariate Linear Regression significance

I've been researching all afternoon trying to get a better idea of what the Likelihood ratio test and the Wald test are actually doing. I have a bunch of covariates and I'm testing out like 30 ...
1
vote
0answers
37 views

Confusion with Bayesian Linear Regression

In the book Gaussian Processes for Machine Learning in Chapter 2 p. 11 (see http://www.gaussianprocess.org/gpml/chapters/RW2.pdf), eq. 2.9 states: $p(f_* | X, y) = \int p(f_* | x_*,w) p(w|X, y)dw$ ...
1
vote
0answers
65 views

Minimize correlation between input and output of a linear system

I am not sure if "minimize correlation" is the right title for this issue but I could not find a better sentence to describe what I would like to achieve. Let's say that I have a black box with ...
1
vote
0answers
14 views

How is genetic programming used in Symbolic regression

I am in highschool and have not taken any courses on this. Rather I am working on an outside project. I don't quite understand how Genetic Programming could be used effectively to generate a set of ...
1
vote
0answers
16 views

Approximation technique when data is missing?

I am doing some statistical studies and I would appreciate some guidance to some approximation techniques when not all data is available. I have a model that takes certain input parameters (discrete, ...
1
vote
0answers
33 views

Does gradient descent and normal equation give the same answer?

I tried to optimize for a linear regression model using both approaches and they gave me two completely different answers. My sample data set was: df <- data.frame(c(1,5,6),c(3,5,6),c(4,6,8)) ...
1
vote
0answers
41 views

Odd Ratio and Logistic Regression

Windy and Play Tennis = 9 Windy and Not Playing Tennis = 8 Not Windy and Play Tennis = 14 Not Windy and Not Playing Tennis = 8 I performed logistic regression in Weka and got odd ratio as 0.3448 for ...