Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Join us in building a kind, collaborative learning community via our updated Code of Conduct.

Questions tagged [regression]

Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

0
votes
0answers
13 views

Back substitution and QR Factorisation via Householder

I'm having difficulties getting the same beta values from this OLS regression as I would without the QR decomposition. I believe it has to do with the shape of my Q block. I start as so: A =\begin{...
0
votes
0answers
16 views

Is there a theory for piecewise differentiable regression polynomials?

I have an interesting question, I would like to have answered... I have a very noisy signal $f$, that I want to smoothen out. Using a global regression cannot work, as I don't have a model of the ...
1
vote
0answers
15 views

Choosing regression variables to minimize variance of coefficients

We have the following exercise to solve: A researcher wants to find out how variable Y is dependents on amounts of variable X, with n = 10 samples given. The i-th sample has an amount $x_i$ of X, ...
3
votes
1answer
60 views

Condition number of $AA^T$ when $A$ is polynomial Vandermonde

Suppose I'm doing polynomial regression of degree $m$ $$p(x, \mathbf{w}) = w_0 + w_1x + \dotsb + w_mx^m$$ given training data $(x_1, t_1), \dotsc, (x_N, t_N)$. Suppose I'm using the loss function $...
0
votes
0answers
12 views

Rigorous error bounds for polynomial regression

Consider a set of $N$ points $(x_i , y_i)$. I want to find a $d$ degree polynomial $P_d(x)$ that will minimize the error, $$ e_d = max_{i \in [N]} ~|P_d(x_i) - y_i| $$ The question I have is about ...
0
votes
0answers
21 views

Expressing the following regression in terms of matrices depending on i

I am not sure, if this question belongs more to Cross-Validated or here. It's a mathematical question of a statistical regression. My aim is to express the following regression in terms of matrix ...
1
vote
2answers
42 views

Explaining Why the Zero Conditional Mean Assumption is Important

I am currently relearning econometrics in more depth than I had before. One thing I am trying to make sense of currently is why it is necessary for the assumption of: $$E(u\mid x)=E(u) $$ to be true (...
2
votes
0answers
85 views
+100

Linear regression with feature representation confusion - is design matrix column space the feature space?

I am trying to visualise the geometry of linear regression with feature representation. I have a regression problem with $n$ data pairs $\mathcal{D}:=\{(\mathbf{x},y)_{i}\}_{i=1}^{n}$, independent ...
0
votes
3answers
23 views

Multivariate linear regression with 2 independent variables - formulae

I have regressed y on x1 and x2 in python but I get very different results when I do it by hand. I am using the following formulae: http://faculty.cas.usf.edu/mbrannick/regression/Reg2IV.html I am ...
1
vote
1answer
38 views

Regression and percentile

On a midterm exam, the average is $50$ points (out of $100$) with an SD of $10$ points. On the final exam, the average was $100$ points (out of $200$) with an SD of 30 points. The correlation ...
-3
votes
0answers
54 views

math behind Support Vector Regression

I have been working with support vector machine for both regression and classification problems. But somehow I am not sure if I really get it. Therefore, Let me explain my understanding of SVR in ...
1
vote
1answer
42 views

eigenvalues of fitted value?

Let a $m\times n$ full-column matrix be $A$. Also, denote $m\times m$ diagonal matrix as $D$. Define $P(A)D=A(A'A)^{-1}A'D$ as the orthogonal projection of $D$ onto the column space of $A$, or "...
1
vote
0answers
31 views

Machine of maximum number of support vectors (SVM)?

I have learned a thing or two about Support Vector Machines (SVM) and it seems to me that maximum margin machines are popular. I came to wonder if there exist any flavour of SVM which not only strive ...
0
votes
1answer
26 views

Multidimensional Linear Fit

As shown in the attached picture, I have different measurement points which belong to certain linear functions with different slopes $m$ and different $y$-intercepts $n$. For every curve I have a ...
1
vote
1answer
35 views

How to obtain expressions for coefficients from OLS formula?

Consider the standard linear regression model: $y_i = \alpha + \beta D_i + e_i$ where the coefficients are defined by linear projections and $D_i$ is a dummy variable. In the population, the ...
1
vote
0answers
48 views

Integration by parts with empirical measure

I'm currently reading through the paper Asymptotic normality of nearest neighbor regression function estimates and am struggling to understand the asymptotic equalities that were shown in the proof of ...
0
votes
0answers
24 views

Find contribution of various features/input variables to the variance of the dependent variable / Attribute variance of dependent variable to features

I am working on this problem where I have 20 odd features (input variables) and two dependent variables. The objective is to find the variance structure of one of the dependent variables. More ...
1
vote
0answers
67 views
+50

Minimization problem with latent function and splines

I have a dataset consisting of pairs $(x_i, y_i)$. I want to determine the function $f$, so $$ f(x)f(y) = 1 $$ with the constraint that $f(x) \leq x$, $f'(x) \geq 0$ and $f''(x) \geq 0$. I was ...
0
votes
0answers
53 views

Convert a matrices multiplications to Ax=b

I have a training data matrix $S_{\tau \times n}$ and actual output $y_{1\times P}$. The weighted parameters for a linear model that maps the input to the output is $$ y = \alpha_{1 \times \tau}S_{\...
1
vote
1answer
60 views

Regressions in case of non-normality

Our variables and residuals are not normally distributed. What we found is that regressions are usually quite robust against violations of normality. But we don't know to which degree, because our ...
1
vote
1answer
25 views

Conditional expectation $E(a^t \epsilon+b^t \beta \mid Y)$ in linear regression matrix model

We have usual matrix linear regression model $Y=X \beta+\epsilon$ , where $E(\epsilon) = 0 $ and $\operatorname{Var}(\epsilon)=\sigma^2 I$ and $\hat{\beta}$ is the least squares estimator of the ...
5
votes
3answers
160 views

Understanding the difference between Ridge and LASSO

I've asked this question a few days ago in the statistics site of this network, but although it's received a fair amount of views, I got no answer. If this kind of double posting is inappropriate let ...
0
votes
1answer
46 views

Relation between projection matrix and linear span

Let $X$ be an $m \times n$ matrix. Define the orthogonal projection onto the column space of $X$ as $$P(X):=X(X'X)^{-1}X'$$ Also, define the linear span of a set of $m\times 1$ column vectors of $X$...
0
votes
0answers
24 views

Modeling with Support Vector Machine in regression problem

I have 20 independent variables(explanatory)and 1 dependent(respond) variable and I wish to use SVM as a regression problem to predict my dependent variable. but the point is that my independent(...
-2
votes
2answers
52 views

Is polynomial regression typically linear or non-linear?

Here's my doubt: When we say polynomial regression, do we usually mean linear or non-linear? A simple answer would be very helpful. This is not a duplicate question. I've read through similar ...
2
votes
1answer
38 views

how fit a model with data following asymptotic / sigmoid pattern

I'm trying to fit data. I assume that the association between dependent and indepdent variable is of the form $$T(y)=aR(x)+b$$ I also know that my data are ressemble either an asymptotic function ...
0
votes
0answers
15 views

How well does a model fit a subset of the original dataset?

A linear regression model has been created based on a dataset with observations which can be sorted into several different categories. I have been asked to assess how well this regression model (...
0
votes
0answers
16 views

Variance of $\hat{\beta_i}$ in a special case of linear regression model [duplicate]

We have OLS model $Y= X\beta+ \epsilon$, where rows of matrix $X$ are multivariate normal independent vectors with expected value $0$ and variance $\Sigma$. Let's say that vector $\epsilon$ is also ...
0
votes
0answers
22 views

Handling outliers in estimated forward rates

I am estimating forward interest rates. Historically i observe that the daily change in interest rate is less than 0.05% at 90 percentile. If my estimated rate for a given day is 8.50% and previous ...
0
votes
1answer
35 views

How to minimize a function following an asymptotic pattern?

When running simulation of data following a linear function (with noise) with python, I can find that the linear model gives the best fit according to the standard error of the regression: ...
0
votes
0answers
41 views

Show that estimator is the best linear estimator with smallest MSE

Let's consider we have OLS model $Y=X \beta+\epsilon$ , where $E(\epsilon) = 0 $ and $Var(\epsilon)=\sigma^2 I$. Here $\hat{\beta}$ is the least squares estimator of the parameter $\beta$ . Let $z \...
2
votes
2answers
46 views

Why not learn noise/error in least squares if we know its form?

Let's say I am doing linear regression and I have a data matrix $A$. And, I know the noise $e_i$, is zero mean (and perhaps we know the distribution, too): $$y_i = a_i^T x + e_i$$ Obviously from ...
0
votes
0answers
33 views

Is there a curve fitting algorithm that focuses on fitting one part of the curve well rather than trying to make a good general fit?

One use case for this would be multi-exponential decay fitting, I'm talking about functions of the type $ \displaystyle f(t)=\sum_i^N a_i \cdot e^{-t/\tau_i} $ It would be awesome to have a fit ...
0
votes
1answer
30 views

Show that $\hat{\beta} $ and $\hat{\sigma^2}$ are unbiased in special case of linear regression model [closed]

Let's consider we have OLS model $Y= X\beta+ \epsilon$, where rows of matrix $X$ are multivariate normal independent vectors with expected value $0$ and variance $\Sigma$. Vector $\epsilon$ is ...
1
vote
1answer
34 views

If $H$ is a hat matrix (symmetric and idempotent), then $h_{ii} = \sum^n_{j=1}h_{ij}^2$?

In regression model, $H=X(X^TX)^{-1}X^T$ is a hat matrix (so it is symmetric and idempotent), then we have $$\text{tr}(H^2)=\text{tr}(H)=\text{rank}(H)=\text{rank}(X),$$ We can conclude that $\sum_{...
1
vote
1answer
33 views

Solve Cost Function $J = \sum{(y_n - \hat{y}_n)^2} + \lambda\lVert w \rVert^2 $ Using Linear Algebra

After struggling for a few days I feel a bit more comfortable solving the cost function $J = \sum{(y_n - \hat{y}_n)^2} + \lambda\lVert w \rVert^2 $ for $w$ using vector calculus. However, when I ...
1
vote
0answers
31 views

Least square estimate for matrix A (Y=AX)

Consider the $Y(i),X(i)$ each of an $n\times 1$ matrix(i range from $1$ to $N$), where $A$ an $n\times n$ matrix. Ask to calculate the least square estimate for $A$. From regression, I kind of know ...
0
votes
1answer
22 views

Finding the Variance of a predicted exponentiated linear value

Let me give you some background on my issue. Let's say I have a simple one covariate linear regressor as follows: $$\log\alpha =\alpha_0+\alpha_1x_{i1}.$$ Obviously, to find $\alpha$, we use ...
1
vote
1answer
26 views

Derivative quadratic form for regression

I am interested in multilinear regression for Student distribution. Let $\mu_i=X_i\beta,$ to compute estimators for multivariate Student distribution I need to compute the following derivative ...
0
votes
0answers
30 views

How do they find out the definition of cross entropy error function?

The definition of cross entropy error function is $$J = \sum_i y_i ln (\hat{y}_i) $$ I wonder how can they come up with this definition?
0
votes
0answers
17 views

Problem with linear regression normal equations when a coefficient should be zero

I am having trouble using the linear regression normal equations to minimize a function of several variables. The minimization solution is : $$\beta = (X^TX)^{-1}X^TY$$ where $\beta$ is the ...
0
votes
0answers
16 views

Is it possible to perform the regression algorithm on two dependent variables?

I'm trying to change the 3d coordinates of one reference frame 'a' to other reference frame 'b'. For few points in 'a' i'm having corresponding points in 'b'. So based on this, i want to convert all ...
0
votes
0answers
19 views

relationship between partial correlation and multivariate normal distribution

Partial correlation between $X$ and $Y$ is obtained by regressing $X$ on $Z$ to obtain $\hat{X}$, regressing $Y$ on $Z$ to obtain $\hat{Y}$, and calculating \begin{equation}Corr(X-\hat{X}, Y-\hat{Y})...
1
vote
0answers
17 views

Formula For Centered Coefficient of Determination

I want to show that in the inhomogeneous regression model $y=\alpha+X\beta+u $ $$ R_*^2=max_{z\in \text{col}(X)}r_{x,y}^2, $$ where $r_{x,y}^2:=\frac{S_{xy}^2}{S_xS_y}$ is the squared empirical ...
-1
votes
2answers
24 views

What are the relationships between no. of inputs and no. of outputs in regression?

What are the relationships between no. of inputs and no. of outputs in regression? Particularly, what if $|y| > |x|$? What if $|y|<|x|$? Why not always $|y|=|x|$? Where $|\cdot|$ denotes the ...
0
votes
0answers
13 views

How is the error term of Radial Basis Function (Network) found?

How is the error term of Radial Basis Function (Network) found? Such as in: $$\min_W I(W)=\frac{1}{2N} \sum_{i=1}^N ||W g(x_i)-y_i||^2+ \color{red}{\frac{\beta}{2} \sum_{j,k} |W_{jk}|^2}$$ Then how ...
0
votes
0answers
32 views

formulation of least squares problems

In general, to use the method of least squares, a linear stochastic system is modeled as: \begin{equation} y = ax + \eta \end{equation} where, $y$, is an observed variable, $x$ is an input while $\...
4
votes
3answers
70 views

Why isn't the linear regression coefficient not just the average vector to data points?

I am having trouble intuitively understanding the correctness of the formula to compute the coefficient for the regression line in a linear regression. I know the formula is $$\frac{\sum_{i=1}^N (...
2
votes
4answers
90 views

Does least squares (approximate solution) minimize the orthogonal distance of $b$ to $Ax$, or does it minimize the error projected along the $b$ axis?

I have always been confused about whether the approximate solution to $Ax=b$ is equivalent to minimizing the average distance of all of the $b$ vectors to $Ax$, or whether it is minimizing the ...
1
vote
0answers
40 views

Finding an interval by curve fitting

Suppose I have a function like the one in this graph: I have samples $x_i,y_i$ from this function, with some Gaussian noise (like the black points). The function behaves, in some interval $[a,b]$, ...