4
votes
2answers
68 views

Rewriting the matrix equation $AX = YB$ as $Y = CX$?

Is it possible in general, if $A,B,C,X,Y$ are square and of the same dimensions? If so, does it generalize to non-square matrices (using a pseudoinverse)? I'm doing some curve fitting in which I have ...
1
vote
1answer
34 views

Derivative of logistic loss function

I am using logistic in classification task. The task equivalents with find $\omega, b$ to minimize loss function: That means we will take derivative of L with respect to $\omega$ and $b$ (assume y ...
3
votes
0answers
33 views

Bayesian linear regression cost function

I am studying classification using linear regression . Now, I want to map it in Bayesian regression. Let talk about binary classification using linear regression again. Assume that I have a set ...
0
votes
1answer
15 views

Derivation of SSE gradient for Linear Regression

In my text book it gives the derviation of the gradient for the likelihood of a linear regression model (to minimize the negative log likelihood by minimizing the Sum Squared Error). The first line ...
0
votes
0answers
44 views

Help larry water his tomato plants with math

I have a bit of a real world problem that I believe Math can help me solve. I think it might be easiest to phrase in a manor similar to that of high school textbook. Larry has a device that can ...
1
vote
1answer
32 views

Calculating R-squared with duplicate data

I have the following question regarding the proper usage of R-squared value. Say I have an equation, that predicts energy consumption for the month of a building. One of the input variables accounts ...
0
votes
0answers
32 views

Multiple Linear Regression sample problem:

I am currently studying linear regression, but I am not sure I understand everything correctly. I was trying to solve some of the exercises at the end of my book, and I picked a random one below. I am ...
1
vote
1answer
24 views

Simple linear regression prove variables are uncorrelated:

I am working on the following problem: In a problem of simple linear regression, $$Y = \hat\beta_0 + \hat\beta_1 x(bar),$$ show that the random variables $\hat\beta_1$ and $Y$ are un-correlated (All ...
2
votes
0answers
31 views

Smallest set of Liner equations, which exactly fit a set of points

I have a set of 2-d points,(it can be of any arbitrary dimension n). I want to find the minimum set of straight lines(linear equations) which exactly passes through the given 2-d points (unlike ...
0
votes
0answers
39 views

How do I go from $(X\beta)'Y$ to $Y'(X\beta)$ when solving OLS in matrix form?

When I first learned how to derive the OLS $\beta$ in matrix form, I learned to take the derivative of the summation first, then convert into matrix form. In other words Using calculus, we derive ...
1
vote
0answers
16 views

Regression factors and covariance matrix

I am trying to follow some notes. They have two matrices. One is called comfact (company factors). This is a $580 \times 5$ matrix. The $580$ rows represent $580$ different companies. The $5$ ...
0
votes
0answers
20 views

linear regression optimization

If I have this equation to be minimized (it is from linear regression) and the example is: or in its complete form: so it says that one needs to minimize the function J(theta). The parts of ...
1
vote
0answers
36 views

Linear Regression Question (Linear Algebra) Help!!

Hey guys, I have a quick question. I am trying to prove that the squared sample correlation between fitted and observed values is equal to $R^2$ (coefficient of determination). I am having a lot of ...
2
votes
2answers
84 views

connection between PCA and linear regression

Is there a formal link between linear regression and PCA? The goal of PCA is to decompose a matrix into a linear combination of variables that contain most of the information in the matrix. Suppose ...
0
votes
1answer
68 views

Contrary interpretations of Least Squares for Regression

According to the original thought, our goal is to minimize the quadratic error $$\min\{\frac{1}{2}(Ax-b)^2 \}$$ Then, we search the extremum by the derivation of $x$ $$A^T(Ax-b)=0$$ $$A^TAx=A^Tb$$ ...
0
votes
0answers
24 views

MSE of weighted PCA estimator

I need to calculate the variance of this estimator which is a generalisation of the OLS estimator: OLS: $Y=X\beta+e$ where Y is n*1 vector of responses X is n*p pbserved matrix of regressor ...
0
votes
0answers
55 views

Bias in Principal Components Regression

Assume we have the well known OLS model $Y=X\beta+e$ where Y is n*1 vector of responses X is n*p pbserved matrix of regressor variables $\beta$ is a p*1 vector of unknown parameters e is a n*1 ...
3
votes
1answer
82 views

Solution to linear system of equations

Notation. Let $y$, $a$, and $b$ be $n\times 1$, $p\times 1$, and $q\times1$ real vectors. Let also $X$ and $Z$ be $n\times p$ and $n \times q$ real matrices. Suppose that there is no solution, $a$, ...
0
votes
1answer
157 views

Quadratic form as a ratio of determinants

I am looking for some hints to prove the following equality: $y^{\top}y - y^{\top}X(X^{\top}X)^{-1}X^{\top}y = \dfrac{\det(L^{\top}L)}{\det(X^{\top}X)},$ where $y$ is a $n\times 1$ vector, $X$ is a ...
0
votes
0answers
30 views

How to do regression for an exponential model?

I have data in the form of: (x1, y1, z1 || t1) (x2, y2, z2 || t2) ... (xN, yN, zN || tN) Where (x,y,z) pairs are inputs, are t's are outputs. Using this data I ...
1
vote
0answers
494 views

Least Squares “analytic expression” for fitting a 2D quadratic function to measurements

I have n scattered elevation measurements: $ \{x_i,y_i,z_i\}_{i=1..n} $ that I want to fit a quadratic function to: $ z = ax^2 + by^2 + cxy + dx + ey + f$. The problem can be written as a vector ...
1
vote
0answers
333 views

What is the Moore-Penrose pseudoinverse for scaled linear regression?

The matrix equation for linear regression is: $$ \vec{y} = X\vec{\beta}+\vec{\epsilon} $$ The Least Square Error solution of this forms the normal equations: $$ ({\bf{X}}^T \bf{X}) \vec{\beta}= ...
1
vote
0answers
51 views

Conditioning on $X$ equal to premultiplying by $X'$?

I am coming across similar thing in many problems in econometrics and I have not been able to figure out whether it is some general notion or only a "coincidence". To take two examples: Deriving ...
0
votes
1answer
133 views

Fast way of finding RSS of Multiple Linear Regression

Is there any smarter way to compute Residual Sum of Squares(RSS) in Multiple Linear Regression other then fitting the model -> find coefficients -> find fitted values -> find residuals -> find norm of ...
2
votes
1answer
251 views

Finding uncertainty in the slope/intercept for a non-linear least squares fit

I have the following function: $$M = a(\log_{10}W-2.5)+b$$ I also have a set of data with actual measured values of $W$ and $M$ (each have individual $\pm$ errors). Here's a small sampling of the ...
2
votes
4answers
95 views

Condition for $\det(A^{T}A)=0$

Is it always true that $\det(A^{T}A)=0$, $\hspace{0.5mm}$ for $A=n \times m$ matrix with $n<m$? From some notes I am reading on Regression analysis, and from some trials, it would appear this is ...
0
votes
0answers
59 views

How does this affect the slope

This is a Linear regression question, where I have a list of values for Sales ($Y$) and Property ($X$), I have calculated the Mean for both $Y$ and $X$, resulting in $Y=52.2$and $X=17$, I have ...
2
votes
1answer
60 views

Possibility of Unboundedness in Least Squares Minimization

Suppose we have the quadratic minimization problem \begin{equation} \min_x \frac{1}{2} x^TPx + q^Tx +r \end{equation} We know that when $P$ is symmetric positive semi-definite, but the optimality ...
2
votes
0answers
118 views

Minimizing L4/ L6/ L2N norm for linear regression

OLS regression minimizes the sum of the squared errors. The normal equation for an OLS for $L_2$ minimization is as follows: $$b= (A'A)^{-1}A'y$$ What would be the equation to minimize the $L_4$ norm ...
2
votes
0answers
124 views

Orthonormal Matrix weighted regression

$Q$ is a rectangular matrix with orthonormal columns. A linear system composed of $$Qx= b$$ is really easy to solve as: $$Q'Q=I$$ hence: $$x=Q'b$$ Given that $Q$ is orthonormal can this be used to ...
1
vote
1answer
242 views

QR factorization for ridge regression

I am solving an overdetermined system of equations: $$Ax= b$$ Using QR factorization, we can solve this system easily by posing it as: $$Rx= Q'b$$ I would like to regularize my estimate of $x$. I ...
0
votes
1answer
125 views

Does Principal Component Regression still work in high-dimensional ($N<p$) situation?

I understand that, many classical methods for multiple regression won't work when $N<p$, where $p$ is the dimension of the input space and $N$ is the sample size. For example, LSE for multiple ...
0
votes
0answers
44 views

Derivative of $H$ with respect to $W$

I am trying to solve a generalized linear squares model with the following form: $\hat{Y}= X(X'\Omega^{-1}WX)^{-1}X'\Omega^{-1}WY $ $ H= X(X'\Omega^{-1}WX)^{-1}X'\Omega^{-1}W $ $ \Omega$ is the ...
0
votes
1answer
215 views

Intersection line of planes

There are 2 planes: $Q: 2x-y=2$ $R: x-y-z=1$ How do I find the line $p=(1,2,-1)$, which intersects $Q$ and $R$ plane.
0
votes
1answer
63 views

Finding the value which minimises all residuals

I have a series of observations, measurements made at various times $t$. I now need to determine the most likely value of $R$ (distance) using the model below. The guide says I should find the value ...
1
vote
2answers
129 views

Least Squares Derivation

I was reading this to review the derivation of the ordinary least squares estimator but I'm having trouble differentiating (4). Can someone please help explain why $ \dfrac{\partial ...
8
votes
3answers
223 views

Minimize $||Ax-b||$ but for $A$, not $x$

I have a machine learning regression problem. I need to minimize $$ \sum_i||Ax_i-b_i||_2^2 $$ However I am trying to find matrix $A$, not the usual $x$, and I have lots of example data for $x_i$ and ...
0
votes
2answers
37 views

Where is the square in the Least square regression method?

I'm having a serious doubt in the least square regression problem. I guess its got to do with the notation of norm. Is the least square formulation $||b - \mathbf{A}x||^2$ or is it $||b - ...
0
votes
1answer
65 views

P-value not shown when there are too many variables in a linear regression

x<-c(1,2,3) y<-x^1.1+x summary(lm(y~x+I(x^1.1))) I have this code in R but it just is for the sake of easier understanding of what I am trying to achieve. ...
1
vote
1answer
118 views

Calculating the regression equations

I have four data points $(1,2), (2,4), (3,5), (5,7)$ and Im looking for the least squares regression line that best fits them. I use the normal equation $A^tAx=A^tb$ in this form - ...
5
votes
3answers
1k views

Linear Regression?

How was the formula for Ordinary Least Squares Linear Regression arrived at? Note I am not only looking for the proof, but also the derivation. Where did the formula come from?
1
vote
2answers
358 views

Simple Least Squares Regression?

I have a vector X of 50 real numbers and a vector Y of 50 real numbers. I want to model them as y = ax + b How do I determine a and b such that it minimizes the ...
0
votes
1answer
84 views

Regression on Linear Model?

I have 50 or so training examples involving a set of 200 or so real numbers (x1,x2,...,x200) (normalized to a 0 mean and std dev 1), and a single output real (y) in the range 0.0..1.0. I want to fit ...
1
vote
0answers
372 views

Moore–Penrose pseudoinverse reference

Given the eigendecompositions $AA^{\top}=Q \Lambda Q^{\top}$ and $A^{\top}A=P \Lambda P^{\top}$, where $\Lambda$ is a diagonal matrix (of eigenvalues) and $P$ and $Q$ are unitary eigenvectors matrices ...
0
votes
1answer
134 views

Explain Least-squares plotting with Ones -matrix

I am stuck to this code and particularly the point F=[ones(size(x)) x x.^2]. What does it do mathematically? I cannot understand what the heck the matrix has to do ...
0
votes
1answer
119 views

Linear algebra with a linear model (Matlab)

Given the equation $$r = B + e(r\cos(\theta))$$ and the corresponding data: $\theta: 0.88; 1.1; 1.42; 1.77; 2.14$ and $r: 3; 2.4; 1.65; 1.25; 1.01$ How do you input these data for matlab to ...
1
vote
0answers
684 views

Best-fitting plane

I need to implement the algorithm described below. Everything is fine until the eigenvalues computation. I'm completely new to them and I found a lot of very complicated paper on the net. Is it ...
0
votes
1answer
95 views

Is this expression correct?

sorry I don't know how to post Latex here I have to implement in Matlab the following formula and I'd like to know if the expression given here really corresponds to the minimum indicated, and why: ...
2
votes
0answers
64 views

Accurate computation for Linear Regression case

I am writing a program that inputs a sequence of points $(x_i,y_i)$ based on the user clicking on certain pixels in an image shown. The program should then find the "best -fitting" line in the least ...
0
votes
1answer
2k views

How different is Beta computation using Covariance and Linear Regression?

I wanted to compute Beta for a Stock against an Index (Say Stock X against S&P 500). I computed the daily returns for over one year applied the following logic : Beta = COVAR(X, S&P ...