Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

learn more… | top users | synonyms

3
votes
3answers
114 views

linear solution of curve fitting on multiple linear functions differing by a multiplier

I recently posted this question here but I thought this could be of interest also in mathematics, given I found a partially related question here I am facing the following problem. I know nonlinear ...
3
votes
2answers
625 views

Binary Logistic Regression Model Processing

Thanks for showing interest and wanting to help out. My aim is to develop a model that - as accurately as possible - predicts how entities in a population will either cooperate or defect, as a % of ...
3
votes
2answers
602 views

Fitting a sine function to data

I have a sequence of $n$ points $(x_i,y_i)$, for $i=1,\dots,n$. I would like to find the function, of the form $y=V\sin(x+\phi)$, which best fits the points. Which numerical method could I use? I have ...
3
votes
1answer
180 views

Ridge Regression: $\hat{\beta} \rightarrow \beta$

I'm trying to find the probability limit of $$\hat{\beta} = \left( \sum x_i x'_i + \lambda I_k \right)^{-1} \left( \sum x_i y_i \right) $$ as $n \to \infty$, and $\lambda$ is some positive ...
3
votes
2answers
34 views

Distance between a plane and set of points

Suppose $m$ data points belonging to a class represented by matrix $A$. Therefore, the size of matrix $A$ is $m\times n$. In addition, suppose $w\cdot x + b=0$ be equation of a plane in ...
3
votes
1answer
25 views

Quantile Regression - Linear Loss Minimization

I'm currently reading Quantile Regression by Roger Koenker, and for some reason I'm having a lot of trouble deriving one of his equation (sect. 1.3, p. 5-6). He goes on to demonstrate that $\hat{x}$ ...
3
votes
1answer
72 views

Rates of convergence of an OLS estimator

I have a linear regression model $$ y_t=x_t\beta+e_t,\quad t=1,\ldots,N. $$ Here $x_t$ is non-random and given by $(1,\delta_t t)$ where $\delta_t$ is 1 for odd $t$ and $0$ otherwise. Moreover, ...
3
votes
1answer
255 views

Multicollinearity and SVD

I compute the Singular Value Decomposition of a $n \times n$ matrix. If the matrix is not full rank, and I have 2 collinear columns, I end up with one singular value equal to 0. Is it possible to find ...
3
votes
1answer
946 views

What is the difference between Curve Fitting and Regression(Machine Learning)?

I know that Machine Learning regression algorithms try to find the function of the data. That is, if we have 1000 data points (x,y), to find a general continuous function that follows the trends of ...
3
votes
1answer
125 views

Minimum required data for cosine fit

With a minimum amount of (noisy) data-points, I need to find the amplitude of a simple cosine y=A*cos(x), where x is an angle from 0:2pi. I know how to fit data to the function, and I know how to ...
3
votes
1answer
90 views

Solution to linear system of equations

Notation. Let $y$, $a$, and $b$ be $n\times 1$, $p\times 1$, and $q\times1$ real vectors. Let also $X$ and $Z$ be $n\times p$ and $n \times q$ real matrices. Suppose that there is no solution, $a$, ...
3
votes
1answer
108 views

Constraining estimated linear regression coefficients over several regressions

I'm trying to run a series of simultaneous linear regressions, and I want to constrain the regression coefficients. For the standard ordinary least squares regression, the specification of the ...
3
votes
1answer
76 views

Why is $\sum x^2 _t \times \text{Var}(\beta)=\frac{\sum x^2 _t \times \sigma^2}{ \sum x^2 _t} = \sigma^2$?

I do not get this connection. Is is reliable to divide this equation by $\sum x^2 _t$ to get just $\sigma^2$ ? $$\sum x^2 _t \times E(\hat \beta - \beta)^2=\sum x^2 _t \times ...
3
votes
2answers
132 views

Can I consider shooting% as an independent variable

First time poster in the math section (a few posts in the stats section) and I am looking for clarification on a variable query that I have. Basically I enjoy sports and enjoy putting a mathematical ...
3
votes
2answers
567 views

Confidence interval of a random variable for an ordinary linear regression

I have a small problem. With my limited stats background I am not sure I am getting this one right. After fitting an ordinary linear regression model I get ...
3
votes
0answers
19 views

Regression with many discrete and continuous predictors and few rows

I want to do regression on a dataset. It has one continuous dependent variable that I want to predict. It has many categorical and some continuous predictors. It only has a few rows. A simplified ...
3
votes
0answers
28 views

How to perform nonlinear regression with regressors affected by gaussian error?

I am trying to calibrate a sensor and I have a data set consisting of several observations of a 3-dimensional vector $X_i$, with $X_i=w_i + \epsilon_i$ where $w_i$ is the value that the sensor ...
3
votes
1answer
53 views

Weighted least squares with angular data

Suppose I have a system whose state is $\Theta=(\theta_1,\theta_2,\ldots,\theta_n)$, where $\theta_i\in[-\pi,\pi)$ (i.e., they are angles). I'd like to determine the most likely estimate of $\Theta$ ...
3
votes
2answers
126 views

Regression with error coming from rounding

I am looking at the following model: $c$ is a fixed vector in $\mathbb{R}_+^n$ and for any $x \in \mathbb{R}_+^n$ we obtain a value $y =[c^Tx]$, i.e. rounding $c^Tx$ to the nearest integer. I want ...
3
votes
1answer
145 views

How to perform a monotonic function fitting of data points?

I'm seeking suggestions for general purpose function fitting of a set of data points, where, based on physical intuition, the relationship is expected to be "monotonic", i.e. the function should be ...
3
votes
0answers
127 views

Expected in-sample error of linear regression with respect to a dataset D

In my textbook, there is a statement mentioned on the topic of linear regression/machine learning, and a question, which is simply quoted as, Consider a noisy target, $ y = (w^{*})^T \textbf{x} + ...
3
votes
1answer
35 views

Logarithmic Functions algebra question

This is my first post and I honestly just want a second opinion on my answer to a question I got incorrect on an exam before I go arguing over it with my professor. Basically, is this mathematically ...
3
votes
0answers
206 views

Can the sigmoid function approximate any function (or relation) where 0<y<1

I'm studying Machine Learning and Artificial Neural Networks. Some basic principles of Machine Learning are linear regression, multivariate linear regression, and nonlinear regression. The last of ...
3
votes
0answers
174 views

Multilinear or Tensor Regression?

Given input data $x_t\in \mathbb{R}^n$ and output data $y_t\in\mathbb{R}^m$, the closed form solution to $\min_A \sum_t \|y_t - Ax_t\|^2_2$ is given by $A = (XX^T)^{-1}XY^T$ where $x_t$ form the ...
3
votes
0answers
156 views

Minimizing L4/ L6/ L2N norm for linear regression

OLS regression minimizes the sum of the squared errors. The normal equation for an OLS for $L_2$ minimization is as follows: $$b= (A'A)^{-1}A'y$$ What would be the equation to minimize the $L_4$ norm ...
3
votes
1answer
245 views

Projection Pursuit Regression

This is with reference to projection pursuit regressions. I kind of get the idea behind approximating a continuous function using weighted sums of ridge functions. I am not sure why ridge functions ...
3
votes
0answers
162 views

How to perform nonlinear regression with correlated errors?

I have a nonlinear least squares problem, but the errors are correlated. I could use R's nls function to do the regression if the errors were independent, but I don't know the right way to handle ...
3
votes
0answers
307 views

Least Square Method with Positive Parameters

this is my first post here in the Stack Exchange. A friend told me about this forum and I'm giving it a try. I searched a bit past threads, but couldn't find what I wanted, so I'm posting the problem ...
3
votes
0answers
1k views

Correlation and Regression Question

Two separate tests are designed to measure a students ability to solve problems. Several students are randomly selected to take both tests and the results are: $$ \begin{matrix} \text{Test A}(x) ...
3
votes
1answer
137 views

Bound of linear regression's object function

Randomly uniformly select $n$ numbers from a set $\{1,2,...,U\}$ with/without replacement, $y_i$ is the $i$th number selected, and $x_i$ is the rank of $y_i$ in the $n$ numbers. The rank is the order ...
2
votes
1answer
678 views

Polynomial fitting - how to fit and what is _polynomial fitting_

I don't understand what is polynomial fitting. Can anyone explain me how to fit a curve to given points?
2
votes
3answers
188 views

least squares regression in 3space

robjohn is giving me a hand with this, but in case anybody else knows... I need to do a least-squares regression for linearity on a set of coordinates in 3space. If the dataset is linear, I need to ...
2
votes
4answers
564 views

Fitting curve for exponential: $y = A - B\mathrm{e}^{-t/\tau}$

I have some data that follows a saturation or charging profile such as $y = A - B\mathrm{e}^{-t/\tau}$. To begin with, is there a proper name for this function? I have seen it many times, including: ...
2
votes
2answers
297 views

Closed form for coefficients in Multiple Regression model

I want to find $\hat{\beta}$ in ordinary least squares s.t. $\hat{Y} = \hat{\beta}_0 + \hat{\beta}_1 X_1 + \cdots + \hat{\beta}_n X_n $. I know the way to do this is through the normal equation using ...
2
votes
1answer
563 views

Intuition and the math behind normalization

What exactly is the purpose of normalization. From what I read, it is to adjust two different sets of values so you can compare them, but I don't understand why, nor the math behind it. Could anyone ...
2
votes
3answers
7k views

Fitting exponential curve to data

If I have a collection of data points that follow an exponential curve relationship, how can I manually construct the equation that defines the best-fit exponential curve for the data?
2
votes
2answers
299 views

design matrices

Given a linear model $Y = X\beta + \epsilon$ with three treatments and six subjects where $X$ is the design matrix, suppose $X = \begin{matrix}1 & 1 & 0\\ 1 & 1 & 0\\ 1 & 0 ...
2
votes
1answer
708 views

Linear Regression with 3x3 Matrices

Here's my Homework Problem: We can generalize the least squares method to other polynomial curves. To find the quadratic equation $y=a x^2+b x+c$ that best fits the points $(-1, −3)$, $(0, 0)$, ...
2
votes
1answer
268 views

How do I do a least squares fit of $a x + b y = 1$?

How do I do a least squares fit of the line equation $a x + b y = 1$, so that the points are as close to the line as possible? (Not just vertically close) If I use the matrices $$X = ...
2
votes
2answers
120 views

Linear Models - Regression Analysis

As a student learning Applied Regression Analysis, I come from a background with very little information about this topic. I understand that given $y = \beta_0 + \beta_1x_1 + \epsilon$ $E(y\mid x) = ...
2
votes
1answer
819 views

What is $Cov(\hat{Y},Y)$?

If $\hat{Y}$ is the OLS linear regression model for $Y$, what can I say about $Cov(\hat{Y},Y)$? Is this value 0?
2
votes
1answer
1k views

Hat Matrix Identities in Regression

I need to show that $\bar h= \sum{h_{ii}/n} = \operatorname{Tr}[H]/n = (p+1)/n$ Using the fact that $\operatorname{Tr}[AB]=\operatorname{Tr}[BA]$ and $H=X(X^TX)^{-1}X^T$. But I have no idea how to ...
2
votes
1answer
3k views

Construct / find the simplest function based on data

Let's say I have these 7 natural numbers (all between 0 and 255): 255, 23, 45, 32, 87, 52, 146 How can I find a function F(x) that, once computed, gives me back ...
2
votes
3answers
381 views

Residuals of regression model

Let's suppose I do a regression between earnings and age (and suppose I do not know the distribution of earnings). Would it be possible for the residuals to be normally distributed? I am thinking it ...
2
votes
2answers
443 views

Calculate Line Of Best Fit Using Exponential Weighting?

I know how to calculate a line of best fit with a set of data. I want to be able to exponentially weight the data that is more recent so that the more recent data has a greater effect on the line. ...
2
votes
1answer
203 views

How to handle constant term in Least Squares Regression?

In the well known matrix form of a least squares regression where I am trying to solve for B in Y = B1X1 + B2X2 + B3 I might be given X and Y sample data as something like $X$ = $\begin{bmatrix} ...
2
votes
1answer
39 views

Calculate trend and represent in text?

First off, I'm terrible at math. I'm writing a script that monitors transactions from clients daily over a 7 day period. Given a set of numbers like below, I would like to calculate a trend and ...
2
votes
1answer
36 views

Question on regression

So I've been given this formula For regression $R^2=1 - \sum \frac{{(y_i - \hat{y}_i)}^2}{(y_1-\bar{y})^2}$ Now an obvious question that has come to me is why $R^2$ stays the same in certain ...
2
votes
1answer
38 views

How to take the derivative of Matrices

I was browsing the derivation of the Least Squares estimates and stumbled about this problem. It said that: $$E = (Y + XB)^2$$ $$\frac{dE}{dB} = -X^TY + X^TXB$$ It is to my understanding that the ...
2
votes
1answer
172 views

Isn't the Hat Matrix just an identity matrix?

In Linear regression $y = X\beta + \epsilon$ The Hat matrix is defined to be $H = X(X^TX)^{-1} X^T$ . However. If I compute the equation for Hat matrix, I just get an identity matrix. My calculation ...