Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

learn more… | top users | synonyms

4
votes
1answer
70 views

Weighted least squares with angular data

Suppose I have a system whose state is $\Theta=(\theta_1,\theta_2,\ldots,\theta_n)$, where $\theta_i\in[-\pi,\pi)$ (i.e., they are angles). I'd like to determine the most likely estimate of $\Theta$ ...
4
votes
0answers
74 views

Bayesian linear regression cost function

I am studying classification using linear regression . Now, I want to map it in Bayesian regression. Let talk about binary classification using linear regression again. Assume that I have a set ...
4
votes
1answer
462 views

Determine whether ARMA(p,q) is stationary and/or invertible?

Determine whether an ARMA(p,q) process is stationary and invertible such that $y_t = \sum_{i=1}^{p} \phi_i y_{t-i} + \sum_{i=1}^{p} \theta_{i} \epsilon_{t-i}$ with the restriction that ...
4
votes
0answers
263 views

Multilinear or Tensor Regression?

Given input data $x_t\in \mathbb{R}^n$ and output data $y_t\in\mathbb{R}^m$, the closed form solution to $\min_A \sum_t \|y_t - Ax_t\|^2_2$ is given by $A = (XX^T)^{-1}XY^T$ where $x_t$ form the ...
4
votes
1answer
43 views

Regression model for a shearing process

30 Widgets are randomly assigned to a shearing process. There are 3 such processes, each getting 10 widgets. The lengths of each widget are recorded before undergoing the shearing. The amount that ...
3
votes
1answer
3k views

Why can we assume that the expected value of the error term is zero? [closed]

Why can we assume that the expected value of the error term in a linear regression model is zero? This is with regard to a simple linear regression.
3
votes
4answers
844 views

Fitting curve for exponential: $y = A - B\mathrm{e}^{-t/\tau}$

I have some data that follows a saturation or charging profile such as $y = A - B\mathrm{e}^{-t/\tau}$. To begin with, is there a proper name for this function? I have seen it many times, including: ...
3
votes
2answers
157 views

How do I use specific data points on a graph to determine an equation?

I need to find a function $f(x)$ such that the following data points would fit on it: $$f(1) = 0 \\ f(2) = 0.5 \\ f(4) = 1.0 \\ f(8) = 1.5 \\ \cdots $$ and so on. So the pattern is every time $x$ ...
3
votes
3answers
690 views

Correlation between variables

I asked this question on stats SE but did not find a suitable answer so far. Maybe someone can help. Given n random variables x1,...,xn (one-dimensional). The following is known (corr() = Pearson ...
3
votes
1answer
2k views

Analytic solution for matrix factorization using alternating least squares

The standard form for ridge regression aims to minimize the following cost function. $$ \min\ \ \sum_i(y_i-x_i^T\beta)^2 + \lambda\sum_j\beta^2_j $$ As described here, it's possible to differentiate ...
3
votes
2answers
3k views

Examples of logistic regression and multinomial logistic regression

I've been studying to understand the concept of logistic regression and I think I understand the idea more or less, but there are still some gaps to fill. What I'm looking for is an example of ...
3
votes
3answers
11k views

Fitting exponential curve to data

If I have a collection of data points that follow an exponential curve relationship, how can I manually construct the equation that defines the best-fit exponential curve for the data?
3
votes
1answer
641 views

QR factorization for ridge regression

I am solving an overdetermined system of equations: $$Ax= b$$ Using QR factorization, we can solve this system easily by posing it as: $$Rx= Q'b$$ I would like to regularize my estimate of $x$. I ...
3
votes
1answer
4k views

Construct / find the simplest function based on data

Let's say I have these 7 natural numbers (all between 0 and 255): 255, 23, 45, 32, 87, 52, 146 How can I find a function F(x) that, once computed, gives me back ...
3
votes
1answer
70 views

Calculate trend and represent in text?

First off, I'm terrible at math. I'm writing a script that monitors transactions from clients daily over a 7 day period. Given a set of numbers like below, I would like to calculate a trend and ...
3
votes
1answer
784 views

Find x,y & z (xyz+xyz=zyx)

I saw this problem the other day at work and found it pretty interesting: $$xyz + xyz = zyx$$ Find $x, y, z$ and the base(s) which this is true. Note that $x,y,z$ are simply digits concatenated, ...
3
votes
2answers
2k views

Assumption of a Random error term in a regression

In one of my recent statistics courses, our teacher introduced the linear regression model. The typical $y=\alpha + \beta X + \epsilon$, where $\epsilon$ is a "random" error term. The teacher then ...
3
votes
1answer
354 views

Isn't the Hat Matrix just an identity matrix?

In Linear regression $y = X\beta + \epsilon$ The Hat matrix is defined to be $H = X(X^TX)^{-1} X^T$ . However. If I compute the equation for Hat matrix, I just get an identity matrix. My calculation ...
3
votes
2answers
188 views

How many points to find a polynomial?

I would like to fit a formula $ax^b + cx^d+ e$ to a set of points. I have two questions. If my data were perfect, how many points do I need in the worst case to get $a,b,c,d,e$ exactly? If my data ...
3
votes
2answers
767 views

Binary Logistic Regression Model Processing

Thanks for showing interest and wanting to help out. My aim is to develop a model that - as accurately as possible - predicts how entities in a population will either cooperate or defect, as a % of ...
3
votes
2answers
749 views

Fitting a sine function to data

I have a sequence of $n$ points $(x_i,y_i)$, for $i=1,\dots,n$. I would like to find the function, of the form $y=V\sin(x+\phi)$, which best fits the points. Which numerical method could I use? I have ...
3
votes
1answer
224 views

Ridge Regression: $\hat{\beta} \rightarrow \beta$

I'm trying to find the probability limit of $$\hat{\beta} = \left( \sum x_i x'_i + \lambda I_k \right)^{-1} \left( \sum x_i y_i \right) $$ as $n \to \infty$, and $\lambda$ is some positive ...
3
votes
2answers
40 views

Distance between a plane and set of points

Suppose $m$ data points belonging to a class represented by matrix $A$. Therefore, the size of matrix $A$ is $m\times n$. In addition, suppose $w\cdot x + b=0$ be equation of a plane in ...
3
votes
1answer
37 views

Quantile Regression - Linear Loss Minimization

I'm currently reading Quantile Regression by Roger Koenker, and for some reason I'm having a lot of trouble deriving one of his equation (sect. 1.3, p. 5-6). He goes on to demonstrate that $\hat{x}$ ...
3
votes
2answers
42 views

Likelyhood function analysis

I've done some calculations on a large number of data, and created the following graph in excel representing the data: How do I go about analysing this regression in order to find the formula that ...
3
votes
1answer
167 views

Rates of convergence of an OLS estimator

I have a linear regression model $$ y_t=x_t\beta+e_t,\quad t=1,\ldots,N. $$ Here $x_t$ is non-random and given by $(1,\delta_t t)$ where $\delta_t$ is 1 for odd $t$ and $0$ otherwise. Moreover, ...
3
votes
1answer
152 views

Minimum required data for cosine fit

With a minimum amount of (noisy) data-points, I need to find the amplitude of a simple cosine y=A*cos(x), where x is an angle from 0:2pi. I know how to fit data to the function, and I know how to ...
3
votes
1answer
95 views

Solution to linear system of equations

Notation. Let $y$, $a$, and $b$ be $n\times 1$, $p\times 1$, and $q\times1$ real vectors. Let also $X$ and $Z$ be $n\times p$ and $n \times q$ real matrices. Suppose that there is no solution, $a$, ...
3
votes
1answer
130 views

Constraining estimated linear regression coefficients over several regressions

I'm trying to run a series of simultaneous linear regressions, and I want to constrain the regression coefficients. For the standard ordinary least squares regression, the specification of the ...
3
votes
1answer
78 views

Why is $\sum x^2 _t \times \text{Var}(\beta)=\frac{\sum x^2 _t \times \sigma^2}{ \sum x^2 _t} = \sigma^2$?

I do not get this connection. Is is reliable to divide this equation by $\sum x^2 _t$ to get just $\sigma^2$ ? $$\sum x^2 _t \times E(\hat \beta - \beta)^2=\sum x^2 _t \times ...
3
votes
2answers
144 views

Can I consider shooting% as an independent variable

First time poster in the math section (a few posts in the stats section) and I am looking for clarification on a variable query that I have. Basically I enjoy sports and enjoy putting a mathematical ...
3
votes
2answers
596 views

Confidence interval of a random variable for an ordinary linear regression

I have a small problem. With my limited stats background I am not sure I am getting this one right. After fitting an ordinary linear regression model I get ...
3
votes
1answer
73 views

Interpolation and mapping between scattered vectors in two unequally dimensioned spaces

Imagine two spaces: An ‘input’ space with dimension $m$. An ‘output’ space with dimension $n$. $m \geq n$ There are points in each of these spaces defined such that some characteristic is ...
3
votes
1answer
46 views

Detrending sine waves accurately

I am doing some data analysis where I look at electricity demand over the course of a day, but need to separate the intra-day (constant and periodic) components from daily changes (assumed linear). At ...
3
votes
0answers
31 views

Regression with many discrete and continuous predictors and few rows

I want to do regression on a dataset. It has one continuous dependent variable that I want to predict. It has many categorical and some continuous predictors. It only has a few rows. A simplified ...
3
votes
0answers
42 views

How to perform nonlinear regression with regressors affected by gaussian error?

I am trying to calibrate a sensor and I have a data set consisting of several observations of a 3-dimensional vector $X_i$, with $X_i=w_i + \epsilon_i$ where $w_i$ is the value that the sensor ...
3
votes
2answers
188 views

Regression with error coming from rounding

I am looking at the following model: $c$ is a fixed vector in $\mathbb{R}_+^n$ and for any $x \in \mathbb{R}_+^n$ we obtain a value $y =[c^Tx]$, i.e. rounding $c^Tx$ to the nearest integer. I want ...
3
votes
2answers
302 views

How to perform a monotonic function fitting of data points?

I'm seeking suggestions for general purpose function fitting of a set of data points, where, based on physical intuition, the relationship is expected to be "monotonic", i.e. the function should be ...
3
votes
1answer
292 views

Expected in-sample error of linear regression with respect to a dataset D

In my textbook, there is a statement mentioned on the topic of linear regression/machine learning, and a question, which is simply quoted as, Consider a noisy target, $ y = (w^{*})^T \textbf{x} + ...
3
votes
1answer
36 views

Logarithmic Functions algebra question

This is my first post and I honestly just want a second opinion on my answer to a question I got incorrect on an exam before I go arguing over it with my professor. Basically, is this mathematically ...
3
votes
0answers
375 views

Can the sigmoid function approximate any function (or relation) where 0<y<1

I'm studying Machine Learning and Artificial Neural Networks. Some basic principles of Machine Learning are linear regression, multivariate linear regression, and nonlinear regression. The last of ...
3
votes
0answers
857 views

What is the Moore-Penrose pseudoinverse for scaled linear regression?

The matrix equation for linear regression is: $$ \vec{y} = X\vec{\beta}+\vec{\epsilon} $$ The Least Square Error solution of this forms the normal equations: $$ ({\bf{X}}^T \bf{X}) \vec{\beta}= ...
3
votes
0answers
215 views

Minimizing L4/ L6/ L2N norm for linear regression

OLS regression minimizes the sum of the squared errors. The normal equation for an OLS for $L_2$ minimization is as follows: $$b= (A'A)^{-1}A'y$$ What would be the equation to minimize the $L_4$ norm ...
3
votes
1answer
305 views

Projection Pursuit Regression

This is with reference to projection pursuit regressions. I kind of get the idea behind approximating a continuous function using weighted sums of ridge functions. I am not sure why ridge functions ...
3
votes
0answers
183 views

How to perform nonlinear regression with correlated errors?

I have a nonlinear least squares problem, but the errors are correlated. I could use R's nls function to do the regression if the errors were independent, but I don't know the right way to handle ...
3
votes
0answers
336 views

Least Square Method with Positive Parameters

this is my first post here in the Stack Exchange. A friend told me about this forum and I'm giving it a try. I searched a bit past threads, but couldn't find what I wanted, so I'm posting the problem ...
3
votes
1answer
138 views

Bound of linear regression's object function

Randomly uniformly select $n$ numbers from a set $\{1,2,...,U\}$ with/without replacement, $y_i$ is the $i$th number selected, and $x_i$ is the rank of $y_i$ in the $n$ numbers. The rank is the order ...
2
votes
1answer
748 views

Polynomial fitting - how to fit and what is _polynomial fitting_

I don't understand what is polynomial fitting. Can anyone explain me how to fit a curve to given points?
2
votes
3answers
195 views

least squares regression in 3space

robjohn is giving me a hand with this, but in case anybody else knows... I need to do a least-squares regression for linearity on a set of coordinates in 3space. If the dataset is linear, I need to ...
2
votes
3answers
86 views

Proof of Gauss-Markov theorem

Theorem: Let $Y=X\beta+\varepsilon$ where $$Y\in\mathcal M_{n\times 1}(\mathbb R),$$ $$X\in \mathcal M_{n\times p}(\mathbb R),$$ $$\beta\in\mathcal M_{n\times 1}(\mathbb R ),$$ and ...