Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

learn more… | top users | synonyms

0
votes
0answers
9 views

LASSO with equivalent quadratic costs

Is there any fundamental difference between the solutions obtained by minimizing following LASSO cost functions, if any? ( $A_{N \times n }$ and $ N >> n$) $ J=\Vert y-Ax \Vert_{2}^{2} + \...
2
votes
5answers
92 views
+50

Why work with squares of error in regression analysis?

In regression analysis one finds a line that fits best by minimizing the sum of squared errors. But why squared errors? Why not use the absolute value of the error? It seems to me that with squared ...
1
vote
2answers
8k views

Unconditional expectation vs conditional expectation in regressions - does it really matter?

I refer here to a simple linear regression whose true representation is given by the equation: $y_i=x_i'\beta+u_i$, where as usual $x_i$ is a $K\times1$ vector of independent explanatory variables, $\...
1
vote
0answers
12 views

Nonlinear Multivariate Regression

Assuming I know exactly my forward model, which is represented by $n$ non-linear functions, or some probability models: $\vec{R}=f(x,y,z)$ , $f:\mathbb{R}^3\to\mathbb{R}^n$ Where each item in $R_i$ ...
3
votes
1answer
165 views

Minimum required data for cosine fit

With a minimum amount of (noisy) data-points, I need to find the amplitude of a simple cosine $y=A*cos(x)$, where x is an angle from $0$:$2\pi$. I know how to fit data to the function, and I know ...
1
vote
1answer
24 views

How to find an equation that describes a non-linear correlation of multiple parameters

I tried to search for this problem but I don't know what exactly I'm looking for. I found some empirical parameters that correlate but not linearly. An example follows: $Y_1$ and $Y_2$ are the values ...
0
votes
1answer
1k views

Derivative of logistic loss function

I am using logistic in classification task. The task equivalents with find $\omega, b$ to minimize loss function: That means we will take derivative of L with respect to $\omega$ and $b$ (assume y ...
0
votes
0answers
18 views

Exponential curve fit with MATLAB's fit function does not deliver good fit

I am trying to use MATLAB's fit function to fit a curve through a data set which obviously shows an exponential decay. These are the commands I use: ...
0
votes
1answer
22 views

“Best” solution to incompatible system of linear equations

I'm comparing data to a theory I've developed and right now I have to do some parameter fitting. Say I have two unknown parameters $x$ and $y$ such that $a_{1}x+b_{1}y=c_{1}$, $a_{2}x+b_{2}y=c_{2}$, $...
1
vote
2answers
35 views

Regression without linearity [closed]

Given two independent, standard-normally distributed random variables $x,y\sim \mathcal{N}(0,1).$ I would like to do an univariate linear regression without intercept $Y = X \cdot \beta + \epsilon.$ R ...
0
votes
0answers
23 views

Solving position vector/constants given the acceleration vector is a function of the velocity vector

Recently I've been working on a problem and it's consuming all my free time. I've cobbled together a small model that takes into account lift and drag and now I'm trying to fit it to real data. The ...
1
vote
0answers
12 views

The relationship between simultaneous equations model and seemingly unrelated regression model?

recently I try to solve a equations-system. So after read few pieces of paper, I want to use SUR model. Based on what I read, those paper notes that usually equations-system has two method to be ...
1
vote
0answers
24 views

Probability distribution database

Is there some kind of probability distribution database? Very often I'm faced with a problem of fitting a distribution to data. And it also happens very often that exponential-looking data doesn't fit ...
1
vote
1answer
32 views

Why is the regression line an estimate of the average value of y for each value of x?

The regression line, passing through the point of averages with a slope equivalent to r, is said to be a good estimate of the average value of y for each value of x. I can see why this is the cases ...
0
votes
0answers
7 views

What is the relationship between regression line and graph of averages?

Why is the regression line regarded as a smoothed version of the line of averages, and why is it that when the graph of averages falls on a straight line, that line is also the regression line?
1
vote
1answer
40 views

Please show that $f(\beta_0,\beta_1)=\log(1+\operatorname{exp}(-y_1(\beta_0+\beta_1 x_1)))+\log(1+\operatorname{exp}(-y_2(\beta_0+\beta_1 x_2)))$

I would like to show that the following result is indeed true. I am very new with this subject, so I ask for a hint to get me started please. Please show that $f(\beta_0,\beta_1)=\log(1+\...
0
votes
0answers
26 views

choose between L1 and L2 normalization in logistic regression regularization

Wondering what are the pros and cons comparing to L1 and L2 normalization in logistic regression regularization part, For example, in below formula, it is use L2 normalization (in squared form of ...
0
votes
1answer
21 views

This is a question about the best type of regression analysis to use in my software

Let me start by saying that I am not a mathematician and I am not very good at math. I am mainly interested in obtaining the best possible results. I am currently doing trial and error with my ...
2
votes
1answer
29 views

Local quadratic approximation

I wanted to implement some penalized regression parameter estimation algorithm by Fan&Li (http://sites.stat.psu.edu/~rli/research/penlike.pdf, section 3.3, [1]), but cannot catch the idea of some ...
1
vote
2answers
38 views

Force minimum of quadratic fit to certain data point

I want to fit some data $(x_i, y_i)$ with quadratic function. No problem till there. However, I want the polynomium minimum of the fitted curve to be at certain point $(x_k, y_k)$. If it is possible, ...
1
vote
1answer
22 views

Average Percent Rate of Change

Excuse the png equations, still a MathJax newbie. I am analyzing data I have computed: Alcohol content and Caffeine content retention after a duration of 8 hours for each. I had gotten the data in ...
1
vote
0answers
25 views

Linear regression from function

I have a function given by its analytical form $f(x,y,z)=0$. Is is possible to calculate linear regression directly from this function on a selected neighborhood of a point? I want to "approximate" ...
0
votes
0answers
51 views

Least square method

$4$ arbitrary points $(x_1,y_1)$ $(x_2,y_2)$ $(x_3,y_3)$ $(x_4,y_4)$ are given in the $xy$ plane using the method of least squares. If regression of $y$ upon $x$ give the fitted line $y=ax+b$ , and ...
3
votes
2answers
418 views

How to perform a monotonic function fitting of data points?

I'm seeking suggestions for general purpose function fitting of a set of data points, where, based on physical intuition, the relationship is expected to be "monotonic", i.e. the function should be ...
0
votes
1answer
428 views

Fast way of finding RSS of Multiple Linear Regression

Is there any smarter way to compute Residual Sum of Squares(RSS) in Multiple Linear Regression other then fitting the model -> find coefficients -> find fitted values -> find residuals -> find norm of ...
0
votes
0answers
18 views

Linear regression formula derivation

I always thought linear regression was numerically calculated by bruteforcing solutions and comparing them with some $r^2$ error accuracy. However, I recently stumbled upon this formula for ...
3
votes
2answers
28 views

intuition behind having a unique regression line

I understand this mathematically. we have function of 2 variables represents the sum of square errors. We have to find the $a$ and $b$ that minimize the function. there is only one minimum point. But ...
4
votes
1answer
505 views

Determine whether ARMA(p,q) is stationary and/or invertible?

Determine whether an ARMA(p,q) process is stationary and invertible such that $y_t = \sum_{i=1}^{p} \phi_i y_{t-i} + \sum_{i=1}^{p} \theta_{i} \epsilon_{t-i}$ with the restriction that $\theta_{0} = ...
2
votes
0answers
24 views

What is the relationship between the function $\mathbb{E}(Y \mid X = x)$ and linear regression?

Consider the function $$ r(x) = \mathbb{E}(Y \mid X = x) $$ This has been called the regression function in a textbook I'm using. I'm trying to figure out the relationship between this function ...
0
votes
1answer
30 views

Cubic Polynomial fitting with defined ranges for coefficients

Is there a way, given a set of values $(x,y)$, to find a cubic polynomial $f(x)$ that fits the values? My cubic polynomial is defined as $c_0 + c_1x +\frac {1}{2} c_2 x^2 +\frac {1}{6}c_3 x^3$ ...
2
votes
0answers
13 views

Derive the Hat Matrix to map actual response to estimated resposne

In order to measure the quality of a regression we can calculate the Hat Matrix. Using it we can estimate the response variable as if we used the predictor variables to regress them. For linear ...
0
votes
0answers
20 views

MLE of heteroscedastic model?

Given the regression model where our and are identically and independently distributed. I'm trying to find the MLE B-hat and the unbiased estimator sigma-hat^2. I haven't dealt with any models in ...
0
votes
1answer
33 views

Showing sum of squared residuals is zero?

I have the model $$y_i = B_0+\sum\limits_{i=0}^pB_kX_{ik} + e_i$$ I'm looking to show the sum of squared residuals is zero if $p = (n-1)$. I have tried expanding it quite in depth and I haven't been ...
1
vote
1answer
39 views

Showing Residual Sum of Squares for Multiple Linear Regression is 0

Problem: I have the linear regression model: $y_i=\beta_0+\sum_{k=1}^p \beta_kx_{ik}+\epsilon_i$ where $\epsilon_i\sim N(0,\sigma^2)$, for $i = 1,2,\ldots ,n$. I want to prove that the residual sum ...
3
votes
2answers
2k views

Assumption of a Random error term in a regression

In one of my recent statistics courses, our teacher introduced the linear regression model. The typical $y=\alpha + \beta X + \epsilon$, where $\epsilon$ is a "random" error term. The teacher then ...
1
vote
0answers
15 views

Name of the field of study that details extrapolation of a series based on subset sample data

Apologies if my notation and/or terminology is way off - I'm not well versed in mathematics. I'm looking for the name of the field of mathematics that might help me solve my problem. Here's my problem:...
1
vote
1answer
26 views

Derivative of dot product of Residual Sum Square in matrix notation

I am trying to derive the following expression w.r.t. $\beta$: \begin{equation} RSS(\beta) = (\mathbf{y} - \mathbf{X} \beta)^T (\mathbf{y} - \mathbf{X} \beta) \end{equation} I know that the ...
3
votes
0answers
77 views

Two dimensional (discrete) orthogonal polynomials for regression

This question How to work out orthogonal polynomials for regression model and the answer http://math.stackexchange.com/a/354807/51020 explain how to build orthogonal polynomials for regression. ...
0
votes
1answer
20 views

error term in time-series Seasonal AR model

I am reading a paper related to timeseries forecasting in which I have a question regarding the seasonal AR model described in equation (1.2) namely: $log(y_t)$~$log(y_{t-1}) + log(y_{t-12}) + x^{(1)}...
1
vote
1answer
42 views

Can I ignore multicolinearity problem if all the regression coefficients are highly significant?

Can I ignore multicolinearity problem if all the regression coefficients are highly significant? My data is large enough and all the resulting coefficients are significant enough in less than 0.01 ...
1
vote
0answers
17 views

Confused about solution to the piecewise constant regression model

I am confused about the solution to the following solution to fitting piecewise constants: Specifically, are we minimising the sum of squares, that is, finding the vector $\beta = (\beta_1,\beta_2, ...
0
votes
0answers
19 views

Is there a way to determine the most most orthagonal variable or the “most powerful variable” in a Logistical Regression in the statstical software R?

I am curently working on a Logistical (Binary) Regression, and I am using R to create ROC curves based on the data. I cannot seem to determine exactly how I can determine variables to change to change ...
0
votes
1answer
34 views

How do I find first two steps of Newton's method?

How to find given by the points $(x_0,y_0)=(0.8,2.1)$ first two steps of newton method ,in order to approximate for $f(x,y)=x^3+14x+x^2y^2-5y$ one result of system of equation $\nabla f(x,y)=(0,0)$ ?
0
votes
0answers
15 views

How to get best fitting model decision for data X and Y in e.g. Matlab?

I have two sources of data, X and Y, which are basically counts, from 23 individual origins (3D ROIs in my case). For example: ...
0
votes
1answer
564 views

Convert odds ratio based on unit change to several unit changes

Imagine to have two groups of people, the first one more strongly exposed to a pollutant than the second one, and the first one developing a certain disease more often. Having measurements of the ...
2
votes
2answers
929 views

connection between PCA and linear regression

Is there a formal link between linear regression and PCA? The goal of PCA is to decompose a matrix into a linear combination of variables that contain most of the information in the matrix. Suppose ...
0
votes
0answers
23 views

Software to run non-linear regression

We currently use a very old version of StatGraphics and unfortunately it doesn't run on x64 systems. So I am looking for open-source software that may do the same thing as a replacement. Namely we run ...
0
votes
1answer
27 views

coefficient of determination: absence of cross products [closed]

With regard to the coefficient of determination, why is the total variation equal to the sum of the explained variation and the unexplained variation and there are no cross-products?
1
vote
0answers
103 views

showing SSE of simple regression model is larger than or equal to SSE of multiple regression model

Lets say we have 2 linear regression models: $y_i = B_0 + B_1x_{i1} + \epsilon_i,$ where $\epsilon_i$ follows $N(0,σ_1^2)$ $y_i = B_0 + B_1x_{i1} + B_2x_{i2} + \lambda_i,$ where $ \lambda_i$ follows ...
1
vote
0answers
14 views

Numerical Method for fitting parameters of an explicit integration to actual data

I have a heat transfer system described by, $$\{\dot{T}\} = [C^{-1}]\left([K]\{T\} + \{F\} \right)$$ where ${T}$ is a vector of the nodal temperatures of the system. From initial conditions I am able ...