Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [regression]

Questions on (linear or nonlinear) regression, the fitting of functions that best approximate empirical data.

0
votes
0answers
18 views

Mathematical derivation of why Bagging reduces variance

I am having a problem understanding the following math in derivation that bagging reduces variance. The math is shown but can not work it out as some steps is missing. link
-1
votes
0answers
25 views

Fitting a sinusoid through three arbitrary points

Let's say there are three arbitrary points on the x-y plane. Does there always exist a function of the form y = A.sin(Bx+C) that satisfies all the three points for real A, B and C? P.S. No pair among ...
0
votes
2answers
18 views

Linear Regression Assumption

the following website states that linear regression assumes a linear relationship between dependent and independent variables: "First, linear regression needs the relationship between the independent ...
1
vote
2answers
31 views

Multivariate Quadratic Regression

I would like to make a polynomial regression, but for multivariate input data. In the univariate case, one can write polynomial regression as a multivariate linear regression problem and can come up ...
0
votes
2answers
27 views

Necessity of standardizing data in regularized regression.

It is well known that in Ridge or LASSO regression we add a regularization term to penalize large regression coefficients. What if the true relationship between the response and covariates relies on a ...
0
votes
1answer
20 views

Proof of Batch Gradient Descent's cost function gradient vector

In the book Hands-On Machine Learning with Scikit-Learn & TensorFlow, the author only showed the formula for the Batch Gradient Descent method, such as: $ \dfrac{\partial}{\partial \theta_{j}} ...
0
votes
0answers
12 views

How to select the important feasures from lasso model?

I used different initial value and applied the interactive method to fit the lasso model (1000 sample size, 5000 features), but the non-zero coefficients are different every time I changed the initial ...
0
votes
0answers
54 views

Expectation of the Error

Suppose $X_1,\ldots,X_n, Y_1,\ldots, Y_n,U_1,U_2,V_1,V_2$ are i.i.d. uniformly random variables in $[-1,1]$. Let $M_1 = \frac{V_2-V_1}{U_2-U_1}$ and $B_1 = V_1-U_1M_1$. Let $M_2,B_2$ be the ...
1
vote
0answers
16 views

How to find the correspondence between ridge coefficients to modified ridge?

The ridge coefficients minimize a penalised residual sum of squares: $\hat{w}_{ridge}= \arg\min(\sum_{i=1}^{N}(y_{i}-w_{0}-\sum_{j=1}^{d}x_{ij}w_{j})^{2}+\lambda\sum_{j=1}^{d}w_{j}^{2}),\,\lambda\...
0
votes
0answers
14 views

Multiple Regression Assumption

If I am running a multiple linear regression model with six independent variables against dependent variable, do the assumptions of multiple regression need to be satisfied? or does this only applies ...
0
votes
2answers
30 views

Formulating the polynomial regression

I'm trying to formulate a regression problem such that $y=ax^{b}$. Previously, I formulated the $y=ax+b$ like $y=Ac+e$ where $c= \begin{bmatrix} a\\ b \end{bmatrix}$ and $A= \begin{bmatrix} x_1 &...
1
vote
1answer
36 views

Finding the M.L.E estimates of weight maximizing the Likelihood Function of a Linear Regression

I am reading this book "Pattern Recognition and Machine Learning " - Christopher M. Bishop . My question is around equation $3.13$ page $141$.In the book , he talks of Maximum likelihood estimate of ...
1
vote
0answers
26 views

How to do a sensitivity analysis on a non-linear equation?

In the company, it is very difficult to actually do quotations for our customers properly because we do not have perfect information regarding the factors that affect the cost and profit. So I created ...
0
votes
0answers
16 views

Name to specific fit

I have been looking for the name of a specific fit. I was in need of a function that minimized the error (L1) between the function and the observed values but the function must be greater than or ...
1
vote
0answers
15 views

How to calculate $\text{cov}(\hat{Y}_{ij}, \hat{Y}_{kj})$ if $Y_{ij} = \mu + a_i + b_j + e_{ij}$?

Let's assume we have the model following Two-Factor model without replications : $$Y_{ij} = \mu + a_i + b_j + e_{ij}, \; i=1,\dots,p \; \text{and} \; j=1,\dots, q $$ I am interested in calculating ...
0
votes
2answers
19 views

Joint Linear Regression hypothesis test

Suppose I have the following joint model: $$ y_{f_i} = \beta_1 + \beta_2 t_i + \beta_3 s_i +\epsilon_i $$ $$ y_{m_i} = \beta_4 + \beta_5t_i + \beta_6 s_i +\epsilon_i $$ where $y_{f_i}$ corresponds ...
0
votes
1answer
27 views

How to find quadratic regressions by hand

Currently I am working on an assignment for which I have to calculate the quadratic regression and linear regression (I know how to do this one) of some data points by hand. Nonetheless, I do not know ...
1
vote
1answer
34 views

Existence and uniqueness of least square fits

I just noticed that my pythons scipy library gave me another solution to an exponential curve fit than my pocket calculator. If you plot both solutions both match the data (visually) very well. This ...
0
votes
0answers
16 views

Product of two conditional distributions in Bayesian modelling

I am currently reading an introduction to Bayesian Modelling. Im trying to understand why the following is true: $$p(w|x,t,\alpha,\beta)\propto p(t|x,w,\beta)p(w|\alpha)$$ where: $w$ is are the ...
0
votes
1answer
14 views

Interpreting linear regression coefficients for a covariate that's correlated with other covariates

The interpretation of linear regression coefficients that I learned is that the coefficient is the change in outcome associated with a unit change in that covariate, assuming all other covariates stay ...
0
votes
1answer
20 views

Estimate the mean of the independent variable (regression)

If I have the simple linear model: $y=\alpha+\beta x+\epsilon$, and I know that $\hat{\alpha}= \hat{\overline{y}}-\hat{\beta}\hat{\overline{x}}$. Then my estimate og the mean is actually: $\hat{\...
0
votes
0answers
16 views

What can be used to measure the fitness of a curve to a set of data?

I'm essentially looking for a style of polynomial regression, but in an ideal world would be able to handle sinusoidal functions as well. I know my TI-84 can do cubic and quartic regression, but I'm ...
1
vote
2answers
27 views

In a normal linear model (with intercept), show that if the residuals satisfy $e_i = a + \beta x_i$, then each residual is equal to zero.

In a normal linear model (with intercept), show that if the residuals satisfy $e_i = a + \beta x_i$, for $i = 1\dots n$, where $x$ is a predictor in the model, then each residual is equal to zero. I'...
0
votes
2answers
23 views

Show that $\hat{\beta} - \hat{\beta}_{(i)} = \frac{(X^TX)^{-1}\vec{x}_{i}^{T}(y_i-\hat{y}_i)}{1-h_{ii}}$

$$\hat{\beta} - \hat{\beta}_{(i)} = \frac{(X^TX)^{-1}\vec{x}_{i}^{T}(y_i-\hat{y}_i)}{1-h_{ii}}$$ where $h_{ii} = \vec{x}_{i}^T (X^TX)^{-1} \vec{x}_{i}^{T}$ and $\hat{\beta}_i$ is the estimator of $\...
1
vote
0answers
29 views

Any evaluatory cheap polynomial regression alternatives for bounded input?

Background: I was trying to approximate $\ f(x) =1/2^x,(f(x) <10^{-38}):127>x>0 \\$. The polynomial had to be of relatively low degree such that when evaluated using horner's method to be ...
0
votes
0answers
26 views

Probability of minimum of distance between random points

I am reading "A Distribution-Free Theory of Nonparametric Regression" by Laszlo Gyorfi et al. and I have problem with one exercise. Let $X, X_1, \dots X_n$ be independent and uniformly distributed ...
0
votes
0answers
21 views

How do I fit multiple curves with same fitting parameters?

I have three different curves for three different temperatures. The expression I'm trying to fit the data to looks like this: $E_b=E_0(1-\tau/\tau_0)^{\alpha} (1-T/T_m)$. So I'm trying to fit the ...
0
votes
0answers
7 views

Prediction of time series based on lagged correlations

I have several questions. I will split the text up in one high-level description of the goal of my exercise, a detailed description of my potential solution and finally my actual questions. Please ...
0
votes
2answers
31 views

Proof by induction of a regression formula (series)

As part of the curriculum, we have recently been working on proofs by induction. During the exercises I encountered an exercise that I do not know how to solve, I would be happy to help. Sorry ...
1
vote
1answer
13 views

Transforming variable to change a variable so you could use it in a model?

If you had a variable where the functional form was incorrect and unknown (not squared, or exponential, or square root, etc.), what would be one way to “fix” the variable so you could use it in ...
1
vote
1answer
40 views

Ridge regression objective function gradient

Considering ridge regression problem with given objective function as: $$f (W) = \|XW - Y\|_F^2 + \lambda \|W\|_F^2$$ Having convex and twice differentiable function results into: $$\nabla f (W) = ...
1
vote
1answer
41 views

Stupid question but unsure on Regression.

When I want to do a regression between daily change of currency prices and a daily stock index changes which one shall I use as the dependent and independent variable. Thanks for any help.
0
votes
0answers
14 views

Can a Bayesian regression problem be regarded two sub consecutive regression?

Assume I have $n$ observations for y and X, which leads to the following regression, $y=X*\beta+e, e\sim N(0,\sigma^2)$ Assume priors: $\sigma^2\sim \frac{\nu_0 s_0^2}{\chi^2_{\nu_0}}$ and $\beta\...
0
votes
0answers
16 views

Formula for ordinal complementary log-log regression

I have a model developed in R using the polr() function in the MASS package. The model is an ordinal regression with a complementary log-log link (method=cloglog). The model uses four predictors ...
0
votes
0answers
21 views

Regression through linear Fourier coefficient fitting?

Basically suppose on was given an unknown function/data and expected to write a function so that $Y=f(X)$, this can be done by linear regression in simple cases very easily. However, suppose that the ...
1
vote
1answer
22 views

Regression model + expected value, variance and autocorrelation of the error term

Consider this regression model $$Y_t=X_t\beta+\epsilon_t, ~~~~~~~~~~\epsilon_t \sim WN(0, \sigma^2_{\epsilon})$$ with 3 different specifications of the error term: $\epsilon_t=\alpha_1\epsilon_{t-1}+...
1
vote
1answer
34 views

Linear regression - find $b$, $s^2$, $R^2$

Suppose you want to fit the model $Y=\alpha+\beta X+\epsilon$ but you don't have the full data set $\left[\begin{matrix}y&X\\\end{matrix}\right]=C$. Instead you only have: $$C'C=\begin{bmatrix} ...
0
votes
0answers
26 views

Expectation of random variable with omitted variable bias

I am given: $y_i = \beta_0 + \beta_1range_i + \beta_2dist_i + \epsilon_i$ $E[\epsilon_i | range_i, dist_i]=0$ Then in the case where $dist_i$ is omitted: $y_i = \beta_0 + \beta_1range_i + \eta_i$ ...
1
vote
3answers
40 views

Linear Regression and Ridge Regression

In Linear regression, we have $\textbf{$\theta$} = \textbf{$\left (X^TX\right )^{-1} X^T y$}$. In Ridge regression, we have $\textbf{$\theta$} = \textbf{$\left ( \lambda I+ X^TX\right )^{-1} X^T y$}$...
0
votes
1answer
17 views

Feature matrix as the Kronecker product of two feature matrices. How to build an alternative?

I have two feature matrices $\textbf{X}$ and $\textbf{Y}$ which I encoded through one-hot encoding the rows of two feature matrices $\textbf{X'}$ and $\textbf{Y'}$. Thus, they are sparse with a few $1$...
1
vote
1answer
41 views

How is the log-likelihood for a multinomial logistic regression calculated?

In a multinomial logistic regression, the predicted probability $\pi$ of each outcome $j$ (in a total of $J$ possible outcomes) is given by: $ \pi_j = \frac{e^{A_j}}{1+\sum_{g \neq j}^Je^{A_j}} $ ...
0
votes
0answers
8 views

Trying to generate two equations that describe debt growth from numerous samples

I hope this isn't too stupid a question... I have a set of 15 U.S. government unfunded liabilities debt values I've pseudo-randomly collected over 685 days... I'm hoping to generate two equations: One ...
1
vote
1answer
48 views

How to derive the variance of the mean of predictions from a linear regression model?

The context is linear regression analysis for estimating a sample mean. Assume the usual multivariate linear model: $$Y = X\beta + \epsilon$$ with $X$ a $n \times p$ covariate matrix with intercept, $...
0
votes
0answers
12 views

Gauss-Markov Mobility Model query

I have a query in understanding a little point, please if you could help me understand that. Thanks In the Gauss-Markov Mobility Model, the current speed and position are calculated based on speed ...
0
votes
0answers
17 views

Relationship between different types of correlation coefficients

Let, $r_{1(2.34...p)}$ = Correlation between $x_1$ and $x_{2.34...p}$. The latter being the residuals after regressing $x_2$ on $x_3 , x_4 ....x_p$. $r_{1.234..p}$ = Multiple correlation coefficient ...
0
votes
1answer
33 views

“At what point does another year of age reduce the probability of smoking?”

Say I have something like: $$pSmoke = .656 - .069log(cigprice)+.012log(income)-.029educ + .02age - .00026age^2 - .101restaurn - .026white$$ where $pSmoke$ is the probability of smoking and $age$ is ...
0
votes
1answer
37 views

Find an equation of the form $y= a e^ {x ^2} + bx^3$ which best fits the points $(−1, 0), (0, 1), (1, 2)$

Find an equation of the form $y= a e^ {x ^2} + bx^3$ which best fits the points $(−1, 0), (0, 1), (1, 2)$ I am given the hint don’t form a summation – explicitly form S with three terms, and find ...
0
votes
0answers
17 views

Why is this algorithm for PLS correct?

I am studying Partial Least Square Regression (PLS), and I am not able to understand how the algorithm for performing a PLS factorization works. PLS is composed of two parts. First, we factor two ...
0
votes
1answer
24 views

Calculating error on slope of graph

I'm trying to find the rate of change and the error on that rate based on 7 measurements points and the assumption that the trend is linear. My calculations are below: $$ \begin{array}{|c|c|c|c|} \...
0
votes
1answer
20 views

Understanding the lower and upper limit of a graph

I was recently presented the following graph: I was given very little context and I have to recreate it. I don't want it to be done for me, but some context would help. Any idea what these upper and ...