Questions tagged [logistic-regression]
For questions about logistic regressions, a regression model where the dependent variable is categorical.
248
questions
0
votes
0
answers
23
views
Why in logistic regression for every threshold the decision boundary is a hyperplane?
I'm a beginner in machine learning study and I can't figure it out an exercise:
The function h(x) = θ(w ̃x) is used to approximate ...
0
votes
0
answers
9
views
Normality assumption for predictors?
In data science, should we ensure that all variables are normally distributed? I do understand that the y-value would need to be normally distributed as a logistic regression assumption. For instance, ...
- 1
0
votes
0
answers
50
views
Compute Lipschitz constant of Hessian of logistic loss
Convergence theory of Newton's method assumes that the Hessian $\nabla^2 l(w)$ of the loss function l is Lipschitz continuous, i.e. $\big\Vert \nabla^2 l(w_1) - \nabla^2 l(w_2)\big\Vert_2 \leq L \big\...
- 217
0
votes
0
answers
27
views
Gradient descent calculation split in 2 steps
In a machine learning course I am taking we have an assignment with a notebook (In linear models). So far I have calculated the cost and the sigmoid in a logistic regression and the next exercise ...
1
vote
0
answers
12
views
How to explain covariance in logistic regression + analogy to linear regression
Introduction
Linear model
In linear regression we predict continuous variable $Y \in R^n$ with use of $n \times p$ deterministic plan matrix $X$ and theoretical model (let's ignore intercept ...
- 73
0
votes
0
answers
16
views
Condition logit model: Weighted mean of ratios of coefficients of subgroups does NOT equal the ratios of coefficients of the whole sample, why?
I am well aware that when one splits the sample into subgroups (e.g. sex, country, whatever), and then estimates any logistic regression model, the coefficients are not comparable between (otherwise ...
0
votes
0
answers
18
views
Understanding derivation of Logitboost
I am trying to understand the derivation of LogitBoost.
I looked at the original paper by (Friedman, Hastie and Tibshirani, 2000) "Additive Logistic Regression:
A Statisitcal View OF Boosting&...
- 304
1
vote
0
answers
37
views
Intuition of Negative entropy as a Fenchel conjugate of logistic cost function
Let us first recall the logistic cost function
$$f_1(x)=\log(1+e^x)$$
and the negative entropy function
$$f_2(y)=y\log(y)+(1-y)\log(1-y).$$
Here $\log$ denotes the natural logarithm and $x\in \mathbb ...
- 1,208
0
votes
0
answers
31
views
How do you transform a Derivative to a Recurrence Equation?
In the logistic equation, they say $\frac{dx}{dt} = rx(1-x)$ is equal to $x_{n+1} = rx_{n}(1-x_{n})$. How do they do that? And if you were to transform $\frac{dx}{dt} = 2$ to the recurrence equation, ...
- 312
-1
votes
1
answer
55
views
Machine learning - Cost function for non linear functions [closed]
The cost function is some indication of the 'cost'/how the predicted value differs from the actual value.
In linear regression, this can be measured using MSE.
In the case of the logistic function, ...
- 77
2
votes
0
answers
102
views
Proof that the MLE of logistic regression doesn't have a closed-form solution
It is well known that the maximum likelihood estimator of logistic regression does not admit a closed form solution, at least in the general case where the predictors are not binary or categorical. ...
- 156
0
votes
1
answer
24
views
explanation for cross entropy for logistic regression
as far as I know, cross entropy of two distributions is:
$$
C(p,q) = -\sum_{s \in classes}p(s)\log(q(s))
$$
however, the loss function for logistic regression (called "crossentropy loss") it'...
0
votes
1
answer
22
views
What is the best way to estimate the parameters of a logistic regresion model?
I recently read about logistic regression model.
$$y=\frac{1}{1+e^{-(\beta_0+\beta_1x)}}$$
where y is a categorical variable with either 0 or 1 output.
What seems to be perplexing to me is, I can see ...
- 154
3
votes
1
answer
54
views
How to determine labeled balls for logistic regression
I am studying this paper about logistic regression. In section 4.2 (Randomly Generated Problems) on page 1534, they say "Features of positive (negative) examples are independent and identically ...
- 931
0
votes
1
answer
38
views
Working out the derivative of the log-likelihood for group LASSO
I'm following the working of the sparse group LASSO in the paper 'A Sparse-Group LASSO' by Simon. For the linear case, we have the problem given as
$$\text{min}_\beta \frac{1}{2}||y-\sum_{l=1}^m X^{(l)...
- 13
0
votes
1
answer
51
views
Mathematical derivation from Deep learning book
In https://www.deeplearningbook.org/contents/mlp.html p. 179 this derivation is made but lacks details that I need to understand it:
$P(y)= \frac{exp(yz)}{\sum_{y’=0}^{1}exp(y’z)} = \sigma((2y-1)z)$
...
- 237
0
votes
0
answers
10
views
Does every neuron in a feedforward neural network produce a binary output?
My question is very simple, but different sources seem to contradict each other.
Given a neuron of a simple feedforward neural network, we know it takes the scalar product of its input vector and its ...
0
votes
0
answers
12
views
Binary to Multiclass logistic regression and vice versa
As I was working on a problem, I came across the mention of logistic regression being used for binary and multiclass problems.
Specifically, I am very keen on the problem with the below equations. How ...
- 1
0
votes
0
answers
40
views
Logistic Regression Problem
You are testing a new drug and have gathered binary data on whether the drug performed its desired effects. From the control trial, $102$ people saw improvement with a placebo and $241$ did not. With ...
- 35
0
votes
0
answers
27
views
Find z as $\tau\to0$
Let $x$ be $(x_1, x_2, ..., x_K)^T$ and $z$ be $(z_1, z_2, ..., z_K)^T$ be 2 K-dimensional vectors. Each dimension of z is defined as $$z_j=\frac{e^{x_j/\tau}}{\sum_{k=1}^{K}e^{x_k/\tau}},\tau>0, 1\...
- 145
2
votes
3
answers
143
views
Implementing multiclass logistic regression from scratch
This is a sequel to a previous question about implementing binary logistic regression from scratch.
Background knowledge:
To train a logistic regression model for a classification problem with $K$ ...
- 48k
2
votes
1
answer
59
views
Why there is no $y_i$ term in $\frac{d^{2} J(\boldsymbol{\alpha})}{d \alpha_{i}^{2}}$?
$$\frac{d^{2} J(\boldsymbol{\alpha})}{d \alpha_{i}^{2}}=\lambda^{-1} \mathbf{x}_{i}^{\mathrm{T}} \mathbf{x}_{i}+\frac{1}{\alpha_{i}\left(1-\alpha_{i}\right)}$$
I cannot understand why there is no $...
- 371
0
votes
0
answers
35
views
Single variable logistic regression
I am trying to write out the code for using Newton-Raphson to solve for a coefficient and intercept in a single variable logistic regression.
However in both the math and in the code I see that the ...
- 101
3
votes
1
answer
68
views
Implementing binary logistic regression from scratch
Background knowledge:
To train a logistic regression model for a classification problem with two classes (called class $0$ and class $1$), we are given a training dataset consisting of feature vectors ...
- 48k
1
vote
1
answer
31
views
Solving for the Logistic Regression Coefficients
I have been searching for a source on how to compute the coefficients for logistic regression, but haven't found any. I suppose it is very easy to find like linear regression.
Can you kindly suggest a ...
- 1,608
0
votes
0
answers
57
views
How to compute the gradient of this multinomial logistic loss function?
How to compute the first and second derivative of the multinomial logistic loss function with respect to $w_k$ that is minimized in this statement?
$$
\begin{array}{ll}
\min _{\mathbf{w}} & \frac{...
- 371
0
votes
2
answers
132
views
How to plot a decision boundary for binary logistic regression in matlab
let me preface by saying this is from a homework question, but the question is not to plot the decision boundary, just to train the model and do some predictions. I have already done that and my ...
- 145
0
votes
1
answer
22
views
confused about beta weights in logistic regression
I've recently started getting into machine learning, and I saw that making a simple classifier using logistic regression was a good place to start.
I was following through this article and got to the ...
1
vote
0
answers
16
views
How to estimate the covariance matrix for outcome of logistic mixed-effects model
I have a logistic mixed-effects model below.
$\mathbf{y}=(y_1, y_2, ..., y_n)^T$ is a n dimensions vector.
$\mathbf{p}=(p_1,p_2, ...,p_n)^T$ is a n dimensions vector.
$logit(\mathbf{p}) = (logit(p_1),...
- 155
1
vote
1
answer
34
views
For the Logistic model, why is the objective function unbounded below if two sets are linearly seperated?
I am reading Approximate linear discrimination via logistic modeling in the Section 8.6.1 of B & V's Convex Optimization book. On Page 428,
$$
\operatorname{minimize} \ -l(a, b) \tag{8.27}
$$
...
- 194
0
votes
0
answers
14
views
Logistic discrete equation: analitic solution
I have to proof that the solution for the logistic discrete equation: $x_{n+1}=4x_n\left(1-x_n\right)$ with $r=4$, and $x_0 \in (0,1)$ when $n=0$, it's of the form $$x_n=\sin^2\left(\beta\alpha^n\...
- 9
0
votes
0
answers
31
views
Why is softmax normalization so popular?
Is any theoretical grounding that suggests that the softmax normalization performs better than other normalization functions?
So far, the only valid reasons I've seen are that it scales values to $[0, ...
- 221
0
votes
0
answers
9
views
Interpretation of Proportion change in regression
I have two equations:
expenditure = 10 + 20*income
where income is measured as a proportion between 0 and 0.7
ln(expenditure) = 3.5 + 1.2 * income
Assumption: expenditure measures in thousand ...
- 13
2
votes
1
answer
108
views
Show that the joint distribution of $X_1, X_2, \dots, X_n$ belongs to a two-parameter exponential family.
Let $X_1, X_2, \dots, X_n$ be ind $ \sim\text{Ber}(\theta_i)$ where
\begin{equation}
\theta_i = P(X_i=1)=\frac{\exp(\alpha+\beta t_i)}{1+\exp(\alpha + \beta t_i)}
\end{equation}
where $t_1, ...
- 77
0
votes
0
answers
24
views
Which classification method to choose?
I am working on a statistics project in which I analyze 4 different classification methods (mostly for binary classification and with quantitative input variables).
On the one hand I'm studying ...
- 59
0
votes
1
answer
63
views
Fitting data to a logistic function
I've spent my whole entire weekend struggling with this problem. If we are given a set of data and are asked to fit it into a logistic equation:
$\frac{dP}{dt} = bP(\frac{a}b-P)$, where a and b is ...
- 169
0
votes
0
answers
26
views
How probabilities and predictions are calculated in logistic regression?
I have these values of y and x:
y x
0 3
0 4
0 7
1 8
1 11
0 14
1 15
1 16
1 17
I want to calculate prediction for values 9 and 20 using logistic regression....
2
votes
1
answer
735
views
What is the derivative of binary cross entropy loss w.r.t to input of sigmoid function?
I want to compute the derivative of binary cross entropy loss w.r.t to the input of the sigmoid function and was wondering if there's a closed form expression? I've seen derivations of binary cross ...
- 145
1
vote
1
answer
45
views
Comparing two explanatory variables explanatory variables
I am new to stats, I am working on a problem and I need some direction.
I have a binary dataset and I have three explanatory variables x,y, and z that I want to analyze. There is no guarantee of ...
- 13
0
votes
1
answer
63
views
How can we derive the Normal equations for Logistic Regression?
I was wondering from numerical linear algebra point of view that since solving OLS $\|b-Ax\|_{2}=\min_{w\in\mathbb{R}^{n}}\|b-Aw\|$ is equivalent to solving the normal system
$$
A^{T}Ax=A^{T}b
$$
then ...
- 331
1
vote
0
answers
25
views
How to know a classifier is a linear or not linear in classification problems?
I read a lot , but still not able to get the following concepts -:
(1) If a classifier is given, how do we know whether its a linear or non linear classifier? (Interested in step by step procedure to ...
1
vote
1
answer
60
views
Likelihood values from Sigmoid
There are multiple doubts of mine associated around this theme:
In MLE, we try to find the PDF parameters ($\theta$) which maximise the likelihood of the observed data ($L(\theta | data)$). To get ...
- 324
0
votes
0
answers
7
views
Variance Covaraince Matrix of Parameters in Logistic Regression
The given below image is taken from book Introduction to Linear Regression Analysis (Douglas C Montgomery) My apologies in advance for not typing , I just want to understand the concept.
(1) First ...
1
vote
2
answers
93
views
Deriving Probability from logit or log-odds
How can I motivate the derivation of the $p$ below?
From: https://en.wikipedia.org/wiki/Logistic_regression#Logistic_model
$l = log_b\frac{p}{1-p}=\beta_0 + \beta_1x_1+\beta_2x_2
$
$p = \frac{b^{\...
- 225
1
vote
0
answers
17
views
Account Resurrection as a function of Days Dead and Time to Resurrect
I am a data scientist working for a company that takes user deposits. I wanted to answer the question of how likely an account that's dropped to $0 on deposit - or dies, in other words - would refund ...
- 111
0
votes
0
answers
4
views
Creating adversarial sample by projecting my data on the other side of the hyperplane
I performed a binary classification using logistic regression.
My goal is the following:
I know the coefficient w of the hyperplane equation y = wTx + b. What I would like to do is create opposing ...
- 1
0
votes
0
answers
25
views
Why is the sigmoid of a linear model equal to the probability of the target being $1$?
From this resource, the writer starts with a linear model:
$$
y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + ... + \beta_m x_m
$$
and then makes the RHS sigmoidal. This must then make the LHS sigmoidal to ...
- 1,871
0
votes
1
answer
15
views
Why does linear regression output numbers but logistic regression output probabilities?
From reading this resource, the writer wrote:
A linear model does not output probabilities, but it treats the classes as numbers (0 and 1) and fits the best hyperplane (for a single feature, it is a ...
- 1,871
0
votes
0
answers
28
views
Binary Logistic Regression with labels $\{-1, 1\}$
I am reading about binary logistic regression and I have found an equivalent formulation given by
$$\min_{w} \frac{1}{m}\sum_{i} \log \left(\exp(-y_iX_iw) + 1\right),$$
where $X \in \mathbb{R}^{m \...
- 721
1
vote
0
answers
65
views
How to prove the non convexity of logistic regression?
In linear Regression we have the following loss function :-
$$L(x)=\frac1{n}\left(\sum_{i=1}^n ((a+bx_i)-y_i)\right)^2$$
Hence, we can observe that L(x) is having a convex graph but when we use the ...
- 13