3
$\begingroup$

I've been studying to understand the concept of logistic regression and I think I understand the idea more or less, but there are still some gaps to fill. What I'm looking for is an example of logistic regression and multinomial logistic regression to take the point home. Could someone perhaps give me a very simple examples to show me the fundamental steps one needs to take when doing logistic regression?

For example could someone show me what would I have to do If my data set consisted of the following:

$$\textbf{x}_1 = x_{11}, x_{12}, ..., x_{1n}$$ $$\textbf{x}_2 = x_{21}, x_{22}, ..., x_{2n}$$ $$\vdots$$ $$\textbf{x}_m = x_{m1}, x_{m2}, ..., x_{mn}$$

So $\textbf{x}_i$ is the $i$th variable and the observations from that variable I have denoted as $x_{ij}$, $1 \leq j \leq n$. I have $m$ variables and $n$ observations on each of them. Let's say the data described different kind of physiological features from $n$ different persons and I would like to use logistic regression to get the probability that person $A_k$, $1 \leq k \leq n$ gets an heart attack. In this example there are only two possible events: "heart attack" and "no heart attack". Could someone give me a pencil and paper like example what I would need to do to estimate the parameters using maximum likelihood etc. just to make it absolutely clear to me. It would be very nice if I could get examples in the case of two possible events and in the general cases where the number of possible events is $\geq 3$, because I'm using logistic regression in a project work where I have more than three possible events. Hope my question is clear.

The examples don't have to be long or anything. Just a very simple, short, showing all the steps,(e.g. calculating parameter estimates etc.). I have been watching some examples from the web, but in many sources the actual calculations I would like to see are just replaced by the words: "And computer gives us the parameters" :(. I want to see what is going on :)

Thank you for any help :)

P.S. You don't have to present a numeric example (you can if you want ;)). It is enough just to show all the necessary steps so that I'll be able to program the steps in Matlab if I wanted to :) Both logistic and multinomial logistic regression :)

UPDATE: If anyone is interested I found a very good site explaining logistic regression covering the details, etc. The site can be found here:

http://www.real-statistics.com/logistic-regression/basic-concepts-logistic-regression/

$\endgroup$

2 Answers 2

3
$\begingroup$

A bit of context on logistic regression. When dealing with logistic regression, you want to compute the probability $p$ of the binomial variable "YES" / "NO", or "heart attack" / "no heart attack" etc...by stating that such probability depends on a certain number of variables, let us say

$$x_1,\dots,x_p$$

through

$$\log(\frac{p}{1-p}):=\beta_0+\beta_1 x_1+\dots+\beta_p x_p.$$

This is a choice (we could have used other models, like the probit) motivated by the fact that $p=p(\beta,x)$ is a probability (i.e. is a real number between 0 and 1) and the logit transformation

$$\operatorname{logit}(p):=\log(\frac{p}{1-p})$$

is invertible, with inverse

$$p(\beta,x)=\frac{1}{1+exp(-\beta_0-\beta_1 x_1-\dots-\beta_p x_p)}.$$

Given a set of $n<p$ measurements $(y_{1},x_{1,1},\dots,x_{1,p})$, $\dots$, $(y_{n},x_{1,n},\dots,x_{1,n})$, with $y_{n}$ either equal to $0$ (no event) or $1$ (event), we want to estimate the parameters $(\beta_0,\beta_1,\dots,\beta_p)$, i.e. the missing piece in our regression scheme. This task is usually performed using maximum likelihood methods, which end up to the numerical algorithm called "Newton Raphson" for the parameter estimation.

The recursive algorithm provides you with an answer in most of the cases; divergences can occur for different reasons, however. One particularly interesting reason is called "complete separation" of one or even more variable. You will surely meet this topic in applications.

  • Example:

Try to perform a logistic regression for the following easy vectors of data (in this order! Please, do not shuffle the components...)

$$y=(0,0,0,0,0,0,0,0,0,0,1,1,1,1,1)$$ $$x=(0,0,0,0,0.1,0.2,0.3,0.4,0.5,0.6,0,0,0,0.9,1)$$

...what do you get, as an answer?

The logistic regression can be theoretically motivated by the principle of maximum entropy: in fact, if we are supposed to use it on the binomial variable "YES" / "NO", or "heart attack" / "no heart attack" in presence of certain constraints,it is possible toshow that the probability distribution for such variable that maximizes the (Shannon) entropy is the logistic distribution. In this sense, the logistic regression, which is so common in applications, plays a special role.

$\endgroup$
12
  • $\begingroup$ +1 Thank you @Avitus :) Life saver, sorry if my questions are silly and too simplistic. I'm new on the field and lacking confidence on the subjects so I need someone to tell me what do at first x) $\endgroup$
    – jjepsuomi
    Jul 11, 2013 at 13:00
  • 1
    $\begingroup$ Your questions are interesting! Please, let me know the results of the logistic regression with the data in my answer, if you run it ;) $\endgroup$
    – Avitus
    Jul 11, 2013 at 13:06
  • 1
    $\begingroup$ I'll give the answer soon, I'm just gonna get more familiar with this first :) I found an excellent post on logistic regression: real-statistics.com/logistic-regression/…. $\endgroup$
    – jjepsuomi
    Jul 12, 2013 at 10:32
  • 1
    $\begingroup$ @jjepsuomi I saw the comment with your output, but now disappeared :/ Anyway: Matlab gives you a finite answer, does it? Could you please specify 'maxiter' (max. number of iterations of the Newton Raphson algorithm) equal to 1,2,3,4... and see what does it happen? $\endgroup$
    – Avitus
    Jul 12, 2013 at 10:34
  • 1
    $\begingroup$ @jjepsuomi welcome back :) The coefficients you get are false friends...in the Matlab code, while running "glmfit", try to add "maxiter=1", then "maxiter=2", then "maxiter=3" etc etc...you should get a surprise :). You can find the exact syntax here stat.umn.edu/macanova/htmlhelp/node133.htm $\endgroup$
    – Avitus
    Jul 22, 2013 at 11:03
1
$\begingroup$

These are roughly the steps:

  1. Set up your data set appropriately (e.g., one matrix for covariates, one vector for dependent variable, i.e., 0/1 for binary logit)
  2. create a function in Matlab that computes the likelihood or log-likelihood for a coefficient input using the data (note: if you use a numerical minimizer, this should be the negative of the likelihood function)
  3. use a numerical minimizer, e.g., minFunc, that calls your likelihood function and feeds it coefficient-candidates to find the log-likelihood maximizing coefficients
  4. if desired, calculate standard errors of the coefficients

This is the main approach for both binary and multinomial logit; the difference is just that the likelihood function changes.

Since logit is very widespread, I would first look if there isn't already a Matlab implementation for this. If not, almost every statistical package has it and does the maximizing for you. You then won't have to agonize over problems like which starting values for the numerical maximizer to use, which minimization method etc.

ADDENDUM

Here is a Matlab log-likelihood function for binary logit that I used years ago.

function[l]=logit_ll(param,data)
% defining y,x,n,k 
y1 = data(:,1); %dependent variable
k=size(data,2);
x = data(:,2:k,:); %covariates
[n k]=size(x);

% coefficients 
beta=param;

% Probability 1
pa1=exp(x(:,:)*beta)./(1+exp(x(:,:)*beta));
% Probability 2
pa2=1./(1+exp(x(:,:)*beta));

% prevent 0 or 1 for technical reasons
pa1=min(pa1,0.9999999999999999999999999999);
pa1=max(pa1,0.0000000000000000000000000001);

pa2=min(pa2,0.9999999999999999999999999999);
pa2=max(pa2,0.0000000000000000000000000001);

% log-likelihood
Lik = (log(pa1)).*y1(:,:) + (log(pa2)).*(ones(n,1)-y1(:,:));

% 1) Sum over individuals
% 2) Multiply by -1 because we use a minimizer
l=-sum(Lik);
$\endgroup$
3
  • $\begingroup$ +1 thank you for your answer :) Very nice. It would be nice to perhaps see some formulas as well , like if you didn't have a computer :) My point is not to invent the wheel again, but to just understand the mechanics and in order to do that I must look under the hood u know ;D $\endgroup$
    – jjepsuomi
    Jul 11, 2013 at 11:46
  • $\begingroup$ I added the code of the likelihood function, which contains the relevant expressions. For more theory on this, I recommend elsa.berkeley.edu/books/choice2.html $\endgroup$
    – Nameless
    Jul 11, 2013 at 12:32
  • $\begingroup$ +1 Thank you for your help @Nameless very nice! Thnx a million! :) $\endgroup$
    – jjepsuomi
    Jul 11, 2013 at 12:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .