Examples of logistic regression and multinomial logistic regression

Question

I've been studying to understand the concept of logistic regression and I think I understand the idea more or less, but there are still some gaps to fill. What I'm looking for is an example of logistic regression and multinomial logistic regression to take the point home. Could someone perhaps give me a very simple examples to show me the fundamental steps one needs to take when doing logistic regression?

For example could someone show me what would I have to do If my data set consisted of the following:

$$\textbf{x}_1 = x_{11}, x_{12}, ..., x_{1n}$$ $$\textbf{x}_2 = x_{21}, x_{22}, ..., x_{2n}$$ $$\vdots$$ $$\textbf{x}_m = x_{m1}, x_{m2}, ..., x_{mn}$$

So $\textbf{x}_i$ is the $i$th variable and the observations from that variable I have denoted as $x_{ij}$, $1 \leq j \leq n$. I have $m$ variables and $n$ observations on each of them. Let's say the data described different kind of physiological features from $n$ different persons and I would like to use logistic regression to get the probability that person $A_k$, $1 \leq k \leq n$ gets an heart attack. In this example there are only two possible events: "heart attack" and "no heart attack". Could someone give me a pencil and paper like example what I would need to do to estimate the parameters using maximum likelihood etc. just to make it absolutely clear to me. It would be very nice if I could get examples in the case of two possible events and in the general cases where the number of possible events is $\geq 3$, because I'm using logistic regression in a project work where I have more than three possible events. Hope my question is clear.

The examples don't have to be long or anything. Just a very simple, short, showing all the steps,(e.g. calculating parameter estimates etc.). I have been watching some examples from the web, but in many sources the actual calculations I would like to see are just replaced by the words: "And computer gives us the parameters" :(. I want to see what is going on :)

Thank you for any help :)

P.S. You don't have to present a numeric example (you can if you want ;)). It is enough just to show all the necessary steps so that I'll be able to program the steps in Matlab if I wanted to :) Both logistic and multinomial logistic regression :)

UPDATE: If anyone is interested I found a very good site explaining logistic regression covering the details, etc. The site can be found here:

http://www.real-statistics.com/logistic-regression/basic-concepts-logistic-regression/

Answer 1

A bit of context on logistic regression. When dealing with logistic regression, you want to compute the probability $p$ of the binomial variable "YES" / "NO", or "heart attack" / "no heart attack" etc...by stating that such probability depends on a certain number of variables, let us say

$$x_1,\dots,x_p$$

through

$$\log(\frac{p}{1-p}):=\beta_0+\beta_1 x_1+\dots+\beta_p x_p.$$

This is a choice (we could have used other models, like the probit) motivated by the fact that $p=p(\beta,x)$ is a probability (i.e. is a real number between 0 and 1) and the logit transformation

$$\operatorname{logit}(p):=\log(\frac{p}{1-p})$$

is invertible, with inverse

$$p(\beta,x)=\frac{1}{1+exp(-\beta_0-\beta_1 x_1-\dots-\beta_p x_p)}.$$

Given a set of $n<p$ measurements $(y_{1},x_{1,1},\dots,x_{1,p})$, $\dots$, $(y_{n},x_{1,n},\dots,x_{1,n})$, with $y_{n}$ either equal to $0$ (no event) or $1$ (event), we want to estimate the parameters $(\beta_0,\beta_1,\dots,\beta_p)$, i.e. the missing piece in our regression scheme. This task is usually performed using maximum likelihood methods, which end up to the numerical algorithm called "Newton Raphson" for the parameter estimation.

The recursive algorithm provides you with an answer in most of the cases; divergences can occur for different reasons, however. One particularly interesting reason is called "complete separation" of one or even more variable. You will surely meet this topic in applications.

• Example:

Try to perform a logistic regression for the following easy vectors of data (in this order! Please, do not shuffle the components...)

$$y=(0,0,0,0,0,0,0,0,0,0,1,1,1,1,1)$$ $$x=(0,0,0,0,0.1,0.2,0.3,0.4,0.5,0.6,0,0,0,0.9,1)$$

...what do you get, as an answer?

The logistic regression can be theoretically motivated by the principle of maximum entropy: in fact, if we are supposed to use it on the binomial variable "YES" / "NO", or "heart attack" / "no heart attack" in presence of certain constraints,it is possible toshow that the probability distribution for such variable that maximizes the (Shannon) entropy is the logistic distribution. In this sense, the logistic regression, which is so common in applications, plays a special role.

Answer 2

These are roughly the steps:

1. Set up your data set appropriately (e.g., one matrix for covariates, one vector for dependent variable, i.e., 0/1 for binary logit)
2. create a function in Matlab that computes the likelihood or log-likelihood for a coefficient input using the data (note: if you use a numerical minimizer, this should be the negative of the likelihood function)
3. use a numerical minimizer, e.g., minFunc, that calls your likelihood function and feeds it coefficient-candidates to find the log-likelihood maximizing coefficients
4. if desired, calculate standard errors of the coefficients

This is the main approach for both binary and multinomial logit; the difference is just that the likelihood function changes.

Since logit is very widespread, I would first look if there isn't already a Matlab implementation for this. If not, almost every statistical package has it and does the maximizing for you. You then won't have to agonize over problems like which starting values for the numerical maximizer to use, which minimization method etc.

ADDENDUM

Here is a Matlab log-likelihood function for binary logit that I used years ago.

function[l]=logit_ll(param,data)
% defining y,x,n,k 
y1 = data(:,1); %dependent variable
k=size(data,2);
x = data(:,2:k,:); %covariates
[n k]=size(x);

% coefficients 
beta=param;

% Probability 1
pa1=exp(x(:,:)*beta)./(1+exp(x(:,:)*beta));
% Probability 2
pa2=1./(1+exp(x(:,:)*beta));

% prevent 0 or 1 for technical reasons
pa1=min(pa1,0.9999999999999999999999999999);
pa1=max(pa1,0.0000000000000000000000000001);

pa2=min(pa2,0.9999999999999999999999999999);
pa2=max(pa2,0.0000000000000000000000000001);

% log-likelihood
Lik = (log(pa1)).*y1(:,:) + (log(pa2)).*(ones(n,1)-y1(:,:));

% 1) Sum over individuals
% 2) Multiply by -1 because we use a minimizer
l=-sum(Lik);