I took a deep dive into logistic regression recently. I was bothered by the fact that the likelihood formula often explicitly incorporates the values of the coding (0 or 1) into the calculation. In the actual problem we usually have a sample space of something like {H, T}^N. It turns out that when we are trying to find the ML estimate for a parameter we can view each choice of the parameter (beta) as corresponding to a pmf on this sample space, call this p(beta). Each p(beta) corresponds to a point in the open N-dimensional unit cube (at least for logistic regression). That is, the corners can't be reached for any choice of beta with the logit or probit as the link function. These corners are what folks typically use for their coding scheme, even if they don't know why. The corners correspond to the 2^N possible responses, but the actual responses are of course some sequence of H's and T's in the probability space. We can map beta to p(beta) in the interior of the cube and we can also map our response (Y) to a corner. But the corner is now a pmf with all its mass at the point Y, call this p(Y). Now since the set of all pmf's over a probability space is a metric space we can ask whether the ML solution is the p(beta) closest to the point mass pmf(Y). That is, beta minimizes

D(p(beta), p(Y))

where D(u,v) is the total variation distance.

It turns out that the p(beta) that achieves this minimum is also the ML solution. The math is the same, but this approach connects ordinary least squares to ML is the sense that both approaches minimize the distance between the observation, Y, and the span of the design matrix, typically some linear subspace. In OLS we use euclidean distance. But once we move to ML we use the metric for all probability measures on the sample space. Then we can keep the response is the same metric space as the span of the design matrix. This approach extends nicely to other link functions like Poisson and other discrete responses. It also extends to continuous responses but we have to be careful with how we define the distance between p(beta) and p(Y) since p(Y) is now a delta function.

Does anyone know where there may be more written about this concept.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.