# How to differentiate discrete probabilities?

Assuming that I have a function $$f(p(x),p(c),p(x,c)) = \ln (p(x)p(c)) + \ln (p(x,c))$$

where $p(\cdot)$ are discrete probabilities, $x \in X, c \in C$ are random variables. So $p(x)=p(X=x)$ denotes probability of $x$ occurring, $p(x,c)=p(X=x,C=c)$ denotes probability of $x,c$ occurring together.

If I differentiate this function with respect to $p(x,c)$, how do I differentiate the term $ln (p(x)p(c))$ wrt $p(x,c)$?

Do I have to do partial differentiation $$\frac{\partial \ln (p(x)p(c))}{\partial p(x)}\frac{\partial (p(x))}{\partial p(x,c)}+\frac{\partial \ln (p(x)p(c))}{\partial p(c)}\frac{\partial (p(c))}{\partial p(x,c)}$$

or differentiation of this term wrt $p(x,c)$ is zero?

-
What are the discrete probabilities? Are you familiar with partial derivatives? What such differentiation shall give you as an outcome? – Ilya Jan 18 '13 at 9:27
The discrete probabilities are simply obtained from a discrete data. I am not familiar with partial derivatives. My guess is differentiate $ln (p(x)p(c))$ wrt $p(c,x)$ is zero, but it seems that $p(x)p(c)$ and $p(x,c)$ are related, but they are not governed by a formula or equation, so I do not know how to do partial differentiation on them – Michael Jan 18 '13 at 9:47
If those are partial derivatives, it is zero, but since there is certainly a dependence between these values - I asked you why do you need to differentiate them. Clarifying that point would help. – Ilya Jan 18 '13 at 9:50
@Ilya my function is to measure a characteristic of the data, and I am wondering what will be the optimal probabilities to get the maximum of the function, or if given a probability, what is the gradient of the function to reach the maximum of the function, To get the maximum of the function, I am thinking of getting its gradient – Michael Jan 18 '13 at 10:08
Presumably $p(x,c)$ and $p(c,x)$ are meant to be the same thing? It would be slightly less confusing if you stick to one of them. – joriki Jan 18 '13 at 10:13

The answer below is very general and doesn't take into account what emerged in later clarifications of the question. It treats $p(x)$, $p(c)$ and $p(x,c)$ as unknown entities. With the clarifications provided, $f$ needs to be considered as a function of many more variables, namely all $p(x,c')$ and all $p(x',c)$. All derivatives should then be partial derivatives, and we can take the derivative of $f$ with respect to $p(x,c)$ while keeping all other $p(x',c)$ and $p(x,c')$ constant. With $p(x)=\sum_cp(x,c)$ and $p(c)=\sum_xp(x,c)$, we have $\partial p(c)/\partial p(x,c)=1$ and $\partial p(x)/\partial p(x,c)=1$, so

$$\frac{\partial f}{\partial p(x,c)}=\frac1{p(x,c)}+\frac1{p(x)}+\frac1{p(c)}\;.$$

(You'll need a Lagrange multiplier for the normalization constraint if you take this approach.)

It depends entirely on how you're viewing these probabilities. If you have a model where $p(x)$, $p(c)$ and $p(x,c)$ can be varied separately, you can treat them as distinct independent variables and just take the partial derivative with respect to any of them, e.g. $\partial f/\partial p(x,c)=1/p(x,c)$.

On the other hand, if your model posits some relationship between them, you have to take that relationship into account. If $p(x)$ and $p(c)$ can be expressed as functions of $p(x,c)$, you can form the total derivative with respect to $p(x,c)$, and the result is similar but not quite the same as what you wrote:

\begin{align} \frac{\mathrm df}{\mathrm dp(x,c)}&=\frac{\partial f}{\partial p(x,c)}+\frac{\partial f}{\partial p(x)}\frac{\mathrm dp(x)}{\mathrm dp(x,c)}+\frac{\partial f}{\partial p(c)}\frac{\mathrm dp(c)}{\mathrm dp(x,c)} \\ &=\frac1{p(x,c)}+\frac1{p(x)}\frac{\mathrm dp(x)}{\mathrm dp(x,c)}+\frac1{ p(c)}\frac{\mathrm dp(c)}{\mathrm dp(x,c)}\;. \end{align}

Theoretically you could also consider two of the three as independent and have only one relationship expressing the third in terms of them; in that case, "differentiating with respect to $p(x,c)$" is only well-defined if you specify which of the two others you're holding constant. Then you can form a partial derivative with respect to $p(x,c)$, which you can also obtain with the chain rule. For instance, if you want to differentiate with respect to $p(x,c)$ while holding $p(x)$ constant, that would be

\begin{align} \left.\frac{\partial f}{\partial p(x,c)}\right|_{p(x)}&=\frac{\partial f}{\partial p(x,c)}\left.\frac{\partial p(x,c)}{\partial p(x,c)}\right|_{p(x)}+\frac{\partial f}{\partial p(x)}\left.\frac{\partial p(x)}{\partial p(x,c)}\right|_{p(x)}+\frac{\partial f}{\partial p(c)}\left.\frac{\partial p(c)}{\partial p(x,c)}\right|_{p(x)} \\ &= \frac{\partial f}{\partial p(x,c)}+\frac{\partial f}{\partial p(c)}\left.\frac{\partial p(c)}{\partial p(x,c)}\right|_{p(x)} \\ &= \frac1{p(x,c)}+\frac1{p(c)}\left.\frac{\partial p(c)}{\partial p(x,c)}\right|_{p(x)}\;, \end{align}

where the unqualified partial derivatives are the standard partial derivatives of $f$ with respect to its three arguments while keeping the other two constant.

-
I only know the $p(x,c)$ is the joint probability of random variable $x,c$. Is there a way to express $p(x),p(c)$ as function of $p(x,c)$? – Michael Jan 18 '13 at 10:25
in discrete case $p(x) = \sum_c p(x,c)$ – Ilya Jan 18 '13 at 10:26
@Michael: Does Ilya's comment reflect what you had in mind? If so, the formulas can be simplified considerably. By the way, you should have mentioned right away that $x$, $c$ are random variables; I was taking them to be events. – joriki Jan 18 '13 at 10:34
@joriki: I apologise for missing the details. So given that I have $p(x)=\sum_c p(x,c)$ (thank you Ilya), I can differentiate $\frac{\partial p(x)}{\partial p(x,c')}=\frac{\partial \sum_c p(x,c)}{\partial p(x,c')}$? – Michael Jan 18 '13 at 10:37
@joriki: I think your total derivative is what I needed, and with the relationship $p(x) = \sum_c p(x,c)$, $p(c) = \sum_x p(x,c)$, I can plug these relationships into your total derivative to get my gradient – Michael Jan 18 '13 at 10:44

In case you have only a finite number of $p(x)$, $p(c)$ and $p(x,c)$ it's possible just to find the maximum of their function $f$ directly, however I agree it may not be optimal. In such case it's worth looking where $f(p,q,r)$ has its maximum and picking up discrete values closest (in some sense) to the argument of the maximum $(p^*,q^*,r^*)$. Provided the fact that $\ln$ has some Lipschitz properties, the found solution shall not be bad.

However, it matters here where are you looking for the maximum. In case all three value are independent, you shall just look for it on $[0,1]^3$. In such case you have just a maximum of a sum of 3 independent $\ln$ and it is easy to find.

However, in case there is a certain dependence - you shall look for the conditional maximum. E.g. if you know that $|p(x,c)-p(x)p(c)|\leq \varepsilon$, you shall look for the maximum only on $$[0,1]^3\cap \{(p,q,r):|r-pq|\leq \varepsilon\}.$$ Or if you know that $p(x,c) = g(p(x),p(c))$ then you shall look for the maximum of $$f(p,q,g(p,q))$$ on $[0,1]^2$. Without knowing the underlying statistical problem, it's hard to say which kind of dependence you shall have.

-
Thank you very much for your help, much appreciated – Michael Jan 18 '13 at 13:48