In many books, Kullback-Leibler Divergence is equal to Relative Entropy. $$ D_{kl}(u,v) = \sum_{i=1}^n(u_ilog(u_i/v_i). $$ However, I find in the book, Convex Optimization (Stephen Boyd) page 90, the KL Divergence is defined as, $$ D_{kl}(u,v) = \sum_{i=1}^n(u_ilog(u_i/v_i)-u_i+v_i). $$ Why KL Divergence has these two different definition? Which one is correct?

  • 1
    $\begingroup$ The book (also written by Vandenberghe) clearly mentions "Note that the relative entropy and the Kullback-Leibler divergence are the same when u and v are probability vectors, i.e., satisfy 1Tu = 1Tv = 1.)" $\endgroup$ – LinAlg Dec 20 '18 at 2:45

They're equivalent because $\sum_i u_i=\sum_i v_i=1$.

  • $\begingroup$ thank you so much. $\endgroup$ – Alan Dec 19 '18 at 23:15
  • $\begingroup$ equivalent only for probability vectors, as the book mentions $\endgroup$ – LinAlg Dec 20 '18 at 2:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.