There is a post that someone answered it already, but I think his/her answer is how to validate the formula, not the way to derive the formula. A dual norm optimization problem

This formula comes from the paper : https://arxiv.org/pdf/2010.01412.pdf, it's about SAM optimizer.

$\epsilon^*(w)= \underset{ ||\epsilon||_p<=\rho }{argmax}$ ${\epsilon^T\nabla_wL_S(w)} $ (1)

With classical dual norm prolem and the solution is:

$\hat{\epsilon}(w)=\rho sign(\nabla_WL_S(w))|\nabla_WL_S(w)|^{q-1}/(||\nabla_wL_S(w)||_q^q)^{1/p}$ (2)

when $1/p + 1/q = 1$ and $|.|^{q-1}$ denotes elementwise absolute value and power.

What I understand is I can use Holder's inequality to derive that

$\epsilon^Tx<=||\epsilon||_p ||x||_q<=\rho||x||_q$ when $x = \nabla_wL_S(w)$

So it seems to me the formula should be

$\epsilon^*(w)=\underset{ ||\epsilon||_p<=\rho }{argmax}$ $\rho||x||_q$

I don't know where sign and ${||\nabla_WL_S(w)||_q^q}^{1/p}$ come from. Can anyone show me how I can dervie first formula to the second one?



You must log in to answer this question.