I am trying to understand some derivations in a machine learning paper. Suppose I have a function R_j = max(0,\sum_i x_iw_{ij} + b_j) ( A Relu Neuron ), and I want to find the closest root {x^i} of this function to point {x_i}. The paper does so by searching for a root in a particular search direction: {x^i} = {x_i} + t{v_i}. I understand that the direction we should move in the unconstrained space is the gradient of R_j in an analogue way to gradient descent and thus why for the unconstrained input space case the direction is w_ij ( gradient of R_j) along line x_i. However, I cannot understand why the paper to find a root in the positive real space must search instead of line x_i on the segment ({x_i 1{w_ij <0} }, {x_i}) and especially why in this case the search direction becomes v_i = x_i 1{w_ij >0}.

Any help or resources I can find to understand would be much appreciated. The section of the paper in question is 1C in the supplementary material. ( https://arxiv.org/pdf/1512.02479.pdf )

Thank you in advance!



You must log in to answer this question.