For simplicity I start with a 1-layer feed-forward neural network $F$, its formula is: $$ F: \mathbb{R}^n \rightarrow \mathbb{R}^n $$ $$ y \mapsto F(x) = ⎎(W x) $$ where ⎎ is the sigmoid function, $W$ is the $n \times n$ weight matrix.

The network learns to associate inputs $x$ to outputs $y$, but it is only given reward signals (both + and - rewards). This means, if an output is correct, the network will be given a positive reward, and vice versa.

The learning algorithm generates random $x$ values, outputs its $y$ value, and get the + or - reward, adjusts its weights accordingly.

My idea is: let $y_0 = F(x_0)$. Define an $\epsilon$-neighborhood of $y_0$ as $U(y_0)$. The pre-image of this neighborhood is $F^{-1}(U(y_0))$.

Upon getting a + reward, we want the learning algorithm to adjust the weights such that the volume of the pre-image becomes bigger, ie more points near $x_0$ will be mapped to or near $y_0$. This is a form of generalization based on proximity.

If $\epsilon$ is small, the volume of $F^{-1}(U)$ is approximately equal to the volume of $U$ scaled by the Jacobian determinant:

$$ \det J = \left| \frac{\partial F^{-1}(y)}{\partial y} \right|$$

Using gradient descend (as we do in back-propagation), we want to adjust weights to maximize (or minimize if negative reward) the Jacobian, ie, in the direction of $\nabla_W \cdot |J| = \left[\frac{\partial \det J}{\partial W} \;\right]$. This is my reasoning.


I'm not sure if this step is correct: $$ \det J = \left| \frac{\partial W^{-1}⎎^{-1}(y)}{\partial y} \right| $$ $$ \stackrel{?}{=} \left| W^{-1} S \right| $$ where $S$ is a diagonal matrix with entries $1/({ ⎎'(⎎^{-1}(y)) })$.

We seek the matrix $\frac{\partial \det J}{\partial W}$ whose entries are $\frac{\partial \det J}{\partial w}$

This involves finding the derivative of a determinant: $$ \frac{\partial}{\partial w} |J| = tr( |J| \cdot J^{-1} \cdot \frac{\partial J}{\partial w})$$ where the last factor in the above RHS is: $$ \frac{\partial J}{\partial w} = \frac{\partial W^{-1} S }{\partial w} $$ $$ \stackrel{?}{=} \frac{\partial W^{-1}}{\partial w}S + W^{-1} \frac{\partial S}{\partial w} $$ $$ \stackrel{?}{=} -W^{-1} \frac{\partial W}{\partial w} W^{-1} S + W^{-1} \hat{S} $$ where $\hat{S}$ is a diagonal matrix with entries: $$ -\frac{⎎''(⎎^{-1}(y))}{⎎'(⎎^{-1}(y))^3} ⎎'(Wx) \frac{\partial W}{\partial w} x$$

Is the above correct?



You must log in to answer this question.

Browse other questions tagged .