0
$\begingroup$

I am trying to implement an algorithm to perform gradient descent in a $n$-dimensional space in the context of an RBF network. My network has 5 inputs and 1 output. It has the following Gaussian radial basis function: $$f(x_i)=\sum_{k=1}^nw_ke^{(-\lambda_k||x_i - \mu_k||^2)}$$

where:

  • $n$ is the number of centroids in the network
  • $w_k$ are the weights between the hidden layer and output layer for each centroid $n$
  • $\lambda_k$ are the function width parameters for the $k^{th}$ centroid
  • $x_i$ is the $i^{th}$ training data input (vector)
  • $\mu_k$ is the $k^{th}$ centroid (vector)

My goal is the find optimal values of $w_k$ and $\lambda_k$ that minimize the squared error. I plan on doing that iteratively by:

  1. Fix $\lambda_k$ and use pseudo-inverse to solve for $w_k$
  2. Fix $w_k$ and solve for $\lambda_k$ using gradient descent by minimizing the error w.r.t. $\lambda_k$
  3. Repeat steps 1 and 2 until convergence is achieved (or any other criteria).

I have already found the weights by fixing each $\lambda_k$ to 1 (randomly) and using the pseudo-inverse matrix. Now I need to find values for each $\lambda_k$ that minimize the squared error. From what I understand, I need to:

  1. Find the partial(w.r.t. $\lambda_k$) derivative of the error function: $$S=\sum_{i=1}^p(y_i - f(x_i))^2$$where $f(x_i)$ is the function above, $y_i$ is the actual output from the $i^{th}$ training data and $p$ is the number of entries in the training set.
  2. Equate that partial derivative to zero.
  3. Solve for all the $\lambda_k$.

I do not understand well how to perform gradient descent w.r.t. $\lambda_k$.

Can anyone help me clarify how to perform gradient descent in a $n$-dimensional space? I need to find a vector of gradients $\delta_k$ to apply to each $\lambda_k$, so that $\lambda_k := \lambda_k + \delta_k$

My current algorithm for one iteration of gradient descent:

for each item x in training data set
   feed forward x in the network and get the output
   for each centroid k
      λk = λk - μ*(gradient of the cost function)    // μ is a constant for the learning rate
   end loop.
end loop.

where:

$\frac{\delta S}{\delta\lambda_k} = \sum_{i=1}^n2d_i^2(y_i - e^{-d_i^2\lambda_k})e^{-d_i^2\lambda_k} = \sum_{i=1}^n2d_i(y_i-f(x_i))f(x_i)$

where $d_i = ||x_i - \mu_k||$ = euclidean distance and $y_i$ = output from the training set, both known values.

Thank you very much!

$\endgroup$
  • $\begingroup$ you have to compute $\frac{\partial S}{\partial f(x_i)}$ then $\frac{\partial f(x_i)}{\partial \lambda _k}$ and finally $$\frac{\partial S}{\partial \lambda_k} = \sum_i \frac{\partial S}{\partial f(x_i)}\frac{\partial f(x_i)}{\partial \lambda _k}$$ (see en.wikipedia.org/wiki/Total_derivative ) $\endgroup$ – reuns May 31 '16 at 5:08
  • $\begingroup$ the total derivative is how to compute $\frac{\partial u}{\partial t}$ when $u(t) = h(f(t),g(t))$. you have to understand why the result is $$\frac{\partial u}{\partial t} = \frac{\partial h}{\partial f}\frac{\partial f}{\partial t} + \frac{\partial h}{\partial g}\frac{\partial g}{\partial t}$$ $\endgroup$ – reuns May 31 '16 at 5:10
  • $\begingroup$ Thanks for the reply. I knew I had to find the derivatives, so would you mind explaining how this relates to a n-dimensional space and how I can compute the vector of gradients? $\endgroup$ – Vincent L May 31 '16 at 12:10
  • $\begingroup$ what ?? the gradient descent is $\lambda_k \leftarrow \lambda_k-\eta \frac{\partial S}{\partial \lambda_k}$ ... $\endgroup$ – reuns May 31 '16 at 13:16
  • $\begingroup$ Do you mean I have to find each $\lambda_k$ one by one by iterating over each one and finding the minimum? Or is there a way to perform the operations on vectors and find all $\lambda_k$ at the same time? Also, can you help me find the derivative of this function? $\endgroup$ – Vincent L May 31 '16 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.