Optimization, Gradients, and Multivariate Data

I would like to learn gradient based optimization for multivariate data.

For example, assume the data I have is $X = (x_0, ..., x_n)$ where $x_i$ are some random variables and $f$ a function measuring (Pearson, if you like) correlation. Then, I would like to minimize the value of $f(X)$ i.e. make the variables $x_0, ..., x_n$ uncorrelated. How could this be achieved using gradient based methods?

After I have learnt this, the next thing is that I would like to implement the procedure in MATLAB. If you have any tips for that, I would like to hear those as well.

-
What are the optimization variables here? Surely you don't want to change the input data arbitrarily, or I could throw away the original values and give you back random values for all the $x_i$. –  Rahul Jan 5 '13 at 17:26
The example I gave is analogous to principal component analysis (PCA), which I know from statistics (and know how to do based on an eigenvalue decomposition). However, I am here interested to solve the problem I provided through gradient based methods, which are the subject of my interest. –  user55285 Jan 5 '13 at 17:31
Are some variables random and some deterministic? Are you minimizing an expectation of sorts? –  copper.hat Jan 5 '13 at 17:51

1 Answer

You can use Gradient Descent method for optimization. It is one of the most common methods for optimization and learning and you can learn about it from various online resources. Further since you want to code this in matlab, their are two methods:

• You start from point zero and write code based on mathematical algorithms you get online
• Or you can refer to already written MATLAB code on gradient descent (http://www.mathworks.in/matlabcentral/fileexchange/38588-gradient-descent). And then extend on this code.

I have have good experience on such problems, so if you don't find anything productive online then please revert back as I can give a detailed algorithm with explanation to you.

-
I have already searched for resources online. After reading the material I could find, I did not succeed to implement the example I gave. That's why I am here, I would like to learn how the gradient based optimization methods would work for multivariate data starting from the beginning and assuming the (simple?) example I gave. –  user55285 Jan 5 '13 at 17:26
Fine! I will get back to you with a detailed document/explanation in some time until then you can look on the gradient descent code, whose link I gave to you. As I have seen this code is quite self explanatory! –  Akshay Jan 5 '13 at 17:33