1
$\begingroup$

Consider following two optimization problems where a quadratic order objective function needs to be minimized as given below:

$$\min_\textbf{x} f(\textbf{x}) = \textbf{x}^TA\textbf{x} + \textbf{b}^T \textbf{x} \tag{1}$$

$$\min_\textbf{x} g(\textbf{x}) = \textbf{x}^TC\textbf{x} + \textbf{d}^T \textbf{x} \tag{2}$$

Let say, I wish to optimize them with some optimization algorithm that has some hyper-parameters (for eg. gradient descent with learning rate as one hyper parameter). Is it to possible to define some relation between $(A, b)$ and $(C, d)$ variables such that same hyper-parameter will give "good" (possibly optimal) solution on both the problems? Can this be extended to general convex optimization problems?

$\endgroup$
8
  • $\begingroup$ What are the domains of $f$ and $g$? $\endgroup$
    – Rodrigo de Azevedo
    Commented Jul 28, 2019 at 21:38
  • $\begingroup$ @RodrigodeAzevedo I meant the objectives to represent a general optimization problem where x could be any discrete or continuous vector. I was primarily looking for any general notion that might define the above mentioned "similarity". Please let me know if I am not clear enough. $\endgroup$
    – randomprime
    Commented Jul 28, 2019 at 21:42
  • $\begingroup$ With minor modifications to $f$ and $g$, the minimizers will be the solutions of the linear systems $A x = b$ and $C x = d$. How close they are is then linear algebra. $\endgroup$
    – Rodrigo de Azevedo
    Commented Jul 28, 2019 at 21:49
  • $\begingroup$ @RodrigodeAzevedo This "linear algebra" is the part I am not aware of. Let say, if norm of (A, b) and (C, d) are quite close, would that mean the optimization will be quite similar. Taking the gradient descent analogy further, would that mean same learning rate, momentum parameter etc. will work good enough for both problems then? $\endgroup$
    – randomprime
    Commented Jul 28, 2019 at 21:55
  • 1
    $\begingroup$ I think, hyper parameters are related to structure of the optimization surface. For example, in the given linear system case, the singular values determine the surface. So if the largest and smallest singular values are similar, i.e. their sums are equal for both A and C, then the optimal step sizes will be equal. As for general convex cases, you can check the Hessian matrices of the f(x) and g(x) functions, if use first order methods; but as for non-convex cases, I do not think that we can find a way to measure similarity. $\endgroup$
    – Kurban
    Commented Jul 31, 2019 at 9:07

0

Reset to default

You must log in to answer this question.