0
$\begingroup$

I am studying a linear quadratic control problem with discounting. For $\gamma \in (0,1)$, $Q \succeq 0$ and $R \succ 0$ and linear dynamics $s_{t+1}=As_t + B a_t$, let the total cost starting in state $s_0$ and using the control $a_t = \theta s_t$ be:

$$ J_{\theta}(s_0) = \sum_{t=0}^\infty \gamma^t \left( s_t^\top Q s_t + a_t^\top R a_t \right) = s_0^\top \left[ \sum_{t=0}^\infty \gamma^t ((A+B\theta)^t)^\top (Q + \theta^\top R \theta) (A+B\theta)^t \right] $$

Assume $\theta$ to be a stabilizing control, i.e. $\rho(A+B\theta) < 1$. I want to show the following:

Let $\theta'$ be another linear control such that $J_{\theta'}(s_0) < J_{\theta}(s_0)$ for all starting states $s_0$. Then, is it true that $\theta'$ is also stabilizing. That is, is $\rho(A+B\theta') < 1$? If so, how does one prove this kind of result? The discount factor $\gamma$ is stumping me.

Edit: What I am studying here is called Policy Iteration in dynamic programming. Also assuming the optimal controller, $\theta^*$ is stable. Policy iteration produces an 'improved policy' in terms of costs from every state but I am not sure if this policy is stable.

$\endgroup$
1
$\begingroup$

It is not true that $\theta'$ has to be a stabilizing state feedback controller. This would even not be true without the discounting factor $\gamma^t$.

For example consider

$$ A = \begin{bmatrix} 2 & 0 \\ 0 & 0.5 \end{bmatrix}, \quad B = \begin{bmatrix} 1 \\ 1 \end{bmatrix}, $$

$$ Q = \begin{bmatrix} 0 & 0 \\ 0 & 1 \end{bmatrix}, \quad R = 1, \quad s_0 = \begin{bmatrix} 1 \\ 0 \end{bmatrix}. $$

It can be shown that $\theta =\begin{bmatrix}-2 & 0\end{bmatrix}$ is a stabilizing controller and would yield some positive total cost. However, if you would use $\theta' =\begin{bmatrix}0 & 0\end{bmatrix}$ one would get that the total cost is zero.

Things don't have to change if one would also include the constraint that $(A,Q)$ is detectable. For example consider the following scalar system with $A=2$, $B=1$, $Q=1$, $R=1$ and $\gamma=¼$. Because the system is scalar by using the geometric series the cost function can be simplified to

\begin{align} J_\theta(s_0) &= y_0^2 (Q + \theta^2 R) \sum_{k=0}^\infty \left(\gamma (A + B\,\theta)^2\right)^k, \\ &= \frac{y_0^2 (Q + \theta^2 R)}{1 - \gamma (A + B\,\theta)^2}, \\ &= \frac{y_0^2 (1 + \theta^2)}{1 - ¼(2 + \theta)^2}. \end{align}

In order for the geometric series to converge it should hold that $|¼(2 + \theta)^2|<1$ which implies $-4<\theta<0$. It can be shown that in that interval $J_\theta(s_0)$ is minimized at $\theta^*=¼(1-\sqrt{17})\approx -0.78$. It can be noted that $\rho(A+B\,\theta^*)\approx 1.22 \nless1$ and since $\theta^*$ is the minimizer of $J_\theta(s_0)$ it follows that any stabilizing $\theta$ yields a larger value for $J_\theta(s_0)$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Hi @kwin van der Veen, I am assuming $\theta'$ to be such that it improves the cost from all states, not just a particular $s_0$. For example, let's assume the state space to be entire $\mathbb{R}^2$ or a compact subset of $\mathbb{R}^2$. $\endgroup$ – Jalaj May 24 at 0:30
  • $\begingroup$ @Jalaj my second counter example should then still hold. I suspect that for higher order systems one can also come up with similar counter examples. However, for higher order systems the cost function is not as easy to evaluate. $\endgroup$ – Kwin van der Veen May 24 at 5:01
  • $\begingroup$ The second example you give has an unstable optimal controller. Consider the situation where I have a stable controller and then I come up with another controller that improves costs from all states (I am coming from the Dynamic programming perspective where this is called 'Policy Iteration/Improvement'). It can be shown that Policy Iteration improves costs from all starting states. What is unclear if the 'improved policy' is stable. $\endgroup$ – Jalaj 11 hours ago
  • $\begingroup$ @Jalaj it doesn't have to be that the improved cost function yields a stable control policy as shown by my second example. $\endgroup$ – Kwin van der Veen 43 mins ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.