# Stabilizing controls in linear quadratic regulator

I am studying a linear quadratic control problem with discounting. For $$\gamma \in (0,1)$$, $$Q \succeq 0$$ and $$R \succ 0$$ and linear dynamics $$s_{t+1}=As_t + B a_t$$, let the total cost starting in state $$s_0$$ and using the control $$a_t = \theta s_t$$ be:

$$J_{\theta}(s_0) = \sum_{t=0}^\infty \gamma^t \left( s_t^\top Q s_t + a_t^\top R a_t \right) = s_0^\top \left[ \sum_{t=0}^\infty \gamma^t ((A+B\theta)^t)^\top (Q + \theta^\top R \theta) (A+B\theta)^t \right]$$

Assume $$\theta$$ to be a stabilizing control, i.e. $$\rho(A+B\theta) < 1$$. I want to show the following:

Let $$\theta'$$ be another linear control such that $$J_{\theta'}(s_0) < J_{\theta}(s_0)$$ for all starting states $$s_0$$. Then, is it true that $$\theta'$$ is also stabilizing. That is, is $$\rho(A+B\theta') < 1$$? If so, how does one prove this kind of result? The discount factor $$\gamma$$ is stumping me.

Edit: What I am studying here is called Policy Iteration in dynamic programming. Also assuming the optimal controller, $$\theta^*$$ is stable. Policy iteration produces an 'improved policy' in terms of costs from every state but I am not sure if this policy is stable.

It is not true that $$\theta'$$ has to be a stabilizing state feedback controller. This would even not be true without the discounting factor $$\gamma^t$$.

For example consider

$$A = \begin{bmatrix} 2 & 0 \\ 0 & 0.5 \end{bmatrix}, \quad B = \begin{bmatrix} 1 \\ 1 \end{bmatrix},$$

$$Q = \begin{bmatrix} 0 & 0 \\ 0 & 1 \end{bmatrix}, \quad R = 1, \quad s_0 = \begin{bmatrix} 1 \\ 0 \end{bmatrix}.$$

It can be shown that $$\theta =\begin{bmatrix}-2 & 0\end{bmatrix}$$ is a stabilizing controller and would yield some positive total cost. However, if you would use $$\theta' =\begin{bmatrix}0 & 0\end{bmatrix}$$ one would get that the total cost is zero.

Things don't have to change if one would also include the constraint that $$(A,Q)$$ is detectable. For example consider the following scalar system with $$A=2$$, $$B=1$$, $$Q=1$$, $$R=1$$ and $$\gamma=¼$$. Because the system is scalar by using the geometric series the cost function can be simplified to

\begin{align} J_\theta(s_0) &= y_0^2 (Q + \theta^2 R) \sum_{k=0}^\infty \left(\gamma (A + B\,\theta)^2\right)^k, \\ &= \frac{y_0^2 (Q + \theta^2 R)}{1 - \gamma (A + B\,\theta)^2}, \\ &= \frac{y_0^2 (1 + \theta^2)}{1 - ¼(2 + \theta)^2}. \end{align}

In order for the geometric series to converge it should hold that $$|¼(2 + \theta)^2|<1$$ which implies $$-4<\theta<0$$. It can be shown that in that interval $$J_\theta(s_0)$$ is minimized at $$\theta^*=¼(1-\sqrt{17})\approx -0.78$$. It can be noted that $$\rho(A+B\,\theta^*)\approx 1.22 \nless1$$ and since $$\theta^*$$ is the minimizer of $$J_\theta(s_0)$$ it follows that any stabilizing $$\theta$$ yields a larger value for $$J_\theta(s_0)$$.

• Hi @kwin van der Veen, I am assuming $\theta'$ to be such that it improves the cost from all states, not just a particular $s_0$. For example, let's assume the state space to be entire $\mathbb{R}^2$ or a compact subset of $\mathbb{R}^2$. – Jalaj May 24 at 0:30
• @Jalaj my second counter example should then still hold. I suspect that for higher order systems one can also come up with similar counter examples. However, for higher order systems the cost function is not as easy to evaluate. – Kwin van der Veen May 24 at 5:01
• The second example you give has an unstable optimal controller. Consider the situation where I have a stable controller and then I come up with another controller that improves costs from all states (I am coming from the Dynamic programming perspective where this is called 'Policy Iteration/Improvement'). It can be shown that Policy Iteration improves costs from all starting states. What is unclear if the 'improved policy' is stable. – Jalaj 11 hours ago
• @Jalaj it doesn't have to be that the improved cost function yields a stable control policy as shown by my second example. – Kwin van der Veen 43 mins ago