Does the "Riccati LMI" implies boundedness? Assume a matrix $P\succeq0$ satisfies the "Riccati LMI"
\begin{align}
\begin{pmatrix}
FPF^T - P + M & FPH^T + S\\
HPF^T+S^T& HPH^T + I
\end{pmatrix}\succeq0
\end{align}
with $M\succ0$ and arbitrary $S$. Is the matrix $P$ has to be bounded when the pair $(F,H)$ is detectable?
Detectability def.: There exists a matrix K s.t. $\rho(F-KH)<1$ ($\rho$ is the spectral radius).
An alternative (simpler?) formulation is that for all eigenvectors $Fv=\lambda v$ with $|\lambda|\ge1$, $Hv\neq0$.
Detectability is necessary:
Choose $F=2,H=0,S=0,M=1$, then the LMI is
\begin{align}
\begin{pmatrix}
P + 1 & 0\\
0& 1
\end{pmatrix}\succeq0,
\end{align}
and is satisfied with arbitrarily large $P$.
 A: The claim is true, even without assuming $M,P\succeq0$.  I do, however, need that $\rho(P)=\|P\|_2$; this is true as long as you mean "finite-dimensional matrix" when you say "matrix", but fails for operators in general.
For simplicity of notation, let $$B=\begin{bmatrix}1&-K\end{bmatrix}\begin{bmatrix}M&S\\S^{\mathsf{T}}&1\end{bmatrix}\begin{bmatrix}1\\-K^{\mathsf{T}}\end{bmatrix}$$ and $$J=F-KH\text{;}$$ we know $\rho(J)<1$.  Also, let $r(P)=\sup_{\|w\|\leq1}{w^{\mathsf{T}}Pw}$ denote the numerical range of $P$; by property 13 in the link, $r(P)$ is unbounded iff $P$ is.
Testing the Riccati LMI against $\begin{bmatrix}1\\-K^{\mathsf{T}}\end{bmatrix}v$ and rearranging, we have \begin{align*}
v^{\mathsf{T}}Pv-v^{\mathsf{T}}Bv&\leq v^{\mathsf{T}}(F-KH)P(F-KH)^{\mathsf{T}}v \\
&=v^{\mathsf{T}}JPJ^{\mathsf{T}}v \\
&\leq r(P)\rho(J)^2\|v\|^2\tag{1}
\end{align*}  Inequality (1) is the only step in this proof that requires $\rho(P)=\|P\|_2$.
Now take a sequence of unit vectors $\{v_n\}_n$ such that $v_n^{\mathsf{T}}Pv_n\to r(P)$; by passing to a subsequence, we may also assume that $v_n^{\mathsf{T}}Bv_n\to c$ (for some $c\in\mathbb{R}$).  Taking limits in (1) and rearranging, $$r(P)(1-\rho(J)^2)\leq c$$  This bounds $r(P)$.
(This proof also shows a little more once you assume $P\not\preceq0$.  Since $r(P)\geq0$, we must have $c\geq0$, which implies that $B$ is not negative semidefinite either.  One should then have $S$ "smaller" than $M$…but that's another question.)
