Is there a closed form for this geometric-like series of two matrices? Obviously, if for a square matrix $A$ we have $|A|<1$, then $\sum_{i=0}^\infty A^i = (I-A)^{-1}$ which is defined. Obviously also holds for a product of two (same-sized, square) matrices $AB$ with an analogous norm condition: $\sum_{i=0}^\infty (AB)^i = (I-AB)^{-1}$.
But since $A, B$ do not necessarily commute, I am now left wondering what happens for a sum $\sum_{i=0}^\infty A^i B^i$, or a sum $\sum_{i=0}^\infty A^i C B^i$ where $A, B$ are still both square but not the same size, and there's a $C$ in the middle to bridge the size gap. Clearly this sum will converge if $|A|, |B| < 1$ both since matrix norms are (taken to be) submultiplicative.
Are there closed form expressions for these sums, and what are these series called? I believe there may be a Kronecker product or vectorization of a matrix involved but I don't know the details.
 A: There is a closed-form formula if you allow vectorisation:
\begin{aligned}
\operatorname{vec}\left(\sum_kA^kCB^k\right)
&=\sum_k\operatorname{vec}\left(A^kCB^k\right)
=\sum_k\left((B^k)^T\otimes A^k\right)\operatorname{vec}(C)
=\sum_k(B^T\otimes A)^k\operatorname{vec}(C)\\
&=(I-B^T\otimes A)^{-1}\operatorname{vec}(C).
\end{aligned}
A: As shown in the comments this sum is a representation of the solution of an almost Sylvester type equation (with certain assumptions on the invertibility of $A,B$). I will not delve into the solution of the Sylvester equation further, but I would like to show that there is a meaningful way to perform the infinite sum using the eigenvalue decompositions of the matrices (whenever they exist), although it doesn't have a pretty form in terms of the operations mentioned. Assume that the matrices can be decomposed as follows
$$A=S\Lambda S^{-1}~,~ B=TMT^{-1}$$
with $\Lambda_{ab}=\lambda_a \delta_{ab}$ , $M_{ab}=\mu_a \delta_{ab}$ diagonal matrices with the eigenvalues of A and B as entries. Writing the sum in component form one can show that, whenever the sum converges,
$$\sum_{n=0}^\infty (A^n B^n)_{ad}=S_{ab}\frac{(S^{-1}T)_{bc}}{1-\lambda_b \mu_c}T^{-1}_{cd}$$
The middle matrix could be potentially written as a Hadamard product between two matrices, but I don't see any conceptual simplification in rewriting it this way.
A solution of the Sylvester equation given any matrices $A,B$ is presented in a detailed study here, which includes studying the Jordan decomposition of $A,B$.
The second sum can be performed similarly:
$$\sum_{n=0}^\infty (A^n B^n)_{ad}=S_{ab}\frac{(S^{-1}CT)_{bc}}{1-\lambda_b \mu_c}T^{-1}_{cd}$$
It would be interesting to see if there is an expression of this sum in terms of some sort of BCH type formula or an compact expression involving the commutator of $[A,B]$.
