For $A\in\mathbb{R}^{n\times n}$ and $B\in\mathbb{R}^{n\times k}$, does $(I_{n}-aA)^{-1}Bb$ determine $a\in\mathbb{R}$ and $b\in\mathbb{R}^k$? Let $A$ be a real $n\times n$ matrix, and $B$ be a real $n\times k$
matrix of rank $k<n$, with $\mathrm{col}(B)$ (the column space of $B$) not an
invariant subspace of $A$. We assume $A\neq0,I_n$. Consider $$f(a,b)=(I_{n}-aA)^{-1}Bb,$$ for real
scalar $a\in S$ and $b\in\mathbb{R}^k$. $S$ is the set of values of $a$ such that
$(I_{n}-aA)$ is invertible. 
We say that $f(a,b)$ identifies $a$ and $b$ over $\Omega$ if $f(a,b)=f(a^{\ast
},b^{\ast})$ implies $(a,b)=(a^{\ast},b^{\ast})$ for any two pairs
$(a,b),(a^{\ast},b^{\ast})\in\Omega$.
The problem is to describe the set over which $f(a,b)$ does not identify $a$ and $b$.

What I have done so far:
Rewrite $f(a,b)=f(a^{\ast},b^{\ast})$ as
$$
B(b-b^{\ast})+AB(ab^{\ast}-a^{\ast}b)=0.\tag{1}\label{1}
$$
Thus, $f(a,b)$ identifies $a$ and $b$ over $\Omega$ if $\eqref{1}$ implies
$(a,b)=(a^{\ast},b^{\ast})$ for all $(a,b),(a^{\ast},b^{\ast})\in\Omega$.
So far I understand only two particular cases:  


*

*If $\mathrm{rank}(B,AB)=2k$, then $\eqref{1}$ is satisfied iff $b-b^{\ast}=0$ and
$ab^{\ast}-a^{\ast}b=0$, from which $(a,b)=(a^{\ast},b^{\ast})$ provided that
$b\neq0$. Hence, in this case $f(a,b)$ identifies $a$ and $b$ over
$S\times\mathbb{R}^{k}/\{0\}$.

*Partition $B$ as $(B_{1},B_{2})$ where $B_{1}$ is $n\times k_{1}$ and
$B_{2}$ is $n\times k_{2}$, with $0<k_{1}<k$, and suppose that $AB_{1}=B_{1}L$
for some $k_{1}\times k_{1}$ matrix $L$ ($\mathrm{col}(B_{1})$ is an invariant
subspace of $A$), and that $\mathrm{rank}(B,AB)=k+k_{2}.$ Then, letting
$\left(  b_{1}^{\prime},b_{2}^{\prime}\right)  $ be the partition of
$b^{\prime}$ conformable with that of $B,$ $\eqref{1}$ becomes
$$
B_{1}(b_{1}-b_{1}^{\ast}+L(ab_{1}^{\ast}-a^{\ast}b_{1}))+B_{2}(b_{2}%
-b_{2}^{\ast})+AB_{2}(ab_{2}^{\ast}-a^{\ast}b_{2})=0.
$$
Since the columns of $(B_{1},B_{2},AB_{2})$ are linearly independent,
$\eqref{1}$ is satisfied if and only if $b_{1}-b_{1}^{\ast}+L(ab_{1}^{\ast
}-a^{\ast}b_{1})=0$, $b_{2}-b_{2}^{\ast}=0$, and $ab_{2}^{\ast}-a^{\ast}%
b_{2}=0$. Combining the second and third equalities, gives $b_{2}=b_{2}^{\ast
}$ and $a=a^{\ast}$, provided that $b_{2}\neq0.$ Hence the first equality
becomes $\left(  I_{k_{1}}-aL\right)  \left(  b_{1}-b_{1}^{\ast}\right)  =0$,
which is equivalent to $b_{1}-b_{1}^{\ast}$ because $I_{k_{1}}-aL$ is
invertible for any $a\in S$. This means that $f(a,b)$ identifies $(a,b)$
provided that not all entries of $b$ associated to columns of $B$ not in
$\mathrm{col}(A)$ are zero (if they were we would essentially be in the
case when $\mathrm{col}(B)$ is an invariant subspace of $A$--in that case it is clear that
$\left(  a,b\right)  $ cannot be identified from $f(a,b)$)



Update:
Let's look at the case $k=2.$ Write $B=(B_{1},B_{2})$, so $B_1$ and $B_2$ are vectors. Assume
$\mathrm{rank}(B,AB)=\mathrm{rank}(B,AB_{2})=k+1$ (the case $\mathrm{rank}(B,AB)=2k$ is trivial, see case 1 above, and so is the case $\mathrm{rank}(B,AB)=k$), and write $$AB_{1}%
=l_{1}B_{1}+l_{2}B_{2}+l_{3}AB_{2},$$ for $l_{1},l_{2},l_{3}\in\mathbb{R}$.
From $\eqref{1}$,
\begin{equation}
\left[  \beta_{1}-\beta_{1}^{\ast}+l_{1}(a\beta_{1}^{\ast}-a^{\ast}\beta
_{1})\right]  B_{1}+\left[  \beta_{2}-\beta_{2}^{\ast}+l_{2}(a\beta_{1}^{\ast
}-a^{\ast}\beta_{1})\right]  B_{2}+\left[  a\beta_{2}^{\ast}-a^{\ast}\beta
_{2}+l_{3}(a\beta_{1}^{\ast}-a^{\ast}\beta_{1})\right]  AB_{2}=0.
\end{equation}
Since $\mathrm{rank}(B,AB_{2})=k+1,$ this equality is satisfied iff
$$\left\{
\begin{array}
[c]{c}%
b_{1}-b_{1}^{\ast}+l_{1}(ab_{1}^{\ast}-a^{\ast}b_{1})=0\\
b_{2}-b_{2}^{\ast}+l_{2}(ab_{1}^{\ast}-a^{\ast}b_{1})=0\\
ab_{2}^{\ast}-a^{\ast}b_{2}+l_{3}(ab_{1}^{\ast}-a^{\ast}b_{1})=0
\end{array}
\right.$$
As a linear system in the unknowns $b_{1}^{\ast},b_{2}^{\ast},a^{\ast}$, this
is
$$C\left[
\begin{array}
[c]{c}%
b_{1}^{\ast}\\
b_{2}^{\ast}\\
a^{\ast}%
\end{array}
\right]  =\left[
\begin{array}
[c]{c}%
b_{1}\\
b_{2}\\
0
\end{array}
\right],\tag{1}\label{2}$$
where
$$C=\left[
\begin{array}
[c]{ccc}%
1-l_{1}a & 0 & l_{1}b_{1}\\
-l_{2}a & 1 & l_{2}b_{1}\\
-l_{3}a & a & l_{3}b_{1}+b_{2}%
\end{array}
\right]  .$$
If $\mathrm{rank}(C)=3$, the only solution to the system is $\left(  b^{\ast
},a^{\ast}\right)  =\left(  b,a\right)  $. If $\mathrm{rank}(C)<3$, there may
be other solutions.
Note that
$$
\det(C)=\left(  l_{1}a-1\right)  b_{2}-\left(  l_{2}a+l_{3}\right)  b_{1}.
$$
Note that for fixed $a$, the set of $(b_1,b_2)$ such that $\det(C)=0$ is a line in $\mathbb{R}^2$. 
Incidentally note that case 2. above (with $k=2$) obtains when $l_{2}=l_{3}=0.$ In that case,
$\det(C)=\left(  l_{1}a-1\right)  b_{2}$, so $\det(C)=0$ only if $b_{2}=0$
($l_{1}a-1=0$ is impossible as $l_{1}$ must be an eigenvalue of $A$, so $l_{1}a-1=0$ would contradict invertibility of $(I_n-aA)$). That is,
in that case, we find again the result that $b,a$ is identifiable provided
that $b_{2}\neq0$.
 A: Here are some ideas.
We consider the relation $(I_n-aA)^{-1}Bb=v$ where $A,B$ and the vector $v$ are known and $a,b=[b_i]$ are unknown. Note that $n\geq k,\operatorname{rank}(B)=k$ and $1/a\notin \operatorname{spectrum}(A)$.
Then $Bb=(I-aA)v=v-aAv$, that is $Bb+aAv=v$. Let $B=[B_1,\cdots,B_k]$.
Assume that $A,B,v$ are randomly chosen. Then the $(B_i)_i,Av=w,v$ are random vectors in $\mathbb{R}^n$.
The above equation can be rewritten $\sum_{i\leq k}b_iB_i+aw=v$, that is, we want to decompose the vector $v$ on the $k+1$ vectors $(B_i)_i,w$; moreover, the decomposition must be unique. 
It is possible with probability $1$ when $n=k+1$; otherwise, it is possible with probability $0$.
When $A$ is random, it's invertible with probability $1$. Here, $A$ is not assumed to be invertible and the question is: how to choose $v$ so that the decomposition exists and is unique? In particular, the vectors $v\in\ker(A)$ are not convenient. I think that there are many such subcases and I stand prudently on the sidelines.
