I'm trying to understand a cryptanalysis of a variant of the Hill cipher using an unkown alphabet through a known-plaintext attack.
The classic Hill cipher use an $n\times n$ inversible matrix $K = \begin{vmatrix} a & b \\ c & d \\ \end{vmatrix} $ as the cipher key : the plaintext data is split into vectors of size $n$ using the alphabet $$ \begin{matrix} A & B & C & \cdots & Y & Z \\ \hline \ 0 & 1 & 2 & \cdots & 24 &25 \end{matrix} $$ (e.g. "YAZ" would become $\begin{pmatrix}24\\0\\25\end{pmatrix}$) and for each vector $P$ we calculate the encrypted block $Q$ with $$Q = K P \mod 26$$ ($26$ being the alphabet size). Knowing the key matrix, one can decipher the message $Q$ via $$P = K^{-1}Q \mod 26.$$
I've read a ton of documents dealing with this classic Hill cipher : as long as an attacker have at least $n^2$ plaintext/cipher pairs ($n$ beeing the key matrix size) he can easily set-up a linear system which should be resolvable.
My problem here is to extend these results to a Hill cipher using an permutation of the previous alphabet, e.g.
$$ \begin{matrix} A & B & C & \cdots & Y & Z \\ \hline \ x_a & x_b & x_c & \cdots & x_y & x_z \end{matrix} $$
where $\{x_a, x_b, \cdots, x_z\}$ is a permutation of the set $\{0, 1, ..., 25\}$.
The key required to decipher an encrypted message is now composed of the key matrix $K$ and the alphabet, thus the equations are not anymore linear : if $f$ is the function associating a single character to its alphabet value (e.g. $f(A) = x_a$) the typical system to resolve given a plaintext input "AB" and the ciphered output "XY" in order to find the $2\times 2 $ key matrix $K$ and the alphabet becomes $$\{ \begin{eqnarray} f(X) = a f(A) + b f(B) \mod 26 \\ f(Y) = c f(A) + d f(B) \mod 26 \end{eqnarray} $$
How can such a cipher be reversed using known-plaintext attack (I know multiple $(A, B)$ and $(X, Y)$ pairs and want to find $a, b, c, d$ and $f^{-1}$) ? I've found an old research article (it's behind a paywall so I could just read the front page unfortunately) stating that such a non-linear system could be reduced to a linear form using a larger set of unknowns, but I'm not able to figure out the process which leads the author to these conclusions. Can someone give me a hint ?
