When can we express a matrix in a smaller basis? Let's suppose that we have two orthonormal basis, $V=\{|v_i\rangle \}$, $i=1,...n$ (ket notation) and $U=\{|u_j\rangle\}$, $j=1,..,m$, with $m<n$ (i.e. $\dim U< \dim V$), where $|u_j\rangle=\sum_{i=1}^{n}c_i^j |v_i\rangle$. Given that $[A]_V$ is the matrix representation of some transformation $A$ in the basis $V$, under which conditions $[A]_U$ is the matrix representation of the same transformation $A$ in the basis $U$?
$$[A]_{U}=\left[
\begin{array}{c}
\langle u_1| \\
 \vdots\\
\langle u_m|
\end{array} \right] [A]_{V}\left[
\begin{array}{c}
|u_1\rangle & \cdots & |u_m\rangle|
\end{array} \right]$$
 A: A linear transformation $A$ acts on a vector space $X$. If the $X$ is finite dimensional, then every basis of $X$ has the same number of vectors (which is equal to $\dim X$). Hence, your question doesn't make sense from the start: you cannot have two (orthonormal) bases of $X$ with different numbers of basis vectors.
A: There $\nexists$ a linear transformation between two Vector Spaces with different dimensions. If there was, it would be a homomorphism and two Vector Spaces $M, N$ are homomorphic $\iff \dim M = \dim N$. 
A: I figured out the answer to my badly posed question, so please allow me to post it.
For simplicity I use $n=3$ and $m=2$. The basis vectors $|v_i\rangle$ are orthonormal and the new basis vectors $|u_j\rangle$ are defined as linear combintions of $|v_i\rangle$:
$$|u_j\rangle=\sum_{i=1}^{3}c_i^j|v_i\rangle$$
In order them to be orthonormal too, the condition is: if $c_i^1\neq 0$ then $c_i^2 = 0$, $\forall i$. Let's suppose, without loss of generality, that
\begin{align}
|u_1\rangle &=c_1|v_1\rangle+c_2|v_2\rangle=\left[
\begin{array}{c}
c_1 \\
c_2\\
0
\end{array} \right]_V\tag{1}\\
|u_2\rangle &=c_3|v_3\rangle=\left[
\begin{array}{c}
0 \\
0\\
c_3
\end{array} \right]_V \tag{2}
\end{align}
Next, consider some operator $A$, with the following matrix representation in the basis $V$:
$$[A]_{V}=\left[
\begin{array}{ccc} A_{11} & A_{12} & A_{13}  \\
A_{21} & A_{22} & A_{23}\\
A_{31} & A_{32} & A_{33}
\end{array} \right]_V=\sum_{i,j=1}^{3}A_{ij}|v_i\rangle\langle v_j|$$
where $A_{i,j}\equiv \langle v_i|A|v_j \rangle$, $i,j=1,2,3$. 
Let's suppose that we can write:
$$[A]_{U}=\left[
\begin{array}{cc} A'_{11} & A'_{12}  \\
A'_{21} & A'_{22}
\end{array} \right]_U=\sum_{i,j=1}^{2}A'_{ij}|u_i\rangle\langle u_j|$$
where $A'_{i,j}\equiv \langle u_i|A|u_j \rangle$, $i,j=1,2$. From $(1)$ and (2) we can express $|u_i\rangle \langle u_j|$ in terms of $|v_k\rangle \langle v_{\ell}|$, $k,\ell=1,2,3$. Therefore
\begin{align}
[A]_{V} &=A'_{11}\left(|c_1|^2|v_1\rangle \langle v_1|+c_1 c_2^*|v_1\rangle \langle v_2|+c_2 c_1^*|v_2\rangle \langle v_1|+|c_2|^2|v_2\rangle \langle v_2|\right)\\
&+A'_{12}\left(c_1 c_3^*|v_1\rangle \langle v_3|+c_2 c_3^*|v_2\rangle \langle v_3|\right)\\
&+A'_{21}\left(c_3 c_1^*|v_3\rangle \langle v_1|+c_3 c_2^*|v_3\rangle \langle v_2|\right)\\
&+A'_{22}|c_3|^2|v_3\rangle \langle v_3|\\
\end{align}
and we get the following conditions:
\begin{align}
\frac{A_{11}}{|c_1|^2}&=\frac{A_{12}}{c_1 c_2^*}=\frac{A_{21}}{c_2 c_1^*}=\frac{A_{22}}{|c_2|^2}=A'_{11} \\
\frac{A_{13}}{c_1 c_3^*}&=\frac{A_{23}}{c_2 c_3^*}=A'_{12} \\
 \frac{A_{31}}{c_3 c_1^*}&=\frac{A_{32}}{c_3 c_2^*}=A'_{21}\\
 \frac{A_{33}}{|c_3|^2}&=A'_{22}
\end{align}
If they are all satisfied, we can express $A$ in the basis $U$:
$$
[A]_U=\left[
\begin{array}{c}
\langle u_1| \\
\langle u_2|
\end{array} \right]_V [A]_V \hspace{0.2cm}\left[
\begin{array}{cc}
|u_1\rangle & |u_2\rangle
\end{array} \right]_V=\left[
\begin{array}{cc} A'_{11} & A'_{12}  \\
A'_{21} & A'_{22}
\end{array} \right]_U$$
This of course can be generalized to any $n$ and $m$, $n>m$.
Edit: I spotted the mistake in my answer: the vectors $|u_i\rangle$, $i=1,2$ of course do not form another basis of the $3\times 3$ vector space $V_A$ on which the operator $A$ acts, $A:V_A\to V_A$. They form an incomplete orthogonal set, $S\subset V_A$ (which is a plane in $V_A$). If the mentioned conditions are all satisfied, this just means that $A$ projects all the vector of $V_A$ to $S$, $A:V_A\to S$. But this doesn't mean that we can write $A$ as $2\times 2$ matrix because this would restrict its action to $S$, i.e. $A:S\to S$, which is not true for $A$, because by definition it acts on whole space $V_A$.
