Find basis so Transformation Matrix will be diagonal $e_1,e_2$ will be basis for $V$. $W$ has a basis $\{e_1+ ae_2,2e_1+be_2\}$. Choose an $a,b$ s.t. that the basis for $W$ will have a transformation matrix $T$ will be in diagonal form.
$T(e_1) = 1e_1+5e_2$
$T(e_2) = 2e_1+4e_2$
$V$ and $W$ are linear spaces of dimension $2$.
 A: With your assumptions you have: $\mathcal{T}$:$\mathcal{V}$->$\mathcal{W}$.
If you choose a=5, b=4 then $\mathcal{T}$=$\mathcal{I}$
A: In the basis for $V$,
$$T_{[V]} = \left(\begin{matrix}
1 & 2 \\
5 & 4 \end{matrix} \right)$$
If you want the transformation $T$ written in $W$'s basis to be diagonal, then you want each basis vector of $W$ to be mapped to some multiple of itself:
$$T_{[W]} = \left(\begin{matrix}
\lambda_1 & 0 \\
0 & \lambda_2 \end{matrix} \right)$$
You know that $T : e_1 \mapsto e_1 + 5 e_2$ and $T : e_2 \mapsto 2e_1 + 4e_2$. Using this, you can solve for $a$ and $b$ by stipulating that:
$$e_1 + a e_2 \mapsto \lambda_1 e_1 + \lambda_1 a e_2,$$ 
$$2 e_1 + b e_2 \mapsto \lambda_2 2 e_1 + \lambda_2 b e_2$$
Appling the map $T$:
$$T_{[V]} \left( \begin{matrix}
1 \\
a \end{matrix} \right) = \lambda_1 \left( \begin{matrix}
1 \\
a \end{matrix} \right) = \left( \begin{matrix}
1 \\
5 \end{matrix} \right) + a\left( \begin{matrix}
2 \\
4 \end{matrix} \right)$$
$$T_{[V]} \left( \begin{matrix}
2 \\
b \end{matrix} \right) = \lambda_2 \left( \begin{matrix}
2 \\
b \end{matrix} \right) = 2\left( \begin{matrix}
1 \\
5 \end{matrix} \right) + b\left( \begin{matrix}
2 \\
4 \end{matrix} \right)$$
We've ended up with simultaneous equations for $\lambda_1$ and $\lambda_2$:
$$\begin{cases}
\lambda_1 = 1 + 2a \\
a \lambda_1 = 5 + 4a
\end{cases},\quad \quad \begin{cases}
2\lambda_2 = 2 + 2b \\
b \lambda_2 = 10 + 4b
\end{cases}$$
Rearranging these equations gives two quadratics:
$$2a^2 - 3a - 5 = 0$$
$$b^2 - 3b - 10 = 0$$
Our solutions are $a = 5/2$ or $-1$ and $b = 5$ or $-2$. Any of these four possible configurations will make $T$ written in $W$'s basis be a diagonal matrix.
As an example, set $a = -1, b = 5$. Then:
$$\underbrace{\left( \begin{matrix}
1 & 2 \\
5 & 4 \end{matrix} \right)}_{T_{[V]}} \left( \begin{matrix}
1 & 2 \\
-1 & 5 \end{matrix} \right) = \left( \begin{matrix}
-1 & 12 \\
1 & 30 \end{matrix} \right)$$
It can be seen that the first basis vector of $W$ is an eigenvector of $T$ with eigenvalue of $-1$, and the second is an eigenvector of $T$ with eigenvalue $6$.
Thus:
$$T_{[W]} = \left( \begin{matrix}
-1 & 0 \\
0 & 6 \end{matrix} \right)$$
With this choice for $a$ and $b$.
A: It helps to write this up in matrices. If we use the standard basis $\{e_1,e_2\}$ for $V$ and $W$, the linear transform $T:V\to W$ is given by
$$
T = \begin{pmatrix}
1 & 2 \\ 5 & 4
\end{pmatrix}.
$$
Next, the matrix
$$
P = \begin{pmatrix}
1 & 2 \\ a & b
\end{pmatrix}
$$
takes vectors written in the $\{e_1 + a e_2, 2 e_1 + b e_2\}$ basis and returns them in the standard $\{e_1,e_2\}$ basis. (To see this, observe that in the $\{e_1 + a e_2, 2 e_1 + b e_2\}$ basis, the vector ${1\choose0}$ represents the element $e_1 + a e_2$ which is written ${1\choose a} = P{1\choose 0}$ in the $\{e_1,e_2\}$ basis.)
So $P^{-1}$ takes vectors written in the $\{e_1,e_2\}$ basis and returns them in the $\{e_1 + a e_2, 2 e_1 + b e_2\}$ basis. 
If we want $T:V \to W$ to be diagonal in the $\{e_1 + a e_2, 2 e_1 + b e_2\}$ basis, this is the same as requiring that $P^{-1}T$ be diagonal.
Now use the formula for matrix inverses to get $P^{-1}$, multiply this with $T$ to get $P^{-1}T$, and solve for $a$ and $b$ such that the resulting matrix is diagonal.
As some of the commenters to the question have observed, setting $a=5, b=4$ gives $T =P$, so $P^{-1}T = I$, which is diagonal. However, this approach works for more general cases where $T$ may not be so nice.
Edit: I realize I might have been hasty in assuming we want $T:V\to W$. If we really want $T:W \to W$ to be diagonal, then we should solve for $a,b$ such that $P^{-1}TP$ is diagonal, or equivalently $$TP = P\begin{pmatrix}* & 0 \\ 0 & *\end{pmatrix}.$$
A: you want to find $a, b$ so that $$T(e_1+ae_2) = k_1(e_1+ae_2), 
T(2e_1 + be_2) = k_2(2e_1 + be_2) \text{ for some } k_1, k_2. \tag 1  $$
we are given $$Te_1 = e_1 + 5e_2, Te_2 = 2e_1 + 4e_2. \tag 2 $$ subbing $(2)$ in $(1)$ and equating the coefficients, we get $$ 1 + 2a = k_1, 5 + 4a = k_1a, 2+2b = 2k_2, 10 + 4b = k_2b \tag 3 $$
that is $$5+4a = a(1+2a)=a + 2a^2\to 2a^2-3a - 5 = 0 \to 
a = -1, 5/2.$$  and $$10 + 4b = b(1+b) \to b^2 - 3b -10 = 0\to b = -2, 5. $$
A: The matrix 
$$T=
\begin{bmatrix}
1&2\\
5&4
\end{bmatrix}
$$
 has eigenvalues: $\lambda_1=-1$ and $\lambda_2=6$, with corrisponding eigenspaces:
$$
\lambda_1=1 \rightarrow v_1=
\begin{bmatrix}
x\\
-x
\end{bmatrix}
$$
$$
\lambda_2=6 \rightarrow v_2=
\begin{bmatrix}
2y\\
5y
\end{bmatrix}
$$
So we can find a diagonal form of $T$ with any transformation that has as new basis two vectors in these eigenspaces. I.e. matrices of the form:
$$P=
\begin{bmatrix}
x&2y\\
-x&5y
\end{bmatrix}
$$
or
$$P'=
\begin{bmatrix}
2y&x\\
5y&-x
\end{bmatrix}
$$
Since we want a transformation matrix of the form
$$Q=
\begin{bmatrix}
1&2\\
a&b
\end{bmatrix}
$$
we have two possibilities:
$$
P=
\begin{bmatrix}
1&2\\
-1&5
\end{bmatrix}
\Rightarrow a=-1 \quad \land \quad b=5
$$
or
$$
P'=
\begin{bmatrix}
1&2\\
\dfrac{5}{2}&-2
\end{bmatrix}
\Rightarrow a=\dfrac{5}{2} \quad \land \quad b=-2
$$
