Matrix multiplication using Galois field $$\begin{bmatrix}1 &1 &6\\4& 3& 2\\5 &2& 2\\5& 3& 4\\4& 2& 4\end{bmatrix}\begin{bmatrix}4\\5\\6\end{bmatrix} = \begin{bmatrix}3\\5\\4\\3\\2\end{bmatrix}. $$
I am not getting that how come this result is possible ?
[Editor's comment #1: The question makes sense, but the asker forgot to explain their notation - possibly because they have not been exposed to any alternatives (happens regrettably often when programmers and/or telecommunication majors are introduced to finite fields). Below please find an elaboration of my educated guess, JL.]
Here the notation is using a common way of writing polynomials with binary coefficients as integers. We are working over the field $GF(2^3)$ aka $\Bbb{F}_8$ defined as $\Bbb{F}_2[\alpha]$, where $\alpha$ is a zero of an irreducible cubic. We then compactly represent an arbitrary element 
$$
z=a_0+a_1\alpha+a_2\alpha^2\in GF(8), \ a_0,a_1,a_2\in GF(2),
$$
as the sequence of bits
$$
z=a_2a_1a_0
$$
that is then (internally to the computer program) stored as the integer
$$
i(z)=(a_2a_1a_0)_2
$$
in base two. 
For example, the element $\alpha^2+\alpha$ is converted to $6$, because
$$
\alpha^2+\alpha=1\cdot\alpha^2+1\cdot\alpha^1+0\cdot\alpha^0=110_2=6.
$$
[Editor's comment #2: A popular choice is that $\alpha$ is a zero 
of the polynomial $x^3+x+1$. In other words, we have the equation
$$
\alpha^3+\alpha+1=0.
$$
Unless I made a mistake, the first component of the matrix product above matches with this minimal polynomial in the sense that the calculation
$$
1\cdot4+1\cdot5+6\cdot6=1\cdot\alpha^2+1\cdot(1+\alpha^2)+(\alpha+\alpha^2)^2=\alpha+1=3,$$
is true when $\alpha^3+\alpha+1=0$, JL.]
 A: Posting the details of the calculations for the first couple of matrix entries. I use the assumption that $\alpha$ is a zero of the primitive polynomial $x^3+x+1$, and hence $\alpha$ is generator of the multiplicative group of $GF(8)$. Below we have a table of base-$\alpha$ discrete logarithms of non-zero elements of $GF(8)$
$$
\eqalign{
\alpha^0&=1&&=\color{red}1,\\
\alpha^1&=\alpha&&=\color{red}2,\\
\alpha^2&=\alpha^2&&=\color{red}4,\\
\alpha^3&=1+\alpha&&=\color{red}3,\\
\alpha^4&=\alpha+\alpha^2&&=\color{red}6,\\
\alpha^5&=1+\alpha+\alpha^2&&=\color{red}7,\\
\alpha^6&=1+\alpha^2&&=\color{red}5,\\
\alpha^7&=1.
}$$
where I use $\color{red}{\mathrm{red}}$ to list how an element of $GF(8)$ is stored as a bitfield interpreted as an integer (see my comments in the OP).
I refer you to my on-site table of discrete logarithms for the derivation of this table as well as first examples of its use. A reader can view this answer as another example in the use of discrete log tables for carrying out arithmetic in a binary field.
For the first entry we have
$$
\begin{aligned}
\color{red}{1\cdot4+1\cdot5+6\cdot6}&=1\cdot\alpha^2+1\cdot(\alpha^2+1)+(\alpha^2+\alpha)\cdot(\alpha^2+\alpha)\\
&=\alpha^2+(\alpha^2+1)+\alpha^4\cdot\alpha^4\\
&=1+\alpha^8\\
&=1+\alpha=\color{red}{3}.
\end{aligned}
$$
For the second entry it goes like
$$
\begin{aligned}
\color{red}{4\cdot4+3\cdot5+2\cdot6}&=\alpha^2\cdot\alpha^2+(\alpha+1)\cdot(\alpha^2+1)+\alpha\cdot(\alpha^2+\alpha)\\
&=\alpha^4+\alpha^3\cdot\alpha^6+\alpha\cdot\alpha^4\\
&=\alpha^4+\alpha^9+\alpha^5\\
&=\alpha^4+\alpha^2+\alpha^5\\
&=(\alpha+\alpha^2)+\alpha^2+(1+\alpha+\alpha^2)\\
&=1+\alpha^2=\color{red}{5}.
\end{aligned}
$$
For the third entry 
$$
\begin{aligned}
\color{red}{5\cdot4+2\cdot5+2\cdot6}&=(1+\alpha^2)\cdot\alpha^2+\alpha\cdot(\alpha^2+1)+\alpha\cdot(\alpha^2+\alpha)\\
&=\alpha^6\cdot\alpha^2+\alpha\cdot\alpha^6+\alpha\cdot\alpha^4\\
&=\alpha^8+\alpha^7+\alpha^5\\
&=\alpha+1+\alpha^5\\
&=\alpha+1+(1+\alpha+\alpha^2)\\
&=\alpha^2=\color{red}{4}.
\end{aligned}
$$
Leaving the last two as exercises. Ask, if you have problems.
