$Y´=QY$, How come this to be the derivative? 
Let $Q=(q_{ij})(i,j=1,...,n)$ be a square matrix of (infinitely differentiable real-valued ) functions $q_{ij}$ of a real variable. Let $Y=(y_1,...,y_n)^t$ denote a column vector of functions. Let $S_Q$ be the set of all $Y$ such that:
$Y´=QY$.
Show that $S_Q $is a vector space(over $\mathbb{R}$). Let $\Phi=(\varphi_1,...,\varphi_n)$ be a column vector of functions, and let
$\langle \Phi,Y\rangle=\varphi_1y_1+...+\varphi_ny_n$.
Let $D$ be the derivative. Define $D$ applied to a matrix of functions to be the matrix obtained by applying $D$ to each component. Show that:
$D\langle \Phi,Y\rangle=\langle D\Phi,Y\rangle+\langle \Phi,DY\rangle=\langle (D+Q^t)\Phi,Y\rangle$.
By induction, prove that for any positive integer k,
$D^k\langle\rangle=\langle (D+Q^t)^k\Phi,Y\rangle$.

The first proof was simple and I accomplished it. Because it is too lengthy I am not going to write it down(definition of Vector space).
I was able to prove $D\langle \Phi,Y\rangle=\langle D\Phi,Y\rangle+\langle \Phi,DY\rangle=\langle (D+Q^t)\Phi,Y\rangle$.
First I recalled the product derivative rule $(f(x)g(x))´=f(x)´g(x)+f(x)g(x)´$.
Since $\langle \Phi,Y\rangle=\varphi_1y_1+...+\varphi_ny_n$ is assumed.
We have $D\langle \Phi,Y\rangle=D(\varphi_1y_1+...+\varphi_ny_n)=(\varphi_1y_1+...+\varphi_ny_n)´=\\\varphi_1´y_1+\varphi_1y_1´+...+\varphi_n´y_n+\varphi_ny_n´=\langle D\Phi,Y\rangle+\langle \Phi,DY\rangle$
Now that we have, by assumption we know $Y´=QY$
$\langle D\Phi,Y\rangle+\langle \Phi,DY\rangle=\langle D\Phi,Y\rangle+\langle \Phi,QY\rangle=\langle D\Phi,Y\rangle+\langle Q^t\Phi,Y\rangle=\langle (D+Q^t)\Phi,Y\rangle$
The third proof follows from the last one.
Despite the fact ,I guess, I was able to answer the questions. I do not understand the following:
Questions:
1) Is $S_Q$ a subspace of the matrix $Q$? What is the matrix $Q$?
2) What is the intuition behind the derivative $Y´=QY$? Why multiply matrix $Q$, if it is not the differential operator matrix?
 A: Lets start with your second question.
Let $\mathfrak{F}(\Bbb R)$ be the collection of smooth functions $\Bbb R\to\Bbb
R$ and let $Q$ be a $n\times n$ matrix over $\mathfrak{F}(\Bbb R)$. This means
that $Q$ is of the form
$$
Q=
\begin{bmatrix}
q_{11} & q_{12} & \dotsb & q_{1n} \\
q_{21} & q_{22} & \dotsb & q_{2n} \\
\vdots & \vdots & \ddots & \vdots \\
q_{n1} & q_{n2} & \dotsb & q_{nn} \\
\end{bmatrix}
$$
where each $q_{ij}$ is a smooth function $q_{ij}:\Bbb R\to\Bbb R$.
Next, let $\{y_1,\dotsc,y_n\}$ be functions $\Bbb R\to\Bbb R$ satisfying
$Y^\prime=QY$ where
$$
Y=
\begin{bmatrix}
  y_1 \\ y_2 \\ \vdots \\ y_n
\end{bmatrix}\tag{1}
$$
Note that the equation $Y^\prime=QY$ gives the system of first-order
differential equations
$$
\begin{array}{rcrcrcrcrcrcrcrcrc}
  y_1^\prime & = & q_{11}\cdot y_1 & + & q_{12} \cdot y_2 & + & \dotsb & + & q_{1n}\cdot y_n \\
  y_2^\prime & = & q_{21}\cdot y_1 & + & q_{22} \cdot y_2 & + & \dotsb & + & q_{2n}\cdot y_n \\
  \vdots     &   & \vdots          &   & \vdots           &   & \ddots &   & \vdots          \\
  y_n^\prime & = & q_{n1}\cdot y_1 & + & q_{n2} \cdot y_2 & + & \dotsb & + & q_{nn}\cdot y_n \\
\end{array}\tag{$\ast$}
$$
So, solving $Y^\prime=QY$ is equivalent to solving the system ($\ast$).

Example. Let $Q=\begin{bmatrix}  x^2 & e^x \\  4   & \sin(x)\end{bmatrix}$. Then $Y^\prime=QY$ is the system of first-order differential equations $$\begin{array}{rcrcrcrcrcrcrcrcrc}  y_1^\prime & = & x^2\cdot y_1 & + & e^x\cdot y_2 \\  y_2^\prime & = & 4\cdot y_1  & + & \sin(x)\cdot y_2\end{array}$$

Now, note that any given $Y$ of the form (1) is an element of the vector space
$\mathfrak{F}(\Bbb R)^n$. Let $S_Q$ be the collection of all $Y$ satisfying
$Y^\prime=QY$. We wish to show that $S_Q$ is a subspace of $\mathfrak{F}(\Bbb
R)^n$. To do so, we may use the one-step vector subspace test, noting that
$S_Q\neq\varnothing$ since $\mathbf{0}\in S_Q$. To use the one-step vector
subspace test, suppose that $Y,Z\in S_Q$ and that $\lambda\in\Bbb R$. Then
$$
(Y+\lambda\cdot Z)^\prime
= Y^\prime + \lambda\cdot Z^\prime
= QY + \lambda\cdot QZ
= Q(Y+\lambda\cdot Z)
$$
Hence $Y+\lambda\cdot Z\in S_Q$, proving that $S_Q$ is indeed a subspace of
$\mathfrak{F}(\Bbb R)^n$.
So, to summarize, the answers to your questions are:


*

*$S_Q$ is not ``a subspace of the matrix $Q$'' (this language does not make sense). Rather, $S_Q$ is a subspace of the vector space $\mathfrak{F}(\Bbb R)^n$.

*The equation $Y^\prime=QY$ is a succinct way to write the system of first-order linear differential equations ($\ast$).
