A confusion about rotation group. Definition: Let $G$ be a group and $S$ a set. Suppose to each group element $g$ there corresponds a function $f_g: S\to S$ in such a way that $f_g(f_h(s))=f_{gh}(s)$ for any $s\in S$ and any ${g,h}\subset G$. Then we say that the group $G$ acts on $S$. In this case we will write $gs$ or $g\cdot s$ instead of $f_g(s)$.
Suppose a group $G$ acts on sets $S_1$ and $S_2$(in particular, $S_1$ and $S_2$ may coincide). Then there is a natural way to define the action of $G$ on the set of functions $f:S_1\to S_2$:$$g\cdot f(s)=(g\cdot f)(g\cdot s)$$
If I have a 3-D rotation matrix group $SO(3)$. Let $A$ be a $3\times3$ matrix, which can also be viewed as a linear transformation on $\Bbb R^3$. How to prove that $R \cdot A=RAR^{-1} $ for $R\in SO(3)$ ?
 A: Recall a function $S_1\to S_2$ (or any relation) is formalized as a subset of $S_1\times S_2$. If $G$ acts on both domain and codomain then it acts on functions by replacing every point $(x,y)$ on its graph with the point $(gx,gy)$. That is, $(g\cdot f):gx\mapsto gy$ whenever $f:x\mapsto y$. Or as you put it,
$$(g\cdot f)(gs)=gf(s) \quad \textrm{for all }s\in S_1.$$
(I will use concatenation of letters for $G$ acting on $S_1$ and $S_2$ but use $\cdot$ for $G$'s action on the functions $S_1\to S_2$ so this doesn't get too confusing.)
As $s$ varies over all of $S_1$, so does $gs$ because $g$ acts bijectively. Substitute $\sigma=gs$ so that
$$(g\cdot f)(\sigma)=gf(g^{-1}\sigma) \quad \textrm{for all }\sigma\in S_1. $$
This is precisely what you wanted to prove as a special case with matrices.
A: Pick an arbitrary vector $\mathbf{x} \in \mathbb{R}^3$ and set $\mathbf{y} = A\mathbf{x}$. A rotation sends the vector $\mathbf{x}$ to $R\mathbf{x}$, and the vector $\mathbf{y}$ to $R\mathbf{y}$.
Just as the linear transformation $A$ sends $\mathbf{x}$ into $\mathbf{y}$, we would like the new "rotated" transformation $A'$ to send $R\mathbf{x}$ into $R\mathbf{y}$. So we have 
$$A'R\mathbf{x} = R\mathbf{y} = RA\mathbf{x}$$
Can you continue from here?
