There are infinitely many equivalent irreducible representations of $SO(3)$ on $\mathbb R^3$ The irreducible representation of $SO(3)$ on $\mathbb R^3$ is the set of the matrices $M$ such that $MM^T=I$ and  $\det(M)=1$. But this is not the only one, indeed if $A$ is an invertible matrix then the set of the matrices $A^{-1}MA$ is isomorphic to the previous one. Thus the set of the matrices $A^{-1}MA$ is also an irreducible representation of $SO(3)$ on $\mathbb R^3$.
What I mean is that I can find infinite sets of $3 \times 3$ matrices that are irreducible representations of $SO(3)$ on $\mathbb R^3$ and, even if all these sets are the same up to an isomorphism, only the elements of the set $M$ have the meaning of rotation transformations.  Before I noticed this fact I thought that the irreducible representation of $SO(3)$ on $\mathbb R^3$ was unique and it was the set of the rotations matrix, but now I'm confused because that set is one of many. 
So how do I tell what is the meaning of the irreducible representations of $SO(3)$ on $\mathbb R^3$? If I consider the orthogonal matrices with determinant equal to one they act on $\mathbb R^3$ as rotation transformations but the matrices $AMA^1$ act on $\mathbb R^3$ differently
 A: When we say that the 3-dimensional irreducible representation of $SO(3)$ is unique this is exactly what we mean. Precisely, it means that if you have two sets of 3-dimensional matrices that act as a group representation then you know automatically that those two sets are isomorphic. They may be distinct as sets but they are still identical as representations.
This applies even to vector spaces that are very different. Consider the following two vector spaces. The first is just $\mathbb{R}^3$. The second space is a subset of real functions on a sphere, spanned by the real spherical harmonics that have $\ell = 1$, $\mathcal{Y_1} = \operatorname{span}(Y_1^0(\theta,\phi),Y_1^{+1}(\theta,\phi),Y_1^{-1}(\theta,\phi))$. This is also a three-dimensional vector space, but clearly a very different set than the first. Both vector are representations of $SO(3)$ in the sense that there are linear operators on these spaces that represent the group. The orthogonal matrices you mention in your question are the operators for the first group. The operators that act on the second group are a little more abstract, but for every rotation $g$ that sends $(\theta,\phi)\mapsto (\theta',\phi')$ we can find a matrix much that $Y_1^m(\theta',\phi') = X^m_{\,\,m'}Y_1^{m'}(\theta,\phi)$
Despite the apparent difference between these two sets of matrices, $M$ and $X$, they are isomorphic as representations. That is, there is a map $T$ from $\mathbb{R^3}$ to $\mathcal{Y}_1$ that we can use to map between the two different sets of matrices: $M_g = T^{-1}X_g T$.
This is pretty abstract but it's important. It allows us to say that even though $\mathbb R^3 $ and $\mathcal{Y}_1$ are very different on some levels they are exactly the same when it comes to their behavior under rotations. Being clear about the difference between representations as sets and representations as abstract categories of behavior under rotations is important to getting a deeper understanding of representation theory.
A: The set of matrices with those properties is unique and contains infinitely many matrices. If you apply the transformation you described you stay in the same set, so I don't see why you get a family of representations rather than just the one.
The argument can be made once we fix a set of generators: the set of such matrices is generated by a finite number of matrices. And these generators might not be mapped to themselves under the transformation you propose. However, how to choose the generators is up to you. So there is no contradiction there as well. Just like for $\mathrm{SU}(2)$ sometimes we use $J^1,J^2,J^3$ and sometimes we use $J^+,J^-,J^3$.
What you found goes under the name of automorphism. If the matrix $A$ is itself a matrix of $\mathrm{SO}(3)$, then this automorphism is called inner.

A valid point was raised in the comments and I was indeed too quick. Call $f_A$ the map
$$
f_A\colon M \mapsto A^{-1}MA\,.
$$
Now $f_A$ as a map on $\mathrm{SO}(3)$ maps 
$
M^T\mapsto f_A(M^T) = A^{-1}M^TA
$, which is different than $f_A(M)^T$. But we note that $f_A(M)^T$ satisfies a different equation
$$
f_A(M)^T \,A^TA\,\,f_A(M)= A^TM^T(A^{-1})^T A^T A A^{-1}M A  = A^TA
$$
So let us call $\mathrm{SO}(3)$ the set
$$
\mathrm{SO}(3)\equiv\{M \in M_{3\times3}(\mathbb{R})\;\colon M^TM=\mathbb{1}\,,\det M = 1\}\,,
$$
and $\mathrm{SO}(3)_A$ the set
$$
\mathrm{SO}(3)_A\equiv\{M \in M_{3\times3}(\mathbb{R})\;\colon M^TA^TAM=A^TA\,,\det M = 1\}\,.
$$
It is evident that the map $f_A\colon \mathrm{SO}(3)\to \mathrm{SO}(3)_A$ is an isomorphism between these two spaces. So do we get two different inequivalent $\mathrm{SO}(3)$'s? Of course not, the former is the space of isometries of the standard scalar product on $\mathbb{R}^3$, while the second is the space of isometries of the bilinear form $A^TA$, which is symmetric by construction and non degenerate by hypothesis. Since the form is non degenerate, there is always a change of basis where $\mathrm{SO}(3)_A$ is the same as $\mathrm{SO}(3)$.
