Use quaternion to represent rotation matrix I know the unit quaternion can represent the 3D rotation. For example, $Rp=q*p*q^{-1}$ where $R$ is the rotation matrix of body frame with respect to inertial frame, $q$ is the unit quaternion, $*$ is the quaternion multiplication, $p$ is pure quaternion whose first element is $0$ and rest elements are a $3 \times 1$ vector.
Right now I have a formula like $RDR^{T}$ where $R$ is the rotation matrix of body frame with respect to inertial frame, $D$ is a $3 \times 3$ coefficient matrix. Could I use quaternion $q$ to represent this formula in a similar way that I showed in my above example?
 A: Given a pure rotation matrix $\mathbf{R}$, its inverse is $\mathbf{R}^{-1} = \mathbf{R}^T$.  Let
$$\mathbf{R} = \left[ \begin{matrix}
X_x & X_y & X_z \\
Y_x & Y_y & Y_z \\
Z_x & Z_y & Z_z \\
\end{matrix} \right] , \quad \mathbf{D} = \left[ \begin{matrix}
x_x & x_y & x_z \\
y_x & y_y & y_z \\
z_x & z_y & z_z \\
\end{matrix} \right]$$
Then,
$$\mathbf{R} \mathbf{D} \mathbf{R}^{-1} = \left[ \begin{matrix} \chi \\ \gamma \\ \zeta \end{matrix} \right]$$
where
$$\begin{array}{rclclcl}
\chi   & = & x_x X_x X_x & + & x_y Y_x X_y & + & x_z Z_x X_z \\
 ~     & + & y_x X_x Y_x & + & y_y Y_x Y_y & + & y_z Z_x Y_z \\
 ~     & + & z_x X_x Z_x & + & z_y Y_x Z_y & + & z_z Z_x Z_z \\
\gamma & = & x_x X_y X_x & + & x_y Y_y X_y & + & x_z Z_y X_z \\
 ~     & + & y_x X_y Y_x & + & y_y Y_y Y_y & + & y_z Z_y Y_z \\
 ~     & + & z_x X_y Z_x & + & z_y Y_y Z_y & + & z_z Z_y Z_z \\
\zeta  & = & x_x X_z X_x & + & x_y Y_z X_y & + & x_z Z_z X_z \\
 ~     & + & y_x X_z Y_x & + & y_y Y_z Y_y & + & y_z Z_z Y_z \\
 ~     & + & z_x X_z Z_x & + & z_y Y_z Z_y & + & z_z Z_z Z_z \\
\end{array}$$
The $*$ in $q * p * q^{-1}$ refers to Hamilton product, which is very different from matrix product, and for unit quaternion $q = q_r + q_i\mathbf{i} + q_j\mathbf{j} + q_k\mathbf{k}$ (i.e. where $q_r^2 + q_i^2 + q_j^2 + q_k^2 = 1$) and a general quaternion $p = p_r + p_i\mathbf{i} + p_j\mathbf{j} + p_k\mathbf{k}$,
$$\begin{aligned}
q * p * q^{-1} &= p_r ( q_r^2 + q_i^2 + q_j^2 + q_k^2 ) \\
  ~            &~+ \biggr( p_i ( q_r^2 + q_i^2 - q_j^2 - q_k^2 ) - 2 p_j ( q_r q_k - q_i q_j ) +2 p_k ( q_r q_j + q_i q_k ) \biggr) \mathbf{i} \\
               &~+ \biggr( p_j ( q_r^2 - q_i^2 + q_j^2 - q_k^2 ) + 2 p_i ( q_r q_k + q_i q_j ) - 2 p_k ( q_r q_i - q_j q_k ) \biggr) \mathbf{j} \\
               &~+ \biggr( p_k ( q_r^2 - q_i^2 - q_j^2 + q_k^2 ) - 2 p_i ( q_r q_j - q_i q_k ) + 2 p_j ( q_r q_i + q_j q_k ) \biggr) \mathbf{k} \\
\end{aligned}$$
Because $p$ has at most four components, you cannot simply use $q * D * q^{-1}$.
We can, however, examine how we can decompose $D$ into vectors, so we can rotate and sum those.  Since $\mathbf{R}\mathbf{D}\mathbf{R}^{-1} = (\mathbf{R}\mathbf{D})\mathbf{R}^{-1} = \mathbf{R}(\mathbf{D}\mathbf{R}^{-1})$,
$$\mathbf{D} \mathbf{R}^{-1} = \left [ \begin{matrix}
x_x X_x + y_x Y_x + z_x Z_x \\
x_y X_y + y_y Y_y + z_y Z_y \\
x_z X_z + y_z Y_z + z_z Z_z = d\\
\end{matrix} \right ]$$
we have
$$\mathbf{R} \mathbf{D} \mathbf{R}^1 = \mathbf{R} \mathbf{d}$$
and we can express the same vector using $q d q^{-1}$ (with $d$ and the result quaternions with real component zero) with $q$ corresponding to the rotation matrix $\mathbf{R}$.
Unfortunately, an unit quaternion represents a rotation or orientation, via $q = \cos(\varphi/2) + x \sin(\varphi/2)\mathbf{i} + y \sin(\varphi/2)\mathbf{j} + k \sin(\varphi/2) \mathbf{k}$, where $(x, y, z)$ is the rotation axis unit vector ($x^2 + y^2 + z^2 = 1$) and $\varphi$ is the rotation angle, and recovering the basis vector components ($X_x$, $X_y$, $X_z$, and so on) is non-trivial.
So, the answer to OP's question, if $\mathbf{R}\mathbf{D}\mathbf{R}^{-1}$ can be represented via a quaternion $q$ corresponding to the rotation matrix $\mathbf{R}$, the answer is not really, the expression would be very, very long and complicated.

Another approach to examining the question is to realize that Hamilton product between two quaternions $q = q_r + q_i \mathbf{i} + q_j \mathbf{j} + q_k \mathbf{k}$ and $p = p_r + p_i \mathbf{i} + p_j \mathbf{j} + p_k \mathbf{k}$ can be expressed as a matrix product, via
$$q * p = \left [ \begin{matrix} q_r & q_i & q_j & q_k \end{matrix} \right ] \left [ \begin{matrix}
p_r & p_i & p_j & p_k \\
-p_i & p_r & -p_k & p_j \\
-p_j & p_k & p_r & -p_i \\
-p_k & -p_j & p_i & p_r \\
\end{matrix} \right ] = \left[ \begin{matrix}
q_r & q_i & q_j & q_k \\
-q_i & q_r & q_k & -q_j \\
-q_j & -q_k & q_r & q_i \\
-q_k & q_j & -q_i & q_r \\
\end{matrix} \right] \left[ \begin{matrix} p_r \\ p_i \\ p_j \\ p_k \\ \end{matrix} \right ]$$
and for an unit quaternion $q$ and a "vector" p ($p = x \mathbf{i} + y \mathbf{j} + z \mathbf{k}$, a quaternion with real component zero),
$$q p q^{-1} = \left[ \begin{matrix}
q_r & q_i & q_j & q_k \\
-q_i & q_r & q_k & -q_j \\
-q_j & -q_k & q_r & q_i \\
-q_k & q_j & -q_i & q_r \\
\end{matrix} \right] \left[ \begin{matrix} 0 & x & y & z \end{matrix} \right ] \left[ \begin{matrix}
q_r & -q_i & -q_j & -q_k \\
q_i & q_r & -q_k & q_j \\
q_j & q_k & q_r & -q_i \\
q_k & -q_j & q_i & q_r \\
\end{matrix} \right]$$
whereas for a rotation matrix $\mathbf{R}$ (so $\mathbf{R}^{-1} = \mathbf{R}^T$) we have
$$\mathbf{R} \mathbf{D} \mathbf{R}^{-1} = \left[ \begin{matrix}
X_x & X_y & X_z \\
Y_x & Y_y & Y_z \\
Z_x & Z_y & Z_z \\
\end{matrix} \right] \left[ \begin{matrix}
x_x & x_y & x_z \\
y_x & y_y & y_z \\
z_x & z_y & z_z \\
\end{matrix} \right] \left[ \begin{matrix}
X_x & Y_x & Z_x \\
X_y & Y_y & Z_y \\
X_z & Y_z & Z_z \\
\end{matrix} \right]$$
So, the similarity is in the notation, not in the operations done.
