What is the definition of rotation tensor R? The book A First Course in Continuum Mechanics says the rotation tensor, R, is implicit in F.
The matrix presentation of rotation is here.
However, I am interested in its tensor representation.
How can you define rotation tensor R?
The question is discussed in connection with right stretch tensor U which is defined by $U = \sqrt{F^T F}$ where $F$ is the deformation gradient.
It is possible that this can help to define the rotation tensor.
 A: It takes a few steps to get to $\mathbf{R}$ given the deformation gradient $\mathbf{F}$. Here's what I would do:
1) Compute the right Cauchy-Green tensor : $\mathbf{C} = \mathbf{F}^T\mathbf{F} = \mathbf{UR}^T\mathbf{RU}=\mathbf{U}^2$. Note that $\mathbf{C}$ is a symmetric, positive-definite tensor.
2) Next compute the eigenvalues and eigenvectors of $\mathbf{C}$ (let's call them $\omega_i$ and $\mathbf{r}_i$). Recall that the eigenvectors of $\mathbf{C}$ are the same as those of $\mathbf{U}$, and that the eigenvalues of $\mathbf{U}$ are the square roots of the eigenvalues of $\mathbf{C}$.
3) Now, compute $\mathbf{U}=\sqrt{\mathbf{C}}$ by using the spectral decompotition of $\mathbf{U}$.
$$
\mathbf{U} = \sum_{i=1}^3 \sqrt{\omega_i} \left( \mathbf{r}_i \otimes \mathbf{r}_i \right)
$$
4) After this, return to your polar decomposition and compute
$$
\mathbf{R} = \mathbf{FU}^{-1}
$$
The algebra can be a bit tedious, but once you know the process, it's pretty mechanical to get to the answer. Most continuum mechanics textbooks should have a rigorous proof of everything I just showed. One of the best I've read is called "The Mechanics and Thermodynamics of Continua". There's only one book out there by that title, so a quick search should find it.
A: One way to define a rotation matrix $R$ is so that for any vector $x$, the vector $Rx$ satisfies $|Rx| = |x|$.  Here $Rx$ represents the vector $(\sum_j R_{ij} x_j)_i$, and $|x| = \sqrt{\sum_i |x_i|^2}$.  To exclude reflections, add the requirement that $\text{det}(R) = 1$.  So a rotation matrix is a matrix that preserves the distance between points.  (I'm not quite sure what you mean by the difference between a tensor and a matrix, since a matrix is a rank two tensor.)
A book that I think really covers this well is "Fundamentals of Matrix Computations" by D. Watkins.  Read the beginning of the chapter on the least squares method/QR factorization.  Also, look at the chapter on the SVD decomposition, and I think that will help you understand the "stretching tensor."  I know this book is really meant for people writing code, but I think it explains some fundamental linear algebra concepts better than many linear algebra textbooks.
A: I would define the rotation tensor R in the context of continuum mechanics as
Any square matrix $F$ can be presented as $F = RU$, where $R$ is unitary and $U$ is positive definite symmetric matrix.
This presentation is useful because the rotation of the object (= choice of coordination) does not affect energy. 
