Could someone explain the notation of the average of quaternions equation? The equation has some notation that is difficult to find the meaning for. It is equation (3) in the paper 'Quaternion Averaging' by F. Landis Markley, et al. on page 3 under 'The Average Quaternion'. Here is the equation (forgive the poor Latex):

Using the definition of the Frobenius norm, the orthogonality of $A(\mathbf{q})$ and $A(\mathbf{q}_i)$, and some properties of the matrix trace (denoted by $\text{Tr}$) gives

$$
\begin{eqnarray}
\lVert A(\mathbf{q}) - A(\mathbf{q}_i) \rVert ^2_F && = \text{Tr } \{[A(\mathbf{q}) - A(\mathbf{q}_i)]^T[A(\mathbf{q}) - A(\mathbf{q}_i)]\} \\ 
&& = 6-2 \text{ Tr }[A(\mathbf{q})A^T(\mathbf{q}_i)]
\end{eqnarray}
$$
Specifically, with reference to $A(\mathbf{q})$, the brackets suggest this is a function, but $A$ would usually be the symbol for a matrix. $\mathbf{q}$ is likely a quaternion, so maybe a quaternion matrix?
Also, with reference to $A(\mathbf{q}_i)$, the $_i$ would be used in summation, but there is a lack of summation in the equation itself that would use it. Elsewhere in the paper it suggests $\mathbf{q}_i$ is a set of n attitude estimates, so a set of quaternions?
Curly braces would usually denote a set, but is it hard to tell whether ${[A(\mathbf{q}) - A(\mathbf{q}_i)]^T[A(\mathbf{q}) - A(\mathbf{q}_i)]}$ is a set.
Likewise, square brackets may denote any number of things, including equivalence class, or the floor, and so on.
 A: Each $\mathbf q_i$ is an attitude estimate ("in quaternion form," so I'd say it is a quaternion).
$A(\mathbf{q})$ is a matrix. More specifically, the paper says on page 3, it is the attitude matrix of a quaternion $\mathbf{q}$.
The fact that the equation doesn't have a summation and the use of an arbitrary index $i$ also tell you that the expression is valid for any attitude estimate. The equation is plugged into a summation in equation (2). 
The expression $[A(\mathbf{q}) - A(\mathbf{q}_i)]^T[A(\mathbf{q}) - A(\mathbf{q}_i)]$ is the product of two matrices; one is the transpose of the matrix $A(\mathbf{q}) - A(\mathbf{q}_i)$ and the other is the matrix $A(\mathbf{q}) - A(\mathbf{q}_i)$. Square brackets are used, I think, to avoid nested parentheses.
The trace is a function of a matrix; a reason for the use of the curly braces could be that there are parenthesis and square brackets in the argument.

I found another paper that explains the notation (page 2, especially): http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20030093641.pdf
