is there any statistical test for independence in this situation? I wonder if there is a statistical test which fits my need.
Let $X=(X_1,...,X_n) ,Y=(Y_1,...,Y_n)$ be $n$-dimensional random vectors.
Is there any statistical test to check whether $X_1,...,X_n$ are more mutually independent than $Y_1,...,Y_n$?
 A: As @Robert Israel already stated, this depends on what you mean by "more mutually independent".
In the strict sense of probability theory, random variables are either independent or not. So, the only way $X_1,X_2$ can be more mutually independent than $Y_1,Y_2$ is if the former are independent and the latter aren't.
But if you formally define "independence" to be some quantitative criterion, this will generally get you a test.
Which definition is the right one depends on your problem.
For example, you could be interested in whether the mean-squared correlation in the first group is larger than in the second.
You might use the following statistic:
$$
T=\frac{2}{n(n-1)}\left(\sum_{i=1}^n\sum_{j\neq i}E[(X_i-EX_i)(X_j-EX_j)]^2-\sum_{i=1}^n\sum_{j\neq i}E[(Y_i-EY_i)(Y_j-EY_j)]^2\right)
$$
The empirical version of $T$, say $t$, can be calculated readily.
If $t$ is significantly different from zero, you could say with high probability that one group is more dependent than the other, according to this definition.
Which distribution it follows under $H_0$ depends on your model.
In general, by quantitatively defining what it means to be "more independent", you create a mapping from the random variables $(X_1,\dots,X_n)$ into $\mathbb{R}$.
So you'll get two "independence scores" $T(X_1,\dots,X_n)$ and $T(Y_1,\dots,Y_n)$, one of which will be (weakly) larger.
To test whether this difference is significant, you could then perform a statistical test under distributional assumptions adequate to your situation.
