I was reading this paper and could need some help understanding an inequality in Sec. 3.2.

The starting point is the following matrix $A_{i}$ from Eq. (13):

$$A_{i} = QW_{i}^Q{W_{i}^{K}}^{T}K^{T},$$ where $Q, K\in \mathbb R^{l\times d_m}$, $W_{i}^Q, W_i^K \in \mathbb R^{d_m \times d_k}$, and hence $A_i \in \mathbb R^{l\times l}$.

Now, the paper defines $W_i^{QK} := W_i^{Q}{W_{i}^{K}}^{T} \in \mathbb R^{d_m\times d_m}$ and makes the following chain of inequalities in Eq. (14):

$$\text{rank}(W_i^{QK}) \leq \min(\text{rank}(W_i^Q, \text{rank}(W_i^K)) \leq \min(d_m, d_k) = d_k,$$ since $d_k = d_m / h$, where $h > 0$.

From the inequality they conclude that $\text{rank}(A_i) \leq d_k$, but I don't quite see how the conclusion follows.



You must log in to answer this question.

Browse other questions tagged .