# Efficient evaluation of the inverse of a triangular matrix on a vector

I have this matrix that interests me. It arises when we try to express the norm of a $(p,p)$-form on an $n$-dimensional vector space in terms of (squares of) traces of the form with respect to the Lefschetz operator on the space. It's coefficients are these: $$a_{jk} = \begin{cases} (-1)^k\tbinom{p-k}j \tbinom{n-j-k-p}{n-2p} \tbinom{n-k-p+j}{j} & \hbox{ if j + k \leq p,} \\ 0 & \hbox{else.} \end{cases}$$ Here $j,k$ run between $0$ and $p$, so this is a $(p+1)\times(p+1)$ matrix. (We assume $2p \leq n$ so we don't run into trouble in the binomial coefficients.) For $p = 1$ it looks like this: $$A = \begin{pmatrix} n-1 & -1 \\ n & 0 \end{pmatrix}$$ In general $A$ is a triangular matrix (we could rearrange the bases here so it would be upper-triangular - wrong, but it wasn't important) and thus has a triangular inverse (its determinant is always nonzero).

I really want to calculate $v = A^{-1}(1,\ldots,1)$. I hope (against hope) that the mess of binomial coefficients in the result simplifies (there's some evidence for this when $p=1,2$) but to see that I have to be able to actually calculate $v$. I stress that I really don't care about $A^{-1}$ itself or its value on any other vector than $(1,\ldots,1)$.

Does anyone have an idea how to do this? Calculating $A^{-1}$ directly is probably out of the question, but maybe there's a nice trick to get the value of an inverse on a given vector?

• I don't think you can make, say, $\pmatrix{0&1\cr-1&0\cr}$ upper-triangular by changing the basis (unless you're willing to go complex). – Gerry Myerson Nov 9 '13 at 23:06
• Solving a triangular system... so this would be a substitution method? – Algebraic Pavel Nov 9 '13 at 23:09
• Gerry: Oops. No matter, at least the inverse is easy to calculate. :) Pavel: I suppose, but actually carrying it out with those coefficients would be monstrous. – Gunnar Þór Magnússon Nov 9 '13 at 23:28
• In the end I think Cramer's rule (see en.wikipedia.org/wiki/Cramer's_rule) and brute force calculations do what I want since the matrix is (skewed) triagonal. – Gunnar Þór Magnússon Nov 10 '13 at 0:24