Property of order 2 automorphism of simple Lie algebra

If $$\sigma$$ is an automorphism of simple Lie algebra, of order 2, such that we fixed : $$\sigma (x_\beta)=-y_\beta$$ for each simple root of a given base $$\Delta$$, is it also necessarily true that $$\sigma (x_\gamma)=-y_\gamma$$ for nonsimple root $$\gamma$$ ? (Humphreys Lie algebra book, exercise 14.4)

And this is what I tried :

I first looked for a counter-example with $$sl(3,\mathbb{C})$$, 6 roots, a base contains 2 simple roots and I called the third (positive) one (with height 2) $$\alpha_3$$ : but after some calculations I found that : $$\sigma (x_{{\alpha}_3})=-y_{{\alpha}_3}$$ . $$sl(3,\mathbb{C})$$ was not a counter-example.

So I decided to try to prove that the property is true : we know that $$[L_{\alpha}L_{\beta}]=L_{\alpha+\beta}$$. Suppose $$\alpha$$ and $$\beta$$ are two simple roots, and call $$\gamma={\alpha+\beta}$$. Let's choose $$x_\gamma=[x_{\alpha}x_{\beta}]$$. If we prove the property for $$x_\gamma$$, we're done by induction on height of the root.

$$\sigma(x_\gamma)=[\sigma(x_\alpha)\sigma(x_\beta)]=[-y_{\alpha}-y_\beta]=[y_{\alpha}y_\beta]$$

We know that $$y_\gamma=t[y_{\alpha}y_\beta]$$ for some t. But is $$t$$ necessarily $$-1$$ ? I'm stuck here. Thank you for help.

• IIRC that's an annoying but unavoidable technicality. It depends on how you define the $x_\alpha$ and $y_\alpha$. The issue becomes quite clear once you write out explicitly e.g. the $3\times 3$ matrices in $\mathfrak{sl}_3$ and see where the minus sign comes in when bracket-multiplying. Commented Feb 17, 2022 at 16:54
• $x_\alpha \in L_\alpha$ and $y_\alpha \in L_-\alpha$ such that $[x_{\alpha}y_\alpha]=h_\alpha \in H$ ($H$ is a Cartan subalgebra) and $Sp(x_\alpha,y_\alpha,h_\alpha)$ is isomorphic to $sl_2$ . @TorstenSchoeneberg, thank you for your comment Commented Feb 17, 2022 at 19:30
• Well that alone does not determine either the $x_\alpha$ or the $y_\alpha$ to the level needed for deciding this question. The isomorphism to $\mathfrak{sl}_2$ has to send them to specifically chosen elements. Note also that even when you say $x_\gamma= [x_\alpha, x_\beta]$, and we care about signs, you are making a choice which is no longer invariant under flipping $\alpha$ and $\beta$. Commented Feb 17, 2022 at 20:00

Too long for a comment, and can maybe be upgraded to an answer: When we do the calculations in $$\mathfrak{sl}_3$$, and call

$$x_\alpha := \pmatrix{0&1&0\\0&0&0\\0&0&0}, \qquad x_\beta :=\pmatrix{0&0&0\\0&0&1\\0&0&0}$$

$$y_\alpha := \pmatrix{0&0&0\\1&0&0\\0&0&0}, \qquad y_\beta :=\pmatrix{0&0&0\\0&0&0\\0&1&0}$$

$$h_\alpha := \pmatrix{1&0&0\\0&-1&0\\0&0&0}=[x_\alpha, y_\alpha], \qquad h_\beta :=\pmatrix{0&0&0\\0&1&0\\0&0&-1} =[x_\beta, y_\beta]$$

then the issue is that $$[x_\alpha, x_\beta]= \pmatrix{0&0&1\\0&0&0\\0&0&0}$$ but $$[y_\alpha, y_\beta] = \pmatrix{0&0&0\\0&0&0\\\color{red}{-1}&0&0}$$, consequently

$$[[x_\alpha, x_\beta],[y_\alpha, y_\beta]]= \pmatrix{\color{red}{-1}&0&0\\0&0&0\\0&0&\color{red}{1}}$$ which does not act on $$[x_\alpha, x_\beta]$$ as $$h_{\alpha+\beta}$$ (which should be $$=h_\alpha + h_\beta$$) usually should. So if you want things to be consistent, and you want to call $$x_\gamma= [x_\alpha, x_\beta]$$ (in this order), then you should call $$y_\gamma = \color{red}{-}[y_\alpha, y_\beta] = [y_\beta, y_\alpha]$$.

(Too make things more annoying, some sources would call $$y_\alpha$$ and $$y_\beta$$ the negatives of my $$y_\alpha$$ and $$y_\beta$$ above. But then you get the same sign issue just with now an unexpected $$+$$ instead of a $$-$$.)

You say you checked what the order two automorphism $$A \mapsto -A^T$$ (negative transpose) does. It looks good if we follow the above definitions.

The point is you can choose $$x_\gamma$$ wherever you want in $$L_\gamma$$ ($$=\{x\in L| [hx]=\gamma(h)x$$ for all $$h \in H\}$$) for some nonsimple root $$\gamma$$. But there exists, for this $$x_\gamma$$, a unique $$y_\gamma \in L_{-\gamma}$$ such that $$\{x_\gamma,y_\gamma,[x_{\gamma}y_\gamma]\}$$ will form a $$sl_2$$-triple (Humphreys, prop. 8.4). In your example, if we are choosing $$x_\gamma=[x_{\alpha}y_\beta]$$ or $$x_\gamma=[x_{\beta}y_\gamma]$$ we don't have a counter-example because $$\sigma(x_\gamma)=-y_\gamma$$ in both cases.

However, if we take : $$x_\gamma = \pmatrix{0&0&2\\0&0&0\\0&0&0}$$ then $$y_\gamma =\pmatrix{0&0&0\\0&0&0\\\frac{1}{2}&0&0}$$ so that $$[{x_\gamma}y_\gamma]=h_\alpha+h_\beta$$ (with your notation) and $$\{x_\gamma,y_\gamma,h_\alpha+h_\beta\}$$ is a $$sl_2-$$triple

But here : $$\sigma(x_\gamma)=\sigma(2[{x_\alpha}x_\beta])=2[{y_\alpha}y_\beta]\ne -y_\gamma$$ This indeed seems to be a counter-example !

• Good point! Yes, if this is an allowed choice for the $x_\gamma$, then this works. Note that this scaling through of an originally chosen $x_\alpha$ with $k$ and $y_\alpha$ with $k^{-1}$ is no longer possible if we demand the $x_\alpha$ to be in a Chevalley basis, or other normalizations of Cartan-Weyl bases, cf. encyclopediaofmath.org/wiki/Cartan-Weyl_basis . It seems like if we impose the condition $N_{\alpha, \beta}=−N_{-\alpha, -\beta}$ (which I was getting at with my comment-answer), and use the terminology $y_\alpha=x_{-\alpha}$, then the assertion is true, right? Commented Feb 18, 2022 at 17:04
• Yes @TorstenSchoeneberg, you're right ! $$\sigma(x_{\alpha+\beta})=\frac{1}{N_{\alpha,\beta}}\sigma([x_{\alpha}x_\beta])=\frac{1}{N_{\alpha,\beta}}[x_{-\alpha}x_{-\beta}])=\frac{-N_{\alpha,\beta}}{N_{\alpha,\beta}}x_{-\alpha-\beta}=-x_{-\alpha-\beta}$$ Thank you for your comment. Now I understand your answer. In Humphreys exercise we don't deal with Chevalley basis. Commented Feb 20, 2022 at 17:21