Geometric center of convex set of positive semidefinite matrices

Consider the set of $$d\times d$$ matrices that are positive semidefinite and have unit trace. This is a convex set, $$S$$. Is it possible to think of a geometric center of this set? The criterion for geometric center is an element $$\rho$$ such that if $$\rho + X\in S$$, then $$\rho - X\in S$$

For $$d = 2$$, it is the identity matrix. For $$d>2$$, it is not the identity matrix. Indeed $$\begin{pmatrix} 1 & 0 & 0\\ 0 & 0 &0 \\ 0 & 0 & 0 \end{pmatrix} = \frac{1}{3}\begin{pmatrix} 1 & 0 & 0\\ 0 & 1 &0 \\ 0 & 0 & 1 \end{pmatrix} + \begin{pmatrix} 2/3 & 0 & 0\\ 0 & -1/3 &0 \\ 0 & 0 & -1/3 \end{pmatrix} \in S$$ but $$\frac{1}{3}\begin{pmatrix} 1 & 0 & 0\\ 0 & 1 &0 \\ 0 & 0 & 1 \end{pmatrix} - \begin{pmatrix} 2/3 & 0 & 0\\ 0 & -1/3 &0 \\ 0 & 0 & -1/3 \end{pmatrix} \notin S$$

So is it possible to find such an element for $$d>2$$ or if not, why not?

When $$d\ge3$$, there cannot be any geometric centre. Since the trace of a matrix and the property of being a centre are preserved by unitary similarity, we may consider only the case where $$\rho$$ is a nonnegative diagonal matrix.
If $$\rho=\frac1nI$$, then a similar example to yours shows that $$\rho$$ is not a centre.
If $$\rho$$ contains two different diagonal entries $$a>b$$ instead, $$\rho$$ is also not a centre because $$\pmatrix{a\\ &b\\ &&c\\ &&&\ddots} +\pmatrix{-\frac{a+b}2\\ &\frac{a+b}2\\ &&0\\ &&&\ddots} =\pmatrix{\frac{a-b}2\\ &\frac{a+3b}2\\ &&c\\ &&&\ddots}\succeq0$$ but $$\pmatrix{a\\ &b\\ &&c\\ &&&\ddots} -\pmatrix{-\frac{a+b}2\\ &\frac{a+b}2\\ &&0\\ &&&\ddots} =\pmatrix{\frac{3a+b}2\\ &\frac{b-a}2\\ &&c\\ &&&\ddots}\nsucceq0.$$
• Thank you, that's a nice proof. However, could you add on a little about whether there exists a better way to define the center? For instance math.stackexchange.com/questions/1260067/… and math.stackexchange.com/questions/1377209/… talk about this "center" of a convex set. Is there an operation that satisfies: If $C \odot X\in S \implies C\odot^{-1} X \in S$, where I have replaced + and - in my original question with some operation $\odot$ and a natural inverse operation $\odot^{-1}$ – user1936752 Feb 28 at 11:31