A name for this property? Let $*$ be an operation such that $(xy)^* = y^*x^*$, e.g. if $x,y$ are $2\times2$ matrices and $*$ is "take the inverse" or if $x,y$ are operators and if $*$ is the adjoint.
Is there a name for such a property ?
 A: I'm not sure if there's a general name for this (other than "order-reversing homomorphism", or "a homomorphism $R \to R^{op}$", as said in the answer of G. Sassatelli)  but many operations of this nature (including both of the ones you mention) are involutions.
Other examples include:


*

*The operation of inversion in any ring: if $a$ and $b$ are invertible elements, then $(ab)^{-1}=b^{-1}a^{-1}$.

*The operation of transposition on matrices:  if $M$ and $N$ are any two matrices for which the product $MN$ is defined, then $(MN)^T = N^T M^T$.


Note that any involution necessarily takes the identity to itself: $M^* = (1 \cdot M)^* = M^* \cdot 1^* $, hence $1^*=1$.  

Edited to add:  Just to clarify, not all involutions have the desired property; as Dmitry Rubanovich points out in the comments, an involution does not necessarily reverse order (although most of the interesting ones do).  And conversely not every order-reversing homomorphism is an involution; involutions are all bijective and satisfy $x^{**}=x$, which need not be the case for an arbitrary order-reversing homomorphism.  But (as I wrote originally) many operations of this nature -- including both of the examples in the OP -- are involutions, and *-algebras (per Federico Poloni's comment) provide a rich source of additional examples.
A: Anti- (automorphism | endomorphism | homomorphism | isomorphism) of (groups | rings | monoids | semigroups | algebras... ).   
Sample sentences:
The inversion operation in a group is an antiautomorphism of that group.  Matrix transpose is an antiautomorphisms of the ring of matrices.   Conjugation is an antiautomorphism of quaternions and octonions. [ More at https://en.wikipedia.org/wiki/Antihomomorphism ]
Those are the most common examples and are also involutions (performing operation twice returns to initial state).  
It is more common to say anti*morphism or order-reversing morphism, than to talk about a morphism to the op-structure on the same object.
A: I don't believe this is standard, but my abstract algebra textbook (Contemporary Abstract Algebra by Gallian) refers to the fact that $(ab)^{-1} = b^{-1}a^{-1}$ as the "Socks-Shoes Property" (you put your socks on before your shoes, but if you want to undo that operation, you need to take your shoes off before taking off your socks). So a "socks-shoes operation" might be a possible, if informal, term for this.
A: In noncommutative ring theory (which might be applied to real matrices), a map $\phi:R\to R$ such that $\phi(ab)=\phi(b)\phi(a)$ and $\phi(a+b)=\phi(a)+\phi(b)$ may be called a ring-homomorphism $R\to R^{\operatorname{op}}$.
What's $R^{\operatorname{op}}$? Basically if $(R,+,\cdot)$ is a ring, $R^{\operatorname{op}}$ is the ring $(R,+,\cdot^{\operatorname{op}})$ given by $a\cdot^{\operatorname{op}}b:=b\cdot a$, while the sum remains the same.
A: Maybe you would want to call it antidistributivity, simply.
