Deriving the formula $\det(AB)=\det(A)\det(B)$ from the geometric property of a determinant Suppose we are given that the determinant satisfies the following property for any $X\subset\mathbb{R}^n$: $$\widehat{\operatorname{vol}}(\alpha (X))=\det A\cdot\operatorname{vol}(X).$$ Here $\alpha\colon\mathbb{R}^n\to\mathbb R^n$ is any linear operator which under any fixed basis yields the matrix $A$. Further $\widehat{\operatorname{vol}}(\alpha (X))$ represents the signed volume, i.e. it is the volume with the negative sign if $\alpha$ involves reflection. $\operatorname{vol}(X)$ represents the ordinary volume.
Now we wish to show that for any two matrices $A,B$ we have $\det(AB)=\det(A)\det(B)$. To that end we let $\alpha,\beta$ be the operators corresponding with $A,B$ and for any set $X$ with nonzero volume and note that $$\det(AB).\operatorname{vol}(X)=\widehat{\operatorname{vol}}(\alpha\beta (X))=\det A\cdot\operatorname{vol}(\beta X)$$
Now I would like that $\det A\cdot\operatorname{vol}(\beta X)=\det A\cdot\det B\cdot\operatorname{vol}(X)$ so that cancelling out $\operatorname{vol}(X)$ we get that $\det(AB)=\det(A)\det(B)$ as required. My problem is that $\operatorname{vol}(\beta X)\ne\det B\cdot\operatorname{vol}(X)$ in general as $\operatorname{vol}(\beta X)$ represents the ordinary volume and not signed volume.
What am I missing here?
 A: $\DeclareMathOperator\vol{vol}%
\newcommand\volhat{\widehat{\operatorname{vol}}}$You could start with the unsigned equation
$$
\vol(\alpha(X)) = \left|\det \alpha\right|\cdot\vol(X)
$$
from which we get
$$
\left|\det (\alpha\circ\beta)\right|\cdot \vol X = \vol(\alpha(\beta(X))) = \left|\det\alpha\right|\cdot\vol(\beta(X)) = \left|\det\alpha\right|\cdot\left|\det\beta\right|\cdot\vol(X)
$$
and therefore
$$
\left|\det(\alpha\circ\beta)\right| = \left|\det\alpha\cdot\det\beta\right|.
$$
Your oriented volume $\volhat(Y)$ is not well defined, since $\volhat(Y)$ for $Y=\alpha(X)$ seems to depend on properties of $\alpha$ but we also have $Y=\operatorname{id}(Y)$, so $\volhat$ can't decide the sign without knowing about the map involed.
In order to make a rigorous statement including the orientation, you first have to define some notion of an oriented set (i.e. a structure that consists of (i) the set and (ii) an orientation of the set) and then define oriented volumes for oriented sets.

Let me explain why you really need the notion of a set carrying an orientation:
If you want to read $\volhat(\alpha(X))$ as a map that takes the pair $(\alpha, X)$ and gives you oriented volume by looking at $\alpha$ for the sign and $X$ for the volume, you have
$$
\det(\alpha\circ\beta)\cdot\vol(X) = \volhat((\alpha\circ\beta)(X))
$$
where now $\volhat$ looks at the pair $(\alpha\circ\beta, X)$. But what you want is $\volhat(\alpha(\beta(X)))$ in the next step, where you now look at the pair $(\alpha,\beta(X))$, where everything $\beta$ does to the orientation is now lost, since $\volhat(\alpha(\beta(X))$ is only concerned about the map $\alpha$ and the unoriented set $\beta(X)$.
Define oriented sets and you always carry orientation from $X$ to $\beta(X)$ to $\alpha(\beta(X))$ and the problem of losing information doesn't occur anymore.

Let me try to do this:

Definition: An oriented set is a pair $\widehat X=(X,\epsilon)$ where $X\subseteq\mathbb R^n$ is some set and $\epsilon\in\{+,-\}$ is the orientation. Define $\epsilon(\widehat X):=\epsilon$.
Let $\mathcal O(\mathbb R^n)$ be the set of all oriented subsets of $\mathbb R^n$.
Definition: A linear map $\alpha:\mathbb R^n\to\mathbb R^n$ gives rise to a map
  \begin{align}
\widehat\alpha:\mathcal O(\mathbb R^n) &\longrightarrow \mathcal O(\mathbb R^n), \\
\widehat X &\longmapsto (\alpha(X), \alpha(\epsilon(\widehat X))),
\end{align}
  where $\alpha(\epsilon)=\epsilon$ if $\alpha$ doesn't flip the space and $\alpha(\epsilon)=-\epsilon$ if it does. Note that $\widehat{\alpha\circ\beta}=\widehat\alpha\circ\widehat\beta$.
Definition: The oriented volume of an oriented set $\widehat X$ is given by
  $$\volhat(\widehat X) := \epsilon(\widehat X)\cdot\vol(X).$$

With this definitions, we have for an oriented set $\widehat X$ and a linear map $\alpha$
$$
\volhat(\widehat\alpha(\widehat X)) = \det\alpha \cdot \volhat(\widehat X) = \alpha(+)\cdot\left|\det\alpha\right|\cdot\volhat(\widehat X).
$$
We conclude
\begin{align}
\det(\alpha\circ\beta)\cdot\volhat(\widehat X) &= \volhat((\widehat\alpha\circ\widehat\beta)(\widehat X)) = \volhat(\widehat\alpha(\widehat\beta(\widehat X)) = \det \alpha \cdot \volhat (\widehat\beta(\widehat X)) \\&= \det \alpha \cdot\det \beta \cdot\volhat(\widehat X)).
\end{align}
