How do you create a “stretch” transformation while keeping volume constant?

A stretch transformation can be represented as:

$$\begin{bmatrix} k & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix}$$

However, this changes the volume of any object which it operates on by a factor of $k$. I'm looking for a way to perform such a stretch while keeping the volume constant, but I'm not sure how exactly to go about doing this. I'm considering something like:

$$\begin{bmatrix} k^{2/3} & 0 & 0 \\ 0 & k^{-1/3} & 0\\ 0 & 0 & k^{-1/3} \end{bmatrix}$$

Unfortunately, I can't find any good justification for why I should reduce each dimension by a factor of $k^{1/3}$. I also considered simply keeping the factor of $k$ in my volume and later renormalizing by dividing the volume of each object in the "new" coordinate system by $k$, however this doesn't address how I should treat a vector that transforms into the new coordinate system.

Your intuition indeed gives you the right result. The volume change factor after applying the matrix is precisely the determinant of the matrix. If you want a transformation which retains volume then you will want a determinant of $1$. The determinant of the original matrix is $k$ and what you've done is scale the matrix to give it determinant $1$.