Presumably $A$ has full row rank (or else $\det(AB)$ is always zero). Therefore $A$ admits a polar decomposition $PU^T$, where $P$ is positive definite and $U^T$ has orthonormal rows (i.e., $U$ has orthonormal columns). Then $\det(AB)=\det(P)\det(U^TB)$. Since $U^T$ has orthonormal rows and the columns of $B$ are unit vectors, each column of $U^TB$ has norm $\le1$. It follows from Hadamard’s determinant theorem that $\det(U^TB)\le1$. Consequently, $\det(AB)\le\det(P)$. The optimal solutions are the matrices $B$ that make equality hold. In particular, $B=U=A^T(AA^T)^{-1/2}$ is an optimal solution.