Let $A,B\in \mathcal{M}_n(\mathbb{C})$ such that $AB=BA$ and $\det B\neq 0$ 
Let $A,B\in \mathcal{M}_n(\mathbb{C})$ such that $AB=BA$ and $\det B\neq 0$. If $|\det(A+zB)|=1$ for any $z\in \mathbb{C}$ such that $|z|=1$, then show $A^n=O_n$.

We know $A,B$ are simultaneously triangularizable. From this it is enough to show that all the eigenvalues of $A$ are zero, but could not proceed further.
 A: $\det(A + B z)$ is an analytic function in a neighbourhood of the closed  unit disk which has absolute value $1$ on the unit circle, so it is a finite Blaschke product.  But if this is a polynomial (i.e. has no finite poles),  we must have $\det(A+Bz) = \det(B) \det(AB^{-1}+zI) = c z^n$  for some constant $c$.  By the Cayley-Hamilton theorem, $(AB^{-1})^n = 0$.  And then since $B$ is invertible and $A$ commutes with it, that implies $A^n = 0$.
EDIT: Here's a somewhat more self-contained version.  Suppose $f$ is a polynomial such that $|f(z)| = 1$ for $|z| = 1$.  Now $g(z) = \overline{f(1/\overline{z})}$ is analytic on $\mathbb C \backslash \{0\}$, and $f(z) g(z) = f(z) \overline{f(z)} = 1$ on the unit circle.  Therefore 
$f(z) g(z) = 1$ for all $z \ne 0$.  If $f(z)$ has a root
at $p \ne 0$, then $g(z)$ has a pole there, which would mean $f(z)$ has a pole at $1/\overline{p}$.  But $f$ is a polynomial, so it has no poles.  Thus $0$ is the only possible root of $f$, which means $f(z) = c z^m$ for some $m$. In your case 
$f$ has degree $n$, so $m=n$.
A: As you said, $A, B$ are simultaneously triangulizable. So we can assume that both are upper triangular matrices. Then $p(z) = \det(A+Bz) = \prod(b_{i}z + a_{i})$ is a polynomial where $a_{i}, b_{i}$ are diagonal elements of $A, B$. 
To show $a_{i} = 0$ for all $i$, it is enough to show that 
$$
|p(z)| = 1 \quad \forall |z| = 1 \Rightarrow p(z) = cz^{n}\quad (\exists c\in \mathbb{C})
$$
for a polynomial $p(z)$. 
Honestly, I don't know an easy way to show this, but there's one possible solution here. Let $p(z) = c_{n}z^{n} + \cdots + c_{1}z + c_{0}$ and $z = e^{i\theta}$. Then 
$$
|p(z)|^{2} = |p(e^{i\theta})|^{2} = b_{0} + 2\sum_{k=1}^{n}\Re(b_{k}e^{ik\theta})
$$
where
$$
b_{k}= \sum_{l=0}^{k} c_{l}\overline{c_{l+k}}
$$
(sorry for abusing notation). If we write $b_{k} =\alpha_{k} + i\beta_{k}$, then 
$$
|p(e^{i\theta})|^{2}= b_{0} + 2\sum_{k=1}^{n} (\alpha_{k} \cos k\theta - \beta_{k} \sin k\theta)
$$
is a constant function on $\theta$, so we have $\alpha_{k} = \beta_{k} = 0$, i.e. $b_{k} = 0$ for all $k\geq 1$. (This follows from uniqueness of Fourier series). So 
$$
c_{n}\overline{c_{0}} = 0 \\ c_{n} \overline{c_{1}} + c_{n-1}\overline{c_{0}} = 0 \\ c_{n}\overline{c_{2}} + c_{n-1}\overline{c_{1}} + c_{n-2}\overline{c_{0}} = 0 \\ \vdots \\ c_{n}\overline{c_{n-1}} + \cdots + c_{1}\overline{c_{0}} = 0
$$
Since $c_{n} \neq 0$, $c_{0} = 0$, and this gives $c_{1} = 0$ by the second equation, and so $c_{2} =0$ by the third equation, and so on. Hence $c_{0} = \cdots = c_{n-1} = 0$ and $p(z) = c_{n}z^{n}$. 
