Showing $b^2 \notin \langle a \rangle$ for $V_{8n}=\langle a,b:a^{2n}=e, b^4 = e, ba=a^{-1}b^{-1},b^{-1}a=a^{-1}b\rangle.$ For $n \in \mathbb N$, finite group $V_{8n}$  is
$$ V_{8n} = \langle a, b  : a^{2n} = e, b^4 =e, ba = a^{-1}b^{-1}, b^{-1}a = a^{-1}b \rangle. $$
This group is defined in this paper  for all $n \in  \mathbb N.$ 

I want to prove that $b^2 \notin \langle a \rangle$.

If possible,   let $b^2  \in \langle a \rangle$. Then $b^2 = a^t$ for some $t$. I observe that the possibility of $t$ is $n$. Since $ba = a^{2n -1}b^3$ so that by mathematical induction on $i$, we get
$$ ba^i = \left\{ \begin{array}{ll}
a^{2n -i}b & \mbox{if $i$ is even };\\
a^{2n - i}b^3& \mbox{if $i$ is odd}\end{array} \right.$$
If $n$ is odd, then   $b^3 = ba^n =  a^n b^3$ implies $a^n = e$; a contradiction. I am stuck here if $n$ is even.  I would be thankful for your kind help.
 A: From the relations $bab=a^{-1}$ and $ba^{-1}b = a$, we get $b^2ab^2 = a$, and since $b^4=1$, this implies $b^{-2}ab^2 = a$, so $b^2 \in C_G(a)$. Since clearly $b^2 \in C_G(b)$, this gives $b^2 \in Z(V_{8n})$.
Now, the calculation in user1729's answer shows that $V_{8n}/\langle b^2 \rangle$ is isomorphic to the fihedral group of order $4n$. Since $b^4=1$, this proves that $|V_{8n}| \le 8n$ and $|V_{8n}| \ge 4n$. But to prove that $b^2 \not\in \langle a \rangle$, we still have to prove that $b^2 \ne 1$, which would show that $|V_{8n}| = 8n$.
To prove that, we follow the standard procedure of contructing a group $\langle a,b \rangle$ of order $8n$ in which $a$ and $b$ satisfy the relations of $V_{8n}$.
Start with the group $H = \langle a,c \mid a^{2n}=c^2=1, ac=ca \rangle \cong C_{2n} \times C_2$. Then $H$ has an automorphism $\phi$ of order $2$ with $\phi(a) = a^{-1}c$ and $\phi(c)=c$. Let $\langle b \rangle$ with $b$ of order $4$ be cyclic, and let $X$ be the semidirect product $H \rtimes_\phi \langle b \rangle$. So $$X = \langle a,b,c \mid a^{2n}=b^4=c^2=1, ac=ca, b^{-1}ab=a^{-1}c,b^{-1}cb=c \rangle,$$ and $|X| = 4|H| = 16n$.
Now $b^2, c \in Z(X)$, and we define $G = X/\langle b^2c^{-1} \rangle$, so $|G| = 8n$ and 
$$G = \langle a,b,c \mid a^{2n}=b^4=c^2=1, b^2=c, ac=ca, b^{-1}ab=a^{-1}c,b^{-1}cb=c \rangle \cong \langle a,b\mid a^{2n}=b^4=1, ab^2=b^2a,  b^{-1}ab=a^{-1}b^2 \rangle.$$
Note that in $G$ we have $bab = b^2a^{-1}b^2 = a^{-1}$ (since $b^2 \in Z(G))$, and similarly $ba^{-1}b = a$, so the relations of $V_{8n}$ are indeed satisfied in $G$, and since $|G| = 8n$, this proves that $|V_{8n}| = 8n$ and $G \cong V_{8n}$.
Finally note that, in $G$, $b^2 = c \not\in \langle a \rangle$, which is what you wanted to prove.
A: This answer uses semidirect products. I hope you are familiar with them, else see the wikipedia link :-)
The following proves that if $b^2\in\langle a\rangle$ then $b^2=1$ (therefore it is an incomplete answer - but some of the working is applied in Derek Holt's answer, so I'll leave it here). One way to see this partial result is to quotient out $b^2$ from the group (that is, quotient out the normal subgroup generated by $b^2$, which I denote by $\langle\langle b^2\rangle\rangle$):
$$\begin{align*}
V_{8n}/\langle\langle b^2\rangle\rangle
&= \langle a, b  : a^{2n} = e, b^4 =e, ba = a^{-1}b^{-1}, b^{-1}a = a^{-1}b \rangle/\langle\langle b^2\rangle\rangle\\
&= \langle a, b  : a^{2n} = e, b^4 =e, ba = a^{-1}b^{-1}, b^{-1}a = a^{-1}b, b^2 \rangle\\
&= \langle a, b  : a^{2n} = e, b^2 =e, bab^{-1} = a^{-1}, bab^{-1} = a^{-1} \rangle\\
&= \langle a, b  : a^{2n} = e, b^2 =e, bab^{-1} = a^{-1}\rangle
\end{align*}
$$
This is a presentation for the semidirect product of $\mathbb{Z}_{2n}\rtimes\mathbb{Z}_2$. In particular, the image of $\langle a\rangle$ has order $2n$ in the quotient, and so $a$ has the same order both in $V_{8n}$ and in the quotient group. Therefore, the subgroup $\langle a\rangle$ intersects the kernel trivially, $\langle a\rangle\cap\langle\langle b^2\rangle\rangle=_{V_{8n}}1$, and hence if $b^2\in \langle a\rangle$ then $b^2=1$, as required.
