Integral $\int_0^\infty \frac{\log^2 x \cos ax}{x^n-1}dx$ Hi I am trying to calculate
$$
I:=\int\limits_0^\infty \frac{\log^2 x \cos (ax)}{x^n-1}\mathrm dx,\quad \Re(n)>1, \, a\in \mathbb{R}.
$$
Note if we set $a=0$ we get a similar integral given by
$$
\int\limits_0^\infty \frac{\log^2 x }{x^n-1}\mathrm dx=-\frac{2\pi^3\cot(\pi/n)\csc^2(\pi/n)}{n^3},\quad \Re(n)>1.
$$
I was trying to write I as 
$$
I=\Re \bigg[\int\limits_0^\infty  \frac{ e^{i ax}\log^2 x}{x^n-1}\mathrm dx\bigg]=\Re\bigg[\int\limits_\infty^0\frac{e^{iax}\log^2 x}{1-x^n}\mathrm dx\bigg]=\Re\bigg[\int\limits_\infty^0e^{iax}\log^2 x\sum_{m=0}^\infty x^{nm} \mathrm dx\bigg].
$$
But was unsure of where to go from here. How can we calculate $I$? It is clear that this method is not going to work.
 A: This is not a complete solution.
First, we use a simple integral (we can get it by differentiating $x^b$ w.r.t. $b$ two times):
$$\int_0^1 x^b \ln^2 x ~d x=\frac{2}{(b+1)^3}$$
Using this integral, we also get:
$$\int_0^1 x^b \ln^2 x ~dx=\int^{\infty}_1 t^{-b-2} \ln^2 t~ dt=\frac{2}{(b+1)^3}$$
We separate the original integral into two parts:
$$
I=-\int\limits_0^1 \frac{\log^2 x \cos (ax)}{1-x^n} ~ dx+\int\limits_1^\infty \frac{\log^2 x \cos (ax)}{x^n-1} ~ dx=I_1+I_2
$$
Let's deal with the first one by using the modified integral from above:
$$\int_0^1 x^{pn} (a x)^{2q} \ln^2 x ~d x=\frac{2a^{2q}}{(pn+2q+1)^3}$$
Now we immediately get (using Taylor expansions for $\cos (ax)$ and $\frac{1}{1-x^n}$):

$$I_1=-2 \sum_{q=0}^{\infty} \frac{(-1)^q a^{2q}}{(2q)!} \sum_{p=0}^{\infty} \frac{1}{(pn+2q+1)^3}=\frac{1}{n^3} \sum_{q=0}^{\infty} \frac{(-1)^q a^{2q}}{(2q)!} \psi^{(2)} \left( \frac{2q+1}{n} \right)$$

Here $\psi^{(2)}$ is the second derivative of the digamma function.
This is a correct series solution for $I_1$, as we can see by comparing the two sides of the equation. See the plot $I_1(n)$ for $a=3$ below:

However, we can't use this method for $I_2$.
Let's see why (now we expand $\frac{1}{1-1/x^n}$):
$$\int^{\infty}_1 x^{-pn-n} (a x)^{2q} \ln^2 x~ dx=\frac{2a^{2q}}{(pn+n-2q-1)^3}$$
And now we see the problem. The denominator can become $0$ for infinitely many combinations of $p,q,n$, thus our integral would diverge badly.
Let's however, solve the case $a=0$, solution for which was provided in the OP ($n>1$).
$$I_2(a=0)=2 \sum_{p=0}^{\infty} \frac{1}{(pn+n-1)^3}=-\frac{1}{n^3} \psi^{(2)} \left(1- \frac{1}{n} \right)$$
Comparing with $I_1(a=0)$ and using the reflection formula for $\psi^{(2)}$:
$$\psi^{(2)}(1-x)-\psi^{(2)}(x)=2 \pi^3 \cot \left( \pi x \right) \csc^2 \left( \pi x \right)$$
we obtain:
$$I(a=0)=-\frac{2 \pi^3}{n^3} \cot \left( \pi \frac{2q+1}{n} \right) \csc^2 \left( \pi \frac{2q+1}{n} \right)$$


So, I'm not sure how to get the series solution for $I_2$, but I hope my answer helps you in some way.

A: 
This is my failed attempt at this problem. Essentially too long for a comment and contains multiple mistakes. 

So, lets start with the integral:
$$\int_0^\infty \cos ax^2\text{ d}x=\sqrt\frac{\pi}{8a}$$
$$\frac{\partial^{-4mn}}{\partial a^{-4mn}}\int_0^\infty {\cos ax^2}=\int_0^\infty \frac{\cos ax^2}{x^{8mn}}\text{ d}x=\frac{\pi}{\sqrt 8}\frac{a^{4mn-1/2}}{\Gamma (4mn+1/2)}$$
Which is true because of this question: Derivative of a negative order?
$$\sum_{m=1}^\infty \int_0^\infty \frac{\partial^{-4mn}}{\partial a^{-4mn}}\cos ax^2 \, dx=\int_0^\infty \left(\frac{\cos ax^2}{x^{8n}}+\frac{\cos ax^2}{x^{16n}}+\frac{\cos ax^2}{x^{24n}}\cdots\right)\text{ d}x=\int_0^\infty \frac{\cos ax^2}{x^{8n}-1}\text{ d}x$$
Now, substitute $x\to\sqrt x, \text{ d}x\to\frac{\text{ d}x}{2\sqrt x}$ and assume that $n=\frac{1}{4}k$
And we get: $$\frac{1}{2}\int_0^\infty \frac {\cos ax}{\sqrt x (x^k-1)}
\text{ d}x=\frac{\pi}{\sqrt 8}\sum_{m=1}^\infty \frac{a^{mn-1/2}}{\Gamma(mk+1/2)}$$
Maybe there is a part of this proof that you can deviate from. 
