How can I determine the convergence radius of the following power series:

$$F(n,z):=\sum_{j=0}^nz^{3j^2}+5z^{j^3}$$ ?

I've tried with the formulas

$$r=\frac{1}{\limsup_{n \to \infty} \sqrt{|a_n|}}$$ and $$r = \lim_{n \to \infty} |\frac{a_n}{a_{n+1}}|$$ but hadn't any success yet.

Assuming that you mean the series $$\sum_{j=0}^\infty z^{3j^2}+5z^{j^3}$$, then:
• since it diverges when $$z=1$$, the radius of convergence is at most $$1$$;
• since each series $$\sum_{j=0}^\infty z^{3j^2}$$ and $$\sum_{j=0}^\infty5z^{j^3}$$ converges when $$\lvert z\rvert<1$$ (by the ratio test), the sum converges when $$\lvert z\rvert<$$ and therefore the radius of convergence is at least $$1$$.
So, the radius of convergence is equal to $$1$$.