I believe the answer is $1/3$, independent of the values of $a$ and $b$.
To see this, we first show that the box counting dimension of the set
$$E = \{1,1/4,1/9,\ldots,1/n^2,\ldots\}$$
is $1/3$. The computations below very much mimic those of example 3.5 on page 45 of Falconer's Fractal Geometry.
To this end, let $\varepsilon>0$ and choose $N$ to be the unique natural number such that
$$\frac{1}{N^2} - \frac{1}{(N+1)^2}<\varepsilon\leq\frac{1}{(N-1)^2}-\frac{1}{N^2}.$$
Now, since $\varepsilon \leq 1/(N-1)^2-1/N^2$, we need at least $N-1$ sets of diameter $\varepsilon$ to cover $E$ - one for each of the numbers $1,1/4,\ldots,1/(N-1)^2$. Thus, $N_{\varepsilon}(E) \geq N-1$. And since
$$\varepsilon > \frac{1}{N^2} - \frac{1}{(N+1)^2} > \frac{1}{(N+1)^3},$$
$1/\varepsilon<(N+1)^3$. Thus,
$$\frac{\log(N_{\varepsilon}(E))}{\log(1/\varepsilon)} >
\frac{\log(N-1)}{\log((N+1)^3)} \to \frac{1}{3}.$$
This much shows that
$$\liminf_{\varepsilon\to0^+}\frac{\log(N_{\varepsilon}(E))}{\log(1/\varepsilon)} \geq \frac{1}{3}.$$
To show that
$$\limsup_{\varepsilon\to0^+}\frac{\log(N_{\varepsilon}(E))}{\log(1/\varepsilon)} \leq \frac{1}{3},$$
we use the fact that, since $\varepsilon>1/(N+1)^3$, all the points of $E$ in $[0,1/(N+1)^2]$ may be covered with $N+1$ sets of size $\varepsilon$. That leaves us with only $N$ more points so that $E$ may be covered with $2N+1$ sets of size $\varepsilon$ so that $N_{\varepsilon}(E)\geq 2N+1$. Also using the fact that
$$\varepsilon\leq\frac{1}{(N-1)^2}-\frac{1}{N^2}< \frac{6}{N^3}$$
for $n>2$, we get that
$$\frac{\log(N_{\varepsilon}(E))}{\log(1/\varepsilon)} <
\frac{\log(2N+1)}{\log((N+1)^3/6)} \to \frac{1}{3}.$$
Now, it's just a couple of steps to get from here to your set of interest - one small step and one not so small.
First, if we define
$$E_a = \{a,a/4,a/9,\ldots,a/n^2,\ldots\}$$
for a positive number $a$, then we again have a set of box dimension $1/3$ because box dimension is preserved under similarity transformations.
Now, let's denote your set by $E_{a,b}$. The projection of your set onto the $x$-axis does not increase distance. Thus, it cannot increase box dimension so that the lower box dimension of $E_{a,b}$ is at least $1/3$. On the other hand, $E_{a,b}$ is the image of $E_a$ under the function $F:\mathbb R\to\mathbb R^2$ defined by $F(x) = (x,b(x/a)^{3/2})$. Since we're mapping to the graph of a differentiable function, this is a Lipschitz map and, therefore, also does not increase dimension. So the upper box dimension of $E_{a,b}$ is at most $1/3$. Taking these together, we a well defined box dimension of $1/3$.