Your understanding is flawed, I’m afraid. When you count the elements of a set, you count just the elements of that set: you don’t count the elements of those elements separately, and there is nothing special about $\varnothing$ in this context.
Let’s consider the set $x=\Big\{a,\{b\},\big\{c,\{d\}\big\}\Big\}$. It has $3$ elements: $a$, $\{b\}$, and $\big\{c,\{d\}\big\}$, so its cardinality is $3$. That last element happens to be a set with $2$ elements of its own, but it’s still just one member of $x$. We could replace it with the infinite set $\Bbb Z$ of all integers, getting the set $\big\{a,\{b\},\Bbb Z\big\}$, and we’d still have a set of cardinality $3$.
Now the cardinality of $x$ is $3$ no matter what $a,b,c$, and $d$ are.1 In particular, it’s $3$ even if $a=b=c=d=\varnothing$, so that $x=\Big\{\varnothing,\{\varnothing\},\big\{\varnothing,\{\varnothing\}\big\}\Big\}$. It’s also $3$ if $a=b=c=d=\Bbb Z$, and $x=\Big\{\Bbb Z,\{\Bbb Z\},\big\{\Bbb Z,\{\Bbb Z\}\big\}\Big\}$. In the first case the $3$ elements of $x$ are $\varnothing$, $\{\varnothing\}$, and $\big\{\varnothing,\{\varnothing\}\big\}$; in the second they are $\Bbb Z$, $\{\Bbb Z\}$, and $\big\{\Bbb Z,\{\Bbb Z\}\big\}$.
1 That’s not quite true, but the two exceptions involve a technicality that beginners sometimes find confusing. Specifically, if $a=\{b\}$, then $$x=\Big\{a,a,\big\{c,\{d\}\big\}\Big\}=\Big\{a,\big\{c,\{d\}\big\}\Big\}$$ and has only $2$ elements. Similarly, if $a=\big\{c,\{d\}\big\}$, then $$x=\big\{a,\{b\},a\big\}=\big\{a,\{b\}\big\}$$ and again has only $2$ elements. In no case, however, does $x$ have $4$ elements.