How to denote 4th level of nested parentheses? When dealing with equation that contains nested parentheses, they are usually denoted using the {[()]} system, with 'regular' parentheses being the innermost member, square brackets middle member and curly brackets the outermost member. 
But what if we have a equation containing more than three levels of nesting? Is there a convention on brackets denotation in this case?
 A: I'm going to go out on a limb and say that your convention of alternating parenthesis is not a convention at all and that for consistency one should always use parenthesis.  e.g  $((x - 3(z +y))^2 + (57 - q(a+b)))^3$ is the "correct" way to do it.  However clarity is often more important than "correctness".  e.g $(\{x - 3(z + y)\}^2 + \{57 - q(a+b)\})^3$ is easier to read and $[\{x - 3(z + y)\}^2 + \{57 - q(a+b)\}]^3$ is even more so.
At any rate I'm confident in saying there is no convention.  
Heck, if you are able to, use colors.  That's even clearer.
A: *

*You could repeat the pattern

*You could use different sizes of parentheses. This works even with just one type: $$\Bigl((a+b)\cdot(c+d)+\bigl(e+f(x)\bigr)\cdot(g+h)\Bigr)\cdot (i+j)$$

*You could index matching parentheses
$$(_1(_2a+b)_2\cdot(_3c+d)_3+(_4e+f(_5x)_5)_4\cdot(_6g+h)_6)_1\cdot (_7i+j)_7$$


Ultimately, I am not even fond of (over-)using the $\{[()]\}$ scheme. After all, it should be a trivial task to find matching pairs even without special markup. Moreover, most types of parentheses have (additionally) some special meaning. Say, you want to write  the union of the set $\{1,2,3\}$ with the half-open interval $(4,5]$ and intersect that with another set $A$. That would be
$$\bigl( \{1,2,3\}\cup (4,5]\bigr)\cap A$$
I think it is confuding if one has to think twice each time (as opposed to only when seeing $($ and/or $)$) whether that symbol is just for grouping.
Finally, if you really need more than three levels of grouping, you may want to reconsider the expression you are about to write ...
A: Not knowing any better I'd just cycle through again.  I don't know of any other kinds of symbols that would do this.  Pointy brackets $\langle \; \rangle$ usually have a different meaning than run-of-the-mill parentheses.
The different symbols are for clarity.  Unless there's a different meaning to $\{ \; \}$ vs $[ \; ]$ vs $( \; )$ it doesn't matter a whole lot.  For my first symbol I use $( \; )$ and that's a pretty common first choice.
