When dealing with equation that contains nested parentheses, they are usually denoted using the {[()]} system, with 'regular' parentheses being the innermost member, square brackets middle member and curly brackets the outermost member. But what if we have a equation containing more than three levels of nesting? Is there a convention on brackets denotation in this case?

  • 2
    $\begingroup$ I'm pretty sure that there is no "usually" here. It's possible your teacher always uses this convention, but, you can just keep nesting the same parentheses over and over. $\endgroup$
    – James
    Jul 29, 2016 at 17:22
  • 2
    $\begingroup$ No. There is no convention. In fact the convention you state that first ( then [ and then { isn't a convention at all. Just as likely { will follow ( with [ being skipped. Equally often just nested ((((())))) is used. ((((())))) is just fine. We only change for the sake of clarity. $\endgroup$
    – fleablood
    Jul 29, 2016 at 17:23
  • $\begingroup$ From inside to outside an expression, the size of my grouping symbols are monotonic increasing, and use ()s followed by []s. It's okay to use grouping symbols more than once in an expression. I would not recommend using {}s for grouping symbols, as that is notation used for sets (granted we still use () and [] for tuples and intervals, but those use commas in between). $\endgroup$
    – anon
    Jul 29, 2016 at 17:32

3 Answers 3


I'm going to go out on a limb and say that your convention of alternating parenthesis is not a convention at all and that for consistency one should always use parenthesis. e.g $((x - 3(z +y))^2 + (57 - q(a+b)))^3$ is the "correct" way to do it. However clarity is often more important than "correctness". e.g $(\{x - 3(z + y)\}^2 + \{57 - q(a+b)\})^3$ is easier to read and $[\{x - 3(z + y)\}^2 + \{57 - q(a+b)\}]^3$ is even more so.

At any rate I'm confident in saying there is no convention.

Heck, if you are able to, use colors. That's even clearer.

  • You could repeat the pattern
  • You could use different sizes of parentheses. This works even with just one type: $$\Bigl((a+b)\cdot(c+d)+\bigl(e+f(x)\bigr)\cdot(g+h)\Bigr)\cdot (i+j)$$
  • You could index matching parentheses $$(_1(_2a+b)_2\cdot(_3c+d)_3+(_4e+f(_5x)_5)_4\cdot(_6g+h)_6)_1\cdot (_7i+j)_7$$

Ultimately, I am not even fond of (over-)using the $\{[()]\}$ scheme. After all, it should be a trivial task to find matching pairs even without special markup. Moreover, most types of parentheses have (additionally) some special meaning. Say, you want to write the union of the set $\{1,2,3\}$ with the half-open interval $(4,5]$ and intersect that with another set $A$. That would be $$\bigl( \{1,2,3\}\cup (4,5]\bigr)\cap A$$ I think it is confuding if one has to think twice each time (as opposed to only when seeing $($ and/or $)$) whether that symbol is just for grouping.

Finally, if you really need more than three levels of grouping, you may want to reconsider the expression you are about to write ...


Not knowing any better I'd just cycle through again. I don't know of any other kinds of symbols that would do this. Pointy brackets $\langle \; \rangle$ usually have a different meaning than run-of-the-mill parentheses.

The different symbols are for clarity. Unless there's a different meaning to $\{ \; \}$ vs $[ \; ]$ vs $( \; )$ it doesn't matter a whole lot. For my first symbol I use $( \; )$ and that's a pretty common first choice.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .