Your "ternary dot product" is basically equivalent to the Hadamard product (https://en.wikipedia.org/wiki/Hadamard_product_%28matrices%29): given two vectors $\langle a_i\rangle, \langle b_i\rangle$, their Hadamard product is just componentwise multiplication: $$\langle a_i\rangle*\langle b_i\rangle=\langle a_ib_i\rangle.$$ To see why this is equivalent to the thing you describe, note that the ternary dot product of $\alpha, \beta, \gamma$ is just $\alpha\cdot (\beta*\gamma)$. In fact, the Hadamard product can be used to define $n$-ary dot products similarly, whereas you can't obviously do this with just the ternary dot product alone. Because of this, I don't think this operation deserves a specific name.
Now, the Hadamard product probably seems really natural (it certainly did to me), so you might ask, "Why isn't the Hadamard product more ubiquitous?" Basically, see why don't we define vector multiplication component-wise?, especially the answer by Giuseppe Negro; essentially, equations in terms of the Hadamard product aren't invariant under rotations, so it is unlikely to have any phsyical meaning (if something has physical meaning then computations involving it should be independent of exactly how we draw our axes). By contrast, equations in terms of the usual dot product are invariant under a wide range of transformations.