It is customary to use square brackets $[\ddots]$ to refer to the matrix as a matrix. It is also customary to use vertical bars $|\ddots|$ to refer to the determinant of the matrix.
It is not customary to use ordinary brackets $(\ddots)$ for anything.
However, mathematicians are fond of making their own notation. So, look in your text book and/or your lecture notes to see how they define this.
Matrices are equal to each other only if they are the same size and every member is equal to the member in the same place. So, in this case there are no solutions.
Determinants are just numbers and can be equal even if the matrices are not.
Since schools like to give exercises which actually have solutions, they are most likely to be talking about determinants, but it can possibly be a trick question.