Is matrix notation with dots acceptable for papers? What are some alternatives? I am very used to this kind notation when dealing with matrices with arbitrary dimensions:
$$
X_P^T\Lambda X_P=
\begin{bmatrix}
 x_1 & x_2, &\dots, &x_n\\
\end{bmatrix}
\begin{bmatrix}
\lambda_1 & 0 & \dots & 0\\
0 & \lambda_2 & \dots & 0\\
\vdots & \vdots &\ddots \\
0 & 0 & \dots & \lambda_n\\
\end{bmatrix}
\begin{bmatrix}
 x_1 \\ 
 x_2 \\ 
 \vdots\\
 x_n\\
\end{bmatrix}
$$
I like it because it's nice and directly conveys the matrix representation. However, someone mentioned to me that these representations are good for notes and teaching materials but not for publications. In this specific example the matrix is a diagonal matrix, but I am asking more in general, if you have to convey an arbitrary matrix with any dimensions and any entries, how do you convey the pattern in the matrix without doing dot notation?
 A: The description on the right is very common and serves to give the reader a schematic description of the matrices. 
I would write the equation above as follows, although this is my preference: 
\begin{equation}
x^\top \Lambda x = 
\begin{bmatrix}
x_1 & \cdots & x_n 
\end{bmatrix}
\begin{bmatrix}
\lambda_1 & & \\
& \ddots &   \\
& & \lambda_n 
\end{bmatrix}
\begin{bmatrix}
x_1 \\ \vdots \\ x_n 
\end{bmatrix}, \tag{1} \label{myeq}
\end{equation}
where it is understood that the zeros occupy the blank spaces above and below the diagonal.
EDIT: Comments in the sequel address the OP's comments to this answer.
For $n \in \mathbb{N}$, let $\langle n \rangle := \{ 1, \dots, n\}$. If $A:\langle m \rangle \times \langle n \rangle \to \mathbb{F}$ is a function and $\mathbb{F}$ is a set (typically a field), then $A$ is called an $m$-by-$n$ matrix. By convention, the value $A(i,j)$ is abbreviated to $A_{ij}$ or $a_{ij}$.
Given the nature of the function, it is often convenient to represent a matrix via a rectangular array. For instance, the matrix $A=\{a_{11},a_{12},a_{21},a_{22}\}$ is written as
\begin{equation}
A=
\begin{bmatrix}
a_{11} & a_{12 } \\
a_{21} & a_{22}
\end{bmatrix}.
\end{equation}
In general, it is understood that $A=[a_{ij}] \in \textsf{M}_{m \times n}(\mathbb{F})$ is the function $\{a_{ij}\}_{i\in\langle m \rangle, j \in \langle n \rangle}$. 
It is customary in linear algebra/matrix theory journals to define a matrix and give its schematic description as in \eqref{myeq}. 
Here is an example from a 2017 paper I co-authored with Charles R. Johnson, who is widely considered to be the best matrix theorist in the world:

I have published and refereed in the top journals in matrix theory and linear algebra and I have never objected to or had a referee object to the practice you describe in your post.  
