(*) says that the diagonals of $R$ are $1$ and the non-diagonals are the correlation, $p$.
I planned on simplifying both equations until they're equivalent, but I'm not sure how I could go about that. I don't understand what number 5 is asking me to do.
Here are a few hints. For number 4, convince yourself that $R$ can be written
$$R=(1-\rho)I + \rho U,\tag1$$ where $I$ is the identity matrix and $U$ is an $n\times n$ matrix of all ones. Then calculate ${\bf x}R{\bf x}^T$ using (1). (Note that $\bf x$ here is a row vector.)
For number 5, the hint provided is more convoluted than it needs to be. First show that $$ \sum x_i^2=\frac {s^2}n + \sum \left(x_i-\frac sn\right)^2,\tag 2$$ where $s$ is an abbreviation for $\sum x_i$. Note that (2) holds for any vector $\bf x$. Plug (2) into (**) and rearrange to get $$ Q({\bf x}) = \left(\frac{1-\rho}n+\rho\right) s^2 + (1-\rho)\sum\left(x_i-\frac sn\right)^2.\tag3$$ Now stare at (3). Let $\bf x$ be a nonzero vector (which is what you need to consider to prove positive [semi]definiteness). Argue that if $\rho\gt-\frac1{n-1}$, then the RHS of (3) is strictly positive; if $\rho < -\frac1{n-1}$, you can find a nonzero $\bf x$ for which (3) is strictly negative; and if $\rho = -\frac1{n-1}$ you can find a nonzero $\bf x$ for which (3) is zero.
Note that there is a typo in number 5: If $\rho=1$ then $R$ is positive definite.
