$\epsilon$ is the empty string. A Turing machine accepts $\epsilon$ if the start state of its finite control is an accepting state.
$\langle M\rangle$ is a "description" of a Turing machine $M$. One way to think of it is to think of any reasonable method for writing down Turing machines and then to imagine $M$ as a string written down in this way. Another way to think of it is that every possible Turing machine has been enumerated and listed in a big book somewhere, and then $\langle M\rangle$ is the number for the appearance of $M$ in the book, with $\langle 1\rangle$ being the first one on the first page, and so on.
$B_{TM}$ here is the set of strings $S$ such that $S$ describes a Turing machine $M$—that is, such that $S=\langle M\rangle$—and $M$ halts on $\epsilon$.