Regarding a notation related to divisors & elliptic curves Section 5.8 of the book An Introduction to Mathematical Cryptography defines the divisor of a rational function $f(X,Y)$ defined on an elliptic curve $E: Y^2 = X^3 + AX + B$ as the formal sum:
$\text{div}(f) = \sum\limits_{P \in E} n_P [P]$
The positive coefficients $n_P$ correspond to the multiplicity of $P$ as a zero of $f$, and the negative coefficients $n_P$ correspond to the multiplicity of $P$ as a poles of $f$.
The book adds:
In this formal sum, the coefficients $n_P$ are integers, and only finitely many of the $n_P$ are nonzero, so $\text{div}(f)$ is a finite sum.
On page 318, it defines sum of a divisor, by dropping the square brackets:
$\text{Sum} \left( \text{div}(f) \right) = \text{Sum} \left( \sum\limits_{P \in E} n_P [P] \right) = \sum\limits_{P \in E} n_P P$

Actually, I don't understand the notation $[P]$. What do the square brackets mean here?

 A: A divisor on $X$ is, as you said, a formal sum $D=\sum{n_P}\cdot[P]$. You can see a divisor as an element of the free abelian group generated by the points of $X$. The term "formal" is because, in principle, there is no way to actually sum two points on a variety.
On the other hand, an elliptic curve has the structure of an abelian group, so it makes sense to sum two points of $X$. One can also mutiply a point $P \in X$ by an integer $n$: $~n\cdot P = P + \cdots +P~$ ($n$ times) if $n\geq 0$ and $~n\cdot P = -(P + \cdots +P)~$ if $n< 0$.
So the operation Sum (which essentially removes the bracket) takes the formal sum to the actual sum
A: Some explanations on the addition on $E$. Fix a point $\infty$ which will the neutral element of the group. If $P, Q\in E$ (rational points), then the sum $P+Q\in E$ is the unique point on $E$ such that as divisors we have 
$$ [P+Q]+[\infty] \sim [P]+[Q]$$ 
where $\sim$ means linear equivalence. In other words, 
$$ [P+Q]-[\infty] \sim ([P]-[\infty])+ ([Q]-[\infty]).$$ 
Replacing $P$ with $P-Q$ in the above equality we get
$$ [P-Q]-[\infty] \sim ([P]-[\infty])-([Q]-[\infty]).$$ 
By induction, it is easy to see that for any $P_1,\dots, P_r\in E$ and $n_1, \dots n_r\in \mathbb Z$, we have 
$$\left[\sum_{1\le i\le r} n_iP_i\right]-\left(\sum_{1\le i\le r} n_i\right)[\infty] \sim \sum_{1\le i\le r} n_i[P_i].$$ 
For a principal divisor $\mathrm{div}(f)=\sum_P n_P[P]$, the degree $\sum_P n_P$ is zero, so 
$$\left[\sum_P n_P P\right]=\sum_P n_P[P].$$ 
I think the last member in your equalities is $\sum_P n_P P$ (without Sum()) as pointed out by Lucas in the comments.  
