Is that a correct notation? $$\dfrac{1}{n}\sum_{i=1}^{n}1_{[y_{i}f(x_{i})<0]}$$
this gives me the number of how many times the expression in brackets is less then zero. 
But is it correct? I see it for the first time, so where(textbook) I can encounter it in an elucidating form?
Or is it from a particular subject like statistics, and reading on calculus won't help me discover this notation?
 A: I think there is (yet) no universally established notation for what you try to express. It is not uncommon to denote
$$1_{P}=\begin{cases} 1 & \text{if $P$ is true} \\ 0 & \text{otherwise}\end{cases}$$
for some statement $P$. In your case $P_i\equiv [y_i f(x_i)<0]$. This construct (indexed by $i$) is sometimes called "characteristic vector/indicator vector" or "characteristic function/indicator function" of some set or statement. I personally like the Iverson brackets (if no confusion can happen):
$$[P]:=\begin{cases} 1 & \text{if $P$ is true} \\ 0 & \text{otherwise}\end{cases}.$$

Anyways, in your case, the sum is counting the number of pairs $(x_i,y_i)$ for which $y_if(x_i)<0$. But this number is divided by $n$, so in the end it gives you the fraction of these pairs, and not the absolute number.
For counting or determining the size/cardinality of a set, there are other established notations like
$$|\{(x_i,y_i)\mid y_if(x_i)<0\}|\qquad\text{or}\qquad \#\{(x_i,y_i)\mid y_if(x_i)<0\}.$$
So e.g. $\#\{1,5,8\}=3$.
