I came across such a function written by $O$. Can you please tell me what is this? Actually I see this function during proofs or error finding. I am familiar with big-O notation in algorithmic complexity, but I am talking about when it is used in math. Example:
It's used in the exact same way as in complexity theory
$f=O(g)$ if and only if there exists a $k$ and an $N$ such that for all $x>N$, $f(x)<kg(x)$
It's not a new symbol. It's generally used in a manner similar to in complexity theory: to lump together the terms into a single term that dominates them in the analysis.
The main difference in the usage is that in error analysis, it's the slowly growing functions that matter rather than the fast growing ones. When $x$ is small, $x>x^2>x^3>\ldots$. It's usually used in the context of taylor series, which allows you to secure small values of $x-c$ (by centering the series near $x$). Therefore the dominating term is the smallest exponent, in contrast to evaluation at large values (such as in CS run-time) where the dominating term is the large exponent.
There is a discussion of this here
It is the same big-O as in algorithmic complexity. The purpose in the specific context being to simplify the formal handling of terms that vanish in the asymptotic limit.