Let $f$ be a continuous function such that... 
Assuming $f^{-1}$ exists,if i take $f(x)=\sqrt x,$ as given in option $(D)$ then I get $f(x)=f^{-1}(x)=\sqrt x$. Now $f^{-1}(x)=\sqrt x$ gives $f(x)=x^2$ which clearly violets the fact $f(x)=\sqrt x.$So option $(D)$ can not be right.Am i going in the right direction? But I do not have clue about other options.Can someone point me in the right direction? Thanks in advance for your time.
 A: Regarding (D), there is no such thing as a function $\mathbb R\to\mathbb R$ defined by a formula $\sqrt x$.  Thus, strictly speaking, your argument misses the mark, but it would be mostly valid if we restricted to positive real numbers.  If it were instead assumed that we were only working with positive numbers, you could also note that $\sqrt{\sqrt{16}}=2\neq 16$.  
"Assuming $f^{-1}$ exists": I don't know what "assuming" means here.  The hypothesis implies directly that $f^{-1}$ exists and $f^{-1}=f$.  You also noted that $f^{-1}=f$ in your argument.
Regarding monotonicity, you can prove that $f$ is injective (a.k.a. one-to-one), and you can use the Intermediate Value Theorem to show that a continuous injective function $\mathbb R\to\mathbb R$ is monotone.
tetori already answered (B) in a comment with the counterexample $f(x)=-x$.  Functions from $\mathbb R$ to $\mathbb R$ that are their own inverses can be pictured as those whose graphs are symmetric across the diagonal line $y=x$.  For every real number $c$, $f(x)=c-x$ is another example, but there are many more.  For more information, see Examples of involutions on $\mathbb{R}$.
