If $f \circ g$ is the identity, is it always true that g is the inverse of f? It seems logical by definition that $f \circ f^{-1} = I$, but then again, I may not be reading the definition correctly and I am wondering if there are any counterexamples to this? 
 A: You can find a counterexample with functions $\mathbb N\to \mathbb N$. If $f(n)=2n$ and $g(n)=\frac{n}{2}$ for even $n$ and $0$ otherwise, then $g\circ f$ is the identity, but $f\circ g$ is not.
You can make a lot of analogues for this simplistic example. 
For example, if you consider the set of real polynomials $\mathbb R[x]$ as functions, and you take $f(p)=\int_0^x p(t)dt$ and $g(p)=\frac{dp}{dx}$, then the fundamental theorem of calculus says $(g\circ f)(p) =p$ for all polynomials $p$, so that $g$ inverts $f$ on the left. But of course $f\circ g$ is not the identity on the constant polynomials. These two maps are even $\mathbb R$-linear vector space maps.
If you feel you understand linear maps very well, then you can make another example very similar to the one above by fixing a basis $\{b_i\mid i\in \mathbb N\}$ of an infinite dimensional vector space, and then choose $f$ to be the linear map that sends $b_i\to b_{i+1}$ for all $i$, and choose $g$ to be the linear map that sends $b_0\to 0$ and $b_i\to b_{i-1}$ for $i>0$. Again, you get that $g\circ f$ is the identity, but $f\circ g$ is not.
Another thing you can observe in the above examples is that you can tweak $g$ and get infinitely many different $g$'s such that $g\circ f$ is the identity. This is not possible when $f$ has a two-sided inverse: at that point, the inverses on both sides are equal and unique.
A: No, if we only know $f\circ g=1$ we cannot conclude that $f$ and $g$ are inverses, but instead only the weaker conclusion that $g$ is a section or a right-inverse of $f$. (Equivalently $g$ is a retract or left-inverse of $f$)
You need both conditions $f\circ g= 1$ and $g\circ f=1$ to conclude that they are inverses. Actually it is enough to show that $f$ has both a right-inverse and a left-inverse. Then it is guaranteed that the two are equal so $f$ has an inverse.
For example, let $f\colon \mathbb{R}^2\to\mathbb{R}$ project the plane down to the line so $f(x,y) =x$ and let $g$ be the inclusion of the line back into the plane, $g(x)=(x,0)$. Then $f\circ g = 1$ (since $f(g(x))=x$) but $g\circ f$ is not (since $g(f(x,y))=(x,0)\neq (x,y)$ if $y\neq 0$), so they are not inverses. 
A: No. Also $g\circ f$ must be identity.
If $f,g$ are considered in a non commutative group, right and left inverse could be different.
In fact, in the most general case, $g$ is defined as the inverse of $f$ iff $f\circ g=g\circ f=I$
A: Hint: You should be able to prove that a function has an inverse on one side if and only if it's injective (one to one), and on the other side if and only if its surjective (onto). So having an inverse is equivalent to having an inverse on each side, and to bijectivity.
An example for one part of this. Consider the function from $\{1,2\}$ to $\{3\}$ that maps everything to $3$.
