Is infinitely small error the same as $0$ error? I am really confused over the idea.
If I have a circle I can approximate its area by using a regular polygon inside of it, with $n$ sides, and I can just split that polygon into triangles and compute the area. If I want better accuracy then I can use a polygon with more sides. The accuracy becomes better since the area between the polygon and circle (the "error") decreases as $n$ increases.
So it stands to reason that if we could keep adding sides our answer would keep getting better since the error would get closer to $0$.
I don't know what it means to "add infinitely many sides" because to me the idea doesn't make sense. No matter how many sides there are we can always add one more. But it does make sense to ask what's the value we never actually reach but get closer and closer to? For that we use the concept of a limit.
The error becomes "infinitely small" but I don't understand what this really means. Is "infinitely small" the same as $0$? Because by definition it's never quite getting to $0$. But conceptually then how can we say something with nonzero error gives us an exact answer as if it had $0$ error? 
 A: Maybe it helps to look at the problem from another side:
You want to determine the area of the circle (well, to start with, you assume that the circle actually has an area, which strictly speaking isn't guaranteed anyway: There do exist sets which don't have an area). But you only know how to determine the area of polygons.
Now you observe that if a polygon is inscribed in the circle, it certainly cannot have a larger area than the circle. This is true for every such polygon, therefore you know that the area of the circle, if it exists, is an upper bound to the set of areas of inscribed polygons. Therefore it certainly cannot be less than the least upper bound to those areas, that is, their supremum.
Now it turns out that if you calculate the area of an inscribed $n$-gon, it is monotonously growing with $n$, therefore their supremum equals the limit for $n\to\infty$.
Note however that at this point, all you know is that the limit gives a lower bound to the area of the circle. For all you know, the circle might be larger than that.
However you can now do a second observation, namely that the area of a circumscribed polygon is always larger than the area of the circle, and therefore their greatest lower bound (the infimum) is an upper bound of the circle's area.
Now the area of the circumscribed polygons decreases with $n$, therefore now the infimum of those equals their limit for $n\to\infty$.
So now you have a lower bound to the area of the circle, which is given by the limit of the areas of the inscribed polygons, and an upper bound to that area, which is given by the limit of the areas of the circumscribed polygons. And now it turns out that both bounds are the same. Now if the lower bound to the circle's area equals the upper bound, then obviously the area of the circle must equal that. Note that it also proves that area of the circle actually exists.
A: Infinitely small error and $0$ error surely aren't the same thing.
Even though it seems like you already grasp the concept, let me reword part of the train of thought exposed:
Draw a circle of radius $1$ and fix a number $\epsilon > 0$. Make it really as small or as big as you want, and now ask me to create a regular polygon whose area misses the exact area of the circle by less than $\epsilon$. If you make $\epsilon$ really big, then I won't need to draw a polygon with many sides but if you make $\epsilon$ really, really small, then I may need to draw a polygon with a very big number of sides! Either way, we can agree that whatever $\epsilon > 0$ you choose, my task isn't impossible. Right?
Thus you probably can't say that my best approximation to the area of a circle has error $\delta$ with $\delta > 0$. Why not? Just set $\epsilon = \delta$ and build a polygon with approximation error inferior to $\epsilon$. So if you really insisted on finding a number that you could say "the approximations by polygons have error blah", then no positive number would do... so it is just like $0$ is the first number that isn't just wrong... or it is like $0$ is the less wrong answer, in a sense...
That is the way I would explain this relation, but certainly infinitely small and $0$ aren't the same thing.
In a not-so-precise sentence, infinitely small is not $0$, but it is as if it were for all practical purposes.
A: Responding to your last sentence, the answer is not the area of polygons, but rather the area of circle. We have techniques developed to find the limit given the function and point under consideration. By the way the example of circle is bad one to explain the way the answer is obtained.
The simpler option is to consider a triangle with vertices $A(0,0),B(1,0),C(1,1)$ and consider a set of rectangles $R_i$ with vertices $(i/n, 0),((i+1)/n,0),(i/n,i/n),((i+1)/n,i/n)$ for $i=1,2,\dots n-1$. The area of each rectangle $R_i$ is clearly $i/n^2$ and the total area of all such rectangles is $(1+2+\cdots +(n-1))/n^2=(n-1)/2n$ and the desired area of triangle $ABC$ is the limit of this expression which is clearly $1/2$.

From your comments it appears that you think that area of plane regions is a pre-defined concept and we are just trying to approximate it by various methods. No!!
The area of a plane region which is a rectangle is defined as product of its width and height/length. For plane regions which are not rectangles the area is defined by a somewhat lengthy procedure. Essentially first we find a large rectangle say $R$ which contains region of interest say $I$. Next we divide the rectangle $R$ into multiple rectangles by drawing lines parallel to its edges. Suppose the width is divided into $m$ parts (not necessarily equal) and height is divided into $n$ parts (again not necessarily equal. Count all the small rectangles which are contained in $I$ and add their areas to get $s$. Further count all the small rectangles which contain some point of $I$. Add their areas to get another number $S$.
These numbers $s, S$ depend on the region of interest $I$ as well as mode of division of $R$ into $mn$ small rectangles. Consider the infimum $a$ of all numbers like $S$ and supremum $A$ of all numbers like $s$. If $a=A$ then we say that the region $I$ has area $A=a$ otherwise the region can not be assigned any area. Using some more effort we can show that the area $A$ can be obtained as a limit of such approximations like $s, S$. 
A: First there is a technical problem that should be addressed. 
 The accuracy becomes better since the area between the polygon and circle 
 (the "error") decreases as n increases.

Actually it only looks like the "error" decreases as n increases unless you can prove it. This is done by considering polygons that are inscribed in and circumscribed about the circle. In which case the area of the circle must be between those two areas. Then you show that the difference of the areas of the two polygons goes to $0$ as n increases.
Adding "infinitely many sides" is nonsense. It is really just an impractical (colorful ?) way of saying let n approach infinity.
However, there is a field called nonstandard analysis in which the real numbers are augmented with numbers that are not equal to $0$ but whose magnitudes are smaller than any positive real number. It would be too much work to describe how that works but you might want to check it out.
A: When you are dealing with a framework including rigorous infinitesimals, then infinitely small error is indeed the same as zero error in the following sense.  Consider for instance the problem of differentiating $y=f(x)=x^2$.  One chooses an infinitesimal $x$-increment $\Delta x$ and computes the corresponding $y$-increment $\Delta y=f(x+\Delta x)-f(x)$, obtaining $2x+dx$.  
The latter expression already contains all the information about the slope at the point, meaning that the derivative can now be computed by applying the standard part: $f'(x)=\textbf{st}(2x+dx)=2x$ where $\textbf{st}$ is the standard part function. In this sense there is no error involved once you know the answer up to an infinitesimal "error".
