If the function $Q(x,y)=ax^2+2bxy+cy^2$, restricted to the unit circle, attains its max at $(1,0)$, then $b=0$. 
I feel confused about the following proof. Why did the author introduce $\epsilon$ in parametrizing the unit circle? To trace out the circle, one can simply use the closed interval $[0,2\pi]$. Besides, why did the derivative of $Q$ w.r.t. $t$ have to vanish at $t=0$? After all, by the extreme value theorem, a continuous function defined on a closed interval can attain its extremes at endpoints, points where the derivative vanishes, or points where the derivative is not defined.


 A: 
Why did the author introduce $\epsilon$ in parametrizing the unit circle?

The purpose of considering the open interval was so that $t = 0$ becomes an interior point of the interval.

Besides, why did the derivative of $Q$ w.r.t. $t$ have to vanish at $t=0$?

Because $t = 0$ is an interior point of the domain and if a local maximum occurs at an interior point, then the derivative must be $0$. (This is not true for the endpoints. Which is why the author considered $\epsilon$.)

After all, by the extreme value theorem, a continuous function defined on a closed interval can attain its extremes at endpoints, points where the derivative vanishes, or points where the derivative is not defined.

The "extreme value theorem" isn't really applicable anymore since the domain isn't compact. (It isn't closed.)
Similarly, it also doesn't make sense to talk about endpoints since every point is an interior point. One can also see that the derivative is indeed defined everywhere here.
A: Here, it is actually given in the problem that

$Q(x,y)$ is maximized at $(x,y)=(1,0)$,

so when $(x,y)$ is reparametrized as $(\cos(t),\sin(t))$, since $(x,y)$ are restricted to the unit circle, whose points also have the same parametric form with $t\in [0,2\pi)$ giving all the points i.e. $Q(.)$ as a function of $t$, the above is same as saying that

$Q(t)$ is maximized at $t=0$ or $t=2\pi$, and both of these points are valid maximum points as you have noted, by the extreme value theorem, if you consider $t\in[0,2\pi]$

To get to $b=0$, you actually have to perform the derivative with respect to $t$, and then equate it at $t=0$ and $t=2\pi$ to $0$.
This is where the extra region of $(-\epsilon,0)\cup (2\pi,2\pi+\epsilon)$ is needed, because if you only consider $Q(t)$ with the domain restricted to $[0,2\pi]$, then $Q(t)$ is not differentiable at the endpoints of it's domain, because the left hand limit $\lim\limits_{h\rightarrow 0-} \dfrac{Q(0+h)-Q(0)}{h}$ is not defined (i.e. at $t=0$) and the right hand limit $\lim\limits_{h\rightarrow 0+} \dfrac{Q(2\pi+h)-Q(2\pi)}{h}$ is not defined, unless the domain of $t$ contains some region to the left of $t=0$ and to the right of $t=2\pi$, and even if you consider one-sided derivatives, it is not difficult to come up with a function that has derivative $\ne 0$ at the point where it attains maximum, like $f(x)=x^2, \ x\in [-1,2]$. This is because the proof of the fact
$$\textit{derivative of a differentiable function is }0 \textit{ at it's extrema}$$
requires approaching the point of extremum from both sides.
That's why you need the padding of $[0,2\pi)$ with some good old $\epsilon$ thickness on both sides.
