Differential operator which is not self-adjoint, contradictory calculation

Consider the symmetric differential operator $-\frac{d^2}{dx^2}$ with domain $D = \{f \in H^2(0,\pi): f(0)=f(\pi)=0\}$ where $H^2$ is the space of all smooth $f$ such that $f'$ is absolutely continuous, i.e., such that there exists an integrable $h$ with $f'(x) = h(0) + \int^x_0 h(t) dt$ and such that $f''$ is square-integrable. Now, consider the function $f(x) = \frac{1}{2 \sqrt \pi} x(\pi-x) \in D$. What is wrong with the following 'reasoning': $1 = \langle A f, A f \rangle = \langle f, A^2 f \rangle = 0$.

I understand of course, there is an important difference between symmetric and self-adjoint. In order to be well-defined for the operation used for the second equality, we need that $f$ is in the domain of $A^\ast$. What goes wrong here? Is $f$ not in the domain of $A^\ast$? Thanks.

The operator $A$ you have defined is selfadjoint. The problem is that $Af$ is not in the domain of $A$. $Af$ is a constant function which does not vanish at $0$, $\pi$.