As I know it, a periodic function is a function that repeats its values in regular intervals or period. However Bézier curves can also be periodic which means closed as opposed to non-periodic which means open. How is this related or possible?
A curve $C$ parameterised over the interval $[a,b]$ is closed if $C(a) = C(b)$. Or, in simpler terms, a curve is closed if its start point coincides with its end-point. A Bézier curve will be closed if its initial and final control points are the same.
A curve $C$ is periodic if $C(t+p) = C(t)$ for all $t$ ($p \ne 0$ is the period). Bézier curves are described by polynomials, so a Bézier curve can not be periodic. Just making its start and end tangents match (as J. M. suggested) does not make it periodic.
A spline (constructed as a string of Bézier curves) can be periodic.
People in the CAD field are sloppy in this area -- they often say "periodic" when they mean "closed".