From the way you are using terminology, this doesn't seem like it's quite at the level of a technical question, and more of just an "intuitive" thing, so forgive me if this answer is light on details. This isn't even really an answer but a long comment. (If you meant to ask a formal question, I suggest defining in some detail precisely what you want from a "frame" in "is it possible to define a frame," and clarifying what exactly the boundary curve of the Möbius strip ought to have to do with this.)
I don't know what you mean by "fails" in "the Frenet-Serret frame fails". It's a perfectly good space curve, and it has its Frenet-Serret frame just like any other curve. (The fact that it is a closed loop presents no difficulty, in this respect it is really no different from the circle.)
But there is no reason to expect the TNB frame on the boundary curve to give you a globally defined and continuous normal vector field on the surface that the curve bounds.
If you think about it, there is nothing unusual about this. A given space curve has only one TNB frame, but it can bound very different looking surfaces. The unit circle in the $xy$-plane is the boundary of a flat disc-shaped surface in the $xy$-plane, but it is also the boundary of a hemispherical bowl, and any number of other shapes. If you've ever used a large hoop or loop of string with soapy water to create large bubbles, you have seen that a single curve can bound any number of very different looking surfaces. Surfaces can differ quite a lot in their geometry--- e.g. how they are curved, whether or not they are curved--- and even in their topology--- while having the same boundary curve.
It takes a bit of visualization to see it, but if you take a Möbius strip and mentally erase the strip, leaving only the boundary curve, you can "fill in" a new surface, with that same curve as its boundary, that is just a distorted disc. This new surface is orientable (both in the technical and intuitive senses of that term). But its boundary curve is the same as that of the Möbius strip, and of course the TNB frame of the boundary curve has not changed.
So it's actually quite intuitive that you can't expect the geometry of a surface to be in any way "recoverable from" or "determined by" its boundary curve.
As for why the Möbius strip is nonorientable--- if you want a formal mathematical proof of this (and not just an "intuitive" sense of it based on a visual understanding of orientability)--- I think the shortest answer is just to begin studying a book on differential geometry (or differential topology, or wherever) where notions like "surface" and "orientable surface" are given rigorous mathematical definitions, and to go from there. (It turns out to quite a lot of work.) Unfortunately, I can't think of a good reference that doesn't assume a lot of prerequisites. Maybe someone will add one in a comment.