This got too long for a comment but is meant to be an extended one. I'm note quite the guy to say being mostly interested in it from a structural/mathematical perspective. Forgive me if I'm not telling you anything new.
You can definitely do TQFT within the confines of pure math. If what you want is the standard model you'll do well to understand your representation theory, as types of particles correspond to fundamental representations of Lie groups ($U(1)\times SU(2)\times SU(3)$ in the standard model, times the Poincaré group if you do the analysis.) From there a quantum field is a section of a vector bundle associated to the representation over space-time satisfying a variational principle (an extremal of an action) involving suitably equivariant connections (which are incidentally your bosons). Faria-Melo develops this and in fact exhibits the standard model in this framework.
They leave out a clear analysis of how representations tie in with types of particles, but this is done by Baez and Huerta in this text (http://math.ucr.edu/~huerta/guts/). Basically, elements in your fundamental representations are fermionic particle states, generators of the adjoint representation are bosons that act on your fermions in a way that can be represented by Feynmann diagrams.
Quantization is still fluffy to me, but it appears this is where quantum groups come in: You cannot deform a semi-simple Lie algebra and get a reasonable deformation of its representation theory (it's category of representations). You can however deform its universal enveloping algebra (which is a Hopf algebra, i.e., an object with a favourably interacting product and coproduct). There is a master class on this going on right now which talks about this for the purpose of studying 3-manifold invariants using 3-dimensional field theories. Notes about quantum groups may be found on its web page: http://www.math.ku.dk/english/research/conferences/2014/tqft/ They have incidentally a crash course on operator algebras as well, which is part of the theory that allows you to reasonably deal with infinite dimensional representations of the Poincaré group.
How the functor point of view on field theories relate to the "classical" one developed in Faria-Melo a bit fuzzy to me, but I suspect you may find some answers in Segal's article on conformal field theories (http://www.math.upenn.edu/~blockj/scfts/segal.pdf -- a pretty shitty scan but you'll find it in his 60th birth day thing).
Of course this leaves out nitty-gritty computational aspects of the kind a physicists would be able to tell you about, and I have never gotten close enough to what the physicists do to actually wanting to renormalize anything (something you apparently need to do because of self-interacting particles producing diverging integrals). This is definitely a pretty big part of QFT you'll be missing if you don't study the physicists approach as well.
It appears the big unifying idea in any case is that a physical system should be invariant under choice of presentation (gauge) up to a group or automorphisms (gauge transformation) and that this is true for classical systems (Lorentz or Poincaré invariance of space or space-time) as well as quantum systems (other Lie groups acting on a vector bundle of states) and that all of physics are more or less fall out as properties of stuff with the right symmetries. This appears to be what physicists and mathematicians agree on either way, so you can't go wrong studying representations.
Aside from Faria-Melo here are some notes I like to look at:
These notes are pretty explicit about the kind of mathematics they use math.lsa.umich.edu/~idolga/physicsbook.pdf
These notes on Lie groups and representation theory are very good. staff.science.uu.nl/~ban00101/lie2012/lie2010.pdf They come with video lectures. webmovies.science.uu.nl/WISM414
