Where would I start to learn category theory for its use in knot theory? I have a background in physics and Ive read Adams Knot book. I know nothing about category theory. Eventually I want to learn category theory for its use in physics.
I'm not entirely sure that you want to read a category theory book. Category theory can be quite technical and dry, and difficult to understand if you don't have the motivating examples in mind. Moreover, it's a large subject and you'll only need the tools related to the part of knot theory you want to study.
A better idea is to read a short expository article on category theory, and then attack Khovanov homology directly. Some introductions to Khovanov homology are going to slowly feed you the more advanced tools you need. In any event, you can look up more category theory/homological algebra as is necessary, which I find a more effective way to learn.
Some nice introductory writing on category theory can be found on Wikiversity.
For an introduction to Khovanov homology, Sammy Black has given a fine MSE answer here.