I don't know whether it can be done, but I have also had this question recently.
To show that it could be done, we would need to state axioms in terms of "categories, functors, (other primitive notions)" and show that those were bi-interpretable with the typical axioms in terms of "categories, objects, morphisms".
Obviously categories can be interpreted as categories in both directions, and we know how to interpret functors from primitive notions of objects and morphisms. So what we need is some way to interpret objects and morphisms from primitive notions of functors.
For the sake notation, define (in the typical axiomatization)
$$ \mathbf{1} = \{ \ast \} \,, $$
the category with one object (the identity morphism is meant to be implicit above), and
$$ \mathbf{2} = \{\ast \to \ast \} \,,$$
(sometimes called the "interval category", cf. here or here) the category with two objects and one non-identity morphism.
Then we know in the standard axiomatization that objects of any category $\mathcal{C}$ can be identified with functors $\mathbf{1} \to \mathcal{C}$ and likewise morphisms of any category $\mathcal{C}$ can be identified with functors $\mathbf{2} \to \mathcal{C}$.
So if it is possible, I would guess that it likely requires not just having "category" and "functor" as primitive notions, but also "$\mathbf{1}$" and $"\mathbf{2}"$ as primitive notions. Then in this case objects would be interpreted/defined as functors $\mathbf{1} \to \mathcal{C}$, and likewise morphisms would be interpreted/defined as functors $\mathbf{2} \to \mathcal{C}$.
Using $\mathbf{1}$ as a primitive notion shouldn't be difficult, because that would just require an axiom saying something to the effect that it is the terminal object in the "category of all categories" $Cat$ (cf. here or here for rigorous definitions). E.g. something to the effect that,
There exists a category $\mathbf{1}$ such that for every category $\mathcal{C}$, there exists a unique functor $\mathcal{C} \to \mathbf{1}$.
I would guess that $\mathbf{2}$ also has a universal property that could be axiomatized solely in terms of functors, but I'm not sure. Because $Cat$ is apparently a (2-)topos (apparently the "archetypical example"), I would guess that $\mathbf{2}$ is "probably" (possibly) a subobject classifier for $Cat$ (a "Boolean subobject classifier"?). Anyway, if it has a universal property that can be stated solely in terms of categories and functors, and thus serve as an axiom, we are probably in good shape.
That being said, we would then either need to give axioms for the behavior of functors without referencing $\mathbf{1}$ or $\mathbf{2}$, or cheat and basically encode the standard axioms for objects and morphisms using/in terms of functors from $\mathbf{1}$ and $\mathbf{2}$. So whether such a project is possible (in a meaningful way) would seem to depend on how necessary it is to cheat when axiomatizing functoriality.
At this point it seems like what we might be looking for might be an axiomatization for $Cat$. This is definitely not my area of expertise, but it seems like such problems have been studied, and progress is possibly incomplete. In any case, see the following links for more reading that might give a better answer:
None of these seem to directly give axiomatizations in terms of categories and functors, per se, but the idea at least seems related. Again, hopefully someone gives a more clarifying/edifying answer one day.