John Baez once argued that $0$-categories should be preorders rather than sets, but the argument was based on higher category theory. Specifically, that the hierarchy of categories should start at $-2$, and goes:
- There is just one $(-2)$ category.
- $(-1)$ categories are equivalent to truth values -- a $(-1)$ category has objects, and the homsets are $(-2)$-categories. Therefore either a $(-1)$ category is empty, or it is non-empty and all objects are isomorphic.
- $0$ categories should be equivalent to preorders -- a $0$ category would have objects, and the homsets are $(-1)$-categories: i.e. truth values. Working out what the composition law should mean amounts to making the value of $\hom(X,Y)$ mean the truth of $X \leq Y$ where $\leq$ is a preorder.
Then, we recover sets as being analogous to groupoids: they are preorders whose arrows are all isomorphisms. Actually, this gives setoids -- sets with an equivalence relation -- but setoids are equivalent to sets.
(IMO, the rationale that preorders are a better basic notion that set is not implausible even without trying to imagine $(-2)$- and $(-1)$-categories. And I definitely think setoid is a better basic notion than set)
I don't recall him saying anything about duality, though, so this may have nothing to do with what you're remembering.