In topological terms, we could say that set $A$ is inside of set $B$ (where the sets are disjoint) if $A$ is contained in a bounded connected component of $\mathbb R^3\setminus B$. In other words, $B$ separates $A$ from $\infty$. However, this does not help in your plate-cup-spoon example, because the spoon is not inside of the cup in the topological sense. Indeed, if the plate were elastic (as all things are in topology), we would be able to deform it gradually until it becomes a bigger cup with smaller cup inside of it. So there is no clear cut between the plate-cup and the cup-spoon configurations.
If your objects are presented as sets of points in $\mathbb R^3$, you can use a linear SVM with soft margin to find the best separating plane between them. Then look at the quality of separation: if it's not very good, then one of the objects is probably inside of the other. This algorithm is not infallible, but it would work for plate-cup-spoon.
