Why must an interior point of $E$ be an element of $E$? This question takes place in a general metric space $X$.  
Let $x$ be an interior* point of $E \subset X$ iff there exists a deleted neighborhood of $x$ that is contained in $E$.  
This is like the normal definition of "interior point", except it uses "deleted neighborhood" instead of "neighborhood", thus allowing a point not in $E$ to be an interior* point of $E$.
My question is: why is this not the standard definition of "interior point"?  I see a couple reasons that it would make a more elegant system.


*

*"Limit point" and "interior* point" are both defined in terms of deleted neighborhoods ($x$ is a limit point of $E$ iff all deleted neighborhoods of $x$ include some point of $E$).  This is more symmetrical.

*(Note: I do not yet have a general/categorical notion of duality) "Limit point" and "interior* point" are more adequately dual, for $x$ is a limit point of $E$ iff $x$ is not an interior* point of the complement of $E$, whereas this does not hold for "limit point" and "interior point".

*The dual notions of closure and interior are more symmetrically defined using "interior* point".  The closure is defined as the union of $E$ and the set of limit points of $E$, and the interior is defined as the intersection of $E$ and the set of interior* points of $E$.  The duality between closure and interior is harder to see with the standard definition of interior as the set of interior points of $E$.  Also the proof that the complement of the closure of $E$ is the interior of the complement of $E$ reduces to a few applications of DeMorgan's law.


So why do people use "interior point" and not "interior* point"? 
 A: The notions of "interior point" and "limit point", as you point out, are not dual. And if we use interior point to define interior and limit point to define closure, then we've just used two non-dual notions to define two dual notions. This is highly dissatisfying, as you know. Which one should we keep and which one should we modify?
Before you jump to the conclusion that "limit point" is golden and "interior point" is backwater, consider the opposite perspective for a moment.
Let $x$ be a "limit* point" of $E$ if every (non-deleted) neighborhood of $x$ intersects $E$. This is like the definition of "limit point" except that it uses "neighborhood" instead of "deleted neighborhood", thus allowing all points in $E$ to be limit* points of $E$.
Three reasons why limit* points are more elegant than limit points: Now limit* points and interior points are both defined in terms of neighborhoods, which is more symmetrical. Limit* points and interior points are now dual. And, most importantly, the dual notions of closure and interior are more symmetrically defined using limit* points: the closure is defined as the set of all limit* points, and the interior is defined as the set of all interior points.
You see that the entire opposite argument works, as I've shown. In fact, this way is the nicer one, because closure and interior both have simpler definitions with limit* point and interior point than with limit point and interior* point. This way, we can say that interior points are just the points in the interior. And similarly we can say that closure points are the points in the closure. What I've been calling limit* points has a standard name: either adherent points or closure points. People make use of "limit* points" because it is a good way of thinking about closures and closed sets, which is simpler in many contexts than limit points. People tend not to make use of "interior* points" to think about interiors and open sets for the converse reason.
A: I can't speak much for categorical reasons, but here are my opinions from the topological point of view.
I think a large reason is that the closure of a set is a much more fundamental concept than the derived set $A^\prime$ (i.e., the set of all accumulation (limit) points of $A$). This is perhaps evidenced by the following definition of the closure:

$\overline{A}$ is the smallest (with respect to $\subseteq$) closed set including $A$ as a subset.

From this one can easily prove the characterisation that $x \in \overline{A}$ iff every (open) neighbourhood of $x$ meets $A$.
While we have the equality $\overline{A} = A \cup A^\prime$, this is sort of an artificial way to look at it, since the points of $\overline{A}$ left out of $A^\prime$ are exactly the isolated points of $A$: those elements $x$ of $A$ which have an open neighbourhood which intersects $A$ only at $x$. We recover these points by not caring where open neighbourhoods of $x$ meet $A$, but only insisting that they do.
Taking the closure as more primary than the derived set, without too much difficulty we can show that $$X \setminus \mathrm{Int} ( A ) = \overline{X \setminus A}$$ or, equivalently, $$X \setminus \overline{A} = \mathrm{Int} ( X \setminus A )$$ giving a very distinct connection between the concepts of interior and closure. The same connection would not hold with the $\mathrm{Int}^*$ operator.
(More anecdotally, I cannot recall ever seeing the $\mathrm{Int}^*$ concept used. This gives at least circumstantial evidence to the idea that topologists have not found much use for it, which is a good reason for not basing too much on the idea.)
A: A good test of a new definition is its power of expressiveness. What I mean by this is, when introducing the definition into mathematical discourse, does it help you express mathematical ideas or concepts in a way that enhances understanding, aids discovery, quickens comprehension of proofs, and so on? 
As others have said, $\text{interior}^*$ is unfamiliar, so I do not know whether it will pass this test. If you want to know whether it will, try preparing a few lectures of elementary topology using it. Who knows? Maybe it will catch on.
