Is this a 'legal' way to show two sets not equal? Part A.
Suppose that for all $x$, $x$ is in a set $X$ iff it is not in another set $Y$. Neither $X$ nor $Y$ is the empty set. And there is a background set Z in which these sets lurk and to which everything is relative. So when we say 'not in another set Y' this is a relative complement between Y and Z, not an absolute complement. From these assumptions, can we derive that $X \neq Y?$
Proof. If $X=Y,$ then $x$ is a member of $X$ if it is not a member of $X$ -- a contradiction. QED.
From the definition, did we prove that $X$ can't equal $Y$ and put a proper restriction on what those sets can be? Or did we show that $X$ is just not well-defined?
Part B. Can this method be 'scaled' to show a set V is not in a certain set R?
I reference a set of sets $R$ in my definition of a set $V$. Any $v$ is a member of $V$ iff it is not a member of any of the sets in the set $R$. It seems that, similar to how we showed inequality before, by definition, we have forced $V$ not be in the set $R$; otherwise, we have a contradiction: $v$ is a member of $V$ if it is not a member of $V$.
Here is the specific example:
For all $v$, $v$ is in $V$ iff $v$ is not in any of these sets: $(W,S,U,T)$ which form a set called $R$. From this assumption can we derive that $V$ is not in the set $R$?
And again, not being in $W, S, U, T$ is relative to a background set $Z$.
 A: Let me note that there are some technical issues with what you seem to be wanting to define in the first place; I'll get to them later, because I think they are not the main issue you are trying to get to.
For the sake of not saying things that are nonsensical in, say, Zermelo-Fraenkel Set Theory, and given what you mentioned in comments, let us assume that we have some "universal set" $U$, and that everything is happening inside that set; so that when you say "for all $x$", you mean "for all $x\in U$". This is part of the technical issues I mention, and I will delve into them further below.
Let us first look at your definition: you say that "$x\in X$ if $x\notin Y$". (Note: While I was writing this, it was corrected to "iff"; but since a lot of the comments refer to this version, and the paragraph after "Part B" still uses "if" instead of "iff", I will keep this.) Remember that "$A$ if $B$" means $B\implies A$. So the statement that you are using to define $X$ is:
$$\forall x(x\notin Y\to x\in X).$$
This does not completely determine $X$. It only tells you some things that are in $X$. Specifically, it tells you that $X$ contains the (relative) complement of $Y$, but it does not preclude $X$ from containing other things. For example, the universe $U$ will satisfy this condition: if $x\notin Y$, then $x\in U$, because "$x\in U$" is always true, and an implication with a true consequent is true. 
If you want to define $X$ as consisting of exactly the elements that are not in $Y$, then you need to say "if and only if", and not just "if". In that case, you would be saying that $x\notin Y$ implies $x\in X$, and that $x\in X$ implies $x\notin Y$. This would be equivalent to saying
$$X = \{x\in U\mid x\notin Y\}$$
in set builder notation. Because in set builder notation, the convention is that $x\in X$ if and only if it satisfies the given condition(s) (in this case, being in $U$ and not being in $Y$). 
I will continue my answer under the assumption that this is what you were trying to give as the definition of $X$.
Now, to your method of proof: it is at best incomplete. What you are arguing is: let us argue by contradiction and assume that $X=Y$. If $x\in X$, then $x\notin Y$ (by definition of $X$), and therefore $x\notin X$ (since $Y=X$).
That's correct as far as it goes.
But now you conclude that you have a contradiction; well, not really or rather, not quite yet. What you have done is proven that (under the undischarged assumption that $X=Y$) you can establish the implication $x\in X\implies x\notin X$.
But this implication is not by itself a contradiction (a statement that is always false). In fact, such a statement is true when $x\notin X$. In general,
$$(P\longrightarrow\neg P)\longrightarrow\neg P$$
is a tautology. 
So you need to take a few more steps. What you have shown is that, under the assumption that $X=Y$, you can prove that for every $x\in U$, $x\notin X$. Because you can prove that $\forall x\in U\bigl((x\in X)\implies (x\notin X)\bigr)$, and so using the tautology above you can deduce that $x\notin X$. 
But this tells you that $X=\varnothing$, which contradicts our assumption that neither $X$ nor $Y$ are empty. And now we have a contradiction.
Where does the contradiction come from? The undischarged assumption is "$X=Y$", and so we conclude that $X\neq Y$.
Alternatively, you can note that if $X=Y$, then $x\in X$ if and only if $x\notin Y$ (by definition of $Y$) if and only if $x\notin X$ (because we are assuming that $X=Y$). So now what we have is
$$\forall x\in U (x\in X\iff x\notin X).$$
Now, if $U$ is not empty, then  that is a contradiction. The statement cannot hold because "$x\in X\iff x\notin X$" is always false, so provided there is at least one $x\in U$, we get a contradiction.  Note that this holds even if $X$ or $Y$ are empty, provided that $U$ is not empty. 
This is fine, but you need to establish the "if and only if", not just the "if". 
As to your second question, same issue: you want your "if"s to be "iffs". Yes, this is a valid method of showing something is not in a set. In fact, that is the argument used by Halmos in Naive Set Theory to prove that, in what amounts to a basic version of Zermelo-Fraenkel Set Theory, there is no universal set.
To do this, he proves the following theorem:
Theorem. Let $A$ be a set. Then there exists a set $B$ such that $B\notin A$.
Proof. Since $A$ is a set, the Axiom of Separation yields that
$$B = \{x\in A\mid x\notin x\}$$
is a set. We claim that $B$ is not an element of $A$.
Indeed, assume by way of contradiction that $B\in A$.
If $B\in B$, then $B\notin B$, by definition of $B$. Therefore, $B\notin B$.
If $B\notin B$, then $B\in B$, by definition of $B$. Therefore, $B\in B$.
Thus, $B\in B$ and $B\notin B$, which is a contradiction. The contradiction arises from the assumption that $B\in A$. Hence $B\notin A$. $\Box$
Corollary. There is no universal set.

Technical issues: in ZF (and in GBN) Set Theory, you cannot define absolute complements; the absolute complement of a set is never a set, because it is a proper class (it contains sets of arbitrarily large cardinality, and no set can contain elements of arbitrarily large cardinality). You can only define relative complements. Most of the time when one talks about complements, as fleablood notes, there is an implicit or explicit set $U$ from which every relevant element is taken, and so we are actually working with relative complements. Your initial definition, in which $Y$ is a set and you define $X$ as an object with $x\notin Y\implies x\in X$ would yield a collection $X$ that cannot be a set in those theories. 
