What kind of modal logics do you get if you replace the semantics of negation with Routley negation? What kind of "almost modal" logics do you get is you replace the semantics for negation with the semantics for Routley negation in an otherwise-standard Kripke frame?
I.e. if we replace the condition $[\lnot a] = [a]^c$ with $[\lnot a] = \{w^* : w \not\in [a]\}$ where $[a]$ means the set of worlds in which $a$ holds, what kind of generalization of modal logic do we get?

In a Routley-Meyer model, we have an involution on worlds $a \mapsto a^*$, a distinguished world $0$, and a three-place relation $R$.
We can paraphrase the presentation given in Wikipedia slightly, and get the following rules for our model. I'll use $aRb$ as an abbreviation for $R0ab$.
$$ aRa \\
\text{If $aRb$ and $bRc$, then $aRc$} \\ 
\text{If $dRa$ and $Rabc$, then $Rdbc$} \\
a^{**} = a \\
\text{If $aRb$, then $b^*Ra^*$} $$
Then, if $W$ is our set of possible worlds, we can make $2^W$ our set of truth values. We pick the elements of $2^W$ that contain the distinguished world $0$ as our designated values.
$$ [a \land b] = [a] \cap [b] $$
$$ [a \lor b] = [a] \cup [b] $$
$$ [\lnot a] = \{w^* : w \not\in [a]\} $$
$$ [a \to b] = \{w : \forall jk \mathop. Rwjk \to (j \not\in [a] \lor k \in [b]) \} $$
And we impose the additional closure condition on interpretations of wffs.
$$ uRw \land u \in [a] \implies w \in [a] $$

I was wondering the other day why we needed the Routley star, but without it, $A \lor \lnot A$ is a tautology because $A$ definitely holds or doesn't hold in the distinguished world $0$. And likewise, $A \land \lnot A$ is not necessarily false.
I'm curious what happens if we use the Routley star semantics for negation in a Kripke frame.
$$ [a \land b] = [a] \cap [b] $$
$$ [a \lor b] = [a] \cup [b] $$
$$ [\lnot a] = \{ w^* : w \not\in [a] \} $$
$$ [\square a] = \{ w : \forall x \mathop. wRx \to x \in [a] \} $$
$$ [\lozenge a] = \{ w : \exists x \mathop. wRx \land x \in [a] \} $$
And impose the conditions:
$$ a^{**} = a \\
\text{If $aRb$, then $b^*Ra^*$} $$
This kind of logic has the interesting consequence that possibility and necessity are no longer dual.
Suppose $V$ is a subset of $W$, then the negation of $V$ is $V^{*c}$ with $V^*$ being defined as $\{w^* : w \in V\}$. And, since $^*$ is an involution, $V^{*c}= V^{c*}$.
So in this case, if $\sim$ is strong negation, then $[\square a] = [\lozenge {\sim} a]^c$, but the same doesn't hold with $\lnot$.
 A: As it stands your semantics has no validities and so it is not straightforward to compare it to other logics. But it can be expanded to a certain relevant logic. Let me explain.
To tackle the issue of logical onmiscience many epistemic logicians had the idea of basing an epistemic modal logic on a weaker relevant logic. Indeed some of those logicians used the Routley-semantics for negation for exactly this purpose. See, for instance, Fagin / Halpern / Vardi's paper "A non-standard approach to the logical omniscience problem". Artificial Intelligence 79 (1995). In this paper a semantics almost exactly like yours is proposed, with some necessary deviations detailed below.
One problem that confronts your semantics is that as it stands it has no validities. To see that let's fix the kind of structures we want to talk about. I'll follow Fagin et al. by letting a non-standard structure ($NS$-structure) for $n$ agents over atom set $\Phi$ be a tuple $(W, R_1, \ldots, R_n, \pi, \space ^*)$, where $(W, R_1, \ldots, R_n)$ is a labelled transition structure as known from Kripke semantics, $^*$ the Routley star with the restrictions you mentioned and $\pi: W \rightarrow 2^{\Phi}$ a valuation. Now, fix any $\Phi$ and consider the $NS$-structure $\mathcal{M}= (W, R, \pi, \space ^*)$ for 1 agent over $\Phi$, with $W = \{s,t \}$, $R$ the identity on $W$, $s^* = t$ and for every atom $p \in \Phi$ we have: $\pi (s)(p) = 0$ and $\pi(t)(p)=1$. By structural induction it is easy to see that $\mathcal{M}, s \not \models \varphi$, for any formula   $\varphi \in \mathcal{L}^{\Phi}$.
To overcome this difficulty we can introduce what Fagin et al. call a strong implication, i.e. a 2-place connective $\Rightarrow$, whose semantic clause is as follows:

$\mathcal{M}, s \models \varphi \Rightarrow \psi$ iff $M, s \not \models \varphi$ or $\mathcal{M}, s \models \psi$

Now, strong implication immediately yields various validities such as $\varphi \Rightarrow \varphi$ or $\varphi \Rightarrow (\varphi \lor \psi)$. Let's call the set of validities of this expanded language the $NS$-logic.
Strong implication is related to the conditional $\rightarrow$ of the relevant logic $R$. It can be shown for $\Rightarrow$-free formulas $\varphi, \psi$  that: $\varphi \rightarrow \psi$ is a theorem of $R$ iff $\varphi \Rightarrow \psi$ is valid with respect to the class of $NS$-structures. For nested strong implications, however this equivalence does not hold: While $p \Rightarrow (q \Rightarrow p)$ is $NS$-valid, $p \rightarrow (q \rightarrow p)$ is not a  $R$-theorem. This suggests that the $NS$-logic might be a fragment of $FDE$, a relevant logic where only relevant conditionals are allowed whose antecedents and consequents are conditional-free. Indeed, Fagin et al. prove that the $NS$-logic is equivalent to Levesque's and Lakemeyer's epistemic logic for implicit and explicit beliefs. The latter logic itself is provably a fragment of $FDE$. So, $NS$-logic is a sublogic of $FDE$ as well.
