Everything actually works for arbitrary ringed spaces and arbitrary sheaves of $\mathcal O_X$-modules, cf. e.g. Tag 01CM of the Stacks Project (where the proof of this lemma is omitted).
Firstly, you should be aware of three subtleties:
- $M \otimes_{\mathcal O_X} N$ is the sheafification of the presheaf
$$U \mapsto M(U) \otimes_{\mathcal O(U)} N(U).$$
To avoid confusion, I will write $M \odot_{\mathcal O_X} N$ for this presheaf tensor product.
- The sheaf $\mathscr Hom_{\mathcal O_X}(M,N)$ is given on $U$ by $\operatorname{Hom}_{\mathcal O_U}(M|_U, N|_U)$, not $\operatorname{Hom}_{\mathcal O(U)}(M(U),N(U))$. This makes it slightly harder to define the 'obvious map', as we will see below.
- For $M, N$ both quasicoherent, it is not in general true that $\mathscr Hom_{\mathcal O_X}(M, N)$ is quasicoherent (this is the remark you make on the bottom; it is enlightening to try to think of a counter-example). However, it does work when $M$ is coherent (and $X$ is Noetherian, or use the correct definition of coherent sheaves on an arbitrary scheme or even ringed space).
I will define an obvious (but not so obvious) isomorphism
$$f \colon \mathscr Hom_{\mathcal O_X}(M, \mathscr Hom_{\mathcal O_X} (N, K)) \to \mathscr Hom_{\mathcal O_X}(M \otimes_{\mathcal O_X} N, K).$$
We have to construct compatible isomorphisms
$$f_U \colon \operatorname{Hom}_{\mathcal O_U} (M|_U, \mathscr Hom_{\mathcal O_U} (N|_U, K|_U)) \to \operatorname{Hom}_{\mathcal O_U}((M \otimes_{\mathcal O_X} N)|_U, K|_U)$$
for all $U \subseteq X$. Let $U$ be fixed from now on. I will break down what both sides are.
Right hand side. Since sheafification commutes with restriction to opens, the right hand side is
$$\operatorname{Hom}_{\mathcal O_U}(M|_U \otimes_{\mathcal O_U} N|_U, K_U) = \operatorname{Hom}_{\mathcal O_U}^{\operatorname{pre}} (M_U \odot_{\mathcal O_U} N|_U, K|_U).$$
An element of this is a compatible family of maps (for all $V \subseteq U$)
$$\psi_V \colon M(V) \otimes_{\mathcal O(V)} N(V) \to K(V).$$
However, as pointed out by the OP in the comments, you cannot now use the classical tensor-hom adjunction to say that this is the same as giving compatible maps
$$M(V) \to \operatorname{Hom}_{\mathcal O(V)}(N(V), K(V)).$$
See also the remark below.
Left hand side. On the other hand, the left hand side consists of giving a compatible system of maps
$$\phi_V \colon M(V) \to \operatorname{Hom}_{\mathcal O_V}(N|_V, K|_V).$$
The map $f$. Thus, there is an obvious map from left to right given by taking global sections: given $\phi_V$ and $m \in M(V)$, the map $\phi_V(m)\colon N|_V \to K|_V$ induces a map $\chi_V(m)\colon N(V) \to K(V)$ by taking global sections. This gives $\chi_V \colon M(V) \to \operatorname{Hom}_{\mathcal O(V)}(N(V), K(V))$. Using the classical tensor-hom adjunction, this in turn gives a map
$$\psi_V \colon M(V) \otimes_{\mathcal O(V)} N(V) \to K(V).$$
However, having a map $\chi_V(m) \colon N(V) \to K(V)$ does not give you a map $\phi_V(m) \colon N|_V \to K|_V$ in general. It works when $V$ is an affine scheme and $N$ is quasicoherent, but that is not satisfactory. Thus, defining an inverse is not quite so easy.
The inverse of $f$. This is where the compatibility of the various $\psi_V$ comes in. That is, given a system $\psi_V$ from the right hand side, for any $m \in M(V)$, we have to construct a morphism of sheaves
$$\phi_V(m) \colon N|_V \to K|_V.$$
For any $W \subseteq V$, we define
\begin{align*}
\phi_V(m) \colon N(W) &\to K(W)\\
n &\mapsto \psi_W(m|_W \otimes n).
\end{align*}
One checks that these are compatible for varying $W$, so we get $\phi_V(m)$ as claimed. Then check that the $\phi_V$ are compatible for varying $V$, etc. $\square$
I also omitted the verifications that anything was $\mathcal O$-linear, or even that the maps defined above are actually inverses. All of this is relatively straightforward.
Remark. Given a commutative diagram
$$\begin{array}{ccc} M(V) \otimes_{\mathcal O(V)} N(V) & \to & K(V) \\ \downarrow & & \downarrow \\ M(W) \otimes_{\mathcal O(W)} N(W) & \to &\ K(W), \end{array}$$
we do not get a commutative diagram
$$\begin{array}{ccc} M(V) & \to & \operatorname{Hom}_{\mathcal O(V)}(N(V), K(V)) \\ \downarrow & & \downarrow \\ M(W) & \to &\ \operatorname{Hom}_{\mathcal O(W)}(N(W), K(W)).\end{array}$$
In fact, we don't even get the right vertical map, because of the contravariance of $\operatorname{Hom}$ is the first variable. Thus, we cannot simplify the description of the right hand side.
