# How to define the dual sheaf

If $X$ is a scheme and a sheaf $\mathcal F$of modules on $X$, then how can we define the dual $\mathcal F^*$ of $\mathcal F$? Obviously, we set $\mathcal F^*(U)=\rm{Hom}_{\mathcal O_X(U)}(\mathcal F(U),\mathcal O_X(U))$ and use sheafication, but how can we define the restriction map $\rho_{U,V}$ since functor Hom is contravariant in the first variable?

-

Your definition is not correct. If $F,G$ are sheaves of sets on $X$, then the internal hom sheaf $\underline{\hom}(F,G)$ is the sheaf on $X$ defined by $\underline{\hom}(F,G)(U) := \hom(F|_U,G|_U)$. The restriction map is obvious: For $V \subseteq U$, every morphism $F|_U \to G|_U$ may be restricted to a morphism $F|_V \to G|_V$ (since open subsets of $V$ are also open subsets of $U$ and then you just take the given morphism there). It is an easy and good exercise to verify that this presheaf is already a sheaf. Besides, it really deserves to be called internal hom, because there is a bijection $\hom(H,\underline{\hom}(F,G)) \cong \hom(H \times F,G)$.
For sheaves of modules on a ringed space $(X,\mathcal{O}_X)$ (of you course you don't need a scheme to do this), the situation is quite similar, and we have $\hom_{\mathcal{O}_X}(H,\underline{\hom}_{\mathcal{O}_X}(F,G)) \cong \hom_{\mathcal{O}_X}(H \otimes_{\mathcal{O}_X} F,G)$. If $F$ is locally of finite presentation and $G$ is quasi-coherent, then also $\underline{\hom}(F,G)$ is quasi-coherent. In this situation, if $F,G$ are associated to $\mathcal{O}_X(X)$-modules $M,N$, then $\underline{\hom}(F,G)$ is associated to $\underline{\hom}_{\mathcal{O}_X(X)}(M,N)$.
That's not the correct definition. You are supposed to use the sheaf hom, so that $\mathscr{F}^* = \mathscr{H}om_{\mathscr{O}_X}(\mathscr{F}, \mathscr{O}_X)$, where $\mathscr{H}om_{\mathscr{O}_X}(\mathscr{F}, \mathscr{G})$ is the sheaf defined by $U \mapsto \textrm{Hom}_{\mathscr{O}_U}(\mathscr{F} |_U, \mathscr{G} |_U)$. (No sheafification required.) If $U \subseteq V$, the map $\textrm{Hom}_{\mathscr{O}_V}(\mathscr{F} |_V, \mathscr{G} |_V) \to \textrm{Hom}_{\mathscr{O}_U}(\mathscr{F} |_U, \mathscr{G} |_U)$ is the one induced by the sheaf-restriction functor $\mathscr{O}_V \textbf{-Mod} \to \mathscr{O}_U \textbf{-Mod}$.