How to notate the restriction of an inverse of a function? Let $f$ be a function, which is defined on $E\subset\mathbb{R}^n$, and which is mapping to an extended real numbers. That is,
$$f:E\longrightarrow\overline{\mathbb{R}}$$
Then, for a subset $A$ of $E$ $\left(\text{i.e.}~A\subset{}E\right)$,
$$f~\big|_A:A\longrightarrow\overline{\mathbb{R}}$$

For an open set $G$, let $f^{-1}(G)=G^*$.
If I want to restrict $f^{-1}$ to a set $H$, how can I write it?
Which one is correct?
$$f~\big|_H^{-1} = G^* \cap H$$
$$f^{-1}\big|_H = G^* \cap H$$

 A: Writing $f^{-1}(G) = G^{*}$—without specifying that $f$ is invertible—implicitly defines a mapping $f^{-1}$ from the power set of the extended reals to the power set of $E$, not a mapping from the extended reals to $E$. Consequently, "restriction of $f^{-1}$ to $H$" doesn't make sense (unless you intend to restrict to $H$ viewed as an element of a power set, which seems unlikely).
In any case, neither proposed formula is grammatically correct; each has a mapping on the left and a set on the right.
Be that as it may,


*

*$f^{-1}\big|_{H}$ denotes the restriction of the inverse of $f$ to $H$;

*$f\big|_{H}{}^{-1}$ denotes the inverse of the restriction of $f$ to $H$.
I'm guessing you're really asking: "If $H$ is a subset of the extended reals, what is the proper notation for $f^{-1}(H)$?"
If that's right, reasonable answers include $H^{*}$ (if you want the inverse image of $H$ itself) or $G^{*} \cap H^{*} = (G \cap H)^{*}$ (if you want the inverse image of the intersection $G \cap H$, which is not unlike a "restriction of $f^{-1}(G)$ to $H$").
