Relationship between autocorrelation function and wavelet coefficient the autocorrelation function can be represented using the spectral density in Fourier space. Is there a similar relationship between the autocorrelation function and the coefficient in the wavelet space?
 A: $\newcommand{\inprod}[2]{\left\langle{#1}\,|\,{#2}\right\rangle}$
$\newcommand{\eqd}{\triangleq}$
$\newcommand{\setn}[1]{{\left\{{#1}\right\}}}$
$\newcommand{\setu}{\cup}$
$\newcommand{\brp}[1]{{\left(#1\right)}}$
$\newcommand{\brbl}[1]{\left\{#1\right.}$
$\newcommand{\brb}[1]{\left\{#1\right\}}$
$\newcommand{\brs}[1]{\left[#1\right]}$
$\newcommand{\setR}{\mathbb{R}}$
$\newcommand{\setC}{\mathbb{C}}$
$\newcommand{\setX}{\mathsf{X}}$
$\newcommand{\opF}{\mathbb{F}}$
$\newcommand{\opW}{\mathbb{W}}$
As you said, the auto-correlation function $R_{xx}(t,u)\eqd E\brs{x(t+u)x^{\ast}(t)}$ is simply a "function" (a mapping from one set $\mathsf{X}$ to another/same set $\mathsf{Y}$). If $x(t)$ is stationary, this definition can be simplified to $R_{xx}(u)\eqd E\brs{x(u)x^{\ast}(0)}$. For a very wide class of functions on $\mathbb{R}\to\mathbb{C}$, we can use the Fourier Transform $\opF$ (transform—a special class of functions often called operators that map from one vector space to another/same vector space). When we use the operator $\opF$ to map the vector $R_{xx}(u)$ to a new vector $S_{xx}(\omega)\eqd \opF R_{xx}(u)$, $S_{xx}(\omega)$ is often called the spectral density of $x$, or the power spectral density (PSD) of $x$.
Of course $\mathbb{F}$ is not the only transform (vector space mapping) that can be applied to $R_{xx}(u)$—any transform that operates over the domain used by $R_{xx}(u)$ could be used. Example transforms include the Laplace Transform (a generalization of the Fourier Transform), B-spline transforms (note on "s": there are an infinite number of B-splines), and of course Wavelet Transforms. Likewise if $R_{xx}$ was discrete (e.g. "$R_{xx}(m)$") rather than continuous, transforms might include the DFT/FFT, Z-transform, and Hadamard Transform.
But to me, your question is not just if the wavelet transform $\opW$ can be applied to $R_{xx}$; it is more like, "Can $\opW$ be applied in like manner as $\opF$ and get a like manner usefulness?" That is, just as the "Power Spectral Density" can be used to calculate the "power" (or something resembling it) in the "spectrum" of $\opF$, can the projection $\opW R_{xx}$ of $R_{xx}$ also be used to calculate power (or something resembling it)?
Here, "spectrum" is defined as the projection coefficients/"Fourier coefficients" $G(\omega)$ of $\mathbb{F}$
$$\brb{\begin{align*}
  G(\omega)
    &\eqd \brs{\opF R_{xx}(t)}(\omega)
     \eqd\inprod{R_{xx}(t)}{\frac{1}{\sqrt{2\pi}}e^{i\omega t}}
     \eqd \frac{1}{\sqrt{2\pi}}\int_{t\in\setR}R_{xx}(t)e^{-i\omega t}dt
  \\&\text{of $R_{xx}$ onto the basis $\setn{\frac{1}{\sqrt{2\pi}}e^{i\omega t} | \omega\in\mathbb{R}}$ and $\opW R_{xx}$}
  \\
  H(a,b)
    &\eqd \brs{\mathbb{W}R_{xx}}(a,b)
     \eqd \inprod{R_{xx}(t)}{\frac{1}{\sqrt{a}}\psi\left(\frac{\omega-b}{a}\right)}
     \eqd \frac{1}{\sqrt{a}}\int_{t\in\setR} R_{xx}(t)\psi\left(\frac{\omega-b}{a}\right)
\end{align*}}$$
is the projection of $R_{xx}$ onto whatever wavelet basis $\{\psi\}$ (e.g. Haar, Daubechies-$p$, Symmlets-$p$) is used in the definition of wavelet transform $\mathbb{W}$.
The answer is "Yes" and is demonstrated by the Plancherel Formula or more generally by Parseval's Identity in any Hilbert Space:
$$\left\lVert R_{xx}(t) \right\rVert^2 = \sum_{n\in\mathbb{z}}\left|\left\langle R_{xx}(t),\phi_n(t)\right\rangle\right|^2$$
where the sequence $\{\phi_n(t)\}$ is any orthonormal basis (Fourier, Wavelet, Hadamard, etc.) for the space.
The term “spectral power” is arguably a bit of an oxymoron because “spectral” deals with leaving the time-domain for an alternate (call it "frequency" if you wish) domain$\ldots$ howbeit the concept of power is solidly founded on the concept of time in that power = energy per time. However, Plancherel/Parseval demonstrate that "power" $\left\lVert R_{xx}(t) \right\rVert^2$ in time can also be calculated from any Hilbert Space transform that is constructed over an orthonormal basis.
