What is the most general notion of "Fourier transform?" I know the definition of a classical Fourier transform that maps a function f(x) on the real line X to a function F(p) on a dual space (here another real line and borrowing some physics notation) P. This is generalized straightforwardly to multidimensional linear spaces, where we can simply apply the integral transform to each dimension in turn. What about curved spaces? Superspaces? Non-commutative spaces? Is it important that the dual space P could be thought of as the cotangent bundle on X? Can this help us understand notions of dynamics on spaces that we cannot classically understand as a space (i.e. anti- and non-commutative coordinates)? Are there other examples of "Fourier transforms" that are interesting/useful? 
 A: One interesting generalization follows Fourier's original analysis for Ordinary Differential Equations on $\mathbb{R}$. It's easiest to break first into ODEs on a half line $[0,\infty)$ for Sturm-Liouville problems
$$
                       Lf=-\frac{d^{2}f}{dx^{2}}+q(x)f(x) = g(x),\\
                         \cos\alpha f(0)+\sin\alpha f'(0) = 0.
$$
where $q$ is integrable on $[0,r]$ for any $r > 0$, but not necessarily integrable on $[0,\infty)$. It can happen a second condition is required at $\infty$, but I won't describe such conditions here. For all $\lambda \in \mathbb{C}\setminus\mathbb{R}$ there is a unique eigenfunction $\phi_{\lambda}(x)$ such that
$$
         L\phi_{\lambda}=\lambda \phi_{\lambda} \\
     +\cos\alpha \phi_{\lambda}(0)+\sin\alpha\phi_{\lambda}'(0)=0 \\
    -\sin\alpha \phi_{\lambda}(0)+\cos\alpha \phi_{\lambda}'(0)=1 \\
       \phi_{\lambda}\mbox{ satisfies required condition at $\infty$ if any } \\
         \phi_{\lambda} \in L^{2}[0,\infty).
$$
As $\lambda$ tends to $u \in \mathbb{R}$, these functions have pointwise everywhere limits $\phi_{u}$ that are eigenfunctions that are not generally in $L^{2}[0,\infty)$ but are square integrable on any finite interval $[0,r]$ for $r > 0$. $\phi_{u}$ is in $L^{2}[0,\infty)$ for some $u$ iff $\phi_{u}$ is an actual eigenfunction with eigenvalue $u$.
Let the operator $L$ be defined on the domain $\mathcal{D}(L)$ consisting of twice locally absolutely continuous functions on $[0,\infty)$ satisfying the the required condition at $0$ as well as any required condition at $\infty$ (if any.) Then there exists a measure $\mu$ on $\mathbb{R}$ concentrated on the spectrum $\sigma(L)$ of the operator $L$ such that the generalized Fourier transform exists as a limit in $L^{2}(\mathbb{R},\mu)$:
$$
   \hat{f}(u)=L^{2}_{\mu}(\mathbb{R},\mu)\mbox{-}\lim_{r\uparrow\infty}\int_{0}^{r}f(x)\phi_{u}(x)dx
$$
The inverse generalized Fourier transform inverts this operation:
$$
   f(x) = L^{2}[0,\infty)\mbox{-}\lim_{R\rightarrow\infty}\int_{-R}^{R}
         \hat{f}(u)\phi_{u}(x)d\mu(u).
$$
Furthermore, Parseval's equality $\|f\|_{L^{2}}=\|\hat{f}\|_{L^{2}_{\mu}}$, which is to say
$$
               \int_{-\infty}^{\infty}|\hat{f}(u)|^{2}d\mu(u)=\int_{-\infty}^{\infty}|f(x)|^{2}dx.        
$$
In fact, $f\mapsto \hat{f}$ is surjective, which makes the generalized Fourier transform unitary. Furthermore, the Fourier transform turns the operator $L$ into multiplication by $u$. That is, $f \in \mathcal{D}(L)$ iff $u\hat{f}(u) \in L^{2}(\mathbb{R},\mu)$, and, in that case,
$$
                  \widehat{Lf}(u) = u\hat{f}(u).
$$
The ordinary Fourier cosine and sine transforms on $[0,\infty)$ are special cases of this general transform pair.
The measure $\mu$ may generally have discrete components and continuous components; so this general transform looks like a discrete + continuous Fourier transform. The Hankel transform is a special case, as well as other classical transforms of Math Physics.
This general develop represents the fleshing out of Fourier's plan, and wasn't fleshed out until more than century after Fourier's original work. This is Sturm-Liouville Theory, which dealt with the ODEs arising out of Fourier's separation of variables for classical equations. In this context, groups are not needed and the equations can be even more general still. The expansions look like continuous eigenfunction expansions, i.e., instead of
$$
   f=\sum_{\lambda_{n}}(f,\phi_{\lambda_{n}})\phi_{\lambda_{n}}\mu_{n}
$$
where $\mu_{n}$ are normalization constants, one has an integral
$$
                f = \int_{-\infty}^{\infty}"(f,\phi_{u})"\phi_{u}(x)d\mu(u),\\
       "(f,\phi_{u})" = L^{2}_{\mu}\mbox{-}\lim_{r\rightarrow\infty}\int_{0}^{r}f(x)\phi_{u}(x)dx.
$$
There may be discrete and/or continuous components for $\mu$. (I use quotes because $\phi_{u} \notin L^{2}$, which means $(f,\phi_{u})$ is not an actual inner product for all $u$, even though it looks like one, just as for the Fourier sine transform $(f,\sin(ux))=\int_{0}^{\infty}f(x)\sin(ux)dx$.) The Fourier transform $\hat{f}(u)=(f,\phi_{u})$ is a spectral density for $f$ whose magnitudes are interpreted as coefficients of the eigenfunctions $\phi_{u}$ required to reconstruct $f$.
This is the original generalization of Fourier transform theory, and is deep enough and difficult enough to prove that its rigorous development remains difficult to find in modern texts.
A: I'm hardly qualified to answer this question, but you might find the following references useful.
Let's start with the two classical examples of the Fourier transform: the Fourier transforms for $L^2$ functions on the line and the circle.
Generalizing the notion of space
The line and the circle are both topological abelian groups, and the Fourier transform expresses functions on the group in terms of characters. This point of view generalizes to...


*

*Locally compact abelian groups.

*Compact anabelian groups, with the help of the Peter-Weyl theorem.


The line and the circle are both Riemannian manifolds, and the Fourier transform expresses functions on the manifold in terms of eigenfunctions of the Laplace-Beltrami operator. This point of view generalizes to...


*

*Riemannian manifolds with the Laplace-Beltrami operator, leading to the spectral theory of manifolds. This works especially well for compact Riemannian manifolds, and it plays an important role in the study of hyperbolic surfaces.

*Graphs with the graph Laplace operator, leading to spectral graph theory.


Generalizing the notion of function
When the space is an abelian group with the structure of a projective algebraic variety, functions can be generalized to coherent sheaves, leading to the Fourier-Mukai transform.
When the space is a Riemannian manifold, functions can be generalized to...


*

*Differential forms, leading to Hodge theory. (Although Hodge theory only deals with the kernel of the Laplacian, while Fourier analysis involves with all the eigenspaces.) There may also be a spectral graph theory version of this.

*Sections of a hermitian vector bundle, leading to the Bochner Laplacian. (I don't know if you can actually do Fourier analysis in this setting.)

