Connection between SVD and Discrete Fourier Transform for Denoising Denoising signals (in particular, 2D arrays, such as images) can be done by removing the high frequency components of the discrete Fourier transform (which is related to convolution with a Gaussian kernel) or by removing the smallest singular values.
I was wondering if there is a known, specific mathematical connection between  these two approaches.
I've seen a little discussion on the topic here and here, but I didn't really glean much specifically except the mention of circulant matrices.
 A: There is an entire textbook written about this "Introduction to Inverse Problems and Imaging" by Bertaro and Boccacci. Chapter 4 sets up the formalism which intertwines inverse problems and fourier transforms. Chapter 5 shows how tikhonov regularization is related to windowing functions in fourier transforms. Here, I'll flush out one connection, that may appease your curiosity.
--- The Big Picture ---
Specifically, both the SVD and the Fourier transform are characterized by some unitary transform (i.e. a change of basis) (1). Once in some new basis, you can make a choice to remove parts of the basis which are sensitive to small changes in the data (2). For both SVD and Fourier transforms these components are removed by "windowing functions" which select a viable window of either singular values or frequencies. Mathematically they look incredibly similar.
Evidence for assertion (1): For the SVD a matrix $A=U \Sigma V^T$ where $U$ and $V$ are unitary transforms into either the row or column space of $A$. The Fourier transform is also a unitary transform into a new basis, see that for $U = e^{i (t \cdot f)}$, then $U U^* = I$.
Evidence for assertion (2): For the SVD, "parts of the basis sensitive to small changes" means the singular vectors (basis vectors) which correspond to small singular values. The small singular values raise the condition number of the matrix and increase the upper bound on the signal to noise ratio. Truncating the basis (eliminating small singular value basis vectors) is often used to fix the problem. For Fourier Transform, "parts of the basis sensitive to small changes" means basis vectors $\exp[i ( t \cdot f)]$ with large frequency, $f$. A well known example is aliasing and a low pass filter or 'frequency truncation' is needed to fix the problem. In comparison, eliminating large frequencies and eliminating small singular value are both basis truncations.
--- The math for truncation in SVD ---
Consider the a integral transform with kernel $K$
$$g(t) = \int dt' K(t,t') h(t') $$
We will first consider SVD windowing functions.
Start by discretizing the integral domain into $N \times N$ intervals.
In particular, our integrals becomes matrix-vector multiplication.
$$\vec{g} = \Delta t K \vec{h} $$
We can now write the ordinary least squares (OLS) solution for $h$ via singular value decomposition (insert $K = U \Sigma V^T$ and invert equation)
$$\vec{h} = V \Sigma^{-1} U^T \vec{g} $$
or equivalently,
$$\vec{h} = \sum_{i=1}^N \frac{1}{\sigma_i} \vec{v}_i \vec{u}_i^T \vec{g} = \sum_{i=1}^N a_i  \vec{v}_i $$
Where $a_i = \frac{\vec{u}_i^T \vec{g}}{\sigma_i}$.
Notice that $\vec{u}_i^T \vec{g}$ and $\sigma_i$ are scalers.
Thus these terms the weight the the linear expansion of the solution in the column space of K.
To emphasize this, I have grouped them into $a_i$
If we want to cut out some basis vectors, doing it based on $\sigma_i$ is optimal because as it goes to zero the quantity diverges.
We will cut out basis vector using a windowing function with tolerance $\epsilon$,
$$ W^{cut}(a_i | \epsilon) = \frac{\vec{u}_i^T \vec{g}}{\sigma_i} \Theta(\sigma_i - \epsilon)$$
Here $\Theta$ is the step function.
So that if $\sigma_i > \epsilon$ then $W_i= a_i$ and $0$ otherwise.
We can write the cutoff solution as
$$\vec{h}^{cut} = \sum_{i=1}^N W^{cut}(a_i | \epsilon) \vec{v}_i $$
Many different windowing functions can be concocted, for example:
$$ W^{Tik}_i(\sigma_i | \epsilon) = \vec{u}_i^T \vec{g} \frac{\sigma_i}{\sigma_i^2+\epsilon}$$
This is the Tikhonov regularization solution.
Notice that for both windowing functions the $\epsilon \rightarrow 0$ recovers the original the OLS solution.
--- The math for truncation in Fourier transforms ---
If we Fourier transform this problem into frequency space the convolution turns into a multiplication (See E.O. Brigham's ``Introduction to FFT'').
Thus, $g(f) = K(f,f') h(f') $ and we can compute the solution through division.
$$h(f') = g(f)/K(f,f')$$
We can Fourier transform over frequency space $\mathcal{F}$ to arrive at a solution.
$$h(t') = \int_{\mathcal{F}} df' \exp[i ( t' \cdot f')]  h(f')= \int_{\mathcal{F}} df' \exp[i ( t' \cdot f')] \frac{g(f)}{K(f,f')}$$
If you want to introduce some cutoff on high frequency data then you can introduce a windowing function as before:
$$ W^{cut}( f' | \epsilon) = \frac{g(f)}{K(f,f')} \Theta(f' - \epsilon)$$
So that
$$h(t') = \int_{\mathcal{F}} df \: W^{cut}( f' | \epsilon) \exp[i ( t' \cdot f') $$
Notice that I have left the Fourier bases outside the weight function.
I have done this to emphasize that a Fourier transform is really a  linear expansion of the solution in the frequency basis.
As with SVD, you could as use the Tikhonov regularization for the frequency basis vectors,
$$ W^{Tik}(f' | \epsilon) = g(f) \frac{K^*(f,f')}{\vert K(f,f') \vert^2 + \epsilon}$$
Or you could select some other windowing function more broadly used by the signal processing community (e.g. Hamming function)

