Fitting integral of signal to neural network and reconstructing signal itself I want to fit the integral of a signal to a neural network and then reconstruct the original signal numerically from the output of that neural network.
More precisely, I have the discrete non-negative signal $f(x)$, then I calculate $F(x) = \Sigma f(x)$ and fit the $F(x)$ to a neural network. The input to the network is the index of sample ($x_i$) and the output is the value of $\hat{F}(x_i)$, where $\hat{F}(x)$ is the prediction of $F(x)$.
Then, I want to reconstruct the original signal $f(x)$ using $f(x_i) = F(x_i)-F(x_i-1)$.
However, as $F(x)$ is monotonically increasing, it starts from about $0$ and the magnitude grows up to several 10 millions. Then the problem shows up here, because the value $f(x_i)= F(x_i)-F(x_i-1)$ remains relatively small compared to the $F(x_i)$ value, and the network is unable to reach the precision such that $F(x_i) _F(x_i-1)$ is calculated correctly. I have trained the network until the loss has shrunk down to orders of 10^(-10), but still get some wrong values for $f(x_i)$.
Could someone please help me with the problem? Thanks.
 A: As far as I understood the problem starts with an initial dataset $S_1 = \{x_i, f(x_i)\}_{i=1}^n$. From this, you create another dataset with $S_2 = \{x_i, F(x_i)\}_{i=1}^n$ where $F(x_i) = \int_a ^{x_i} f(t) \; dt$. You fit a NN to $S_2$ and get $\hat{F}$. From this you want to obtain $\hat{f}$. Assuming I understood correctly, then you can do much better than $f(x_i) \approx \frac{F(x_{i+1})- F(x_{i})}{x_{i+1}-x_{i}}$.
Remember that the output of a NN is a continuous function with respect to the input. Hence, you can simply propagate back to get the gradient, i.e. $\hat{f}(x_i) = \frac{d \; NN(x_i)}{dx_i}$. Modern frameworks for building NNs (like tensorflow or pytorch) offer simple ways to achieve this through automatic differentiation. Check my answer for an example on how to compute gradient of an NN w.r.t. the input. However; this approach might fail if you have an overfitting problem. In such a case, your learned function would be overtuned to noise and hence its gradient will be way off.
A solution here is to force the gradient of the NN to represent the true gradient as well. In such a case, your training set is: $S_3 = \{x_i, F(x_i), f(x_i)\}$. To learn this dataset you need to modify your feedforward and the loss function. For a single input $x_i$, a single feedforward here involves propagating to compute $\hat{F}(x_i)$ then propagating back to compute the predicted derivative $\hat{f}(x_i)$. Then you need to update your parameters on the gradient of the loss function: $L = l(F(x_i),\hat{F}(x_i)) + \rho l(f(x_i),\hat{f}(x_i))$ where $l$ can be any loss function. $\rho$ controls the balance between how well you want to fit the function, and how well you want to fit the derivative. This is known as Sobolev Training of Neural Networks. This approach is theoretically motivated by the work of Hornik, who proved the universal approximation theorems for NNs in Sobolev spaces. As far as I know, this approach is not implemented in any of the famous frameworks for building NNs. While it is quite straight forward to implement, from my experience it can be tricky to choose the best values for $\rho$. A recent paper provides an implementation for Sobolev Training, it might be helpful for you.
