# Numerical methods (for ODE/PDE) that could take approximate solutions/good initial guesses, and further refine it to an certain accuracy

I am currently playing with an old analog computer, which could solve time-dependent ODE/PDEs pretty fast, without time-stepping; thus there is no convergence issues caused by time-stepping because of its computing nature. But the problem with analog computer's solutions is that they are not accurate due to physical limitations. I am very curious that: is there any numerical methods/solvers which can take analog computer's approximate solution (over the time domain) to further process it, and generate a more accurate solution??

Let me give an example of solving second order ODE describing the motion a mass-spring damper. The equation is the following: $$x'' = -0.2\cdot x' - 0.4\cdot x;\quad x(0)=1, x'(0) =0;\quad t_{stop} = 60s.$$ To solve the above equation on an analog computer, we need to map the above equation to an electrical system. Usually an analog computer could perform several basic arithmetic operation in the continuous-time domain, e.g. addition, subtraction, multiplication, integration etc. The output of an integrator represent an state-variable of the ODE; the input of that integrator represent the corresponding first-order time derivative. By configuring the basic computing blocks in feedback loops, we could map the equation as the following: (I use Simulink) After you load the initial conditions onto the integrators, you can let the analog computer run and solve. If you measure the electrical signal at the output of integrator1, you will get the solution of $x(t)$ over the time domain: But, due to the physical limitations (e.g. electrical noise, offsets), the solution of $x(t)$ is not accurate. What I am looking for is a numerical method that can take the above solution of $x(t)$ by analog computer, e.g. the solutions $x(t=1s), x(t=2s), x(t=3s), x(t=4s)... x(t=60s)$, start from these approximate solution points and further refine these solution $x(t=1s), ... x(t=60s)$ to a much higher accuracy.

(This second order ODE is just a simple case for illustration purpose; it happens to have analytic expression of solutions. The more general case would be nonlinear ODEs with no analytic solution.)

Thanks in advance!! Any thoughts and suggestions are greatly welcome and appreciated!!

• I think you mean that you turn the ODE into something with no time stepping, which would have to be more or less a system of algebraic equations. Plenty of such methods exist; for example, spectral methods are of this type. (In the PDE context, fully spectral methods are of this type, while partially spectral methods are not.) – Ian Jul 14 '15 at 0:34
• @GiuseppeNegro I will add an example later on. I could post figures because I just registered :-) – Nate Jul 14 '15 at 0:57
• @Ian Hi Ian, actually analog computers convert the original ODE into the corresponding electrical dynamic system, and the response of the system is the solution of the ODE. I could not add figures to my thread now because of my low reputation. Here is a link which could provide you more info on analog computers: analogmuseum.org/library/eaiapproach.pdf – Nate Jul 14 '15 at 1:11
• In reality that's actually pretty much the same, because a real electric system does not have an ideal frequency profile (it is impossible to devise a machine which passes all frequencies exactly equally), so an analog solution to, say, $x''+x=\sin(t),x(0)=0,x'(0)=0$ is very much like a spectral solution. – Ian Jul 14 '15 at 1:47
• @Nate Internally they don't, you're correct. But the source of their error is most straightforwardly described in the frequency domain: generally speaking a circuit like yours has the effect of a low pass filter in addition to the actual process at hand. As a result, the qualitative features of the error in an analog computer solution and a spectral method solution are very similar. – Ian Jul 14 '15 at 2:23

## 3 Answers

If you have a good initial estimate, Newton's method is hard to beat. Quadratic convergence means that the number of accurate decimal (binary) places doubles with each iteration. This assumes that the first derivative is changing slowly between your estimate and the real solution, which means the second derivative times your error (between the estimate and the real answer) is small compared to the first derivative. From physical arguments you know your solution is a damped sine wave, so fit it to $A \cos (\omega t) \exp(-\lambda t)$ What you really need for Newton's method is estimates of $A, \omega, \lambda$, not estimates of $y(t)$ which is what you get from your circuit. $A$ is easy, it is $y(0)$. I would take $\omega$ from the last zero crossing I could identify easily and $\lambda$ from the ratio of the first peak to the starting amplitude.

• Hello Ross, thanks for your answer. Actually this second order ODE is a simple example with analytic expression of the solution. What I am looking for is a general case where we do not know the solution beforehand: Is it possible that we take the value of $x(t)$ , for example, at $x(t=1s), x(t=2s), x(t=3s), x(t=4s), x(t=5s) ... x(t=60s)$ as initial guesses, and feed these values to some numerical solver to let it start from these guesses. Does there exists such numerical method? – Nate Jul 14 '15 at 14:39
• You can certainly fit the values to any functional form you want. Choosing the form can be an art. For example, you might choose a polynomial of some order. Having done so, you can use a linear function minimizer to find the coefficients of the polynomial. These are discussed in any numerical analysis text. Chapter 10 of Numerical Recipes has a discussion. If you take the data at equally spaced intervals, you are set for a Fourrier analysis. – Ross Millikan Jul 14 '15 at 15:25
• Hi Ross, thanks for mentioning that book chapter. I will take careful read. Polynomial fitting could be a way to take the solutions from analog computers. Do you think there would be a numerical method, where it still uses classical numerical integration for time-stepping (e.g. Runge-Kutta 4th order) and can take advantage of the approximate solutions from analog computers? Thanks a lot!! – Nate Jul 14 '15 at 17:57
• I wouldn't know how to use an approximate solution in a stepping routine. You can take the approximate solution as a starting point for a relaxation method, which is also discussed in Numerical Recipes as a way to solve two point problems. – Ross Millikan Jul 14 '15 at 19:21
• Hi Ross, thanks. I will take a good read of the Numerical Recipes book, to see if I could take advantage of the approximate solutions from analog computers. – Nate Jul 15 '15 at 0:20

In almost the same spirit as Ross Millikan's answer, let me assume that you know the initial conditions $x(0)=1$, $x'(0)=0$ and that the model is something like $$x(t)=e^{-\alpha t} \big(A \cos (\beta t)+B \sin (\beta t)\big)$$ and from data points $(t_i,x_i)$ taken from the analog computer, you want to refine the values of the four parameters $A,B,\alpha,\beta$ which appear in the model.

This can be considered as a nonlinear regression problem and the key issue is to get "reasonable" estimates.

The first condition gives $A=1$ and this is a definitive value (it is no more a parameter to be tuned).

Now, looking at the derivative $$x'(t)=e^{-\alpha t} (\cos (\beta t) (\beta B-\alpha A)-\sin (\beta t) (A \beta +\alpha B))$$ gives $\beta B-\alpha A=0$ which means that we can impose $B=\frac{\alpha} {\beta}$.

All of the above makes the model to be $$x(t)=e^{-\alpha t} \left(\frac{\alpha \sin (\beta t)}{\beta }+\cos (\beta t)\right)$$ and we are just left with two parameters $\alpha,\beta$ for which we need estimates. Looking at $$x'(t)=-\frac{\left(\alpha ^2+\beta ^2\right) e^{-\alpha t} \sin (\beta t)}{\beta }$$ we see that the first minimum of $x(t)$ will correspond to $\beta t_*=\pi$; this gives an estimate for $\beta$. At this point, we have $$x(t_*)=-e^{-\frac{\pi \alpha }{\beta }}=-e^{-\alpha t_*}$$ from which the estimate of $\alpha$ is easily extracted.

Now, we have all the required elements to start the nonlinear least square fit of the data.

Just looking at the plot in the post, using the fact that the first minimum corresponds more or less to $t_*=5$, $x=-0.6$, we obtain as estimates $\beta_0 = \frac{\pi }{5} \approx 0.628319$ and $\alpha_0= \frac{1}{5} \log \left(\frac{5}{3}\right)\approx 0.102165$ while the exact values should be $\beta=\frac{\sqrt{39}}{10}\approx 0.624500$ and $\alpha=\frac 1 {10}$. The nonlinear regression converges in a couple of iterations.

• Dear Claude, thanks for your detailed answer! I agree with your method for this example, where we know the analytic expression for the solution. For a more general case, the ODE would be nonlinear and has no analytic expression. Is there a numerical method that can directly take the values of $x(t)$ , for example, at $x(t=1s),x(t=2s),x(t=3s),x(t=4s),x(t=5s)...x(t=60s)$ as initial guesses, and can refine the solution to a much higher accuracy, without "human calculation" in the loop? Thanks! – Nate Jul 14 '15 at 14:57
• @Nate. I totally agree with your point : if we know the form of the solution, this method is easy to apply (I worked this kind of problems when I used in the 60's analog computers). In the case where we should not have any idea, smoothing splines are very good. – Claude Leibovici Jul 15 '15 at 3:57
• Wow, I am surprised that you have used analog computer:-) I thought this was a very old simulation tool that very few people knew about it. Analog computers have one feature that I like very much: true continuous-time integration. Actually I am thinking to take advantage of this feature, trying to find a way to accelerate conventional numerical ODE solvers. But it looks like numerical solvers are pretty self-sustaining; the approximate solutions from analog computers could not directly accelerate them... – Nate Jul 15 '15 at 13:56
• Did you look at my profile ? I started 55 years ago with computers. Analog computers are marvels for this class of problems. It has been a crime to kill them as they have been. Cheers – Claude Leibovici Jul 15 '15 at 18:24
• Right! In the 50's / 60's, people were taught to use analog computers for simulation! I read that there were a lot of debates on "analog computer v.s. digital computer" in the 70's, but no clear conclusions were made that digital computers are superior than analog ones in all aspects. Then the whole community embraced digital computers ever since. I think analog computers may still do better jobs at simulating dynamic systems because of the inherent continuous-time nature. – Nate Jul 15 '15 at 20:05

Check out this article from Columbia U. https://www.cs.columbia.edu/2016/back-to-analog-computing-columbia-researchers-merge-analog-and-digital-computing-on-a-single-chip/

They seem to be trying to do nearly the same thing you were thinking of, to use analog computing to get a good initial estimate, and then use digital (basically standard numerical techniques) to get to a final acceptable answer, and all this in a real computer chip. Pretty cool if you ask me.