I have five sets of observations of measured y as some function of measured $x_1, x_2, x_3,\ldots$ and I want to fit five functions to these observations. They have the form $$ y = f(x_1, x_2, x_3,\ldots, a_1, a_2, a_3,\ldots) $$ Where the $a_1, a_2,\ldots$ are fitting parameters.

I can use least squares to find the least-squares best fitting parameters for each individual set of observations, but as it happens, this isn't what I really need. Instead it is thought that the physical meaning behind the fitting parameters is such that for all five sets of observations some of the fitting parameters, say $a_3$ & $a_4$, should be the same, but the $a_1$ and $a_2$ should be different for each set.

So I need to find a "globally" "optimal" (in some sense) $a_3$ and $a_4$, and then the best fit $a_1$ & $a_2$ for each observation set.

Can somebody suggest a way to approach this, and if there are standard techniques for it?

I'm aware that people in some fields approach this by iteratively trying to find the best fit $a_3$, $a_4$ by treating the five sets of observations as one big set, and then they fix those parameters and then split into five sets again and then solve for the $a_1$ & $a_2$. And more elaborate iterative approaches are used, but I've never seen this described in a text on numerical methods, and the method is used by metallurgists and other professionals who may not be appreciating how meaningful, or not, their algorithm to solving the problem. So I have doubts about using that approach (are they just getting the numbers they want to see, rather than physically meaningful numbers?)- unless there is good reason to believe that it is effective?

Thanks, Neil


Here's what I would do. (I am assuming for the moment your $f$ is linear, so we can use OLS. If that's not the case, you have to use nonlinear LS or something appropriate.)

I would throw your 5 sets of data together. Now if you estimate your $a$ coefficients on that big data set, your $a_3,a_4$ will be "global" in the sense you want. But so will be $a_1$ and $a_2$, which you don't want.

What we instead want is that these two are optimal for each subset, so we need to estimate those two for each subset, $10$ coefficients overall (call them $a_{j1}$ and $a_{j2}$, for $j=1,...,5$). We can do this with interactions. Define dummies $d_{ji}$, where $d_{ji}=1$ if and only if observation $i$ is from set of observation $j$, and otherwise zero. Hence, you estimate with OLS $$y_i=\sum_{j=1}^5 (a_{j1} d_{ji} x_{i1}+a_{j2} d_{ji} x_{i2})+a_3 x_{i3}+a_4 x_{i4}+\varepsilon_i.$$ Suppose observation $i$ is from set $3$, then $d_{1i}=d_{2i}=d_{4i}=d_{5i}=0$, and $d_{3i}=1$. Hence, only coefficients $a_{31}$ and $a_{32}$ are influenced by this observation, not the coefficients that are supposed to capture the other sets of observation.

How to estimate this exactly? You will have to prepare the data. Just use your two variables $x_{1}$ and $x_{2}$ (the ones for which you want a different coefficient depending on the set), and create 5 new variables from $x_1$ such that the first is $d_{1i}*x_{i1}$, the second $d_{2i}*x_{i1}$ and so on. Do the same for the second variable and then just estimate the $2*5+2$ coefficients with OLS. Done.

  • $\begingroup$ Thank you. I've used this approach. The actual problem contains 11 sets of data, with four fitting parameters specific to each set, and 5 parameters that are "global". This gives a total of 49 parameters to estimate from ~300 observations. It is nonlinear (a sum of exponentials, where each exponential is itself raised to another exponential, all multiplied by a function of another parameter) and it converges to solutions, but it very sensitive to initial guesses for global parameters. A different experimental technique may reduce the number of fitting parameters required. $\endgroup$ – Neil Francis May 1 '13 at 3:37
  • $\begingroup$ And thank you for using the terminology of dummy variables and interactions - I had a feeling that there was a standard approach to do this, but didn't have the language to search for it effectively. $\endgroup$ – Neil Francis May 1 '13 at 3:48
  • $\begingroup$ Hi Neil. I assume you estimate this stuff with maximum likelihood? Your parameters should not change with starting values if your problem is concave, because then there would be a unique maximum. Maybe you have an error in your likelihood function? An alternative may be nonlinear least squares, although I have never used it myself. $\endgroup$ – Nameless May 1 '13 at 12:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.