Least squares method for general formula Up today I've only been using Least squares method in "standard" applications like power/inverse/linear regression with known formulas.
But now I'd like to use it to calibrate (i.e. determine some coefficients) following model:
$$
T(nprocs, DOF) = \alpha \cdot DOF^\beta + \gamma \cdot \frac{DOF^\delta}{nprocs}
$$
I want to use following data for calibration:

From the table we can see, that
$$
\begin{align}
nprocs &\in \{1,2,3,4,6,8,12,24\}\\
DOF &\in \{3993, 7623, 11253, 14883, 18513\}.
\end{align}
$$
It implies, that we'll have an overdetermined system of 40 equations like this:
$$
\begin{align}
0.656395 &= \alpha \cdot 3993^\beta + \gamma \cdot \frac{3993^\delta}{1}\\
0.2569141 &= \alpha \cdot 3993^\beta + \gamma \cdot \frac{3993^\delta}{2}\\
&\vdots\\
0.407449 &= \alpha \cdot 11253^\beta + \gamma \cdot \frac{11253^\delta}{4}\\
&\vdots\\
0.373136 &= \alpha \cdot 18513^\beta + \gamma \cdot \frac{18513^\delta}{24}
\end{align}
$$
If the system was small, I'd just assemble a function $LS$ like this
$$
LS(\alpha, \beta, \gamma, \delta) = (0.656395 - (\alpha \cdot 3993^\beta + \gamma \cdot \frac{3993^\delta}{1}))^2 + (0.2569141 - (\alpha \cdot 3993^\beta + \gamma \cdot \frac{3993^\delta}{2}))^2 + \cdots
$$
and then I'd make partial derivatives and minimize the system by making them equal to zero.
The problem is, apparently, that here I can't use this method, because the system is too large. I suppose, that it'll go to assembling a quadratic polynomial like this 
$$
x^TAx + b^Tx + c
$$
where $A$ will be a symmetric, positive-definite matrix and so I'll be able to minimize the polynomial by solving the simple system 
$$
Ax = b.
$$
My question is - what's the best way to get the matrix $A$ and the vector $b$ without having to assemble the whole function (polynomial) $LS$?
 A: For this problem, you need, in principle, nonlinear regression or turn the problem into an optimization problem.
But, you already have good estimates of the parameters (after the first step you posted). So, why not to solve the problem using Newton-Raphson for four equations (the derivatives with respect to $\alpha,\beta,\gamma,\delta$ which are equal to zero) for the same four  unknowns.
You could even compute the derivatives using finite differences (this is what I would suggest). All of this could be done using Excel.
Edit
In order to prepare our discussion on chat, let us consider what you started with here. For $A$ and $B$, you had a table of $n$ data points $(x_i,y_i)$ and you wanted to fit according to model $$y=a x^b$$ You can easily get estimates fo the parameters using a logarithm transform $$\log(y)=\log(a)+b \log(x)=c+b\log(x)$$ and a linear regression based on data points $(\log(x_i),\log(y_i))$ gives the estimates of $(c,b)$ and $a=e^c$. Eventually, if you want a better fit, you could use the method described here.
But, since what is measured is $y$ and not any of its possible transforms, you need to continue using nonlinear regression; the goal of it will be to minimize $$F=\frac 12\sum_{i=1}^n(ax_i^b-y_i)^2$$ If you do not access a nonlinear regression tool (remember that you could use Excel solver), let us write the conditions of the minimum which is $F'_a=0$ and $F'_b=0$. Let use write these conditions $$F'_a=\sum_{i=1}^n x_i^b(ax_i^b-y_i)=0\implies a\sum_{i=1}^n x_i^{2b}-\sum_{i=1}^n x_i^by_i=0\tag 1$$ $$F'_b=\sum_{i=1}^n a x_i^b \log (x_i) \left(a x_i^b-y_i\right)=0\implies a\sum_{i=1}^n x_i^{2b}\log(x_i)-\sum_{i=1}^n x_i^b \log(x_i)y_i=0\tag 2$$ So, we have two nonlinear equations in $(a,b)$ which, for commodity, I shall rewrite as $$f(a,b)=a\sum_{i=1}^n x_i^{2b}-\sum_{i=1}^n x_i^by_i=0\tag 3$$ $$g(a,b)=a\sum_{i=1}^n x_i^{2b}\log(x_i)-\sum_{i=1}^n x_i^b \log(x_i)y_i=0\tag 4$$ Since we already have good estimates $a_0$ and $b_0$, we can use Newton-Raphson method to solve them writing $$f(a_0,b_0)+\frac {df(a,b)}{da}\Delta a+\frac {df(a,b)}{db}\Delta b=0 \tag 5$$ $$g(a_0,b_0)+\frac {dg(a,b)}{da}\Delta a+\frac {dg(a,b)}{db}\Delta b=0 \tag 6$$ the derivatives being computed at $a=a_0$ and $b=b_0$. These equations are linear. So, compute $(\Delta a,\Delta b)$ then $a=a0+\Delta a$, $b=b0+\Delta b$ and replace $a_0$ by $a$ and $b_0$ by $b$ until they do not change significantly.
You could even reduce the problem to a single equation if you notice that $a$ is explicitelt given from equation $(3)$ $$a=\frac{\sum_{i=1}^n x_i^by_i }{ \sum_{i=1}^n x_i^{2b}}$$ which makes equation $(4)$ to become $$\frac{\sum_{i=1}^n x_i^by_i }{ \sum_{i=1}^n x_i^{2b}}\sum_{i=1}^n x_i^{2b}\log(x_i)-\sum_{i=1}^n x_i^b \log(x_i)y_i=0\tag 7$$ which is a nonliear equation in $b$ (you can solve it using Newton method).
