Calculating new vector positions I'm using the following formula to calculate the new vector positions for each point selected, I loop through each point selected and get the  $(X_i,Y_i,Z_i)$ values, I also get the center values of the selection ($X,Y,Z$) , call them $(X_c,Y_c,Z_c)$  The distance of each point from the center is $d_i=\sqrt{(X_i-X_c)^2+(Y_i-Y_c)^2+(Z_i-Z_c)^2}$.
The coordinates for the new vector position is:
$g_i=\left(\frac b{d_i}(X_i-X_c)+X_c,\frac b{d_i}(Y_i-Y_c)+Y_c,\frac b{d_i}(Z_i-Z_c)+Z_c\right)$
My problem is I don't think it's averaging properly, it should be a smooth path or an average path the whole way from every axis.
Here's a screen shot before I run the script:

And here is what happens after:

It's perfect on the front axis, but as you can see from the top and the side it's not so smooth.
I'm using Python in MAYA to calculate this, here's the code I'm using:
import maya.cmds as cmds
import math
sel = cmds.ls(sl=1, fl=1)
averageDistance = 0
cmds.setToolTo('Move')
oldDistanceArray = []
cs = cmds.manipMoveContext("Move", q=1, p=1)
for i in range(0, len(sel), 1):
    vts = cmds.xform(sel[i],q=1,ws=1,t=1)
    x = cs[0] - vts[0]
    y = cs[1] - vts[1]
    z = cs[2] - vts[2]
    distanceFromCenter = math.sqrt(pow(x,2) + pow(y,2) + pow(z,2))
    oldDistanceArray += [(distanceFromCenter)]
    averageDistance += distanceFromCenter    
    if (i == len(sel) -1):
        averageDistance /= len(sel)
        for j in range(0, len(sel), 1):
            vts = cmds.xform(sel[j],q=1,ws=1,t=1)            
            gx = (((averageDistance / oldDistanceArray[j]) * (vts[0] -  cs[0])) + cs[0])
            gy = (((averageDistance / oldDistanceArray[j]) * (vts[1] -  cs[1])) + cs[1])
            gz = (((averageDistance / oldDistanceArray[j]) * (vts[2] -  cs[2])) + cs[2])
            cmds.move(gx,gy,gz,sel[j])
            cmds.refresh()

Aditionally, I have found another 'error' here: (before)

After: 
It should draw a perfect circle, but it seems my algorithm is wrong
 A: You are putting those points on a sphere, not a circle, this is why from front it looks alright (more or less, in fact I suspect that even in front it is not a circle), while from the side is not a line. To make it more as you wanted, you need align all the points in some common plane.
As for the "error" you mentioned, it is not a bug, just your algorithm works this way. To view your transform graphically, draw yourself rays from each point to the center. You should see that those are not evenly spaced and thus the effect is some parts of the circle don't get enough points.
If I were to code such a thing, I would make two more inputs: the circle I would like to obtain (a plane shape) and one special point which would tell where the circle "begins" (this is so that the circle won't appear "twisted"). Then, for each point of the transform calculate its destination evenly spaced along the circle, and then make a transform that move each point to its destination. 
Some final comments: you didn't describe any context, so I might be wrong, but the evenly spaced points on the circle might distort the over-all shape of the object (the non-evenly spaced points would disturb it less), so I don't think that is a good idea. It might be better to handle the "density of the points" by hand (e.g. by tessellating the object, whatever), but keep the point at the angles/azimuths they were (of course, you want them in one plane if you need a circle). And the algorithm would be easier (for example, you would not need the "circle starting point" then) ;-)
I hope it explains something ;-)
A: In your second example, at least, if you want it to be a circle with the yellow points that I can see highlighted, simply changing the distance to the centre won't be enough, as there are no points along the green axis to be modified to form the perfect circle. However returning to your original problem, the issue to me seems to be that half of the points selected are skewed to the back of the yellow axis, effectively moving your centre to a position that isn't actually the centre you want. By the looks of it, the centre that you want is the geometric centre - given the points you have, you should find $X_c, Y_c$, and $Z_c$ independently by finding the two points furthest from each other along each axis, and then finding the point, along that axis, that lies between those two points
