I want to solve the following problem. I know it is an optimization problem, but since I don't know anything about optimization, I would appreciate if you would give me some guidance regarding where to look in order to find the answer (what kind of optimization, what kind of techniques can solve this problem etc.)
Let digital image $I$ with distribution (histogram) $h(x)$ for $x\in[0,B]$ and $x, B, h(x) \in \mathbb N$.
Let mapping function $g(A,x)=int\left[\frac{Ax}{B+A-x}\right]$ with $A \in \mathbb R^{+}$ and $g\to[0,B]$, which is applied to all the pixels of image $I$, thus, getting the transformed image $II=g(A,I)$
Let $H(A,x)$ be the histogram of image $II$
$$H(A,x)=\sum^{B}_{k=0} \left[\delta\left(g(A,k)-x\right)h(k)\right]$$
This new histogram is solely depended on the initial image $I$ and in the value $A$. The question is which value of $A$ can maximize/minimize a specific metric over the histogram $H$ of the transformed image $II$, without trying all the possible values of $A$. The metric could be histogram flatness
$$\operatorname{flatness}(A)=\frac{\sum^{B}_{i=0}\left[\frac{S}{B}-H(A,i)\right]^2}{S^2}$$
where $S=\sum^{B}_{k=0}h(k)$ (the total number of pixels in the image).
Thank you in advance for your guidance....
Feb 13 at 7:02, which was before you commented. Since the question got no response in the five days before that, chances are that the OP is not checking it daily. // I'm sure that int is a rounding function, but I also agree that optimization for $A$ should probably disregard rounding in favor of continuity. – user53153 Feb 18 at 20:32