The exercise is about convex functions:

How to prove that $f(t)=\int_0^t g(s)ds$ is convex in $(a,b)$ whenever $0\in (a,b)$ and $g$ is increasing in $[a,b]$?

I proved that

$$f(x)\leq \frac{x-a'}{b'-x}f(b')+\left(1-\frac{x-a'}{b'-x}\right)f(a')$$

when we have

$$x=\left(1-\frac{x-a'}{b'-a'}\right)a'+\frac{x-a'}{b'-a'}b'$$

-
$b'-x$ should be $b'-a'$. –  Yimin May 11 '12 at 0:38
Yeah, but I couldn't pass that stage, I just came 'till there. –  André Lima May 11 '12 at 0:41
The converse is true also. –  copper.hat May 11 '12 at 7:02

I also overcome with a different approach. Assume initially $g(t)\geq 0$.

We will consider the auxiliary function $\int_{a'}^x g(t)dt$ to that this function satisfies the convexity conditions in $a'<x<b$, that is,

$$\int_{a'}^x g(t)dt\leq \frac{x-a'}{b'-a'}\left(\int_{a'}^x g(t)dt+\int_x^{b'} g(t)dt\right).$$

To prove that observe

$$\int_{a'}^x g(x)dt (1-\frac{x-a'}{b'-a'})=\frac{x-a'}{b'-a'}\int_{x}^{b}g(x)dt$$

(Note that the integration is in t and g(x) is constant in that

Because $g$ is increasing we have $\int_{a'}^x g(t)dt\leq \int_{a'}^x g(x)dt$ and $\int_{x}^{b'} g(x)dt\leq \int_{x}^{b'} g(t)dt$

Then $$\int_{a'}^x g(t)dt (1-\frac{x-a'}{b'-a'})\leq\frac{x-a'}{b'-a'}\int_{x}^{b}g(t)dt$$

Rearranging it is the desired result.

-

You could use the fact that a function $f$ is convex on a nonempty, open interval $(a,b)$ if and only if $${f(x)-f(c)\over x-c}\le {f(d)-f(x)\over d-x}$$ whenever $a<c<x<d<b$. This follows from the fact that a chord through two points on the graph of a convex function lies on or above the graph of the function.

Then you need to show that, given $a<c<x<d<b$, $$\tag{1} {\int_c^x g(x)\,dx\over x-c}\le{\int_d^x g(x)\,dx\over d-x}.$$ But, since $g$ is increasing, there are numbers $e$ and $f$ with $g(c)\le e\le g(x)$ and $g(x)\le f\le g(d)$ such that $e={\int_c^x g(x)\,dx\over x-c}$ and $f={\int_d^x g(x)\,dx\over d-x}$; which shows that $(1)$ indeed holds.

-
We need to show $f(x+\lambda(y-x)) \leq f(x) + \lambda (f(y)-f(x))$, with $\lambda \in (0,1)$. Suppose $x<y$. Then $$f(x+\lambda(y-x)) - f(x) = \int_{x}^{x+\lambda(y-x)} g(s) \; ds$$ Using the change of variables $t=\frac{s-x}{\lambda}+x$, we get $$\int_{x}^{x+\lambda(y-x)} g(s) \; ds = \int_{x}^{y} g(\lambda(t-x)+x) \; \lambda \; dt \leq \lambda \int_{x}^{y} g(t) \; dt = \lambda(f(y)-f(x)),$$ where the second to last step follows because $\lambda(t-x)+x \leq t$, and $g$ is increasing.
If $x>y$, let $\mu = 1-\lambda$ (note $\mu \in (0,1)$), then we have already shown that $$f(y+\mu(x-y)) \leq f(y) + \mu (f(x)-f(y)).$$ Since $c+\mu(d-c) = c+(1-\lambda)(d-c) = d+\lambda(c-d)$, the desired result follows.