derivative of a function using cosine transform If we have a periodic function y(x) the derivative of function can be found using Fourier transform as 
$derivative = y'(x)$
taking Fourier transform
$F(derivative) = -kiy(x)$ i is the complex number
$derivative = F^{-1}(-kiy(x))$
Now if the function is real non-periodic function with Neumann boundary condition, can its derivative be found by taking cosine transform?
 A: The Neumann boundary condition asks that we solve a differential equation subject to the condition that the derivative of the function involved takes on some specific value at the boundary of the domain over which we are solving. It follows that we require the derivative to exist at the boundary but other than that it does not effect our ability to represent the derivative as a cosine transform.
Assuming that the Fourier transform of the function exists, the following shows when the Fourier transform representation of the derivative reduces to a cosine transform; it does not require the function to be periodic.
If you define:
\begin{align}\hat{F}(\xi)&=\int_{-\infty}^{\infty}f(x)e^{-i2\pi x \xi}{dx},\tag{1}\\
f(x)&=\int_{-\infty}^{\infty}\hat{F}(\xi)e^{i2\pi x \xi}{d\xi},\end{align}
then:
\begin{aligned} \dfrac{d}{dx}f(x)&=\int_{-\infty}^{\infty}\left(i2\pi\xi\right)\hat{F}(\xi)e^{i2\pi x \xi}{d\xi},\\
\dfrac{d}{dx}f(x)&=\int_{-\infty}^{\infty}\left(i2\pi\xi\right)\hat{F}(\xi)\cos(2\pi x \xi){d\xi}-\int_{-\infty}^{\infty}\left(2\pi\xi\right)\hat{F}(\xi)\sin(2\pi x \xi){d\xi},\end{aligned}
so to reduce the Fourier transform representation of the derivative to the cosine transform only we would require:
$$\int_{-\infty}^{\infty}\left(2\pi\xi\right)\hat{F}(\xi)\sin(2\pi x \xi){d\xi}=0$$
and as $\sin$ is an odd function this integral vanishes when $\left(2\pi\xi\right)\hat{F}(\xi)$ is an even function which occurs when:
$$\xi\hat{F}(\xi)=-\xi\hat{F}(-\xi),$$
$$\hat{F}(\xi)=-\hat{F}(-\xi),\tag{2a}$$
that is when $\hat{F}(\xi),$ is odd, and we see from $(1)$ that this requires $$f(x)=-f(-x)\tag{2b}$$ i.e. it requires $f(x)$ to be odd. But if $(2)$ holds then $(1)$ can be written:
\begin{align}
\hat{F}(\xi)&=-i\int_{-\infty}^{\infty}f(x)\sin(2\pi x \xi){dx},\tag{3}\\
f(x)&=i\int_{-\infty}^{\infty}\hat{F}(\xi)\sin(2\pi x \xi){d\xi},
\end{align}

So to summarise, assuming $(1)$ exists and $(2)$ holds we may define the derivative by the following cosine transform:
$$\dfrac{d}{dx}f(x)=i2\pi\int_{-\infty}^{\infty}\xi\hat{F}(\xi)\cos(2\pi x \xi){d\xi}$$
which we see would also result from applying differentiation under the integral sign to $(3)$. 
 
With that said, you have not specified what it is that you want to cosine transform in order to find the derivative; if this equation holds:
$$\dfrac{d}{dx}f(x)=\int_{-\infty}^{\infty}\left(i2\pi\xi\right)\hat{F}(\xi)e^{i2\pi x \xi}{d\xi},$$
then we could simply define:
$$\hat{F}(\xi)=\hat{G}(\xi)\cos(2\pi x \xi)e^{-i2\pi x \xi},$$
to obtain a cosine transform representation of the derivative but this is being a bit pedantic; some clarification of what it is you hope to cosine transform would help to clear that up.
