The triangle has the following vertices:
$$(0,0),(c,0),(0,c)$$
I drew this figure and found the slope and the line:
$$m=\frac{0-c}{c-0}\frac{-c}{c}=-1$$
And was then able to form the following line:
$$y-0=-1(x-c)$$
$$y=-x+c$$
Since the mass is calculated with the following formula:
$$M=\iint_D \rho(x,y)\;dA = \int^{c}_{0}\int^{-x+c}_{0}x^2+y^2\;dy\;dx$$
I then converted this to cylindrical coordinates for ease of computation:
$$r\sin\theta=-r\cos\theta+c$$
$$r=\sqrt{c}$$
My integral thus becomes:
$$M=\int^{\frac{\pi}{2}}_{0}\int^{\sqrt{c}}_{0}r^2\;r\;dr\;d\theta$$
Is this correct? I'm mainly lost at what the limits of integration should be for my outer integral.