# Gradient of function with index operation

First of all, please let me know if anything is off with my notation. I am a computer scientist and as such I am not that strict about notation.

In my problem I want to optimize an objective function with gradient based methods. For this I have to calculate the gradient of the objective. As my state space is very big, numerical differentiation is infeasible (takes too long). I have already tried symbolic and algorithmic differentiation tools, but these cannot handle my objective.

Now of course the question arises if the function is even differentiable (at least once) or not.

Given is:

• set of $$n$$ homgeneous points $$S := \{p_1, ..., p_n\}; p_i \in R^4$$
• a gray-scale camera image $$I \in R^{h \times w}$$ of width $$w$$ and height $$h$$
• an intrinsic projection matrix of the camera $$P \in R^{3\times 4}$$
• an affine transform $$T \in R^{4\times 4}$$ (only rotation and translation)

The objective looks like this

$$f(T) = \sum_{i=1}^{n} I[ P T p_{i}] \; ||p_{i}||_{2}$$

As you recognize I use this unorthodox index operator $$[\;]$$ to indicate that I want to access the pixel value of $$I$$ at the position of the homogeneous point $$P T p_{i}$$ which is a 2D point on the image ($$\in R^3$$).

My question: Is this objective function, or more narrowed down the index operation, differentiable? Or is there a way of approximating the gradient?

Some pointers to literature that handles this kind of problem would be perfect for me!

I should note that I am interested in the gradient for $$f$$ of $$T$$ where $$T$$ is vectorized row-wise. I.e. $$\nabla f(T)_3 = \frac{\partial f(T)}{\partial T_{13}}$$