# Dynamic Programming: convergence theorems

Sorry my ignorance in advance, but I've failed to find the answer to this question both in Bellman's book and Google.

I have the following optimization problem (similar to Markov decision process), which I want to numerically solve using dynamic programming approach:

$$\pi(s) = \arg \min_{a} \int ds' p_a(s,s')\left(R_a(s,s') + \gamma V(s')\right),$$ $$V(s) = \int ds' p_{\pi(s)}(s,s')\left(R_{\pi(s)}(s,s') + \gamma V(s')\right),$$

where $s$ is continuous state. $R_a(s,s')$ — known transition cost from state $s'$ to state $s$ under action $a$. $\gamma$ is a constant close to $1$, $V(s)$ is a cost function $\mathbf{R} \rightarrow \mathbf{R}$. $\pi(s)$ — optimal policy function $\mathbf{R} \rightarrow A$, where $A$ is finite action space.

It is known that the solution $V(s)$ and $\pi(s)$ exists. However, to solve the problem numerically one needs to operate with discrete state space. Instead of state variable $s$ there would be a finite state grid $\hat s_i$ defined with some step $\Delta s$ and $K$ nodes. Also, instead of $V(s)$, there would be a vector of $\hat V_i$ ($V(s)$ calculated on discrete grid $\hat s_i$ in essence).

So, the following problem is solved:

$$\hat \pi_i = \arg \min_{a} \sum_{j} p_a(\hat s_i,\hat s_j)\left(R_a(\hat s_i,\hat s_j) + \gamma \hat V_j\right),$$ $$\hat V_i = \sum_{j} p_{\hat \pi_i}(\hat s_i,\hat s_j)\left(R_{\hat \pi_i}(\hat s_i,\hat s_j) + \gamma \hat V_j\right),$$

The thing that I missed is the following. There must be a convergence theorem which states that $\hat V_i \rightarrow V(\hat s_i)$ as $K \rightarrow \infty$ and $\hat \pi_i \rightarrow \pi(\hat s_i)$ as $K \rightarrow \infty$. In other simple words, I am looking for prove why discrete numerical solution can be used instead of read continuous solution.