# dependent data in Reinforcement learning

Hello I have a question about reinforcement learning. I was watching RL course by David Silver, and in lecture 6: Value Function Approximation, he says that in reinforcement learning, the data you get can be dependent, like non-iid data. However to my knowledge, we use Markov state to represent the state of agent in reinforcement learning, and a characteristic of Markov state is each state is independent, we don't need to know the state before previous state to make decision, then why is he saying that data can be dependent? Can someone clarify it for me? Thanks