I take this as an advice question: I will explain my experience in regards to what you're discussing, but I will leave it to you to make the decision of whether or not you should switch your concentration to mathematics. Usually (note that this distinction is important and many professors have different ways of conducting their courses), in upper-level math courses, there is definitely a component that is lecture-focused. In other words, lectures are important in some ways: professors are typically very knowledgeable in what they are teaching and can point out the highlights, or the topics and examples that are worth noting in your study of the subject of the course. Professors go through the process of determining what is really crucial to understanding the content before they even begin lecturing, reinforced by their expertise. Thus, attending lecture and focusing on the professor's approach and examples is a powerful resource. However, ultimately in a mathematics course, the learning and work is largely left to the student with the professor as a guide. Often, homeworks serve as an outlet for students to work through the motivation and heart of content - this is where most of your effort and subsequent reward (understanding) will come from. In summary, the lectures and the textbook are considered more of resources in a mathematics course in comparison with courses in other disciplines. However, weighing the benefit of these resources, lectures come out on top because the professor goes through the process of figuring out what you need to know (in his/her class) beforehand. In the end, though, you should evaluate the relative benefit of these on a specific course/professor combination basis. This is only a general discussion based on my experiences.
