How to work on a problem that is too large to fit in mind at once? My question here is about a process.
Anamnesis.
I have been researching a numeric algorithm for a computer program. The idea is not difficult to grasp, and every part of the study — be it proof of termination, or complexity analysis, whatever — is well within my reach. But at some point into it, I started losing the vision of the way all the pieces fit together. So now I have this notebook filled with page upon a page of mathematics, and I am drowning in it, I cannot get it done. I have a simpler algorithm that I can check the numbers against, and I remember having developed a clear understanding of all the separate parts of the work some time ago, so at this point there should not be anything to block my work. Yet my progress stopped.
Hypothesis.
My hypothesis is that this inability to progress is due to the complexity of recovering the meaning that was discovered, put on a page and forgotten. At some point the time and effort that has to be spent to bring knowledge back to memory has become so large that the thought process throttled.
My further inference is that I need to compartmentalize the knowledge base accumulated so far, so that I need not remember how this proof or that analysis proceeds in order to integrate it into the whole of the research, and also that I could put some effort into making every individual compartment more accessible in case I do need to look into it again. Moreover, it would then be preferable to compartmentalize and make accessible as I go, to make this type of activity a part of the research process, on par with the more prominently mathematical activities such as imaginative discovery and proof composition.
Parallel.
In computer programming, we have these tools such as programming languages, file systems and version control repositories. A program would be written in a well specified, highly readable language that is also easy to write and verify, it would be compartmentalized into individual modules, stored in files, and every change made to it would be recorded in the version control. This way, one does not ever need to understand the whole program at once. In the meantime, for mathematics I still use a paper notebook and a pen, and there is nothing to keep the heaps of notes from becoming an unmanageable mess.
Two ways to improvement.
Concerning accessibility, I could rewrite selected parts into Latex, but it is a tedious, mentally taxing task, and I am not sure if it is worth the effort. I actually tried taking notes directly in Latex, but I would then spend more time battling with the markup than making mathematical progress. The other option is to use a cleaner paper, a finer pen, and to write more neatly, but there is a human limit to that.
Concerning compartmentalization, I understand that one can use theorems and lemmas to structure the theory, but I do not understand exactly how to do that. For instance, should I consider every complexity analysis of an algorithmic part a separate theorem? How do I decide what deserves a separate theorem title? How do I connect the pieces and make sure they fit?

Am I thinking in the right direction?
What can be done?

Also see this question on Reddit.
 A: This is not only a mathematical approach but more of an engineering work technique : 
To utilize layers of abstraction to get a better ordering of all the thoughts.


*

*Define different layers with different purposes. In engineering this can be for example 


*

*system layer, 

*software layer, 

*hardware layer : electronics hardware 

*physical layer : physics, optics, electrics, mechanics



Or maybe even more famous, the $7$ layers of a computer network stack : If we tried considering all $7$ layers in one big mess we would quickly get exhausted even for quite small applications. So we separate application layer from the other layers, and hopefully when focusing on one layer at a time, things will go smoother for us. 
Also within one layer we often split into many parts to help reduce the number of things we need to keep in mind all the time: for example splitting application into different libraries, modules et.c.
For a mathematical algorithm we can use similar layers of abstraction:


*

*System level. Public functions exposed by library.

*Templates, concepts, inheritance, interfaces : fields, (matrix) multiplication

*Linear algebra high level routines

*Linear algebra low level routines

*Hand-optimized calculation parts

