Chapter 7. Processing Large Datasets with Limited RAM
In the previous chapter, we learned how to optimize the memory consumption of R programs by reducing the copying of data and by removing temporary data. Sometimes, that is still not enough. We might have data that is too large to even fit into memory, let alone perform any computations on them, or even if the data can fit into memory, there is not much free memory left for the analyses that we need to perform.
In this chapter, we will learn advanced techniques to overcome memory limitations and process large datasets.
This chapter covers:
- Using memory-efficient data structures
- Using memory-mapped files and processing data in chunks
