Calculating values on the fly instead of storing them persistently

While executing an R program, it is sometimes convenient to cache all the data needed by the program, including the results of intermediate computations into a RAM prior to execution. During the execution, as and when the program needs to access any part of the data, it can be done very rapidly as all the data has been loaded into the R workspace. Caching intermediate results in RAM can save computational time significantly, especially when they are accessed frequently, as unnecessary recalculation of the data is avoided.

This is not a problem when the cached data can fit into RAM. However, it becomes a problem when there is not enough memory space to contain the data. The good news is, in many cases, the program does not need all parts of the data at the same time. One solution is to swap in and out portions of the data between RAM and the hard disk. Because disk I/O is slow, as we have established in Chapter 1, Understanding...