Cache prefetching

• Stream buffers were developed based on the concept of "one block lookahead (OBL) scheme" proposed by Alan Jay Smith.[1]
• Stream buffers are one of the most common hardware based prefetching techniques in use. This technique was originally proposed by Norman Jouppi in 1990[6] and many variations of this method have been developed since.[7][8][9] The basic idea is that the cache miss address (and ${\displaystyle k}$ subsequent addresses) are fetched into a separate buffer of depth ${\displaystyle k}$. This buffer is called a stream buffer and is separate from cache. The processor then consumes data/instructions from the stream buffer if the address associated with the prefetched blocks match the requested address generated by the program executing on the processor. The figure below illustrates this setup:

[6] A typical stream buffer setup as originally proposed by Norman Jouppi in 1990

• Whenever the prefetch mechanism detects a miss on a memory block, say A, it allocates a stream to begin prefetching successive blocks from the missed block onward. If the stream buffer can hold 4 blocks, then we would prefetch A+1, A+2, A+3, A+4 and hold those in the allocated stream buffer. If the processor consumes A+1 next, then it shall be moved "up" from the stream buffer to the processor's cache. The first entry of the stream buffer would now be A+2 and so on. This pattern of prefetching successive blocks is called Sequential Prefetching. It is mainly used when contiguous locations are to be prefetched. For example, it is used when prefetching instructions.
• This mechanism can be scaled up by adding multiple such 'stream buffers' - each of which would maintain a separate prefetch stream.[10] For each new miss, there would be a new stream buffer allocated and it would operate in a similar way as described above.
• The ideal depth of the stream buffer is something that is subject to experimentation against various benchmarks[6] and depends on the rest of the microarchitecture involved.[11]

This type of prefetching monitors the delta between the addresses of the memory accesses and looks for patterns within it.

In this pattern, consecutive memory accesses are made to blocks that are ${\displaystyle s}$ addresses apart.[3][12] In this case, the prefetcher calculates the ${\displaystyle s}$ and uses it to compute the memory address for prefetching. E.g.: If the ${\displaystyle s}$ is 4, the address to be prefetched would A+4.

In this case, the delta between the addresses of consecutive memory accesses is variable but still follows a pattern. Some prefetchers designs[9][13][14] exploit this property to predict and prefetch for future accesses.

This class of prefetchers look for memory access streams that repeat over time.[15][16] E.g. In this stream of memory accesses: N, A, B, C, E, G, H, A, B, C, I, J, K, A, B, C, L, M, N, O, A, B, C, ...; the stream A,B,C is repeating over time. Other design variation have tried to provide more efficient, performant implementations.[17][18]

Computer applications generate a variety of access patterns. The processor and memory subsystem architectures used to execute these applications further disambiguate the memory access patterns they generate. Hence, the effectiveness and efficiency of prefetching schemes often depend on the application and the architectures used to execute them.[19] Recent research[20][21] has focused on building collaborative mechanisms to synergistically use multiple prefetching schemes for better prefetching coverage and accuracy.

Compiler directed prefetching is widely used within loops with a large number of iterations. In this technique, the compiler predicts future cache misses and inserts a prefetch instruction based on the miss penalty and execution time of the instructions.

These prefetches are non-blocking memory operations, i.e. these memory accesses do not interfere with actual memory accesses. They do not change the state of the processor or cause page faults.

One main advantage of software prefetching is that it reduces the number of compulsory cache misses.[3]

The following example shows how a prefetch instruction would be added into code to improve cache performance.

Consider a for loop as shown below:

for (int i=0; i<1024; i++) {
    array1[i] = 2 * array1[i];
}

At each iteration, the i^th element of the array "array1" is accessed. Therefore, we can prefetch the elements that are going to be accessed in future iterations by inserting a "prefetch" instruction as shown below:

for (int i=0; i<1024; i++) {
    prefetch (array1 [i + k]);
    array1[i] = 2 * array1[i];
}

Here, the prefetch stride, ${\displaystyle k}$ depends on two factors, the cache miss penalty and the time it takes to execute a single iteration of the for loop. For instance, if one iteration of the loop takes 7 cycles to execute, and the cache miss penalty is 49 cycles then we should have ${\displaystyle k=49/7=7}$ - which means that we prefetch 7 elements ahead. With the first iteration, i will be 0, so we prefetch the 7th element. Now, with this arrangement, the first 7 accesses (i=0->6) will still be misses (under the simplifying assumption that each element of array1 is in a separate cache line of its own).

Coverage is the fraction of total misses that are eliminated because of prefetching, i.e.

${\displaystyle Coverage={\frac {\text{Cache Misses eliminated by Prefetching}}{\text{Total Cache Misses}}}}$,

where, ${\displaystyle {\text{Total Cache Misses}}=({\text{Cache misses eliminated by prefetching}})+({\text{Cache misses not eliminated by prefetching}})}$

Accuracy is the fraction of total prefetches that were useful - i.e. the ratio of the number of memory addresses prefetched were actually referenced by the program to the total prefetches done.

${\displaystyle {\text{Prefetch Accuracy}}={\frac {\text{Cache Misses eliminated by prefetching}}{({\text{Useless Cache Prefetches}})+({\text{Cache Misses eliminated by prefetching}})}}}$

While it appears that having perfect accuracy might imply that there are no misses, this is not the case. The prefetches themselves might result in new misses if the prefetched blocks are placed directly into the cache. Although these may be a small fraction of the total number of misses we might see without any prefetching, this is a non-zero number of misses.

The qualitative definition of timeliness is how early a block is prefetched versus when it is actually referenced. An example to further explain timeliness is as follows:

Consider a for loop where each iteration takes 3 cycles to execute and the 'prefetch' operation takes 12 cycles. This implies that for the prefetched data to be useful, we must start the prefetch ${\displaystyle 12/3=4}$ iterations prior to its usage to maintain timeliness.