Graphic processing units rely on a hierarchy of memory for optimization, where the smallest, fastest memory—registers—is located closest to the core and provides access to data in a fraction of a nanosecond.

Findings

Additional insights we found via Charles Grassi

Registers store the values most frequently used in GPU computations and lower latency by reducing the reliance on slower forms of memory for these values.
The L1 cache lives inside the streaming multiprocessor as private memory and can house data for shared use by the cores that make up the SM, rather than having each pull it from the L2 cache.
The L2 cache is shared across the entire GPU and sits outside the streaming multiprocessors that make up the graphics processing clusters.
By holding data needed by multiple parts of the GPU, the L2 cache can save trips to the slower VRAM, improving processing time.
The largest pool of memory is video random access memory (VRAM), a high-bandwidth type of memory that serves as a holding area for data before it is sent up the hierarchy to the L2 cache.