Accelerating Data Transfer for Throughput Processors.

Abstract

Graphics processing units (GPUs) have become prevalent in modern computing systems. While their highly parallel architectures are traditionally used as accelerators for rendering graphics, GPUs are also adept at handling data parallel workloads when provided large blocks of data for processing.

Extracting performance from a GPU requires the programmer to provide enough work to keep the device fully utilized. Unlike CPUs, which are highly optimized to reduce memory access latency, GPUs are optimized for throughput and tend to have high access latency. The naive approach to obtaining performance is to provide a GPU with hundreds to thousands of threads so that some threads will be able to perform computation while others are waiting for data to arrive. This approach, however, cannot guarantee that there will always be enough computation that can hide the long latency of off-chip memory access.

Common memory access patterns on GPUs further complicate code optimization. These patterns include streaming data that is only used once, tiling data in scratchpad memories to preserve locality and share data among many threads, and irregular accesses where neighboring threads access divergent memory locations. Limitations posed by the microarchitecture of modern GPU cores can hinder the GPUs ability to effectively hide memory access latency. This in turn limits GPU throughput and slows down execution of code on GPUs.

This thesis proposes architectural modifications to GPUs that address the issues and inefficiencies posed by these access patterns through the decoupling of memory requests from threads, the execution pipeline, and limited memory system resources. For streaming accesses, instead of threads requesting their own data, data is delivered to threads in a manner that better utilizes available memory bandwidth. Tiled accesses are offloaded to specialized hardware that implements direct memory access for GPUs, freeing computation resources from generating tile addresses and improving tile transfer times. The cost of divergence in irregular patterns is ameliorated using a scatter-gather mechanism distributed across the memory subsystem, which reduces traffic across the on-chip interconnect. The proposed modifications effectively improve throughput, boosting kernel performance on average by 1.23x, 1.36x, and 1.29x, for streaming, tiled, and irregular accesses respectively.