Enable Simultaneous DNN Services Based on Deterministic Operator Overlap and Precise Latency Prediction

Authors: Weihao Cui, Han Zhao, and Quan Chen (Shanghai Jiao Tong University); Ningxin Zheng (Microsoft Research Asia); Jingwen Leng and Jieru Zhao (Shanghai Jiao Tong University); Zhuo Song, Tao Ma, and Yong Yang (Alibaba Cloud); and Chao Li and Minyi Guo (Shanghai Jiao Tong University)

Abstract: While user-facing services experience diurnal load patterns, co-locating services improve the hardware utilization. Prior work on co-locating services on GPUs run queries sequentially, as the latencies of the queries are neither stable nor predictable when running simultaneously. The input sensitiveness and the non-deterministic operator overlap are two primary factors of the latency unpredictability. Hence, We propose Abacus, a runtime system that runs multiple services simultaneously. Abacus enables deterministic operator overlap to enforce latency predictability. Abacus consists of an overlap-aware latency predictor, a headroom-based query controller, and segmental model executors. The predictor predicts the latencies of the deterministic operator overlap. The controller determines the appropriate operator overlap for QoS guarantee of all the services. The executors run the operators as needed to support the deterministic operator overlap. Our evaluation shows that Abacus reduces 51.3% of the QoS violation and improves the throughput by 29.8% on average compared with state-of-the-art solutions.


Presentation: file


Back to Technical Papers Archive Listing