Challenges and Opportunities of DNN Model Execution Caching

Guin R. Gilman, Samuel S. Ogden, Robert J. Walls, Tian Guo

Research output: Contribution to journalArticlepeer-review


We explore the opportunities and challenges of model execution caching, a nascent research area that promises to improve the performance of cloud-based deep inference serving. Broadly, model execution caching relies on servers that are geographically close to the end-device to service inference requests, resembling a traditional content delivery network (CDN). However, unlike a CDN, such schemes cache execution rather than static objects. We identify the key challenges inherent to this problem domain and describe the similarities and differences with existing caching techniques. We further introduce several emergent concepts unique to this domain, such as memory-adaptive models and multi-model hosting, which allow us to make dynamic adjustments to the memory requirements of model execution. 
Original languageAmerican English
JournalDIDL '19: Proceedings of the Workshop on Distributed Infrastructures for Deep Learning
StatePublished - 2019
Externally publishedYes


  • caching algorithms
  • deep learning
  • edge server


  • Computer Sciences

Cite this