End-to-End Deep Learning of Optimization Heuristics

Chris Cummins, Pavlos Petoumenos, Zheng Wang, Hugh Leather

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

210 Downloads (Pure)


Accurate automatic optimization heuristics are necessary for dealing with thecomplexity and diversity of modern hardware and software. Machine learning is aproven technique for learning such heuristics, but its success is bound by thequality of the features used. These features must be hand crafted by developersthrough a combination of expert domain knowledge and trial and error. This makesthe quality of the final model directly dependent on the skill and availabletime of the system architect.Our work introduces a better way for building heuristics. We develop a deepneural network that learns heuristics over raw code, entirely without using codefeatures. The neural network simultaneously constructs appropriaterepresentations of the code and learns how best to optimize, removing the needfor manual feature creation. Further, we show that our neural nets can transferlearning from one optimization problem to another, improving the accuracy of newmodels, without the help of human experts.We compare the effectiveness of our automatically generated heuristics againstones with features hand-picked by experts. We examine two challenging tasks:predicting optimal mapping for heterogeneous parallelism and GPU threadcoarsening factors. In 89% of the cases, the quality of our fully automaticheuristics matches or surpasses that of state-of-the-art predictive models usinghand-crafted features, providing on average 14% and 12% more performance withno human effort expended on designing features.

Original languageEnglish
Title of host publicationProceedings - 26th International Conference on Parallel Architectures and Compilation Techniques, PACT 2017
Number of pages14
ISBN (Electronic)9781467395243
Publication statusPublished - 31 Oct 2017
Event26th International Conference on Parallel Architectures and Compilation Techniques - Portland, United States
Duration: 9 Sept 201713 Sept 2017

Publication series

NameParallel Architectures and Compilation Techniques - Conference Proceedings, PACT
ISSN (Print)1089-795X


Conference26th International Conference on Parallel Architectures and Compilation Techniques
Abbreviated titlePACT 2017
Country/TerritoryUnited States


  • Compiler Optimizations
  • Heterogeneous Systems
  • Machine Learning
  • Optimization Heuristics


Dive into the research topics of 'End-to-End Deep Learning of Optimization Heuristics'. Together they form a unique fingerprint.

Cite this