Towards efficient mapreduce using MPI

Torsten Hoefler; Andrew Lumsdaine; Jack Dongarra

doi:10.1007/978-3-642-03770-2-30

Towards efficient mapreduce using MPI

Torsten Hoefler, Andrew Lumsdaine, Jack Dongarra

Research output: Chapter in Book/Conference proceeding › Conference contribution

Abstract

MapReduce is an emerging programming paradigm for data-parallel applications. We discuss common strategies to implement a MapReduce runtime and propose an optimized implementation on top of MPI. Our implementation combines redistribution and reduce and moves them into the network. This approach especially benefits applications with a limited number of output keys in the map phase. We also show how anticipated MPI-2.2 and MPI-3 features, such as MPI-Reduce-local and nonblocking collective operations, can be used to implement and optimize MapReduce with a performance improvement of up to 25% on 127 cluster nodes. Finally, we discuss additional features that would enable MPI to more efficiently support all MapReduce applications. © 2009 Springer Berlin Heidelberg.

Original language	English
Title of host publication	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)\|Lect. Notes Comput. Sci.
Publisher	Springer Nature
Pages	240-249
Number of pages	9
Volume	5759
ISBN (Print)	3642037690, 9783642037696
DOIs	https://doi.org/10.1007/978-3-642-03770-2-30
Publication status	Published - 2009
Event	16th European Parallel Virtual Machine and Message Passing Interface Users' Group Meeting, EuroPVM/MPI - Espoo Duration: 1 Jul 2009 → … http://dblp.uni-trier.de/db/conf/pvm/pvm2009.html#HoeflerLD09http://dblp.uni-trier.de/rec/bibtex/conf/pvm/HoeflerLD09.xmlhttp://dblp.uni-trier.de/rec/bibtex/conf/pvm/HoeflerLD09

Publication series

Name	Lecture Notes in Computer Science

Conference

Conference	16th European Parallel Virtual Machine and Message Passing Interface Users' Group Meeting, EuroPVM/MPI
City	Espoo
Period	1/07/09 → …
Internet address	http://dblp.uni-trier.de/db/conf/pvm/pvm2009.html#HoeflerLD09http://dblp.uni-trier.de/rec/bibtex/conf/pvm/HoeflerLD09.xmlhttp://dblp.uni-trier.de/rec/bibtex/conf/pvm/HoeflerLD09

Access to Document

10.1007/978-3-642-03770-2-30

Cite this

Hoefler, T., Lumsdaine, A., & Dongarra, J. (2009). Towards efficient mapreduce using MPI. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)|Lect. Notes Comput. Sci. (Vol. 5759, pp. 240-249). (Lecture Notes in Computer Science). Springer Nature. https://doi.org/10.1007/978-3-642-03770-2-30

@inproceedings{15e59fe8e7864862b9379bb4d4143f27,

title = "Towards efficient mapreduce using MPI",

abstract = "MapReduce is an emerging programming paradigm for data-parallel applications. We discuss common strategies to implement a MapReduce runtime and propose an optimized implementation on top of MPI. Our implementation combines redistribution and reduce and moves them into the network. This approach especially benefits applications with a limited number of output keys in the map phase. We also show how anticipated MPI-2.2 and MPI-3 features, such as MPI-Reduce-local and nonblocking collective operations, can be used to implement and optimize MapReduce with a performance improvement of up to 25% on 127 cluster nodes. Finally, we discuss additional features that would enable MPI to more efficiently support all MapReduce applications. {\textcopyright} 2009 Springer Berlin Heidelberg.",

author = "Torsten Hoefler and Andrew Lumsdaine and Jack Dongarra",

year = "2009",

doi = "10.1007/978-3-642-03770-2-30",

language = "English",

isbn = "3642037690",

volume = "5759",

series = "Lecture Notes in Computer Science",

publisher = "Springer Nature",

pages = "240--249",

booktitle = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)|Lect. Notes Comput. Sci.",

address = "United States",

note = "16th European Parallel Virtual Machine and Message Passing Interface Users' Group Meeting, EuroPVM/MPI ; Conference date: 01-07-2009",

url = "http://dblp.uni-trier.de/db/conf/pvm/pvm2009.html#HoeflerLD09http://dblp.uni-trier.de/rec/bibtex/conf/pvm/HoeflerLD09.xmlhttp://dblp.uni-trier.de/rec/bibtex/conf/pvm/HoeflerLD09",

}

Hoefler, T, Lumsdaine, A & Dongarra, J 2009, Towards efficient mapreduce using MPI. in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)|Lect. Notes Comput. Sci.. vol. 5759, Lecture Notes in Computer Science, Springer Nature, pp. 240-249, 16th European Parallel Virtual Machine and Message Passing Interface Users' Group Meeting, EuroPVM/MPI, Espoo, 1/07/09. https://doi.org/10.1007/978-3-642-03770-2-30

Towards efficient mapreduce using MPI. / Hoefler, Torsten; Lumsdaine, Andrew; Dongarra, Jack.
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)|Lect. Notes Comput. Sci.. Vol. 5759 Springer Nature, 2009. p. 240-249 (Lecture Notes in Computer Science).

Research output: Chapter in Book/Conference proceeding › Conference contribution

TY - GEN

T1 - Towards efficient mapreduce using MPI

AU - Hoefler, Torsten

AU - Lumsdaine, Andrew

AU - Dongarra, Jack

PY - 2009

Y1 - 2009

N2 - MapReduce is an emerging programming paradigm for data-parallel applications. We discuss common strategies to implement a MapReduce runtime and propose an optimized implementation on top of MPI. Our implementation combines redistribution and reduce and moves them into the network. This approach especially benefits applications with a limited number of output keys in the map phase. We also show how anticipated MPI-2.2 and MPI-3 features, such as MPI-Reduce-local and nonblocking collective operations, can be used to implement and optimize MapReduce with a performance improvement of up to 25% on 127 cluster nodes. Finally, we discuss additional features that would enable MPI to more efficiently support all MapReduce applications. © 2009 Springer Berlin Heidelberg.

AB - MapReduce is an emerging programming paradigm for data-parallel applications. We discuss common strategies to implement a MapReduce runtime and propose an optimized implementation on top of MPI. Our implementation combines redistribution and reduce and moves them into the network. This approach especially benefits applications with a limited number of output keys in the map phase. We also show how anticipated MPI-2.2 and MPI-3 features, such as MPI-Reduce-local and nonblocking collective operations, can be used to implement and optimize MapReduce with a performance improvement of up to 25% on 127 cluster nodes. Finally, we discuss additional features that would enable MPI to more efficiently support all MapReduce applications. © 2009 Springer Berlin Heidelberg.

U2 - 10.1007/978-3-642-03770-2-30

DO - 10.1007/978-3-642-03770-2-30

M3 - Conference contribution

SN - 3642037690

SN - 9783642037696

VL - 5759

T3 - Lecture Notes in Computer Science

SP - 240

EP - 249

BT - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)|Lect. Notes Comput. Sci.

PB - Springer Nature

T2 - 16th European Parallel Virtual Machine and Message Passing Interface Users' Group Meeting, EuroPVM/MPI

Y2 - 1 July 2009

ER -

Towards efficient mapreduce using MPI

Abstract

Publication series

Conference

Access to Document

Fingerprint

Cite this