## Abstract

A new massively parallel scheme is developed to simulate free-

surface flows with the meshless method incompressible smoothed particle

hydrodynamics (ISPH) for simulations involving more than 100 million

particles. As a pressure-projection method, ISPH requires the solution

of a sparse matrix for the pressure Poisson equation (PPE) which is non

trivial for large problems where the particles are moving with

continuously evolving connectivity. The new scheme uses a Hilbert space

filling curve with a cell-linked list to map the entire domain so that

domain decomposition and load balancing can be achieved easily to take

advantage of geometric locality in order to reduce latency in memory

cache access. The computational domain can be subdivided into more than

12,000 partitions using the message passing interface (MPI) for

communication between partitions. Load balancing is achieved using the

open-source Zoltan library using a new particle weighting system. To

solve the PPE for large problems using tens of thousands of partitions,

the open-source PETSc library is used which requires the HYPRE BoomerAMG

preconditioner to ensure rapid convergence for ISPH. The performance of

the code is benchmarked on the U.K. National Supercomputer ARCHER. The

results show that domain decomposition with a space filling curve can

efficiently treat irregularly distributed particles creating a well-

balanced scheme demonstrating that the approach is well matched to the

highly irregular subdomains and non-uniform distribution of ISPH free-

surface simulations. The benchmark results show that massively parallel

ISPH code can achieve over 90% efficiency for the solution of the PPE,

but the efficiency of computing matrix coefficients decreases when using

more than 12000 partitions giving overall efficiencies in excess of

43% up to 6144 MPI partitions, highlighting future improvements

surface flows with the meshless method incompressible smoothed particle

hydrodynamics (ISPH) for simulations involving more than 100 million

particles. As a pressure-projection method, ISPH requires the solution

of a sparse matrix for the pressure Poisson equation (PPE) which is non

trivial for large problems where the particles are moving with

continuously evolving connectivity. The new scheme uses a Hilbert space

filling curve with a cell-linked list to map the entire domain so that

domain decomposition and load balancing can be achieved easily to take

advantage of geometric locality in order to reduce latency in memory

cache access. The computational domain can be subdivided into more than

12,000 partitions using the message passing interface (MPI) for

communication between partitions. Load balancing is achieved using the

open-source Zoltan library using a new particle weighting system. To

solve the PPE for large problems using tens of thousands of partitions,

the open-source PETSc library is used which requires the HYPRE BoomerAMG

preconditioner to ensure rapid convergence for ISPH. The performance of

the code is benchmarked on the U.K. National Supercomputer ARCHER. The

results show that domain decomposition with a space filling curve can

efficiently treat irregularly distributed particles creating a well-

balanced scheme demonstrating that the approach is well matched to the

highly irregular subdomains and non-uniform distribution of ISPH free-

surface simulations. The benchmark results show that massively parallel

ISPH code can achieve over 90% efficiency for the solution of the PPE,

but the efficiency of computing matrix coefficients decreases when using

more than 12000 partitions giving overall efficiencies in excess of

43% up to 6144 MPI partitions, highlighting future improvements

Original language | English |
---|---|

Number of pages | 33 |

Journal | Computer Physics Communications |

Early online date | 2 Jul 2018 |

DOIs | |

Publication status | Published - 2018 |

## Keywords

- Incompressible Smoothed Particle Hydrodynamics; Domain Decomposition; Dynamic Load Balancing; Space-filling Curve; Sparse Linear Solver; unstructured communication