New Massively Parallel Scheme for Incompressible Smoothed Particle Hydrodynamics (ISPH) for Highly Nonlinear and Distorted Flow

A new massively parallel scheme is developed to simulate free-
surface flows with the meshless method incompressible smoothed particle
hydrodynamics (ISPH) for simulations involving more than 100 million
particles. As a pressure-projection method, ISPH requires the solution
of a sparse matrix for the pressure Poisson equation (PPE) which is non
trivial for large problems where the particles are moving with
continuously evolving connectivity. The new scheme uses a Hilbert space
filling curve with a cell-linked list to map the entire domain so that
domain decomposition and load balancing can be achieved easily to take
advantage of geometric locality in order to reduce latency in memory
cache access. The computational domain can be subdivided into more than
12,000 partitions using the message passing interface (MPI) for
communication between partitions. Load balancing is achieved using the
open-source Zoltan library using a new particle weighting system. To
solve the PPE for large problems using tens of thousands of partitions,
the open-source PETSc library is used which requires the HYPRE BoomerAMG
preconditioner to ensure rapid convergence for ISPH. The performance of
the code is benchmarked on the U.K. National Supercomputer ARCHER. The
results show that domain decomposition with a space filling curve can
efficiently treat irregularly distributed particles creating a well-
balanced scheme demonstrating that the approach is well matched to the
highly irregular subdomains and non-uniform distribution of ISPH free-
surface simulations. The benchmark results show that massively parallel
ISPH code can achieve over 90% efficiency for the solution of the PPE,
but the efficiency of computing matrix coefficients decreases when using
more than 12000 partitions giving overall efficiencies in excess of
43% up to 6144 MPI partitions, highlighting future improvements