Sozykin A.   Epanchintsev T.   Zverev V.   Bersenev A.  

Using the FEniCS Framework for Heart Simulation on Parallel Computing Systems

Reporter: Sozykin A.

The purpose of this paper is to evaluate how the automated scientific computing frameworks could be used for heart simulation on parallel computing systems. Heart simulation is a complex task that requires multiscale modelling on the cell, tissue, and organ levels [1]. Such problems are computationally intensive and need parallel computing, including the use of modern computational accelerators such as GPU and Xeon Phi. However, porting complicated multilevel simulation code to a new architecture requires a long time (often 3–5 years) during which the architecture may become obsolete. In addition, adaptation for parallel computing architectures often leads to significant changes of code. Consequently, it is very hard to determine which mathematical models and numerical methods are used in the optimized code. Hence, reflecting the changes in a mathematical model in the optimized code can be very complicated. As a result, complex multiscale simulation software is rarely adapted to modern parallel computing architectures.  
An alternative approach is based on the usage of frameworks for automated scientific computing. Such frameworks allow the development of simulation software using programming languages with near-mathematical notation. Traditionally, a significant disadvantage of such frameworks was their low simulation performance. However, some modern implementations use advanced tools such as highly efficient libraries, just-in-time compilers, parallel execution, and so on, which improve their performance. Still, it is not clear if  modern automated scientific computing frameworks are efficient enough for real multiscale simulation tasks such as heart simulation.
We present an overview of existing frameworks for automated scientific computing, including OpenFOAM, OpenCMISS, FEniCS, and Chaste. The FEniCS framework [2] was chosen for our purpose, because it provides the automated solution of differential equations by finite element methods and supports automatic parallel execution using MPI. In addition, FEniCS framework is widely used and well documented.
To evaluate the performance of the FEniCS framework for heart simulation tasks, we use the electrophysiological model of cardiac left ventricle and the cell model that were developed in our group [3]. The model incorporates a description of approximately 20 ionic currents and elements of the intracellular signalling. We focus on the implementation of the largest component of the model that is responsible for the simulation of transmembrane potential. This component is a system of nonlinear differential equations that cannot be solved analytically and is very computationally intensive due to the large amount of variables in the 3D domain. The model was implemented using the FEniCS framework in the Python language. The Python code was converted to C/C++ by the Instant dynamic compiler, which is a part of FEniCS. The parallel execution using MPI was automatically provided by FEniCS with the help of the PETSc library. The computational experiments were executed on the “URAN” supercomputer of the Institute of Mathematics and Mechanics of UrB RAS.
During the computational experiments, we investigated the space propagation of membrane potential alternation over left ventricle of human heart. The simulation of one cardiac cycle (one second of physical time) was performed. We used the symmetric anatomical model of left ventricle [4], which was represented by a tetrahedral mesh with 50000 elements.
Performance and scalability of the FEniCS-based solution was compared with the manual implementation of the same model in the LeVen system [5], which uses the C language and OpenMP for parallelization. The performance of the two implementations was roughly the same, but the FEniCS-based solution provides better scalability: it is scaled up to 64 CPU cores, while LeVen only scaled up to 8 cores.
The FEniCS framework is an efficient tool for heart simulation on parallel computing systems because it provides convenient near-mathematical notation, high simulation performance, and scales well.
References
[1] R. C. P. Kerckhoffs, S. N. Healy, T. P. Usyk, and A. D. McCulloch “Computational Methods for Cardiac Electromechanics” Proceedings of the IEEE 94(4), pp. 769-783 (2006).
[2] A. Logg, K.-A. Mardal, G. N. Wells et al. Automated Solution of Differential Equations by the Finite Element Method, (Springer, Berlin, 2012).
[3] O. Solovyova, N. Vikulova, L. B. Katsnelson, V. S. Markhasin, P. J. Noble, A. Garny, P. Kohl, and D. Noble, “Mechanical interaction of heterogeneous cardiac muscle segments in silico: Effects on Ca2+ handling and action potential,” Int. J. Bifurcat. Chaos, vol. 13, pp. 3757–3782 (2003).
[4] S.F. Pravdin, V.I. Berdyshev, A.V. Panfilov, L.B. Katsnelson, O. Solovyova, V.S. Markhasin “Mathematical model of the anatomy and fibre orientation field of the left ventricle of the heart” Biomed Eng Online. 12:54 (2013).
[5] A. Sozykin, S. Pravdin, A. Koshelev, V. Zverev, K. Ushenin, and O. Solovyova, “LeVen – a parallel system for simulation of the heart left ventricle” 9th International Conference on Application of Information and Communication Technologies, AICT 2015 – Proceedings, pp. 249-252 (2015).