Diagnosing and Fixing Performance Issues in an Open Source Code

The code solves fluid dynamics equations and is very well written and categorized in different files, with ample explanations and debug options included. However, when running this code on a cloud server with access to 500 GB of RAM, it exits with the error “Linux Signal 9 SIGKILL Kill,” even on very low mesh numbers. This indicates a significant problem, likely related to memory management or resource limits, given the ample memory available.

One key observation is that, in a section of the algorithm, the code needs to solve a linear system Ax = B, which it does using the LAPACK library. On extremely low mesh counts, this portion of the code executes efficiently, and the solver progresses to the main algorithm, initiating the time step counting. However, as the mesh size increases to moderate and high levels (which are necessary for accurate simulations), the time required to solve Ax = B increases exponentially. On a local system, the code remains in this portion until it solves it, which, depending on the mesh number, can take hours and is not practical. But on the cloud, it exits with signal 9.

Although it is not certain that the linear solver is the main culprit, it can be a good starting point to diagnose and solve the problem. I can provide an input file that allows you to adjust the mesh size and recreate the issue.