Compiling at OLCF

Summit

In order to compile you will need to swap the xl module with gcc (you need to use atleast gcc/7.4.0 due to C++17 support):

module load gcc/10.2.0

Note

You will need to load the same module you use for compiling in your submissions script, otherwise the code won’t find the shared libraries at runtime.

Then load CUDA:

module load cuda/11.5.2

Note

Presently you will see a warning when you load a CUDA 11 module, but the packages should work fine.

You also need to make sure you have the python module loaded:

module load python

Compile with USE_CUDA=TRUE (and COMP=gnu which is usually the default). Is important to setup USE_OMP=FALSE, since the TRUE option is currently disallowed by Castro. An example compilation line is:

make COMP=gnu USE_MPI=TRUE USE_CUDA=TRUE -j 4

The recommended/tested version pairs are:

  • gcc/10.2.0 + cuda/11.5.2

Note

  • OpenMP offloading to the GPU is controlled by USE_OMP_OFFLOAD, which will default to FALSE, AMReX-Astro doesn’t use this feature.

Frontier

log into: frontier.olcf.ornl.gov

see: https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#programming-environment

load modules:

module load PrgEnv-gnu
module load cray-mpich/8.1.27
module load craype-accel-amd-gfx90a
module load amd-mixed/6.0.0
module unload darshan-runtime

this will load ROCm 6.0.0

Note

Tabulate rates seem to exhibit a strange slow down on Frontier, so it is best to run without rate tabulation.

build via:

make USE_HIP=TRUE

HIP Function Inlining

By default, the ROCm compiler inlines all function calls in device code (for better compatibility with codes that use file- or function-scoped __shared__ variables). This greatly increases the time it takes to compile and link, and may be detrimental for the templated Microphysics networks with lots of compile-time loop unrolling.

This can be disabled by passing flags to hipcc to allow non-inlined function calls:

make USE_HIP=TRUE EXTRACXXFLAGS='-mllvm -amdgpu-function-calls=true'

See also https://rocm.docs.amd.com/en/docs-5.3.3/reference/rocmcc/rocmcc.html#rocm-compiler-interfaces

Note

Inline is automatically disabled via our Microphysics repository when it detects a HIP build.