AMReX basicsΒΆ

Nyx is built on the Adaptive Mesh Refinement (AMR) library AMReX. This section provides a very sporadic description of the main AMReX classes and concepts relevant for Nyx. For more details, please visit the AMReX basics doc page, and the rest of the AMReX documentation.

  • amrex::Box: Dimension-dependent lower and upper indices defining a rectangular volume in 3D (or surface in 2D) in the index space. Box is a lightweight meta-data class, with useful member functions.
  • amrex::BoxArray: Collection of Box on a single AMR level. The information of which MPI rank owns which Box in a BoxArray is in DistributionMapping.
  • amrex::FArrayBox: Fortran-ordered array of floating-point amrex::Real elements defined on a Box. A FArrayBox can represent scalar data or vector data, with ncomp components.
  • amrex::MultiFab: Collection of FAB (= FArrayBox) on a single AMR level, distributed over MPI ranks. The concept of ghost cells is defined at the MultiFab level.
  • amrex::ParticleContainer: A collection of particles; in Nyx the main ParticleContainer contains dark matter particles.

Particles in a ParticleContainer are organized per Box. Particles in a Box are organized per tile (this feature is off when running on GPU). Particles within a tile are stored in several structures, each being contiguous in memory: (i) an Array-Of-Struct (AoS) (often called data, they are the 3D position, the particle ID and the index of the CPU owning the particle), where the Struct is an amrex::Particle and (ii) Struct-Of-Arrays (SoA) for extra variables (often called attribs).

The simulation domain is typically decomposed into multiple boxes, and each MPI rank owns, and performs operations on, the fields and particles defined on the boxes assigned to that rank. However, every rank does have the full metadata, i.e the list of all boxes and which ranks the data on those boxes is assigned to (the DistributionMapping). For convenience, AMReX provides iterators, to easily iterate over all the FArrayBox (with optional logical tiling) in a MultiFab that are owned by that MPI rank (MFIter), or over all particles in a ParticleContainer on a per-box basis (ParIter). These are respectively done in loops such as:

// mf is a MultiFab
for ( amrex::MFIter mfi(mf, TilingIfNotGpu()); mfi.isValid(); ++mfi ) { ... }

and

// *this is a pointer to a ParticleContainer
for (ParIter pti(*this, lev); pti.isValid(); ++pti) { ... }