AMReX basicsΒΆ
Nyx is built on the Adaptive Mesh Refinement (AMR) library AMReX. This section provides a very sporadic description of the main AMReX classes and concepts relevant for Nyx. For more details, please visit the AMReX basics doc page, and the rest of the AMReX documentation.
amrex::Box
: Dimension-dependent lower and upper indices defining a rectangular volume in 3D (or surface in 2D) in the index space.Box
is a lightweight meta-data class, with useful member functions.amrex::BoxArray
: Collection ofBox
on a single AMR level. The information of which MPI rank owns whichBox
in aBoxArray
is inDistributionMapping
.amrex::FArrayBox
: Fortran-ordered array of floating-pointamrex::Real
elements defined on aBox
. AFArrayBox
can represent scalar data or vector data, withncomp
components.amrex::MultiFab
: Collection of FAB (=FArrayBox
) on a single AMR level, distributed over MPI ranks. The concept of ghost cells is defined at theMultiFab
level.amrex::ParticleContainer
: A collection of particles; in Nyx the main ParticleContainer contains dark matter particles.
Particles in a ParticleContainer
are organized per Box
. Particles in a Box
are organized per tile
(this feature is off when running on GPU).
Particles within a tile are stored in several structures, each being contiguous in memory: (i) an Array-Of-Struct (AoS) (often called data, they are the 3D position, the particle ID and the index of the CPU owning the particle), where the Struct is an amrex::Particle
and (ii) Struct-Of-Arrays (SoA) for extra variables (often called attribs
).
The simulation domain is typically decomposed into multiple boxes, and each MPI rank owns, and performs operations on,
the fields and particles defined on the boxes assigned to that rank. However, every rank does have the full metadata, i.e
the list of all boxes and which ranks the data on those boxes is assigned to (the DistributionMapping
).
For convenience, AMReX provides iterators, to easily iterate over all the FArrayBox
(with optional logical tiling)
in a MultiFab
that are owned by that MPI rank (MFIter
), or over all particles in a ParticleContainer
on a per-box basis (ParIter
).
These are respectively done in loops such as:
// mf is a MultiFab
for ( amrex::MFIter mfi(mf, TilingIfNotGpu()); mfi.isValid(); ++mfi ) { ... }
and
// *this is a pointer to a ParticleContainer
for (ParIter pti(*this, lev); pti.isValid(); ++pti) { ... }