Regridding
The details of the regridding strategy are described in § Tagging for Refinement; here we cover how the input parameters can control the gridding.
As described later, the user defines routines which tag individual cells at a given level if they need refinement. This list of tagged cells is sent to a grid generation routine, which uses the Berger-Rigoutsos algorithm [1] to create rectangular grids that contain the tagged cells.
The relevant runtime parameters are:
amr.regrid_file
: name of file from which to read the grids (text; default: no file)If set to a filename, e.g.
fixed_girds
, then list of grids at each fine level are read in from this file during the gridding procedure. These grids must not violate theamr.max_grid_size
criterion. The rest of the gridding procedure described below will not occur ifamr.regrid_file
is set.
amr.n_error_buf
: radius of additional tagging around already tagged cells (integer \(\geq 0\); default: 1)
amr.max_grid_size
: maximum size of a grid in any direction (integer \(> 0\); default: 128 (2-d), 32 (3-d))Note:
amr.max_grid_size
must be even, and a multiple ofamr.blocking_factor
at every level.
amr.blocking_factor
: grid size must be a multiple of this (integer \(> 0\); default: 2)amr.blocking_factor
at every level must be a power of 2 and the domain size must be a multiple ofamr.blocking_factor
at level 0.Note
This can be very important for elliptic problems with multigrid. A higher blocking factor allows the multigrid algorithm to coarsen more at the lowest level, reducing the amount of work required by the bottom solver.
amr.grid_eff
: grid efficiency (Real \(>0\) and \(<1\); default: 0.7)When creating a refined grid, do we make boxes that only include the coarse cells that were explicitly tagged for refinement? or do we allow ourselves to encompass nearby, untagged cells in order to make larger and more regular boxes? This is the grid efficiency.
When
blocking_factor = 1
, grid efficiency is exactly the fraction of refined cells in the fineBoxArray
which correspond to coarse cells which were tagged. For other blocking factors, we actually applygrid_eff
at the level which has been coarsened byblocking_factor
, so it is no longer strictly this fraction, but the idea is still the same.
amr.refine_grid_layout
: refine grids more if # of processors \(>\) # of grids (0 if false, 1 if true; default: 1)
Note also that amr.n_error_buf
, amr.max_grid_size
and
amr.blocking_factor
can be read in as a single value which is
assigned to every level, or as multiple values, one for each level.
As an example, consider:
amr.grid_eff = 0.9
amr.max_grid_size = 64
amr.blocking_factor} = 32
The grid efficiency, amr.grid_eff
, means that during the grid
creation process, at least 90% of the cells in each grid at the level
at which the grid creation occurs must be tagged cells. A higher
grid efficiency means fewer cells at higher levels, but may result
in the production of lots of small grids, which have inefficient cache
and OpenMP performance and higher communication costs.
The amr.max_grid_size
parameter means that the final grids will be
no longer than 64 cells on a side at every level. Alternately, we
could specify a value for each level of refinement as
amr.max_grid_size = 64 32 16
in which case our final grids will be
no longer than 64 cells on a side at level 0, 32 cells on a side at
level 1, and 16 cells on a side at level 2. The
amr.blocking_factor
means that all of the final grids will be
multiples of 32 at all levels. Again, this can be specified on a
level-by-level basis, like amr.blocking_factor = 32 16 8
, in which
case the dimensions of all the final grids will be multiples of 32 at
level 0, multiples of 16 at level 1, and multiples of 8 at level 2.
Getting good performance
These parameters can have a large impact on the performance of Castro, so taking the time to experiment with is worth the effort. Having grids that are large enough to coarsen multiple levels in a V-cycle is essential for good multigrid performance in simulations that use self-gravity.
How grids are created
The gridding algorithm proceeds in this order:
Grids are created using the Berger-Rigoutsos clustering algorithm modified to ensure that all new fine grids are divisible by
amr.blocking_factor
.Next, the grid list is chopped up if any grids are larger than
max_grid_size
. Note that becauseamr.max_grid_size
is a multiple ofamr.blocking_factor
theamr.blocking_factor
criterion is still satisfied.Next, if
amr.refine_grid_layout = 1
and there are more processors than grids, and ifamr.max_grid_size
/ 2 is a multiple ofamr.blocking_factor
, then the grids will be redefined, at each level independently, so that the maximum length of a grid at level \(\ell\), in any dimension, isamr.max_grid_size
[\(\ell\)] / 2.Finally, if
amr.refine_grid_layout = 1
, and there are still more processors than grids, and ifamr.max_grid_size
/ 4 is a multiple ofamr.blocking_factor
, then the grids will be redefined, at each level independently, so that the maximum length of a grid at level \(\ell\), in any dimension, isamr.max_grid_size
[\(\ell\)] / 4.