Regridding

The details of the regridding strategy are described in § Tagging for Refinement; here we cover how the input parameters can control the gridding.

As described later, the user defines routines which tag individual cells at a given level if they need refinement. This list of tagged cells is sent to a grid generation routine, which uses the Berger-Rigoutsos algorithm [1] to create rectangular grids that contain the tagged cells.

The relevant runtime parameters are:

  • amr.regrid_file: name of file from which to read the grids (text; default: no file)

    If set to a filename, e.g. fixed_girds, then list of grids at each fine level are read in from this file during the gridding procedure. These grids must not violate the amr.max_grid_size criterion. The rest of the gridding procedure described below will not occur if amr.regrid_file is set.

  • amr.n_error_buf: radius of additional tagging around already tagged cells (integer \(\geq 0\); default: 1)

  • amr.max_grid_size: maximum size of a grid in any direction (integer \(> 0\); default: 128 (2-d), 32 (3-d))

    Note: amr.max_grid_size must be even, and a multiple of amr.blocking_factor at every level.

  • amr.blocking_factor: grid size must be a multiple of this (integer \(> 0\); default: 2) amr.blocking_factor at every level must be a power of 2 and the domain size must be a multiple of amr.blocking_factor at level 0.

    Note

    This can be very important for elliptic problems with multigrid. A higher blocking factor allows the multigrid algorithm to coarsen more at the lowest level, reducing the amount of work required by the bottom solver.

  • amr.grid_eff: grid efficiency (Real \(>0\) and \(<1\); default: 0.7)

    When creating a refined grid, do we make boxes that only include the coarse cells that were explicitly tagged for refinement? or do we allow ourselves to encompass nearby, untagged cells in order to make larger and more regular boxes? This is the grid efficiency.

    When blocking_factor = 1, grid efficiency is exactly the fraction of refined cells in the fine BoxArray which correspond to coarse cells which were tagged. For other blocking factors, we actually apply grid_eff at the level which has been coarsened by blocking_factor, so it is no longer strictly this fraction, but the idea is still the same.

  • amr.refine_grid_layout: refine grids more if # of processors \(>\) # of grids (0 if false, 1 if true; default: 1)

Note also that amr.n_error_buf, amr.max_grid_size and amr.blocking_factor can be read in as a single value which is assigned to every level, or as multiple values, one for each level.

As an example, consider:

amr.grid_eff = 0.9
amr.max_grid_size = 64
amr.blocking_factor} = 32

The grid efficiency, amr.grid_eff, means that during the grid creation process, at least 90% of the cells in each grid at the level at which the grid creation occurs must be tagged cells. A higher grid efficiency means fewer cells at higher levels, but may result in the production of lots of small grids, which have inefficient cache and OpenMP performance and higher communication costs.

The amr.max_grid_size parameter means that the final grids will be no longer than 64 cells on a side at every level. Alternately, we could specify a value for each level of refinement as amr.max_grid_size = 64 32 16 in which case our final grids will be no longer than 64 cells on a side at level 0, 32 cells on a side at level 1, and 16 cells on a side at level 2. The amr.blocking_factor means that all of the final grids will be multiples of 32 at all levels. Again, this can be specified on a level-by-level basis, like amr.blocking_factor = 32 16 8, in which case the dimensions of all the final grids will be multiples of 32 at level 0, multiples of 16 at level 1, and multiples of 8 at level 2.

Getting good performance

These parameters can have a large impact on the performance of Castro, so taking the time to experiment with is worth the effort. Having grids that are large enough to coarsen multiple levels in a V-cycle is essential for good multigrid performance in simulations that use self-gravity.

How grids are created

The gridding algorithm proceeds in this order:

  1. Grids are created using the Berger-Rigoutsos clustering algorithm modified to ensure that all new fine grids are divisible by amr.blocking_factor.

  2. Next, the grid list is chopped up if any grids are larger than max_grid_size. Note that because amr.max_grid_size is a multiple of amr.blocking_factor the amr.blocking_factor criterion is still satisfied.

  3. Next, if amr.refine_grid_layout = 1 and there are more processors than grids, and if amr.max_grid_size / 2 is a multiple of amr.blocking_factor, then the grids will be redefined, at each level independently, so that the maximum length of a grid at level \(\ell\), in any dimension, is amr.max_grid_size [\(\ell\)] / 2.

  4. Finally, if amr.refine_grid_layout = 1, and there are still more processors than grids, and if amr.max_grid_size / 4 is a multiple of amr.blocking_factor, then the grids will be redefined, at each level independently, so that the maximum length of a grid at level \(\ell\), in any dimension, is amr.max_grid_size [\(\ell\)] / 4.