Restarting from Maestro

Overview

We can now initialize a Castro simulation using data from a Maestro plotfile. This should not be thought of as a restart mode, but rather a new simulation with a special initialization. In order to use this feature, you must make sure the Maestro plotfile has the proper variables, add some new parameters to your inputs file, and add a few subroutines to Prob_Xd.f90. You need to build a special executable with “USE_MAESTRO_INIT=TRUE”, which will add “.MAESTRO” to the executable string. For multilevel problems, there are a few extra steps relating to the fact that you have to supply a grids file consistent with the Maestro grid structure.

MAESTRO Plotfile Requirements

The Maestro plotfile needs to have the following variables:

“x_vel”, “y_vel”, (and “z_vel”, depending on dimensionality of the problem)
“density” (castro.MAESTRO_init_type = 1 and 2 only)
Optional species (such as “X(C12)”) - there is an option to not read any species from the Maestro plotfile. In this case, you must make sure your code manually defines the species cell-by-cell in the initial Castro data
“tfromp”
“pi” (castro.MAESTRO_init_type = 2, 3, and 4 only)
“entropy” (castro.MAESTRO_init_type = 4 only)

Also, model_cc_XXXXX needs to list variables in the following order, which is the default order found in MAESTRO/Source/base_io.f90: r, base_r, rho0, p0, gamma1bar, rhoh0, div_coeff, psi, tempbar, etarho_cc, tempbar_init.

List of Parameters

Here are the additional parameters you must add to your inputs file.

Parameter	Definition	Type	Default
castro.MAESTRO_plotfile	name of the Maestro plotfile	std::string	must be set
castro.MAESTRO_modelfile	name of the Maestro model_cc file	std::string	must be set
castro.MAESTRO_npts_model	number of points in the Maestro model_cc file	int	must be set
castro.MAESTRO_first_species	name of the first species	std::string	must be set or else nothing will be read in
castro.MAESTRO_nspec	number of species in the Maestro plotfile	std::string	NumSpec in Castro
castro.MAESTRO_cutoff_density	controls how we overwrite data at the edge of the star	Real	must be set
castro.MAESTRO_init_type	determines how we initialize the Castro state	int	must be set
castro.MAESTRO_spherical	specifies planar or spherical problem	int	must be set

Examples of Usage

castro.MAESTRO_plotfile = “wd_384_6.25e8K_norotate_plt120578”
castro.MAESTRO_modelfile = “./wd_384_6.25e8K_norotate_plt120578/model_cc_120578”
castro.MAESTRO_npts_model = 1663

This is the number of points in castro.MAESTRO_modelfile. Note that this is not the same thing as “npts_model”, which is the number of points in the initial model file used for standard simulations where we do not initialize from a Maestro plotfile.
castro.MAESTRO_first_species = “X(C12)” If you do not specify this, no species will be read in. You can always manually specify or overwrite the species cell-by-cell later.
castro.MAESTRO_nspec = 3

If you do not specify this, it will default to the number of species in the Castro network, “NumSpec”. We have this here because sometimes Maestro and Castro will use different networks with different number of species.
castro.MAESTRO_cutoff_density = 1.e6

The code will use this density to figure out the radial coordinate, r_model_start, which is the last radial coordinate before rho0 falls below castro.MAESTRO_cutoff_density. It is possible to set castro.MAESTRO_cutoff_density to a tiny value, such that rho0 never falls below this value, in which case we set r_model_start to \(\infty\). In INITDATA_MAKEMODEL, we create a new 1D model integrating outward starting from r_model_start. Then, in INITDATA_OVERWRITE, we overwrite newly initialized Castro data in any cell that maps into a radial coordinate greater than r_model_start by interpolating from the new 1D model.
castro.MAESTRO_init_type = 2

Castro will read in data from the Maestro plotfile, and then call the EOS to make sure that \(\rho\), \(e\), \(T\), and \(X_k\) are consistent. The inputs to the EOS are based on the value of castro.MAESTRO_init_type:
1. \(e = e(\rho,T,X_k)\)
2. \(e,T = e,T(\rho,p_0+\pi,X_k)\)
3. \(\rho,e = \rho,e(p_0+\pi,T,X_k)\)
4. \(\rho,T,e = \rho,T,e(p_0+\pi,s,X_k)\)
castro.MAESTRO_spherical = 1

0 = planar; 1 = spherical.

New Subroutines in Prob_Xd.f90

There are three routines that need to be added to your local copy of Prob_Xd.f90. See Castro/Exec/wdconvect/Prob_3d.f90 for a standard spherical Maestro initialization.

INITDATA_MAESTRO

This fills in the Castro state by taking the Maestro data, calling the EOS, and making the proper variables conserved quantities. Specifically, we need a thermodynamically consistent \(\rho\), \(T\), \(e\), and \(X_k\), and then algebraically compute \(\rho{\bf u}\), \(\rho e\), \(\rho E\), and \(\rho X_k\),
INITDATA_MAKEMODEL

This creates a user-defined 1D initial model starting from r_model_start.
INITDATA_OVERWRITE

This overwrites the initialized Castro data using the new 1D initial model for all cells that map into radial coordinates greater than r_model_start.

Additional Notes

Note that for both single-level and multilevel Maestro to Castro initialization, the Castro base grid structure does not have to match the Maestro base grid structure, as long as the problem domain is the same. For example, if the coarsest level in a Maestro plotfile contains \(64^3\) cells divided into 8-\(32^3\) grids, it is ok to use a Castro base grid structure with 1-\(64^3\) grid, 64-\(16^3\) grids, or anything else you can imagine - the grids don’t even have to be the same size. As is normally the case, the Castro base grid structure is created based on the parameters in the Castro inputs file, such as amr.max_grid_size, amr.blocking_factor, etc.

Multilevel Restart

When initialing from a multilevel Maestro plotfile, there are some extra steps. First, you need to create a Castro-compatible grids file from the Maestro plotfile. This can be done with the BoxLib/Tools/Postprocessing/F_Src/fboxinfo.f90 utility. Compile and run this using the “–castro” option, e.g., “fboxinfo.Linux.gfortran.exe –castro pltxxxxx | tee gr0.maestro”, to generate the Castro-compatible grids file. Note that the base grid structure is still controlled by amr.max_grid_size, amr.blocking_factor, etc., since in C++ AMReX, the grids file only indicates the refined grid structure, whereas in Fortran BoxLib the grids file contains the base grid and refined grid structures.

Now, when you initialize the Castro simulation, you need to specify the grid file using amr.regrid_file = “gr0_3d.128_2levels”, for example. You can happily run this now, but note that the regridding algorithm will never be called (since Castro thinks it’s started a new simulation from scratch with a grids file, thus disabling the regridding). If you wish for the grid structure to be changed, you must do a traditional Castro restart from the Castro-generated checkpoint file (you can still use the same “.MAESTRO” executable or an executable built with USE_MAESTRO_INIT=FALSE), making sure that you do not specity amr.regrid_file (or else the grids will stay fixed). You are free to specify amr.regrid_on_restart, amr.compute_new_dt_on_regrid, and amr.plotfile_on_restart.

Sometimes a Maestro plotfile will only have 1 or 2 total levels, but you ultimately want to run a Castro simulation with many more levels of refinement. My recommended strategy is the following:

Initialize a Castro simulation from the Maestro plotfile while preserving the exact same grid structure and run for 10 time steps.
Do a traditional Castro restart from chk00010, but do not increase amr.max_level, and run for 10 more time steps. This allows a new grid structure with the same effective resolution as before settle in using the C BoxLib regridding algorithm.
Do a traditional Castro restart from chk00020, but increase amr.max_level by 1, and run for 10 time steps.
Repeat the procedure from the previous step (using the most updated checkpoint of course) as many times as desired.