Outputting

Restart Capability

Castro has a standard sort of checkpointing and restarting capability. In the inputs file, the following options control the generation of checkpoint files (which are really directories):

  • amr.check_file: prefix for restart files (text; default: chk)

  • amr.check_int: how often (by level 0 time steps) to write restart files (integer \(> 0\); default: -1)

  • amr.check_per: how often (by simulation time) to write restart files (Real \(> 0\); default: -1.0)

    Note that amr.check_per will write a checkpoint at the first timestep whose ending time is past an integer multiple of this interval. In particular, the timestep is not modified to match this interval, so you won’t get a checkpoint at exactly the time you requested.

  • amr.restart: name of the file (directory) from which to restart (Text; not used if not set)

  • amr.checkpoint_files_output: should we write checkpoint files? (0 or 1; default: 1)

    If you are doing a scaling study then set amr.checkpoint_files_output = 0 so you can test scaling of the algorithm without I/O.

  • amr.check_nfiles: how parallel is the writing of the checkpoint files? (Integer \(\geq 1\); default: 64)

    See the section Parallel I/O for more details on parallel I/O and the amr.check_nfiles parameter.

  • amr.checkpoint_on_restart: should we write a checkpoint immediately after restarting? (0 or 1; default: 0)

  • castro.output_at_completion: should we write a final checkpoint/plotfile? (0 or 1)

  • castro.grown_factor: factor by which domain has been grown (Integer \(\geq 1\); default: 1)

Note

You can specify both amr.check_int or amr.check_per, if you so desire; the code will print a warning in case you did this unintentionally. It will work as you would expect – you will get checkpoints at integer multiples of amr.check_int timesteps and at integer multiples of amr.check_per simulation time intervals.

amr.plotfile_on_restart and amr.checkpoint_on_restart require amr.regrid_on_restart to be in effect.

As an example:

amr.check_file = chk_run
amr.check_int = 10

means that restart files (really directories) starting with the prefix “chk_run” will be generated every 10 level-0 time steps. The directory names will be chk_run00000, chk_run00010, chk_run00020, etc.

If instead you specify:

amr.check_file = chk_run
amr.check_per = 0.5

then restart files (really directories) starting with the prefix “chk_run” will be generated every 0.1 units of simulation time. The directory names will be chk_run00000, chk_run00043, chk_run00061, etc, where \(t = 0.1\) after 43 level-0 steps, \(t = 0.2\) after 61 level-0 steps, etc.

To restart from chk_run00061, for example, then set:

amr.restart = chk_run00061

Plotfile Outputting

Castro has two levels of plotfiles, regular plotfiles and small plotfiles. The idea behind this distinction is that we can output a small number of variables very frequently in the small plotfiles and output a large number (or all variables) less frequently. This helps keep the data sizes down while allowing for fine-grained temporal analysis of important quantities.

A few general controls determines whether we want to output plotfiles and when:

  • amr.plot_files_output : this is set to 1 to output plotfiles

  • amr.plotfile_on_restart : set this to 1 to dump out a plotfile immediately when we restart.

  • amr.write_plotfile_with_checkpoint : always output a plotfile when we dump a checkpoint file.

The frequency of outputting and naming of regular plotfiles is controlled by:

  • amr.plot_file : this is the base name for the plotfile, e.g. plt.

  • amr.plot_per : this is the amount of simulation time between plotfile output

    Note

    amr.plot_per will write a plotfile at the first timestep whose ending time is past an integer multiple of this interval. In particular, the timestep is not modified to match this interval, so you won’t get a checkpoint at exactly the time you requested.

  • amr.plot_int this is the number of timesteps between plotfiles. Set this to -1 to rely on the simulation-time-based outputting.

Similarly, the frequency of outputting and naming of small plotfiles is controlled by:

  • amr.small_plot_file : this is the base name for the small plotfile, e.g. smallplt.

  • amr.small_plot_per : this is the amount of simulation time between small plotfile output

  • amr.small_plot_int this is the number of timesteps between small plotfiles. Set this to -1 to rely on the simulation-time-based outputting.

Additional output options control how the I/O is done:

  • amr.plot_nfiles: how parallel is the writing of the plotfiles? (Integer \(\geq 1\); default: 64)

    See the Software Section for more details on parallel I/O and the amr.plot_nfiles parameter.

All the options for amr.derive_plot_vars are kept in derive_lst in Castro_setup.cpp. Feel free to look at it and see what’s there.

Note

You can specify both amr.plot_int or amr.plot_per, if you so desire; the code will print a warning in case you did this unintentionally. It will work as you would expect – you will get plotfiles at integer multiples of amr.plot_int timesteps and at integer multiples of amr.plot_per simulation time intervals.

As an example:

amr.plot_file = plt_run
amr.plot_int = 10

means that plot files (really directories) starting with the prefix “plt_run” will be generated every 10 level-0 time steps. The directory names will be plt_run00000, plt_run00010, plt_run00020, etc.

If instead you specify:

amr.plot_file = plt_run
amr.plot_per = 0.5

then restart files (really directories) starting with the prefix “plt_run” will be generated every 0.1 units of simulation time. The directory names will be plt_run00000, plt_run00043, plt_run00061, etc, where \(t = 0.1\) after 43 level-0 steps, \(t = 0.2\) after 61 level-0 steps, etc.

Controlling What’s in the PlotFile

There are a few options that can be set at runtime to control what variables appear in the regular plotfile.

  • amr.plot_vars: this controls which of the main state variables appear in the plotfile. The default is for all of them to be stored. But you can specify a subset by name, e.g.:

    amr.plot_vars = density
    

    to only store that subset.

  • amr.derive_plot_vars: this controls which of the derived variables to be stored in the plotfile. Derived variables are created only when the plotfile is being created, using the infrastructure provided by AMReX to register variables and the associated C++ routine to do the deriving.

    By default, no derived variables are stored. You can store all derived variables that Castro knows about by doing:

    amr.derive_plot_vars = ALL
    

or a subset by explicitly listing them, e.g.:

amr.derive_plot_vars = entropy pressure

To not output any derived variable,s this is set to NONE.

For small plotfiles, the controls that lists the variables is:

  • amr.small_plot_vars : this is a list of which variables to include in the small plotfile.

  • amr.derive_small_plot_vars : this is a list of which derived variables to include in the small plotfile.

Plotfile Variables

Native variables

These variables come directly from the StateData, either the State_Type (for the hydrodynamic variables), Reactions_Type (for the nuclear energy generation quantities). PhiGrav_Type and Gravity_Type (for the gravity quantities), and Rad_Type (for radiation quantities).

variable name

description

units

density

Mass density, \(\rho\)

\(\gcc\)

xmom

x-momentum, \((\rho u)\)

\({\rm g~cm^{-2}~s^{-1}}\)

ymom

y-momentum, \((\rho v)\)

\({\rm g~cm^{-2}~s^{-1}}\)

zmom

z-momentum, \((\rho w)\)

\({\rm g~cm^{-2}~s^{-1}}\)

rho_E

Total energy density

\({\rm erg~cm^{-3}}\)

rho_e

Internal energy density

\({\rm erg~cm^{-3}}\)

Temp

Temperature

\({\rm K}\)

rho_X (where X is any of the species defined in the network)

Mass density of species X

\(\gcc\)

omegadot_X (where X is any of the species defined in the network)

Creation rate of species X \(\omegadot_k = DX_k/Dt\)

\({\rm s^{-1}}\)

rho_enuc

Nuclear energy generation rate density

\({\rm erg~cm^{-3}~s^{-1}}\)

phiGrav

Gravitational potential

\({\rm erg~g^{-1}}\)

grav_x, grav_y, grav_z

Gravitational acceleration

\({\rm cm~s^{-2}}\)

rmom

Radial momentum (defined for HYBRID_MOMENTUM)

\({\rm g~cm^{-2}~s^{-1}}\)

lmom

Angular momentum (\(\theta\); defined for HYBRID_MOMENTUM)

\({\rm g~cm^{-2}~s^{-1}}\)

pmom

z-momentum (defined for HYBRID_MOMENTUM)

\({\rm g~cm^{-2}~s^{-1}}\)

Shock

Shock flag (= 1 if a zone has a shock; defined for SHOCK)

rad, rad0, rad1, …

Radiation energy density (for multigroup radiation, each group has its own variable)

Derived variables

variable name

description

derive routine

units

abar

Mean atomic mass

derabar

\(\amu\)

angular_momentum_x, angular_momentum_y, angular_momentum_z

Angular momentum / volume in the x, y, or z dir computed as \([(\rho \ub) \times {\bf r}]_n\) where \({\bf r}\) is the distance from center and \(n\) is either x, y, or z

derangmomx, derangmomy, derangmomz

\({\rm g~cm^{-1}~s^{-1}}\)

diff_coeff

Thermal diffusion coefficient, \(\kth/(\rho c_v)\)

derdiffcoeff

\({\rm cm^2~s^{-1}}\)

diff_term

\(\nabla\cdot(\kth\nabla T)\)

derdiffterm

\({\rm erg~cm^{-3}~s^{-1}}\)

divu

\(\nabla \cdot \ub\)

derdivu

\({\rm s^{-1}}\)

eint_e

Specific internal energy computed from the conserved \((\rho e)\) state variable as \(e = (\rho e)/\rho\)

dereint2

\({\rm erg~g^{-1}}\)

eint_E

Specific internal energy computed from the total energy and momentum conserved state as \(e=[(\rho E)-\frac{1}{2}(\rho \ub^2)]/\rho\)

dereint1

\({\rm erg~g^{-1}}\)

entropy

Specific entropy, \(s\), computed as \(s = s(\rho, e, X_k)\), where e is computed from \((\rho e)\)

derentropy

\({\rm erg~g^{-1}~K^{-1}}\)

enuc

Nuclear energy generation rate / gram

derenuc

\({\rm erg~g^{-1}~s^{-1}}\)

Ertot

Total radiation energy density (for multigroup radiation problems)

derertot

Frcomx, Frcomy, Frcomz

Comoving radiation flux

Radiation.cpp

Frlabx, Frlaby, Frlabz

Lab-frame radiation flux

Radiation.cpp

Gamma_1

Adiabatic index, \(d\log p/d\log \rho|_s\)

dergamma1

kineng

Kinetic energy density, \(K = \frac{1}{2} |(\rho \ub)|^2\)

derkineng

\({\rm erg~cm^{-3}}\)

lambda

Radiation flux limiter

logden

\(\log_{10} \rho\)

derlogten

dimensionless, assuming \(\rho\) is in CGS

MachNumber

Fluid Mach number, \(|\ub|/c_s\)

dermachnumber

maggrav

Gravitational acceleration magnitude

dermaggrav

\({\rm cm~s^{-2}}\)

magmom

Momentum density magnitude, \(|\rho \ub|\)

dermagmom

\({\rm g~cm^{-2}~s^{-1}}\)

magvel

Velocity magnitude, \(|\ub|\)

dermagvel

\(\cms\)

magvort

Vorticity magnitude, \(|\nabla\times\ub|\)

dermagvort

\({\rm s^{-1}}\)

pressure

Total pressure, including ions, electrons, and radiation (for non radhydro problems)

derpres

\({\rm dyn~cm^{-2}}\)

radvel

Radial velocity (measured with respect to center or vertical axis if domain_is_plane_parallel is set) \((xu + yv + zw)/r\)

derradialvel

\(\cms\)

circvel

Circumferential velocity (perpendicular to radvel. If domain_is_plane_parallel is set, then this is in the x-y plane

derradialvel

\(\cms\)

soundspeed

Sound speed

dersoundspeed

\(\cms\)

StateErr

thermal_cond

Thermal conductivity, \(\kth\)

dercond

\({\rm erg~cm^{-1}~s^{-1}~K^{-1}}\)

t_sound_t_enuc

derenuctimescale

uminusc

(only for 1D) x-velocity \(-\) sound speed

deruminusc

\(\cms\)

uplusc

(only for 1D) x-velocity + sound speed

deruplusc

\(\cms\)

X(q)

Mass fraction of species q \(X_k = (\rho X_k)/\rho\)

derspec

x_velocity, y_velocity, z_velocity

Fluid velocity, \(\ub = (\rho \ub)/\rho\)

dervel

\(\cms\)

problem-specific plotfile variables

See the section on Problem_Derives.H for more details about defining your own plotfile variables.

variable name

description

units

analytic

pi

pioverp0

primarymask

secondarymask

Terror

Texact

inertial_angular_momentum_x, inertial_angular_momentum_y, inertial_angular_momentum_z

inertial_momentum_x, inertial_momentum_y, inertial_momentum_z

inertial_radial_momentum_x, inertial_radial_momentum_y, inertial_radial_momentum_z

phiEff

phiEffPM_P

phiEffPM_S

tpert

Screen Output

There are several options that set how much output is written to the screen as Castro runs:

  • amr.v: verbosity of Amr.cpp (0 or 1; default: 0)

  • castro.v: verbosity of Castro.cpp (0 or 1; default: 0)

  • gravity.v: verbosity of Gravity.cpp (0 or 1; default: 0)

  • diffusion.v: verbosity of Diffusion.cpp (0 or 1; default: 0)

  • mg.v: verbosity of multigrid solver (for gravity) (allow values: 0, 1, 2, 3, 4; default: 0)

  • amr.grid_log: name of the file to which the grids are written (text; not used if not set)

  • amr.run_log: name of the file to which certain output is written (text; not used if not set)

  • amr.run_log_terse: name of the file to which certain (terser) output is written (text; not used if not set)

  • castro.do_special_tagging: allows the user to set a special flag based on user-specified criteria (0 or 1; default: 1)

    castro.do_special_tagging = 1 can be used, for example, to calculate the bounce time in a core collapse simulation; the bounce time is defined as the first time at which the maximum density in the domain exceeds a user-specified value. This time can then be printed into a special file as a useful diagnostic.

As an example:

amr.grid_log = grdlog
amr.run_log = runlog

Every time the code regrids it prints a list of grids at all relevant levels. Here the code will write these grids lists into the file grdlog. Additionally, every time step the code prints certain statements to the screen (if amr.v = 1), such as:

STEP = 1 TIME = 1.91717746 DT = 1.91717746
PLOTFILE: file = plt00001

The run_log option will output these statements into runlog as well.

Terser output can be obtained via:

amr.run_log_terse = runlogterse

This file, runlogterse differs from runlog, in that it only contains lines of the form:

10  0.2  0.005

in which “10” is the number of steps taken, “0.2” is the simulation time, and “0.005” is the level-0 time step. This file can be plotted very easily to monitor the time step.

Integral Diagnostics

Castro can calculate integrals of quantities on the grid and other global quantities and output them to both the screen and to a runtime file at regular intervals. By default, this capability is off. To enable it, one of the following runtime parameters can be set:

  • castro.sum_interval: if \(> 0\), how often (in level-0 time steps) to compute and print integral quantities (Integer; default: -1)

    The integral quantities include total mass, momentum and energy in the domain every castro.sum_interval level-0 steps. The print statements have the form:

    TIME= 1.91717746 MASS= 1.792410279e+34
    

    for example.

  • castro.sum_per: how often in simulation time to output integral quantities (this is used as an alternate to castro.sum_interval).

By default, 4 output files are created:

  • amr_diag.out : This includes timestep information, in the following columns:

    • timestep

    • time

    • dt

    • finest level

    • coarse timestep walltime

  • gravity_diag.out : For problems with Poisson gravity, this includes the gravitational wave amplitudes

  • grid_diag.out : This includes integrals of the state data:

    • time

    • mass

    • x-, y-, and z-momentum

    • x-, y-, and z-angular momentum

    • kinetic energy

    • internal energy

    • kinetic + internal energy

    • gravitational potential energy

    • total energy (including gravitational potential energy)

  • species_diag.out : This contains the mass of each of the nuclear species on the grid.

    Note

    The species masses are given in units of solar masses.

Castro/Util/scripts/diag_parser.py contains Python code for parsing these output files into Numpy arrays. Usage instructions are included in the file, along with an example script at Castro/Util/scripts/plot_species.py. This reads a species_diag.out file provided on the command line and makes a plot of the total mass fractions over time.

Some problems have custom versions of the diagnostics with additional information. These are not currently supported by the Python parser.

Parallel I/O

Both checkpoint files and plotfiles are really directories containing subdirectories: one subdirectory for each level of the AMR hierarchy. The fundamental data structure we read/write to disk is a MultiFab, which is made up of multiple FAB’s, one FAB per grid. Multiple MultiFab s may be written to each directory in a checkpoint file. MultiFab s of course are shared across CPUs; a single MultiFab may be shared across thousands of CPUs. Each CPU writes the part of the MultiFab that it owns to disk, but they don’t each write to their own distinct file. Instead each MultiFab is written to a runtime configurable number of files \(N\) (\(N\) can be set in the inputs file as the parameter amr.checkpoint_nfiles and amr.plot_nfiles; the default is 64). That is to say, each MultiFab is written to disk across at most \(N\) files, plus a small amount of data that gets written to a header file describing how the file is laid out in those \(N\) files.

What happens is \(N\) CPUs each opens a unique one of the \(N\) files into which the MultiFab is being written, seeks to the end, and writes their data. The other CPUs are waiting at a barrier for those \(N\) writing CPUs to finish. This repeats for another \(N\) CPUs until all the data in the MultiFab is written to disk. All CPUs then pass some data to CPU 0 which writes a header file describing how the MultiFab is laid out on disk.

We also read MultiFabs from disk in a “chunky” manner, opening only \(N\) files for reading at a time. The number \(N\), when the MultiFab s were written, does not have to match the number \(N\) when the MultiFab s are being read from disk. Nor does the number of CPUs running while reading in the MultiFab need to match the number of CPUs running when the MultiFab was written to disk.

Think of the number \(N\) as the number of independent I/O pathways in your underlying parallel filesystem. Of course a “real” parallel filesystem should be able to handle any reasonable value of \(N\). The value -1 forces \(N\) to the number of CPUs on which you’re running, which means that each CPU writes to a unique file, which can create a very large number of files, which can lead to inode issues.