Build System Overview
Make Parameters
These build parameters control the parallelism, included physics,
etc. In general they can be set to TRUE
or FALSE
. The
Castro-specific ones are interpreted by Make.Castro
.
A lot of optional features are enabled at compile time. This allows Castro to reduce the memory footprint of the state arrays by not allocating space for variables that are not used.
General Build Parameters
These Parameters affect the build (parallelism, performance, etc.) Most of these are parameters from AMReX.
USE_ALL_CASTRO
: compile all of the core Castro directories. This is the default (TRUE
), and should not be changed for general simulations. The purpose of this flag is for unit tests, which do not need all of the Castro directories compiled.
USE_AMR_CORE
: compile all of the core AMReX directories, includingBase/
,AmrCore/
,Amr/
, andBoundary/
. This defaults toTRUE
and should be left set for Castro simulations. The purpose of this flag is for unit tests that don’t need all of AMReX.
USE_MLMG
: use the AMReX multi-level multigrid solver for gravity and diffusion. This should always be set toTRUE
.
USE_HYPRE
: compile in the Hypre library. This will be automatically enabled for radiation. You need to specify the path to the Hypre library via eitherHYPRE_DIR
orHYPRE_OMP_DIR
.
Parallelization and GPUs
The following parameters control how work is divided across nodes, cores, and GPUs.
USE_MPI
: compile with the MPI library to allow for distributed parallelism.
USE_OMP
: compile with OpenMP to allow for shared memory parallelism.
USE_CUDA
: compile with NVIDIA GPU support using CUDA.
USE_HIP
: compile with AMD GPU support using HIP.
General Physics Parameters
The following parameters control how the coupling between hydro and reactions is handled.
USE_SIMPLIFIED_SDC
: use the simplified spectral deferred corrections (SDC) solver for coupling hydro and reactions. At the moment, this works with the CTU hydrodynamics solver. This requires running withcastro.time_integration_method = 3
.
USE_TRUE_SDC
: use the true SDC method to couple hydro and reactions. This can do 2nd order or 4th order accuracy. At the moment, this works on single level only. This requires running withcastro.time_integration_method = 2
.
Radiation Parameters
USE_RAD
: use photon radiation diffusion. Note, For multigroup radiation, you need to set the number of radiation groups. This is controlled by theNGROUPS
parameter.
Gravity Parameters
USE_GRAV
: use gravity (this could be constant or self-gravity)
USE_GR
: use a post-Newtonian approximation for GR gravity for the monopole solver.
USE_POINTMASS
: include a pointmass source to the gravitational potential.
Microphysics Parameters
USE_DIFFUSION
: enable thermal diffusion. The conductivity is set viaCONDUCTIVITY_DIR
, which should be a directory in the Microphysics repo.
USE_REACT
: enable reactions. When reactions are set, we need to specify a network and an integrator. Typically these come from the Microphysics repo, but one common exception is thegeneral_null
network, which just defines a composition. The parameters that come into play here are:
NETWORK_DIR
: the network to use. This is expected to be a subdirectory in the Microphysics repo.
NETWORK_INPUTS
: this is the text file that we read to define the composition if we are using thegeneral_null
network (e.g.,gammalaw.net
). The build system will look for this file in the Microphysics repo.
INTEGRATOR_DIR
: this is the ODE integrator to use to integrate the reaction system. This is expected to be a subdirectory in the Microphysics repo.
USE_REACT_SPARSE_JACOBIAN
USE_SPARSE_STOP_ON_OOB
EOS_DIR
: the equation of state to use. This will be a subdirectory under the Microphysics repo.
Hydrodynamics and Source Term Parameters
USE_ROTATION
: include rotation sources
USE_HYBRID_MOMENTUM
: have Castro evolve angular momentum in addition to linear momentum.
USE_SHOCK_VAR
: include a variable in the State_Type StateData that marks the location of a shock.
Simulation Flow Parameters
USE_POST_SIM
: if this is defined, then Castro will call the user-defined routineproblem_post_simulation()
after the full evolution of the problem has ended.
USE_MAESTRO_INIT
: this enables the code to allow Castro to restart from a Maestro simulation. This will need to be updated in the future to allow for restarts from MAESTROeX.
USE_HDF5
: compile in support for HDF5. This is needed for some tables used by Microphysics routines.
Tracer Particle Parameters
USE_PARTICLES
: compile in support for tracer particles.
Build Process Procedure
Note
At build time, there are a number of source files that are autogenerated based
on the configuration of the problem. Most of these files are output into
tmp_build_dir/castro_sources/Nd.COMP.OPTIONS.EXE/
, where N
is the
dimensionality, COMP
is the compiler name, and OPTIONS
can be any
number of options (MPI
, DEBUG
, …).
This is the current build system process.
set_variables.py
is calledThis processes the Castro
_variables
file and writesstate_indices.H
into thetmp_build_dir/castro_sources/
directory.These are used to define the size of the various state arrays and the integer keys to index each state variable.
The hook for this is in
Make.auto_source
in the build rule forstate_indices.H
You can test this portion of the build system by doing
make test_variables
(for
general_null networks
),network_properties.H
is createdThis is done by
write_network.py
The hook for this is in
$(CASTRO_HOME)/Microphysics/networks/general_null/Make.package
Runtime parameter files for the microphysics routines are parsed by
write_probin.py
This writes the routines that manage the Microphysics runtime parameters:
extern_parameters.cpp
andextern_parameters.H
. This is output intmp_build_dir/castro_sources/
.The hook for this is in
Make.auto_source
in the rule forextern_parameters.H
Castro’s runtime parameters are parsed by
parse_castro_params.py
This writes the C++ header files that manage and read the runtime parameters. These file are output in
tmp_build_dir/castro_sources/
.The hook for this is in
Make.auto_source
in the rule forcastro_params.H
Problem-specific runtime parameters are parsed by
write_probdata.py
If the problem directory defines a
_prob_params
then it is parsed and used to C++ header and source filesprob_parameters.H
andprob_parameters.cpp
. These handle reading theproblem.*
parameters from the inputs file. Even without a problem-specific_prob_params
, all of the variables inCastro/Source/problems/_default_prob_params
will be included.The script
Castro/Util/scripts/write_probdata.py
is usedThe hook for this is in
Make.auto_source
in theprob_parameters.H
rule.These headers are output into
tmp_build_dir/castro_sources/
.
The C/C++ dependencies file is created
This creates the individual
.d
files intmp_build_dir
, one for each source fileA set of rules in
Make.rules
handles this. There is some description of what each line does in the comments of the make file
Output to stdout the git version of the sources, via
describe_sources.py
. This doesn’t affect the build process
For all of this to work, we need the tmp_build_dir/s
directory to
be first in the vpath, so our modified sources are found and used.