Radiative Heating and Cooling

Nyx provides the capability to compute local heating and cooling effects due to radiation. The motivation and algorithm for the heating and cooling components is documented in :raw-latex:`\cite{lukic15}`, and the relevant code is located in the Source/HeatCool subdirectory. The code is activated through the USE_HEATCOOL=TRUE option in the GNUmakefile. Mathematically, the heating and cooling can be described by a single ODE in each cell, to be integrated per time step \(\Delta t\). This ODE exhibits a sensitive relationship to quantities such as temperature and free electron density, and consequently it often requires sophisticated integration techniques to compute correctly.

Nyx provides a few different parallelization strategies techniques for solving this ODE.

For legacy reasons, the integration scheme is selected via the nyx.heat_cool_type input parameter. One method is to use USE_HEATCOOL=FALSE with no ODE solve (selected with nyx.heat_cool_type=0). The other method is to use USE_HEATCOOL=TRUE with a vectorized CVODE solve (selected with nyx.heat_cool_type=11).

Users should note that, when compiling with GNUmake , CVODE must be compiled as a separate library; instructions for compiling CVODE are provided in Getting Started. To link the external CVODE solver into Nyx, one must set USE_HEATCOOL=TRUE as well as USE_SUNDIALS=TRUE in the GNUmakefile.

This vectorized option uses CVODE while treating groups of ODEs in different cells as a single system of coupled ODEs. The purpose of this approach is to enable the evaluation of multiple RHSs simultaneously, for improved parallel performance. This approach can lead to a significant performance gain in the ODE integration (which is the among the most expensive computational kernels in Nyx). The group of cells integrated together is controlled by the tilesize used in the CVODE integration loop. If tiling is not used, the entire box of cells is integrated together. CVODE integrates this resulting system of ODEs with using adaptive time-step control which selects timesteps for the whole system. We set the absolute tolerance required for the ODE integration in CVODE to 1e-4, scaled by the initial value of each cells independent variable in the ODE and the relative tolerance required for the ODE integration in CVODE to 1e-4. These tolerances, in particular the relative tolerance, have different effects depending on whether one is integrating a single ODE at a time, or a system of ODEs simultaneously. One should be mindful of the numerical differences which arise from these, which can be observed with the fcompare tool in AMReX.

Input flags with affect the CVODE integration include:

  • nyx.use_sundials_constraint which when non-zero requires that the internal energy (while the problem is evolved) is positive
  • nyx.use_sundials_fused which when non-zero uses Sundials’s GPU fused operations (which are mathematically equivalent, but reduces GPU kernel launch time overhead)
  • nyx.sundials_alloc_type which has up to 5 different vector memory allocation strategies and only affects executables built for GPUs
  • nyx.use_typical_steps which when non-zero sets CVODE’s adaptive step-size selection (which substeps the total CVODE integration time) to be dt / old_max_steps. This maximum was over the entire problem domain. (In the strang case, the old maximum is taken from the same phase in the previous time-step)
  • nyx.sundials_use_tiling which controls whether the MFIter loop that iterates over the CVODE integration uses tiling
  • nyx.sundials_tile_size which controls the tile size of the MFIter loop that iterates over the CVODE integration
  • nyx.hydro_tile_size which controls the tile size of the MFIter loop that iterates over the hydro advance that makes hydro_src for SDC integration
  • nyx.sundials_reltol which controls the relative tolerance given for the CVODE integration
  • nyx.sundials_abstol which controls the absolute tolerance factor which multiplies the local internal energy to set the absolute tolerance vector for the CVODE integration

A HeatCoolTests directory allows for the testing of the CVODE integration alone, given data from a full step of the LyA executable. At present, it assumes the same tilesizes, boxarray, and distribution mapping between both runs. Note, if you supply your own filename paths you must ensure the directory structure exists. Some additional CVODE statistics will be displayed if nyx.v>1

Input flags to control these tests include:

  • hctest_example_write which when non-zero writes all relevant data for 1 step, including an inputs file (should be used with LyA or similar)
  • hctest_example_write_proc which when set will only write data for boxes owned by that processor number (which doesn’t currently have a read mode that will take this smaller data set)
  • hctest_example_index which represents the part of the filename that should typically denote step number, if unset defaults to nStep()
  • hctest_example_read which when non-zero reads all relevant data for 1 step (should be used with HeatCoolTests)
  • hctest_filename_inputs which sets the name of the inputs file to write, if unset, hctest/inputs.${hctest_example_index}
  • hctest_filename_badmap which sets the name of the BoxArray and DistributionMapping file to write, if unset, hctest/BADMAP.${hctest_example_index}
  • hctest_filename_chunk which sets the name of the Chunk data file prefix to use, if unset, hctest/Chunk.${hctest_example_index} (the mfi.index() is used to index the boxes for the full filepath)