.. _ch:make: ********************** MAESTROeX Build System ********************** Build Process Overview ====================== The MAESTROeX build system uses features of GNU make (version 3.82 or later), which is typically the default on systems. The MAESTROeX executable is built in the problem’s directory (one of the directories under ``SCIENCE/``, ``TEST_PROBLEMS/``, or ``UNIT_TESTS/``). This directory will contain a makefile, ``GNUmakefile``, that includes all the necessary information to build the executable. We use the build system in AMReX in ``amrex/Tools/GNUmake``. A set of MAESTROeX-specific macros are defined in ``MAESTROeX/Exec/Make.Maestro``. MAESTROeX gets the location of the AMReX library through the ``AMREX_HOME`` variable. This should be set as an environment variable in your shell start-up files (e.g. ``.bashrc`` or ``.cshrc``). The AMReX build system separates the compiler-specific information from the machine-specific information—this allows for reuse of the compiler information. The only machine-specific parts of the build system are for the MPI library locations, contained in ``amrex/Tools/GNUmake/sites/``. The compiler flags for the various compilers are listed in the files in ``amrex/Tools/GNUmake/comps/``. The compiler is set via the ``COMP`` variable in the problem’s GNUmakefile. There are several options in addition to the compiler that affect the build process: ``USE_MPI``, ``USE_OMP``, ``USE_CUDA`` for the parallelism, and ``DEBUG`` and ``TEST`` for diagnostic information.. Together, these choices along with the compiler name are reflected in the name of the executable. When the ``make`` command is run, the object and module files are written into sub-directories under ``tmp_build_dir/``. Separating each build into a separate sub-directory under the problem directory allows for multiple builds of MAESTROeX to exist side-by-side in the problem directory. Finding Source Files -------------------- The MAESTROeX executable is built from source distributed across a number of directories. In each of these directories containing source files, there is a ``Make.package`` file. This file has a number of lines of the form: :: CEXE_sources += file.cpp where ``file.cpp`` is a C++ source file that should be built when this directory is added to the list of build directories. The AMReX build system relies on the ``vpath`` functionality of make. In a makefile, the ``vpath`` variable holds search path used to locate source files. When a particular file is needed, the directories listed in ``vpath`` are searched until the file is found. The first instance of a file is used. We exploit this feature by always putting the build directory first in ``vpath``. This means that if a source file is placed in the build directory, that copy will override any other version in the source path. Dependencies ------------ There is no need to explicitly define the dependencies between the source files. Scripts in AMReX are run at the start of the build process and parse all the source files and make an explicit list of the dependency pairs. Files Created at Compile-time ----------------------------- Several files are created at build-time: - ``extern.F90``: This is the module that controls runtime parameters. This is created by the script ``write_probin.py`` in ``amrex/Tools/F_scripts/``. The makefile logic for creating it is in ``Make.Maestro``. At compile time, the problem, main MAESTROeX/, and any microphysics directories (set from the ``EOS_DIR``, ``CONDUCTIVITY_DIR``, and ``NETWORK_DIR`` parameters in the ``GNUmakefile`` are searched for ``_parameter`` files. These files are parsed and the ``extern.F90`` file is output containing the runtime parameters, their default values, and the logic for reading and interpreting the inputs file. - AMReX_buildInfo.cpp: This file contains basic information about the build environment (compiler used, build machine, build directory, compiler flags, etc.). This is created by the script ``makebuildinfo_C.py`` in ``amrex/Tools/C_scripts/`` from ``Make.Maestro`` by passing in a number of makefile variables. This is rewritten every time the code is compiled. The primary use of this module is writing out the ``job_info`` file in the plotfiles. - ``actual_network.F90``: This is generated at compile time *only* for the ``general_null`` network. The ``general_null`` network allows the user to define a list of non-reacting species builds the ``actual_network.F90`` based on this list. The makefile logic for building this is in the ``Make.package`` in ``Microphysics/networks/general_null``. The script ``write_network.py`` in that directory does the actual parsing of the species file and outputs the ``actual_network.f90``. MAESTROex Problem Options ========================= .. _sec:make:otherfiles: Problem-specific Files ---------------------- If a problem has a unique file that is needed as part of the build, then that file should be added to a ``Make.package`` file in the problem’s directory. Note that this is not necessary if you place a custom version of a source file in the problem’s directory. Since that file is already listed in the ``Make.package`` in its original location, the build system will know that it needs to be built. Since the ``vpath`` variable puts the problem’s directory at the start of the search path, the version of the file in the problem’s directory will be found first. Defining EOS, Network, and Conductivity Routines ------------------------------------------------ Each MAESTROeX problem needs to define an equation of state, a reaction network, and a routine to compute the conductivities (for thermal diffusion). This is true even if the problem is not doing reactions of thermal diffusion. These definitions are specified in the problem’s ``GNUmakefile``. - ``EOS_DIR``: This variable points to the directory (by default, relative to ``Microphysics/EOS/``) of the equation of state used for the build. Choices that work with MAESTROeX are: - ``helmholtz`` - ``gamma_law`` - ``multigamma`` - ``CONDUCTIVITY_DIR``: This variable points to the conductivity routine used for the build (by default, relative to ``Microphysics/conductivity/``). Choices that work with MAESTROeX are: - ``constant`` - ``stellar`` If diffusion is not being used for the problem, this should be set to constant. - ``NETWORK_DIR``: This variable points to the reaction network used for the build (by default, relative to ``Microphysics/networks/``). Several options are present in ``Microphysics/networks/``. A network is required even if you are not doing reactions, since the network defines the species that are advected and interpreted by the equation of state. A special choice, ``Microphysics/networks/general_null`` is a general network that simply defines the properties of one or more species. This requires an inputs file, specified by ``GENERAL_NET_INPUTS``. This inputs file is read at compile-time and used to build the ``actual_network.F90`` file that is compiled into the source. Core MAESTROeX modules ---------------------- Several modules are included in all MAESTROeX builds by default. In addition to the AMReX sources, we also include - ``MAESTROeX/Source`` - ``Util/model_parser`` The microphysics used may bring in its own dependencies. For each of these included directories, ``Make.Maestro`` adds the list of source files defined in their ``Make.package`` to the list of files to be compiled. It also adds each of these directories to the ``vpath`` as a directory for the build process to search in for source files. Special Targets =============== ``print-*`` ----------- To see the contents of any variable in the build system, you can build the special target ``print-varname``, where ``varname`` is the name of the variable. For example, to see what the network directory is, you would do:: make print-NETWORK_DIR This functionality is useful for debugging the makefiles. ``file_locations`` ------------------ Source files are found by searching through the make ``vpath``. The first instance of the file found in the ``vpath`` is used in the build. To see which files are used and their locations, do: :: make file_locations This will also show any files that aren’t found. Some are expected (e.g., ``extern.F90`` is created at compile time), but other files that are not found could indicate an incomplete ``vpath``. ``clean`` and ``realclean`` --------------------------- Typing ``make clean`` deletes the object and module files for the current build (i.e., the current choice of ``USE_MPI``, ``DEBUG``, ``COMP``, and ``USE_OMP``). This also removes any of the compile-time generated source files. Any other builds are left unchanged. Typing ``make realclean`` deletes the object and module files for all builds—i.e., the entire ``tmp_build_dir/`` is removed. .. _ch:makefiles:special: Special Debugging Modes ======================= AMReX has several options that produce executables that can help track down memory issues, uninitialized variables, NaNs, etc. - ``DEBUG = TRUE`` : Setting ``DEBUG=TRUE`` on the ``make`` commandline or in the ``GNUmakefile`` generates an executable with debugging information included in the executable (e.g., to be interpreted by the debugger, gdb). This will usually add -g to the compile line and also lower the optimization. This will add several options to catch uninitialize variables, bounds errors, etc. The resulting executable will have ``DEBUG`` in its name. - ``TEST = TRUE`` Setting ``TEST=TRUE`` on the ``make`` commandline or in the ``GNUmakefile`` will enable routines in AMReX to initialize MultiFabs and arrays with signalliing NaNs. This behavior is the same as ``DEBUG=TRUE``, but ``TEST`` uses the same compiler optimizations as a normal build. This can be useful with compiler flags that trap floating point exceptions (FPEs), but checks on floating point exceptions can also be enabled through runtime parameters passed to AMReX’s backtrace functionlity: - ``amrex.fpe_trap_invalid``: enabling FPE trapping for invalid operations (e.g. ``0 * inf``, ``sqrt(-1)``) - ``amrex.fpe_trap_zero``: enable FPE trapping for divide-by-zero - ``amrex.fpe_trap_overflow``: enable FPE trapping for overflow - backtracing When exception trapping is enabled (either via AMReX or the compiler), the code will abort, and the backtrace information will be output to a file ``Backtrace.N``, where N is the processor number. AMReX will also initialize multifabs with signaliing NaNs to help uncover any floating point issues. This is also useful to diagnose deadlocks in parallel regions. If the code is hanging, doing “control-C” will be intercepted and the code will generate a backtrace which will identify where in the code there was a deadlock. Behind the scenes, AMReX implements this capability via the Linux/Unix ``feenableexcept`` function (this is in ``backtrace_c.cpp`` in AMReX). - ``FSANITIZER`` For gcc/g++, setting ``FSANITIZER=TRUE`` in ``GNUmakefile`` will enable the address sanitizer support built into GCC. This is enabled through integration with https://github.com/google/sanitizers in GCC. Note: you will need to have the libraries libasan and libubsan installed on your machine to use this functionality. Extending the Build System ========================== Adding a Compiler ----------------- Properties for different compilers are already defined in ``${AMREX_HOME}/Tools/GNUmake``. Each compiler is given its own file in the ``comps/`` sub-directory. These compiler files define the compiler flags for both optimized and debug compiling. Parallel (MPI) Builds --------------------- When building with MPI, the build system needs to know about the location of the MPI libraries. If your local MPI has the ``mpif90`` and ``mpicxx`` wrappers installed and working, then MAESTROeX will attempt to use these. Otherwise, you will need to add a site to the ``sites/`` sub-directory in the AMReX build system specifying the details of your environment.