Skip to content

Running Simulations

After being built, GUERNICA can be minimally run with:

./guernica [-o output_dir] [-i file_name.cfg | path/to.cfg] [--overwrite]

GUERNICA should be run from the repository root, since mesh and profile file paths in the input file are relative to the working directory, and outputs go to runs/.

Command-line arguments

GUERNICA supports the following command-line arguments.

-i <file_name>

Input file.

  • A bare file name is looked up in configs/, so -i test.cfg reads configs/test.cfg.
  • Anything containing a /, or a file that exists as given, is used as is. This lets scan scripts keep generated configs next to themselves.
  • The default input file is configs/input.cfg.

Examples:

./guernica -i test.cfg
./guernica -i scans/run3/input.cfg

-o <output_dir> or --output <output_dir>

Output directory name.

  • All outputs are written to runs/<name>.
  • If the directory already exists, the run is refused unless --overwrite is included.
  • The default output directory is run, so outputs are written to runs/run.

Examples:

./guernica -o case1
./guernica --output case1

--overwrite

Overwrite an existing output directory.

  • This flag does not take any additional argument.
  • If -o is not provided, this overwrites runs/run.

Example:

./guernica -o case1 --overwrite

Resume behavior

If resume = true is set under [time] in the input file, resume settings take precedence over the command line.

In that case:

  • -o / --output are ignored.
  • --overwrite is effectively forced to false.
  • The output directory is taken from resume_directory under [time].
  • If resume_directory is not specified, the default is runs/run.

Example:

[time]
resume = true
resume_directory = runs/my_previous_run

With the above input settings, a command such as

./guernica -o something_else --overwrite

will still resume from runs/my_previous_run.

When resuming:

  • The run directory must contain a complete set of distribution slabs from its last output, so the original run must have used save_dist = true or save_final_dist = true (see Post Processing).
  • The latest step and time are read from those slabs and from the Step ..., time = ... lines in run.log, and the two must agree. Resuming therefore applies to transient runs.
  • t_final is the additional time to run past the resumed time.
  • [n:init] is ignored, and new output is appended to the existing run.log, moment.gf, and balance.dat.

Common usage examples

Run with defaults:

./guernica

Run with a specific input file:

./guernica -i example.cfg

Run with a specific output directory:

./guernica -o example_run

Run with both a specific input file and output directory:

./guernica -i example.cfg -o example_run

Overwrite an existing output directory:

./guernica -i example.cfg -o example_run --overwrite

Startup checks

Before the first step, GUERNICA validates the whole input file and stops with a message naming the offending section and key if anything is inconsistent (unknown keys are ignored, so check spelling). In transient mode it then prints a CFL estimate to run.log:

CFL estimate: dt_max = 0.0021 (dt = 0.001, ratio = 0.48)

A ratio above 1 adds a note, and above 2 a warning, since such runs have been observed to diverge. In steady mode, the range of local pseudo-time steps is printed instead. A steady run that diverges stops with an error rather than writing non-finite output. See Methods Overview.

Selecting a CUDA device

When running multiple simulations on different CUDA devices, it is often useful to specify which GPU a given simulation should use.

This can be done by prepending the command with CUDA_VISIBLE_DEVICES=<num>:

CUDA_VISIBLE_DEVICES=0 ./guernica -o run0
CUDA_VISIBLE_DEVICES=1 ./guernica -o run1

Running in the background

To run a simulation in the background, it is useful to use nohup:

nohup ./guernica [-o output_dir] [-i file_name.cfg] [--overwrite] &

For example:

nohup ./guernica -i example.cfg -o example_run --overwrite &

Since GUERNICA writes outputs to a log file in the output directory, it is not strictly necessary to additionally redirect output when using nohup. For that reason, a command such as

nohup ./guernica > output.log 2>&1 &

is usually unnecessary.

Profiling with Nsight Systems

To profile a simulation and evaluate GPU efficiency, use nsys:

nsys profile -o prof --trace=cuda,nvtx,osrt --sample=none --force-overwrite=true ./guernica [-o output_dir] [-i file_name.cfg] [--overwrite]

The nvtx trace shows each operator as a named range when the binary was built with PROFILE=1 and NVTX support (see Installation). make nsys ARGS="..." builds that binary and runs this command in one step.

Notes

  • This writes profiling output to files such as prof.nsys-rep.
  • --force-overwrite=true allows existing profiling outputs with the same base name to be overwritten.

Example:

nsys profile -o prof --trace=cuda,nvtx,osrt --sample=none --force-overwrite=true ./guernica -i example.cfg -o profile_run --overwrite

Converting profiling output to text

Once the simulation is complete, profiling details can be written to a text file with:

nsys stats prof.nsys-rep > prof.txt

This writes the profiling summary to prof.txt.

Opening the Nsight Systems GUI

Profiling results can also be viewed with the GUI:

nsys-ui prof.nsys-rep

Profiling and simulation output

When profiling, it can be useful to set

[output]
profile = 1

in the input file.

This prevents writing results at intermediate time steps controlled by vis_steps, which helps avoid profiling interference from repeated output writes. Final results are still saved, and that final write is included in the profiling.

Operator timing without a profiler

For a quick breakdown of where the time goes, set

[output]
operator_timing = true

At the end of the run, a table of total and per-iteration wall time for each operator (advection, each collision operator, moments, BGK Newton solve) is written to run.log and operator_timing.txt. The device is synchronized around every operator to make the attribution correct, which slows the run slightly. Leave it off for production runs.