Case files
A nekRS simulation is referred to as a “case,” which utilises a number of files which are described in this page. An overview of these are presented in the the image below .
There are a minimum of three files required to run a case:
Parameter file, with
.parextension. This sets parameters used by the case and can be modified between runs.Mesh file, with
.re2extension. This file defines the geometry of the case.User-defined host file, with
.udfextension. This is used to set specific equations of the case, initial/boundary conditions, data outputs and other user definable behaviour.
With one optional file
Trigger file, with
.updextension. This file allows modifications to the simulation during execution.
The “case name” is then the common prefix applied to these files - for instance,
a complete input description with a case name of “eddy” would be given by the files
eddy.par, eddy.re2, eddy.udf, and eddy.upd.
The only restrictions on the case name are:
It must be used as the prefix on all simulation files, and
Typical restrictions for naming files for your operating system
The following sections describe the structure and syntax for each of these files for a general case. Because the Nek5000 code is a predecessor to nekRS, some aspects of the current nekRS input file design are selected to enable faster translation of Nek5000 input files into nekRS input files. Because these Nek5000-based approaches require proficiency in Fortran, the inclusion of several additional input files, and in some cases, careful usage of fixed-format text inputs, all Nek5000-based methods for case setup are referred to here as “legacy” approaches. All new users are encouraged to adopt the nekRS-based problem setup.
Tip
Two legacy file types you may encounter are .usr and .co2 files. Please follow the references to the Nek5000 documentation for further information on these file types.
Parameter File (.par)
Most information about the problem setup is defined in the parameter file. This file is organized in a number of sections, each with a number of keys. Values are assigned to these keys in order to control the simulation settings.
The general structure of the .par file is as
follows, where FOO and BAR are both section names, with a number of (key, value) pairs.
[FOO]
key = value
baz = bat
[BAR]
alpha = beta
gamma = delta + keyword=value + ...
The valid sections for setting up a nekRS simulation are:
BOOMERAMG: settings for the (optional) AMG solverGENERAL: generic settings for the simulationMESH: settings for the meshOCCA: backend OCCA device settingsPRESSURE: settings for the pressure solutionPROBLEMTYPE: settings for the governing equationsSCALARXX: settings for theXX-th scalarTEMPERATURE: settings for the temperature solution (N.B. This is equivalent to SCALAR00)VELOCITY: settings for the velocity solutionCASEDATA: custom settings
Each of the keys and value types are described below.
nekRS supports both traditional CPU-based architectures and accelerated
computing on GPUs. In the OCCA section, users can select various backend options,
such as the serial mode for CPU-only computations, CUDA for accelerated
computing with Nvidia GPUs, or HIP for AMD GPUs.
std::string user_occa_backend;
options.getArgs("THREAD MODEL", user_occa_backend);
As shown in snippet above, if you have backend = CUDA in the .par file, then
user_occa_backend would be set to CUDA in the above code.
In the GENERAL section, key settings include configuring the simulation’s polynomial
order (polynomialOrder), restart options, and time iteration parameters. For nekRS
simulations, a recommended polynomial order is p=7, though the software currently
supports orders ranging from 2 to 11. By default, nekRS uses the user-specified initial
conditions provided in the udf file to begin computations. However, the startFrom
option allows users to provide a restart file as the initial condition for continuing
simulations. Additional arguments can be used to specify which fields from the restart
file should be used.
To configure the stop criteria for time iterations in the simulation, the user can set
the stopAt parameter to either numSteps, endTime, or elapsedTime,
depending on whether the simulation should stop after a specific number of time steps,
a certain end time, or the total elapsed time.
The frequency at which checkpoint files are saved is controlled by the checkpointControl
and checkpointInterval parameters. The user can specify this frequency based on
either a set number of time steps or a simulation time interval.
Additionally, the user can select the time-stepping order (i.e., the order of the
backward difference scheme for time discretization) by setting timeStepper
to one of the following: tombo1 for first-order, tombo2 (the default) for
second-order, or tombo3 for third-order accuracy.
There are dedicated sections for configuring the solution of fields such as pressure,
velocity, temperature, and passive scalars. Common input parameters across these
sections include residualTol, boundaryTypeMap, as well as advection and
diffusion coefficients. These coefficients may have different names depending
on the field, such as rho and viscosity for [VELOCITY], or rhoCp
and conductivity for [TEMPERATURE].
The .par file also allows users to apply regularization to simulations.
This provides a convenient way to set up a large eddy simulation (LES) using the hpfrt
(i.e., high-pass filter), or to introduce the artificial viscosity
(avm) to a solution field.
For advanced users, nekRS offers additional pressure preconditioners and coarse
mesh solvers that can be explored and fine-tuned through parameters such as
preconditioner, coarseGridDiscretization, and smootherType to
optimize simulation efficiency. The default settings, however, provide a well-balanced
configuration suitable for basic users.
Generally, most .par settings are not saved to a data structure, so throughout the code
base, whenever information from the .par file is needed, it is simply
extracted on-the-fly via the options structure.
nekRS performs validation of the par file. Invalid sections, invalid keys or values, invalid value combinations, missing values etc. will terminate the NekRS run with a clear error message. Deprecated attributes will be highlighted.
nekRS uses just-in-time compilation to allow the incorporation of user-defined functions
into program execution. These functions can be written to allow ultimate flexibility on
the part of the user to affect the simulation, such as to define custom fluid properties,
specify spatially-dependent boundary and initial conditions, and apply post-processing
operations. Some of the parameters in the sections can be overridden through the use of
user-defined functions - see, for example, the viscosity key in
the VELOCITY section. This parameter is used to set a constant viscosity, whereas
for variable-property simulations, a user-defined function will override the viscosity
input parameter. A full description of these user-defined functions on the host and
device are described in Sections User-Defined Host File (.udf). So, the description of valid (key, value)
pairs here does not necessarily imply that these parameters reflect the full capabilities
of nekRS.
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key Value(s) Description / Comment
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
userSections <string> user defined par sections
[OCCA]
backend SERIAL (CPU), CUDA, HIP, DPCPP default defined in $NEKRS_HOME/nekrs.conf
deviceNumber <int>, LOCAL-RANK [D]
platformNumber <int> only used by DPCPP
0 [D]
[GENERAL]
verbose true, false [D]
polynomialOrder <int>
dealiasing true [D], false
cubaturePolynomialOrder <int>
3/2*(polynomialOrder+1) -1 [D]
redirectOutputTo <string>
startFrom "<string>"
+ time=<float> reset to specified time
+ x read mesh coordinates
+ u read velocity
+ t read temperature
+ s00 s01 s02 ... read scalars
+ int use point interpolation
timeStepper tombo1, tombo2 [D], tombo3
stopAt numSteps [D], endTime, elapsedTime stop criterion
numSteps <int>
endTime <float>
elapsedTime <float> in minutes
dt <float> [D]
+targetCFL=<float> adjust dt to match targetCFL
+max=<float>
+initial=<float>
subCyclingSteps <int>, auto number of OIFS sub-steps for advection
0 [D]
checkpointEngine <string>
nek [D], adios
checkpointPrecision <int>
32[D], 64
checkpointControl steps [D], simulationTime, checkpoint frequency unit
checkpointInterval <int>, <float> checkpoint frequency in writeControl units
0 [D] at the end of the simluation
-1 disable checkpointing
constFlowRate meanVelocity=<float> set constant flow velocity
meanVolumetricFlow=<float> set constant volumetric flow rate
+ direction=<X,Y,Z> flow direction
regularization see COMMON field settings
udf "<string>" name of .udf file
oudf "<string>" name of .oudf file
usr "<string>" name of .usr file
nScalars <int> number of non-temperature scalars
by default, the scalars will have the settings
as outlined by the default [SCALAR] section
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
[NEKNEK]
boundaryEXTOrder <int> boundary extrapolation order
1 [D] >1 may require additional inner (Schwarz iterations)
corrector steps
multirateTimeStepping true, false [D]
+correctorSteps=<int> outer corrector steps
0 [D] NOTE: boundaryEXTOrder > 1 requires correctorSteps > 0
for stability
----------------------------------------------------------------------------------------------------------------------
[PROBLEMTYPE]
equation stokes
navierStokes [D]
+variableViscosity use stress formulation
(required for spatially varying viscosity)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
COMMON field settings:
solver none linear solver
user
cvode
PCG [D]
+combined [D]
+block [D for VELOCITY]
+flexible
PFGMRES [D for PRESSURE]
+nVector=<int> dimension of Krylov space
residualTol <float> absolute linear solver residual tolerance
1e-4 [D]
+relative=<float> use absolute/relative residual (whatever is reached first)
absoluteTol <float> absolute solver tolerance (for CVODE only)
1e-6 [D]
initialGuess previous
extrapolation [D]
projection, projectionAconj [D for PRESSURE]
+nVector=<int> dimension of projection space
preconditioner Jacobi [D]
multigrid [D for PRESSURE] polynomial multigrid + coarse grid correction
+additive
SEMFEM
+nonSmoothed
coarseGridDiscretization FEM [D] linear finite elment discretization
+Galerkin coarse grid matrix by Galerkin projection
SEMFEM linear FEM approx on high-order nodes
coarseSolver/semfemSolver smoother
boomerAMG [D] HYPRE's AMG solver
+cpu [D for multigrid]
+device [D for SEMFEM]
+overlap overlap coarse grid solve in additive MG cycle
pMGSchedule p=<int>, degree=<int>, ... custom polynomial order and Chebyshev order for each pMG level
smootherType Jacobi
ASM [D], RAS overlapping additive/restrictive Schwarz
+Chebyshev 1st Kind Chebyshev acceleration
+FourthChebyshev 4th Kind Chebyshev acceleration
+FourthOptChebyshev [D] 4th Opt Chebyshev acceleration
+minEigenvalueBoundFactor=<float> only for 1st Kind Chebyshev required
+maxEigenvalueBoundFactor=<float>
boundaryTypeMap <bcType for ID 1>, <bcType for ID 1>, ... boundary type for each boundary ID
none no-op (internal or periodic)
interpolation neknek boundary (int)
assign int values in codedFixedValue
zeroValue velocity only (w, wall)
codedFixedValue user specified Dirichlet (v, t, inlet)
specify okl function codedFixedValueVelocity
or codedFixedValueScalar in udf
zeroGradient (I, o, O, insulated, outlet, outflow)
codedFixedGradient user specified scalar Neumann (f, flux)
specify okl function codedFixedGradientScalar in udf
zeroXValue/zeroGradient symmetry x-normal plane (slipx, symx)
zeroYValue/zeroGradient symmetry y-normal plane (slipy, symy)
zeroZValue/zeroGradient symmetry z-normal plane (slipz, symz)
zeroNValue/zeroGradient unaligned symmetry (slip, sym)
zeroYZValue/zeroGradient (onx)
zeroXZValue/zeroGradient (ony)
zeroXYValue/zeroGradient (onz)
zeroXValue/codedFixedGradient traction x-normal plane (tractionx, shlx)
specify okl function codedFixedGradientVelocity in udf
zeroYValue/codedFixedGradient traction y-normal plane (tractiony, shly)
specify okl function codedFixedGradientVelocity in udf
zeroZValue/codedFixedGradient traction z-normal plane (tractionz, shlz)
specify okl function codedFixedGradientVelocity in udf
zeroNValue/codedFixedGradient unaligned traction (traction, shl)
specify okl function codedFixedGradientVelocity in udf
checkpointing true [D], false
regularization
hpfrt HPF stabilization
+nModes=<int> number of modes
+scalingCoeff=<float> filter strength
avm artificial viscosity method
(has to be applied manually in case user user props)
+c0 make viscosity C0
+scalingCoeff=<float>
+noiseThreshold=<float> smaller values will be considered to be noise
+decayThreshold=<float>
+activationWidth=<float> half-width of activation function
----------------------------------------------------------------------------------------------------------------------
[MESH]
partitioner rbc, rsb, rbc+rsb [D] partitioning method
boundaryIDMap <int>, <int>, ... map ids to 1,2,3,...
boundaryIDMapV <int>, <int>, ... required for cht case
connectivityTol <float>
0.2 [D]
file "<string>" name of .re2 file
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
[VELOCITY]
density <float> density
rho
viscosity <float> dynamic viscosity
mu
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
[TEMPERATURE]
heatCapacity <float> rho*Cp
rhoCp
conductivity <float> thermal conductivity
k
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
[SCALAR] default scalar section, see general::nScalars
all non-temperature scalars will have settings
as set from this section, unless explicitly overriden
density <float> density
rho
diffusivity <float> diffusivity
D
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
[SCALAR01]
...
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
[BOOMERAMG]
coarsenType <int>
interpolationType <int>
smootherType <int>
iterations <int>
strongThreshold <float>
nonGalerkinTol <float> set 0 to classical Galerkin sparsification
aggressiveCoarseningLevels <int>
chebyshevRelaxOrder <int>
chebyshevFraction <float>
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
[CVODE]
solver cbGMRES [D], GMRES linear solver
+nVector=<int> dimension of Krylov space
gsType classical [D], modified
relativeTol <float> relative tolerance
1e-4 [D]
epsLin <float> ratio between linear and nonlinear tolerances
0.5 [D]
dqSigma <float> step size for Jv difference quotient
automatic [D]
maxSteps <int>
sharedRho true, false [D] use same "density" field for all but the first scalar
jtvRecycleProperties true [D], false recycle property (freeze) evaluation for Jv
dealiasing true, false
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
NOTES
* section name and key/value pairs are treated as case insensitive
* values enclosed within quotes maintain case sensitivity
* values prefixed with 'env::' are interpreted as references to environment variables
Mesh File (.re2)
The nekRS mesh file is provided in a binary format with a nekRS-specific
.re2 extension. This format can be produced by either:
Converting a mesh made with commercial meshing software to
.re2format, orDirectly creating an
.re2-format mesh with nekRS-specific scripts
There are three main limitations for the nekRS mesh:
nekRS is restricted to 3-D hexahedral meshes.
The numeric IDs for the mesh boundaries must be ordered contiguously beginning from 1.
The
.re2format only supports HEX8 and HEX 20 (eight- and twenty-node) hexahedral elements.
Lower-dimensional problems can be accommodated on these 3-D meshes by applying zero gradient boundary conditions to all solution variables in directions perpendicular to the simulation plane or line, respectively. All source terms and material properties in the governing equations must therefore also be fixed in the off-interest directions.
For cases with conjugate heat transfer, nekRS uses an archaic process
for differentiating between fluid and solid regions. Rather than block-restricting variables to
particular regions of the same mesh, nekRS retains two independent mesh representations
for the same problem. One of these meshes represents the flow domain, while the other
represents the heat transfer domain. The nrs_t struct, which encapsulates all of
the nekRS simulation data related to the flow solution, represents the flow mesh as
nrs_t.mesh. Similarly,
the cds_t struct, which encapsulates all of the nekRS simulation data related to the
convection-diffusion passive scalar solution, has one mesh for each passive scalar. That is,
cds_t.mesh[0] is the mesh for the first passive scalar, cds_t.mesh[1] is the mesh
for the second passive scalar, and so on.
Note that only the temperature passive scalar uses the conjugate heat transfer mesh,
even though the cds_t struct encapsulates information related to all other
passive scalars (such as chemical concentration, or turbulent kinetic energy). All
non-temperature scalars are only solved on the flow mesh.
Warning
When writing user-defined functions that rely on mesh information (such as boundary
IDs and spatial coordinates), you must take care to use the correct mesh representation
for your problem. For instance, to apply initial conditions to a flow variable, you
would need to loop over the number of quadrature points known on the nrs_t meshes,
rather than the cds_t meshes for the passive scalars (unless the meshes are the same,
such as if you have heat transfer in a fluid-only domain).
Also note that the cds_t * cds object will not exist if your problem
does not have any passive scalars.
nekRS requires that the flow mesh be a subset of the heat transfer mesh. In other words,
the flow mesh always has less than (or equal to, for cases without conjugate heat transfer)
the number of elements in the heat transfer mesh. Creating a mesh for conjugate heat
transfer problems requires additional pre-processing steps that are described in the
Creating a Mesh for Conjugate Heat Tranfser section. The remainder
of this section describes how to generate a mesh in .re2 format, assuming
any pre-processing steps have been done for the special cases of conjugate heat transfer.
User-Defined Host File (.udf)
The .udf file is a OKL and C++ mixed language source file where user code
used to formulate the case is placed. This code is placed in various functions
and these can be used to perform virtually any action that can be programmed in
C++. Some of the more common examples are setting initial conditions, querying
the solution at regular intervals, and defining custom material properties and
source terms. The available functions that you may define in the .udf file
are as follows.
OKL block
The .udf typically has a #ifdef __okl__ block near the start which is
where all OKL code will be placed that will run on the compute backed specified to
OCCA. The most frequent use of this block is to provide the functions
for boundary conditions that require additional information, such as a value to
impose for a Dirichlet velocity condition, or a flux to impose for a Neumann
temperature condition. Additional user functions may be placed in this block to
allow advanced modification of the simulation or functionality such as calculating
exact values at a specified time point.
Tip
If the user generated functions are sufficiently large, or in older nekRS examples
you may see a .oudf file which is included within the ifdef block
instead of the functions being in the .udf file.
#ifdef __okl__
@kernel void computeexact(const dlong Ntotal)
{
for (dlong n = 0; n < Ntotal; ++n; @tile(p_blockSize, @outer, @inner)) {
if (n < Ntotal) {
// some code
}
}
}
void velocityDirichletConditions(bcData *bc)
{
// some code
bc->u = u;
bc->v = v;
bc->w = w;
}
void scalarDirichletConditions(bcData *bc)
{
// some code
bc->s = s
}
void scalarNeumannConditions(bcData *bc)
{
bc->flux = tflux;
}
UDF_Setup0
This user-defined function is passed to the nekRS MPI communicator comm and a data
structure containing all of the user-specified simulation options, options. This function is
called once at the beginning of the simulation before initializing the nekRS internals
such as the mesh, solvers, and solution data arrays. Because virtually no aspects of
the nekRS simulation have been initialized at the point when this function is called,
this function is primarily used to modify the user settings. For the typical user,
all relevant settings are already exposed through the .par file; any desired
changes to settings should therefore be performed by modifying the .par file.
This function is intended for developers or advanced users to overwrite any user
settings that may not be exposed to the .par file. For instance, setting
timeStepper = tombo2 in the GENERAL section triggers a number of other internal
settings in nekRS that do not need to be exposed to the typical user, but that perhaps
a developer may want to modify for testing purposes.
UDF_Setup
This user-defined function is passed the nekRS simulation object nrs. This function
is called once at the beginning of the simulation after initializing the mesh, solution
arrays, material property arrays, and boundary field mappings. This function is most
commonly used to:
Apply initial conditions to the solution
Assign function pointers to user-defined source terms and material properties
Any other additional setup actions that depend on initialization of the solution arrays and mesh can of course also be placed in this function.
UDF_ExecuteStep
This user-defined function is probably the most flexible of the nekRS user-defined functions. This function is called once at the start of the simulation just before beginning the time stepping, and then once per time step after running each step.
Trigger Files (.upd)
TODO Full description
Allows modifications to the simulation during execution. Can be edited and then notify of changes through sending a signal MPI rank 0.
Footnotes