Overlapping Overset Grids

This tutorial will describe how to run a case with two overlapping meshes (NekNek) from scratch. We illustrate this by using the perhill case. If you are not familiar with NekRS, we strongly recommend you begin with the periodic hill tutorial first!

The three key steps to running a case with NekNek are:

Setting up the mesh with appropriate boundary conditions for the overlapping interface.

Specifying parameters to control stability and accuracy for the overlapping-Schwarz iterations.

Modifying velocityDirichletConditions to read Dirichlet boundary data for the overlapping interfaces.

Pre-processing

The pre-processing steps for a NekNek calculation differ from a mono-domain NekRS calculation only in terms of ensuring that the appropriate boundary condition int is specified at the overlapping interface. This boundary condition serves as the flag to let the solver know that Dirichlet data for these boundary conditions must come from interpolation of the data in the overlapping mesh.

Mesh generation

In this tutorial, we generate two meshes to span the entire domain, an upper and a lower mesh. The lower mesh is generated by genbox with the following input file:

and the upper mesh is generated by genbox with the following input file:

Note: the lower domain spans \(y \in [0,2.5]\) and the upper domain spans \(y \in [2.25,3]\). We also use int for the boundary surfaces that overlap the other domain. Create two directories, lower and upper representing the two meshes then generate them by running genbox for each case to obtain upper/upper.re2 and lower/lower.re2.

$ genbox <<< upper/upper.box
$ mv box.re2 upper/upper.re2
$ genbox <<< lower/lower.box
$ mv box.re2 lower/lower.re2

Reminder:: genbox produces a file named box.re2, so you will need to rename the files between generating the separate parts.

User Defined Functions file (.udf)

As before we need to create a UDF file, to get started create a file hillnn.udf in the current directory.

Modify mesh and add forcing to the flow

As for the mono-domain periodic hill case, we modify the mesh defining a function UDF_Setup and adding:

void UDF_Setup() {
  // mesh modification
  for(int i{0}; i < mesh->Nlocal; ++i) {
    const dfloat argx{B * (std::abs(x[i] - A) - B)};
    const dfloat A1{C * (1. + std::tanh(argx))};
    y[i] = y[i] + A1 * (3. - y[i]);
  }
}

neknek_mesh — Fig. 9 Modified box mesh graded

Currently, applying a constant mass flux with param(54) and param(55) is not supported with overlapping overset grids. For this case, we drive the flow using a constant acceleration term in userf and using this within UDF_Setup:

void userf(double time) {
  const dfloat ffx{0.052};

  // Get x component of non-linear momentum array
  // using offset of 0
  auto o_FUx = nrs->o_NLT + 0 * nrs->fieldOffset;
  platform->linAlg->fill(nrs->mesh->Nlocal, ffx, o_FUx);
}

void UDF_Setup() {
  // mesh modification
  for(int i{0}; i < mesh->Nlocal; ++i) {
    const dfloat argx{B * (std::abs(x[i] - A) - B)};
    const dfloat A1{C * (1. + std::tanh(argx))};
    y[i] = y[i] + A1 * (3. - y[i]);
  }

  nrs->userVelocitySource = &userf;
}

Specifying NekNek and control parameters

The control parameters for this case are the same as that for the mono-domain periodic hill case and impact the stability and accuracy of the calculation. In addition we also specify the extrapolation order for the boundary conditions of the overlapping interface.

Create a parameter file for the lower mesh lower/lower.par:

[GENERAL]
polynomialOrder = 7
stopAt = endTime
endTime  = 10
variableDT = yes
dt = targetCFL = 0.4 + initial=1e-1
timeStepper = bdf2
checkpointControl = steps
checkpointInterval = 20
udf = "../hillnn.udf"

[PROBLEMTYPE]
equation = navierstokes

[VELOCITY]
residualTol = 1e-8
density = 1
viscosity = -100

[NEKNEK]
boundaryEXTOrder = 2

[MESH]
file = "lower.re2"

[CASEDATA]
interpolated_bc_id = 3

and another for the upper mesh upper/upper.par:

[GENERAL]
polynomialOrder = 7
stopAt = endTime
endTime  = 10
variableDT = yes
dt = targetCFL = 0.4 + initial=1e-1
timeStepper = bdf2
checkpointControl = steps
checkpointInterval = 20
udf = "../hillnn.udf"

[PROBLEMTYPE]
equation = navierstokes

[VELOCITY]
residualTol = 1e-8
density = 1
viscosity = -100

[NEKNEK]
boundaryEXTOrder = 2

[MESH]
file = "upper.re2"

[CASEDATA]
interpolated_bc_id = 3

here the extrapolation is set by the variable boundaryEXTOrder. For this tutorial we have set our polynomial order to be \(N=7\) using the polynomialOrder variable.

Defining the session

The last file we require is a session file which will combine all of the components into a single NekRS run, create a new file hillnn.sess and add the following:

lower/lower:1;
upper/upper:1;

the lines being in the form <relative-path-to-component>:<number-of-mpi-ranks>;.

Compilation

You should now have the following files required to compile and run this case:

hillnn.udf

lower/lower.par

upper/upper.par

If for some reason you encountered an insurmountable error and were unable to generate any of the required files, you may use the provided links to download them. After confirming that you have all eight, you are now ready to compile

$ nekbmpi hillnn.sess 2

Note the number of MPI ranks should match the total of those defined in the session file, in this case 2. If all works properly, upon compilation the executable nek5000 will be generated.