Generify hydro MOL for anisotropic EB

[marchdf]

but don't actually let anyone use it because there are issues I think in amrex EB routines (redistribution) that I haven't fully investigated. But hydro MOL works. And this doesn't do EB diffusion either. But it's a step in the right direction.

[marchdf]

This is C7 with dx = dy/2

velocity

density

pressure

The part that bugs me a bit is the density profile is not quite normal to the wall everywhere. I am not sure if that's a bug in Pele or in amrex. Right now my money is on the redistribute routines. Or maybe it's all fine and it's just what happens with anisotropic EB. But I have to move to something else for now.

[baperry2]

Most of the diffs look like they are EB cases that have machine precision changes, which is reasonable to me. But the sod-4 case has bigger diffs. I believe this case tests coarse/fine crossing the EB. I am not sure why that would be affected, but I don't think we should merge this without understanding those diffs.

[baperry2]

Why are these still dx[0] - maybe they should be dx_max instead?

[marchdf]

I haven't dealt with the grad stencils so I left them as they were. I just changed the call signature to take in the dx array, so that I can change it later. Basically making it more explicit where things will need to change.

[marchdf]

Tracked the sod-4 diff down to the removal of these:

      const amrex::Real ebnorm_mag = std::sqrt(AMREX_D_TERM(
        ebnorm[0] * ebnorm[0], +ebnorm[1] * ebnorm[1], +ebnorm[2] * ebnorm[2]));
      for (amrex::Real& dir : ebnorm) {
        dir /= ebnorm_mag;
      }

which is a silly bit of code because ebnorm is already normalized when it is created. So it is being normalized twice. The hypothesis is that removing this and the use of ebnorm_local introduces small machine diffs that mess with the fluxes just enough that we get higher diffs in sod-4.

We debated and went with the option that removes this extra normalization. The case itself runs fine and to completion: