[WIP] Abstract coordinate systems and general cleanup

[jjaraalm]

Restructures the code base to allow for general, orthogonal coordinate systems. Currently, Cartesian (x, y, z) and cylindrical (r, theta, z) are implemented based on the cpu-based, cylindrical version.

Coordinate system-specific operations are abstracted and specific implementations live in src/grid/geometry. In order to support this, significant and wide spread changes had to be made. species_advance/standard and field_advance/standard were heavily restructured and DRY'ed. Most operations are only available in (renamed) Kokkos versions to avoid having to implement general geometries in both legacy and Kokkos code.

By default, the coordinate system is set to Cartesian to support legacy code. To change the coordinate system, grid->geometry must be set during initialization and before any calls to move_p. E.g.,:

  // In begin_initialization:
  define_periodic_grid( Rmin, Tmin, 0,  // Grid low corner
                        Rmax, Tmax, 1,  // Grid high corner
                        nR, nT, 1,      // Grid resolution
                        1, 1, 1 );      // Processor config

  grid->geometry = Geometry::Cylindrical;

Major changes

• Support for general, orthogonal coordinate systems using a scaled field interpolation, Cartesian push, and then a rotation back into the local frame.
• particle_mover_t->disp{x,y,z} has changed. Previously, disp referred to a scaled displacement in terms of the grid spacing. It now refers to the unscaled Cartesian displacement.
• Checkpoint mechanism now works on Kokkos Views with macros CHECKPT_VIEW to checkpoint the size/type of a view and CHECKPT_VIEW_DATA to checkpoint the contents. Restore has similar macros. Checkpoint/restore code relocated from main simulation class to objects.
• grid_t::neighbor was deprecated and removed in favor of grid_t::k_neighbor_h
• Vacuum field kernels are deprecated in favor of template-based optimizations within the standard field kernels.
• Renamed public *_kokkos functions to *
• hydro_p ported to gpu, similar to hydro_p branch and addresses Port hydro_p to Kokkos  #9

Minor changes

• Removed known bug (from CPU version) in set_region_field macro.
• Kokkos::finalize() moved to halt_services()
• Copy to device/host semantics are unified
• Lots of DRY. E.g., all Borris operations are located in field_advance/standard/borris.h rather than explicitly written.
• Updated sample/bench/advance_p and added it as a build product to create a standard performance benchmark, vpic-benchmark

Test changes

• Removed rho_p test since host-side accumulation was deprecated.
• Added valgrind-based tests for memory leaks, and eliminated existing ones
• Added cylindrical tests for particle and field advance (from @astanier)

Known Issues

• Emitters are currently disabled. There is no technical issue, and should be able to be enabled soon.
• Custom particle boundary conditions are not supported, just as on devel. This branch raises an error.
• Github build is broken. Possible GCC version issue. Builds fine on GCC 8.3.1 with OpenMP backend and on a POWER9/V100 system with GCC 7.3.1 and nvcc 10.2.
• Cylindrical systems do not support r=0, but could be implemented soon
• Divergence cleaning in cylindrical systems is implemented, but untested. Needs work to verify correctness.
• absorb_field boundary conditions are not technically correct in cylindrical systems. The Higdon ABC assumes the wave equation is separable along the orthogonal direction, but this is not true in general coordinates. Not a huge problem, but should be documented.

Remaining tasks

• Fix CI/CD
• Performance testing, hopefully Cartesian doesn't suffer too much
• Move sorter and compressor into species_advance/standard (optional)
• Integrated test for cylindrical (similar to Weibel_driver)
• Merge in collisions from Add collisions #30
• Better documentation of coordinate system interfaces
• Support r=0?
• More cleanup/deprecation?

[jjaraalm]

Current performance benchmarks, all with 1x MPI rank and 1x 32GB V100. Overall, there is a small (at most 5%) difference in performance vs. devel, and cylindrical vs. Cartesian geometries.

Particle advance

Pair plasma in a 50^3 domain with 2000 particles / species / cell, advance for 100 steps. For cylindrical geometry, rmin/dr = 80 since r=0 is not supported yet.

Branch	Geometry	Particle advances / sec
devel	Cartesian	1.53e+09
hierarchical	Cartesian	1.26e+09
geometry	Cartesian	1.48e+09
geometry	Cylindrical	1.46e+09

Not sure why hierarchical is slower. This is a little confusing.

LPI sample

nx = 3584, ny = 1, nz = 56, nppc=448, and num_step=800. Custom pbcs are disabled in the deck (devel and hierarchical seem to silently ignore them, while geometry raises an error)

Branch	Run time (s)
devel	184.8
hierarchical	71.8
geometry	185.3

I have no idea why hierarchical is so fast here, @rfbird said it should take about 120s in this setup. Maybe the 32GB vs 16GB V100 allows for more efficient memory access?