Issues with CRTP `Evaluate` on GPUs

[pdmullen]

I am experiencing an issue in a downstream app wherein an Evaluate call (see

singularity-eos/singularity-eos/eos/eos_base.hpp

Lines 135 to 139 in dcd7a6e

	template <typename Functor_t>
	constexpr void Evaluate(Functor_t &f) const {
	CRTP copy = *(static_cast<CRTP const *>(this));
	f(copy);
	}

) leads to data corruption on GPUs (volta with nvhpc). I can seemingly only encounter the issue when working with Spiner EOS tables.

Printing data at addresses from explicit ptrs inside and outside the Evaluate call show differing values.

@jonahm-LANL asked me to make an issue --- we will work on isolating the problem to see if this is a downstream app or singularity issue. If the latter, I am sure a reproducer is in order...

[jhp-lanl]

Which pointers? The functor? The class pointer? Or data pointers specific to spiner?

[Yurlungur]

I think @pdmullen is refering to the pointers to data accessed. The call pattern that fails looks something like this:

// on device
Real *some_device_pointer = &my_device_view(0);
auto my_functor = [=](auto eos) {
  Kokkos::parallel_for(inner_range, [=](int it) {
    eos.SomeCall(pointer[i]);
  });
};
eos.Evaluate(my_functor);

[jhp-lanl]

I think @pdmullen is refering to the pointers to data accessed. The call pattern that fails looks something like this:

// on device
Real *some_device_pointer = &my_device_view(0);
auto my_functor = [=](auto eos) {
  Kokkos::parallel_for(inner_range, [=](int it) {
    eos.SomeCall(pointer[i]);
  });
};
eos.Evaluate(my_functor);

I assume that should be a KOKKOS_LAMBDA rather than a regular lambda?

[Yurlungur]

Nope because that whole call is inside a kokkos kernel already.

[jhp-lanl]

Nope because that whole call is inside a kokkos kernel already.

Evaluate isn't a PORTABLE_FUNCTION. Does it need to be if it's being called on device?

[Yurlungur]

That's why Evaluate is marked constexpr that's supposed to allow us to evaluate it on device. But it's possible something is going wrong there.

[jhp-lanl]

ahhhhhh gotcha