Can I use ByteArray and NonDirect ByteBuffer with JNR

[furstenheim-goodnotes]

Hi,
This might be a noob question. Can I use ByteArrays and ByteBuffers that are not allocateDirect with JNR?

Something like:

Java code

fun MyFunction(
            src: ByteArray,
            srcSize: Int
        ): Int

fun MyFunction(
            src: ByteBuffer,
            srcSize: Int
        ): Int

C code

int MyFunction(const char* src,
                              int srcSize);

As far as I am aware, if I do ByteBuffer.allocate(size) the underlying memory resides in the heap and can be moved around during GC compactation.

Could this lead to a segfault when using the memory in C? Does JNR protect against this?

Thanks

[headius]

Hello there!

ByteBuffer will be recognized by jffi/jnr-ffi as being a block of memory for native functions to access.

For non-direct (allocated on heap with .allocate()), the data will need to be copied out so it can be passed as a regular pointer.

If it is direct (allocate in native memory with allocateDirect()), it will pass the underlying pointer directly and avoid copying data back and forth.

I don't know what ByteArray is, but judging by your code (which I assume is Scala?) it is equivalent to a byte[]? That will behave the same as a non-direct ByteBuffer and copy data out to native and back in (depending on annotations like @Out and @In.

With proper use of these buffers on the native side (i.e. within bounds) they will not cause a segfault, but of course once you're in native territory then bad code can certainly trigger faults by walking off the end of an array.

There's other quirks like the lifetime of the native side (non-direct shouldn't be saved after the call, direct will remain the same pointer until GCed, etc) but hopefully this answers your general question!

[furstenheim-goodnotes]

Hey, thanks a lot for the answer. What do you mean by "non-direct shouldn't be saved after the call"?

Thanks!

[headius]

In the case of non-direct ByteBuffer (heap-based, via ByteBuffer.allocate()), we have to copy the bytes out to a temporary memory location for the native call. After the call has returned, that temporary block of memory will be deallocated and no longer addressable.

I'm not sure if that deallocation happens immediately or lazily, but either way the native call should not hang onto that address and expect to be able to access it after the call.

Direct ByteBuffer will just use the same native buffer until it is garbage collected, so as long as it is being referenced from the Java side its native block of memory will be accessible by native code.

[headius]

I should clarify: there are JVM APIs that might let us temporarily "pin" the heap-based byte[] backing a non-direct ByteBuffer for the duration of the native call (preventing GC from moving it). In this case, the native code can use that address directly without copying, but it won't remain pinned for long.

[furstenheim-goodnotes]

But in this pin do you actually pass the memory address on the heap? If I understood correctly from the internet, there is no guarantee that the memory is actually linear, it can be split in chunks if the memory allocator decides so.

Thanks again 🙏, I haven't found a nice reference on the internet

[headius]

I have not heard of JVM's segmenting a contiguous byte array on the heap but I suppose it's possible. My understanding of the pinning APIs was that if they succeed and give you back a pointer, it will be a proper pointer into the entirety of the array. If it doesn't succeed you need to fall back on copying. I would have to review our code to see whether we are successfully using those APIs. Best bet if you want to guarantee the native memory is accessible is to always use a direct buffer or use the memory management APIs in jnr-ffi.