AccessViolationException in the native message loop locks all pending operations

[KrzysFR]

When a crash occurs in the native message loop (network thread run by fdb_c.dll), the .NET process can become stuck because all pending Tasks on transactions will never complete.

An example of such a case is a very rare AccessViolationException (seen from time to time on our internal Build server) that crashes the network thread. The build process then hangs and must be terminated manually.

The callstack of the crash looks like this:

[Something.Something.FooBarTest.Test_FooBar] [Test Error Output]
Unhandled exception in remote appdomain: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at FoundationDB.Client.Native.FdbNative.NativeMethods.fdb_run_network()
   at FoundationDB.Client.Native.FdbNative.RunNetwork()
   at FoundationDB.Client.Fdb.EventLoop()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart()

If the network thread is not running, then no more callback will run, including the timeout callbacks. This means that any Task pending on a transaction will never complete, even if a timeout was specified on a transaction. The only way for it to stop, would be if the transactions was created with a host-provided cancellation token (ex: per-request CancellationToken in ASP.NET, etc..). If it was created with CancellationToken.None, then the task is dead.

Questions:

First, is it safe to restart the network thread, would it keep up where it left off, and remember all pending callbacks? If it already crashed with an AccessViolationException, it does not look very safe. This seems like we should abandon this process ASAP.

Second, if we can't restart the network thread and must stop the process, how can we ensure that all pending tasks wil abort even if their callback never fires? One solution would be to trigger the internal CancellationTokenSource of each FdbDatabase instance, which means that the binding would need to keep a list of all these instances. OR have as single master CancellationTokenSource that is linked with all the tokens of each FdbDatabase?

[KrzysFR]

Some news on this subject: the Access Violations seems to be due to some race conditions with the way FdbFuture* from the C API are mapped to Tasks in .NET, in some very high cpu / high contention scenarios, escpecially when there is a high rate of failure or cancellation going on. We've seen this on unit tests that spin up tons of requests, fail on some assertion, and want to abort/dispose everything.

Looks like the current shutdown scenario is not clean. A lot of handles will end up in the GC's Finalizer Queue and be destroyed in some random order, inducing all sorts of badness in the C API itself (which throws the Access Violations in multiple place, because we are passing it outdated handles).

The symptom seen on our CI server is that one test will fail with an AV (either directly, or the network event loop will throw an unhandled exception). Then the event loop is dead in the water, but the next unit test will still want to execute. If the first op it does is a read, it will hang. If the first op it does is write only or commit, then it will throw and somehow continue.

Right now, the decision is that IF access violations occurs then the only possible choice is to FailFast() and terminate the process. This is bad practice to FailFast() inside a library, BUT this is the library that handles communications with the database after all, so this is probably one of the rare exceptions where the alternative (keep running) would be too dangerous (data corruption?)

Now, WHY access violations occurs is a different problem, which is currently being worked on.

[KrzysFR]

When working on the new Future handling code, I've been able to trigger this kind of crashes somewhat reliably, by double-freeing future handles (calling fdb_future_destroy twice) or free a handle concurrently with it being used.

I've also seen the internal allocator in fdb_c.dll rapidly reuse the same handles, sometime back to back, which would make this kind of issue, more frequent. I'm not sure if this is from a change in the allocation strategy in fdb v3 or not.

This definitely put the blame on the way the .NET binding currently deals with the Futures handles. There are some gotchas in the C API which are somewhat specific to the way the .NET platform deals with PInvoke and managed callbacks, as well as some pathological cases where the C API make things a bit more complex, like the fact that fdb_future_set_callback can invoke the callback inline, if the future just transitionned to ready.

[KrzysFR]

No hangs observed since last month, using either the master or refac_tuples branch.

Once #54 is merged, the problem should disappear. And the addition of Environment.FailFast will make sure that if there is a problem, it will terminate the app with an entry added to the Windows Event Log.