Fix crash at exit due to a race condition with new signal handler

Previously, the signal handler would interrupt the main thread (thread
that calls `Server::Run`), so there was no chance for the main thread to
resume before the signal handler was finished. With the new signal
handler implementation in gz::common, the signal handler is now called
from a different thread which means the main thread might resume before
the signal handler is finished. This causes a race condition in which
the `Server` class, and therefore the `ServerPrivate` class are being
destructed while the signal handler is still active. Since
`ServerPrivate` also contains the class that encapsulates the signal
handler implementation, the causes a crash as the signal handler tries
to access data that has been deleted.

The fix here is to ensure that the main thread does not exit until the
signal handler is finished by synchronizing on a mutex.

Signed-off-by: Addisu Z. Taddese <addisu@openrobotics.org>

Author: azeey

src/ServerPrivate.cc

@@ -123,10 +123,20 @@ void ServerPrivate::OnSignal(int _sig)
 /////////////////////////////////////////////////
 void ServerPrivate::Stop()
 {
-  this->running = false;
-  for (std::unique_ptr<SimulationRunner> &runner : this->simRunners)
-  {
-    runner->Stop();
+  // Stop might be called from the signal handler thread (new in Ionic) instead
+  // of the main thread, so we need to ensure that we keep `ServerPrivate` alive
+  // while the signal handler is still active. We do that by synchronizing on
+  // the `runMutex` here in ServerPrivate::Run right after the call
+  // SimulationRunner::Run returns. That way, `ServerPrivate::Run` cannot return
+  // before the signal handler is finished.
+  std::lock_guard<std::mutex> lock(this->runMutex);
+  if (this->running)
+  {
+    this->running = false;
+    for (std::unique_ptr<SimulationRunner> &runner : this->simRunners)
+    {
+      runner->Stop();
+    }
   }
 }
 
@@ -196,6 +206,8 @@ bool ServerPrivate::Run(const uint64_t _iterations,
     result = this->workerPool.WaitForResults();
   }
 
+  // See comments ServerPrivate::Stop() for why we lock this mutex here.
+  std::lock_guard<std::mutex> lock(this->runMutex);
   this->running = false;
   return result;
 }