Signal handling should be signal-safe

Summary:
Before this patch, signal handling wasn't signal safe. This leads to real-world
crashes. It used ManagedStatic inside of signals, this can allocate and can lead
to unexpected state when a signal occurs during llvm_shutdown (because
llvm_shutdown destroys the ManagedStatic). It also used cl::opt without custom
backing storage. Some de-allocation was performed as well. Acquiring a lock in a
signal handler is also a great way to deadlock.

We can't just disable signals on llvm_shutdown because the signals might do
useful work during that shutdown. We also can't just disable llvm_shutdown for
programs (instead of library uses of clang) because we'd have to then mark the
pointers as not leaked and make sure all the ManagedStatic uses are OK to leak
and remain so.

Move all of the code to lock-free datastructures instead, and avoid having any
of them in an inconsistent state. I'm not trying to be fancy, I'm not using any
explicit memory order because this code isn't hot. The only purpose of the
atomics is to guarantee that a signal firing on the same or a different thread
doesn't see an inconsistent state and crash. In some cases we might miss some
state (for example, we might fail to delete a temporary file), but that's fine.

Note that I haven't touched any of the backtrace support despite it not
technically being totally signal-safe. When that code is called we know
something bad is up and we don't expect to continue execution, so calling
something that e.g. sets errno is the least of our problems.

A similar patch should be applied to lib/Support/Windows/Signals.inc, but that
can be done separately.

Fix r332428 which I reverted in r332429. I originally used double-wide CAS
because I was lazy, but some platforms use a runtime function for that which
thankfully failed to link (it would have been bad for signal handlers
otherwise). I use a separate flag to guard the data instead.

<rdar://problem/28010281>

Reviewers: dexonsmith

Subscribers: steven_wu, llvm-commits
llvm-svn: 332496

llvm/include/llvm/Support/Signals.h

@@ -56,10 +56,12 @@ namespace sys {
   // Run all registered signal handlers.
   void RunSignalHandlers();
 
+  using SignalHandlerCallback = void (*)(void *);
+
   /// Add a function to be called when an abort/kill signal is delivered to the
   /// process. The handler can have a cookie passed to it to identify what
   /// instance of the handler it is.
-  void AddSignalHandler(void (*FnPtr)(void *), void *Cookie);
+  void AddSignalHandler(SignalHandlerCallback FnPtr, void *Cookie);
 
   /// This function registers a function to be called when the user "interrupts"
   /// the program (typically by pressing ctrl-c).  When the user interrupts the

llvm/lib/Support/Signals.cpp

@@ -36,19 +36,55 @@
 
 using namespace llvm;
 
-static cl::opt<bool>
+// Use explicit storage to avoid accessing cl::opt in a signal handler.
+static bool DisableSymbolicationFlag = false;
+static cl::opt<bool, true>
     DisableSymbolication("disable-symbolication",
                          cl::desc("Disable symbolizing crash backtraces."),
-                         cl::init(false), cl::Hidden);
+                         cl::location(DisableSymbolicationFlag), cl::Hidden);
 
-static ManagedStatic<std::vector<std::pair<void (*)(void *), void *>>>
-    CallBacksToRun;
+// Callbacks to run in signal handler must be lock-free because a signal handler
+// could be running as we add new callbacks. We don't add unbounded numbers of
+// callbacks, an array is therefore sufficient.
+struct CallbackAndCookie {
+  sys::SignalHandlerCallback Callback;
+  void *Cookie;
+  enum class Status { Empty, Initializing, Initialized, Executing };
+  std::atomic<Status> Flag;
+};
+static constexpr size_t MaxSignalHandlerCallbacks = 8;
+static CallbackAndCookie CallBacksToRun[MaxSignalHandlerCallbacks];
+
+// Signal-safe.
 void sys::RunSignalHandlers() {
-  if (!CallBacksToRun.isConstructed())
+  for (size_t I = 0; I < MaxSignalHandlerCallbacks; ++I) {
+    auto &RunMe = CallBacksToRun[I];
+    auto Expected = CallbackAndCookie::Status::Initialized;
+    auto Desired = CallbackAndCookie::Status::Executing;
+    if (!RunMe.Flag.compare_exchange_strong(Expected, Desired))
+      continue;
+    (*RunMe.Callback)(RunMe.Cookie);
+    RunMe.Callback = nullptr;
+    RunMe.Cookie = nullptr;
+    RunMe.Flag.store(CallbackAndCookie::Status::Empty);
+  }
+}
+
+// Signal-safe.
+static void insertSignalHandler(sys::SignalHandlerCallback FnPtr,
+                                void *Cookie) {
+  for (size_t I = 0; I < MaxSignalHandlerCallbacks; ++I) {
+    auto &SetMe = CallBacksToRun[I];
+    auto Expected = CallbackAndCookie::Status::Empty;
+    auto Desired = CallbackAndCookie::Status::Initializing;
+    if (!SetMe.Flag.compare_exchange_strong(Expected, Desired))
+      continue;
+    SetMe.Callback = FnPtr;
+    SetMe.Cookie = Cookie;
+    SetMe.Flag.store(CallbackAndCookie::Status::Initialized);
     return;
-  for (auto &I : *CallBacksToRun)
-    I.first(I.second);
-  CallBacksToRun->clear();
+  }
+  report_fatal_error("too many signal callbacks already registered");
 }
 
 static bool findModulesAndOffsets(void **StackTrace, int Depth,
@@ -68,7 +104,7 @@ static FormattedNumber format_ptr(void *PC) {
 LLVM_ATTRIBUTE_USED
 static bool printSymbolizedStackTrace(StringRef Argv0, void **StackTrace,
                                       int Depth, llvm::raw_ostream &OS) {
-  if (DisableSymbolication)
+  if (DisableSymbolicationFlag)
     return false;
 
   // Don't recursively invoke the llvm-symbolizer binary.