feat(tvix/eval): use backtrace-on-stack-overflow crate r/5340

The backtrace-on-stack-overflow create provides best-effort stack
traces when a stack overflow happens.  Since it's running on the
(usually tiny) signal alternate stack this isn't easy.

This is guarded by a new `backtrace_overflow` feature flag and never
enabled (even if that feature is selected) for release builds.  This
is strictly for debugging; there's crazy unsafe voodoo in there.

   https://lib.rs/crates/backtrace-on-stack-overflow

Example output:

```
Stack Overflow:
   0: backtrace_on_stack_overflow::handle_sigsegv
             at /home/amjoseph/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-on-stack-overflow-0.2.0/src/lib.rs:93:40
   1: <unknown>
   2: __rust_probestack
   3: tvix_eval::vm::VM::run_op
             at src/vm.rs:399
   4: tvix_eval::vm::VM::run
             at src/vm.rs:388:23
   5: tvix_eval::vm::VM::enter_frame
             at src/vm.rs:360:22
   6: tvix_eval::value::thunk::Thunk::force
             at src/value/thunk.rs:116:25
   7: tvix_eval::vm::VM::run_op
             at src/vm.rs:801:37
   8: tvix_eval::vm::VM::run
             at src/vm.rs:388:23
   9: tvix_eval::vm::VM::enter_frame
             at src/vm.rs:360:22
  10: tvix_eval::value::thunk::Thunk::force
             at src/value/thunk.rs:116:25
...
```

Signed-off-by: Adam Joseph <adam@westernsemico.com>
Change-Id: I1d8a2017f836be7bf91a2223e7adacb86fa1dbb2
Reviewed-on: https://cl.tvl.fyi/c/depot/+/7354
Tested-by: BuildkiteCI
Reviewed-by: tazjin <tazjin@tvl.su>

1 files changed, 9 insertions, 0 deletions

diff --git a/tvix/eval/src/vm.rs b/tvix/eval/src/vm.rs
index aa03ffd1eaae..41640a32b3c1 100644
--- a/tvix/eval/src/vm.rs
+++ b/tvix/eval/src/vm.rs
@@ -148,6 +148,15 @@ macro_rules! cmp_op {
 
 impl<'o> VM<'o> {
     pub fn new(nix_search_path: NixSearchPath, observer: &'o mut dyn RuntimeObserver) -> Self {
+        // Backtrace-on-stack-overflow is some seriously weird voodoo and
+        // very unsafe.  This double-guard prevents it from accidentally
+        // being enabled on release builds.
+        #[cfg(debug_assertions)]
+        #[cfg(feature = "backtrace_overflow")]
+        unsafe {
+            backtrace_on_stack_overflow::enable();
+        };
+
         Self {
             nix_search_path,
             observer,