Skip to content

Deadlock: TS-extended class +initialize (v8::Locker) vs cross-isolate ObjC→JS callbacks wedges the app (root cause of the 600s CI test timeouts) #420

Description

@edusperoni

Summary

The long-standing intermittent CI failure Asynchronous wait failed: exceeded 600 seconds with unfulfilled "Jasmine tests" expectation (recently e.g. run 30567093340, and reproduced locally 1-in-~15 runs) is a cross-isolate lock-order-inversion deadlock between the V8 isolate locks and the Objective-C class-initialization monitor, triggered by the synthesized +initialize that ClassBuilder installs on TypeScript-extended classes.

Captured live with sample on a wedged run (full thread samples available; trimmed stacks below).

The cycle

  • Main thread — holds Locker(main isolate) (outer ObjC→JS entry via ArgConverter::MethodCallback, ArgConverter.mm:386), runs JS which calls -[NSNotificationCenter postNotificationName:...]; the observer is a JS closure belonging to a worker isolate, so the ffi closure (ArgConverter.mm:303) tries to take Locker(worker isolate)blocks.
  • Worker thread — holds Locker(worker isolate) for the whole module run (WorkerWrapper::BackgroundLooperRuntime::RunModule), and its JS first-touches a TypeScript-extended ObjC class: objc acquires the class's +initialize monitor and runs the synthesized +initialize (ClassBuilder.mm:268), which does v8::Locker locker(isolate) on the class-creating (main) isolateblocks.
  • Second worker thread — piles up behind the class-init monitor (initializeNonMetaClass wait side).

Main holds A wants B; worker holds B (plus the objc class-init monitor) wants A. The app stays alive but the main thread never runs again — so the junit report is never delivered and the XCTest harness burns its full 600s. This also explains why the failure is timing-dependent (worker startup racing main-thread activity), why it clusters on slow/overloaded CI VMs, and why same-VM step retries tend to fail together.

Evidence

Main thread — holds main isolate lock, blocked taking a worker isolate's lock
Thread_112388225   DispatchQueue_1: com.apple.main-thread  (serial)
start + 6076
start_sim + 20
main  (in TestRunner) + 336  main.m:28
-[NativeScript runMainApplication] + 64  NativeScript.mm:45
tns::Tasks::Drain() + 116  Tasks.cpp:13
std::function<void ()>::operator()() const + 24  function.h:995
std::__function::__value_func<void ()>::operator()[abi:de200100]() const + 60  function.h:436
std::__function::__func<tns::MetadataBuilder::CFunctionCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0, void ()>::operator()() + 28  function.h:319
std::__function::__alloc_func<tns::MetadataBuilder::CFunctionCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0, void ()>::operator()[abi:de200100]() + 24  function.h:167
std::__invoke_r[abi:de200100]<void, tns::MetadataBuilder::CFunctionCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&>(tns::MetadataBuilder::CFunctionCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&) + 24  invoke.h:273
std::__invoke_void_return_wrapper<void, true>::__call[abi:de200100]<tns::MetadataBuilder::CFunctionCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&>(tns::MetadataBuilder::CFunctionCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&) + 24  invoke.h:251
decltype(std::declval<tns::MetadataBuilder::CFunctionCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&>()()) std::__invoke[abi:de200100]<tns::MetadataBuilder::CFunctionCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&>(tns::MetadataBuilder::CFunctionCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&) + 24  invoke.h:179
tns::MetadataBuilder::CFunctionCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0::operator()() const + 516  MetadataBuilder.mm:1006
tns::Interop::CallFunction(tns::CMethodCall&) + 24  Interop.mm:113
tns::Interop::CallFunctionInternal(tns::MethodCall&) + 972  Interop.mm:1653
ffi_call_int + 1520
ffi_call_SYSV + 80
UIApplicationMain + 124
-[UIApplication _run] + 796
GSEventRunModal + 164
CFRunLoopRunSpecific + 536
__CFRunLoopRun + 1788
__CFRunLoopDoTimers + 284
__CFRunLoopDoTimer + 916
__CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__ + 28
__NSFireTimer + 56
ffi_closure_SYSV + 56
ffi_closure_SYSV_inner + 892
tns::ArgConverter::MethodCallback(ffi_cif*, void*, void**, void*) + 2292  ArgConverter.mm:386
v8::Function::Call(v8::Isolate*, v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) + 452
v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::DirectHandle<v8::internal::Object>, v8::internal::DirectHandle<v8::internal::Object>, v8::base::Vector<v8::internal::DirectHandle<v8::internal::Object> const>) + 136
???  load address 0x106160000 + 0x4bf284
???  load address 0x106160000 + 0x88a5b8
???  load address 0x106160000 + 0x88a8c8
???  load address 0x106160000 + 0x88d04c
???  load address 0x106160000 + 0x88d04c
???  load address 0x106160000 + 0x88d04c
???  load address 0x106160000 + 0x88eb8c
tns::MetadataBuilder::MethodCallback(v8::FunctionCallbackInfo<v8::Value> const&) + 544  MetadataBuilder.mm:766
tns::MetadataBuilder::InvokeMethod(v8::Local<v8::Context>, tns::MethodMeta const*, v8::Local<v8::Object>, tns::V8Args&, std::basic_string<char>, bool) + 456  MetadataBuilder.mm:970
tns::ArgConverter::Invoke(v8::Local<v8::Context>, objc_class*, v8::Local<v8::Object>, tns::V8Args&, tns::MethodMeta const*, bool) + 3376  ArgConverter.mm:115
tns::Interop::CallFunction(tns::ObjCMethodCall&) + 24  Interop.mm:117
tns::Interop::CallFunctionInternal(tns::MethodCall&) + 972  Interop.mm:1653
ffi_call_int + 1520
ffi_call_SYSV + 80
-[NSNotificationCenter postNotificationName:object:userInfo:] + 88
_CFXNotificationPost + 696
_CFXRegistrationPost + 404
___CFXRegistrationPost_block_invoke + 88
__CFNOTIFICATIONCENTER_IS_CALLING_OUT_TO_AN_OBSERVER__ + 120
ffi_closure_SYSV + 56
ffi_closure_SYSV_inner + 892
tns::ArgConverter::MethodCallback(ffi_cif*, void*, void**, void*) + 136  ArgConverter.mm:303
v8::Locker::Locker(v8::Isolate*) + 36  v8-locker.h:114
v8::Locker::Locker(v8::Isolate*) + 36  v8-locker.h:114
v8::Locker::Initialize(v8::Isolate*) + 92
absl::Mutex::LockSlow(absl::MuHowS const*, absl::Condition const*, int) + 76
absl::Mutex::LockSlowWithDeadline(absl::MuHowS const*, absl::Condition const*, absl::synchronization_internal::KernelTimeout, int) + 208
absl::Mutex::LockSlowLoop(absl::SynchWaitParams*, int) + 668
absl::Mutex::Block(absl::base_internal::PerThreadSynch*) + 88
AbslInternalPerThreadSemWait + 76
absl::synchronization_internal::PthreadWaiter::Wait(absl::synchronization_internal::KernelTimeout) + 120
_pthread_cond_wait + 976
__psynch_cvwait + 8
Worker thread — inside synthesized +initialize (holding objc class-init monitor), blocked taking the main isolate's lock
Thread_112394946   DispatchQueue_1636: NSOperationQueue 0x104605d00 (QOS: UNSPECIFIED)  (concurrent)
start_wqthread + 8
_pthread_wqthread + 228
_dispatch_worker_thread2 + 232
_dispatch_root_queue_drain + 336
_dispatch_async_redirect_invoke + 852
_dispatch_continuation_pop + 776
_dispatch_client_callout + 12
_dispatch_block_async_invoke2 + 104
__NSOQSchedule_f + 160
__NSOPERATIONQUEUE_IS_STARTING_AN_OPERATION__ + 12
-[NSOperation start] + 612
__NSOPERATION_IS_INVOKING_MAIN__ + 12
-[NSBlockOperation main] + 92
<deduplicated_symbol> + 16
invocation function for block in tns::WorkerWrapper::Start(std::shared_ptr<v8::Persistent<v8::Value, v8::NonCopyablePersistentTraits<v8::Value>>>, std::function<v8::Isolate* ()>, int) + 84  WorkerWrapper.mm:57
tns::WorkerWrapper::BackgroundLooper(std::function<v8::Isolate* ()>) + 112  WorkerWrapper.mm:109
std::function<v8::Isolate* ()>::operator()() const + 24  function.h:995
std::__function::__value_func<v8::Isolate* ()>::operator()[abi:de200100]() const + 60  function.h:436
std::__function::__func<tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0, v8::Isolate* ()>::operator()() + 28  function.h:319
std::__function::__alloc_func<tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0, v8::Isolate* ()>::operator()[abi:de200100]() + 24  function.h:167
std::__invoke_r[abi:de200100]<v8::Isolate*, tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&>(tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&) + 24  invoke.h:273
std::__invoke_void_return_wrapper<v8::Isolate*, false>::__call[abi:de200100]<tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&>(tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&) + 24  invoke.h:243
decltype(std::declval<tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&>()()) std::__invoke[abi:de200100]<tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&>(tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&) + 24  invoke.h:179
tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0::operator()() const + 1164  Worker.mm:228
tns::Runtime::RunModule(std::basic_string<char>) + 124  Runtime.mm:468
tns::ModuleInternal::RunModule(v8::Isolate*, std::basic_string<char>) + 1532  ModuleInternal.mm:185
v8::Function::Call(v8::Isolate*, v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) + 452
v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::DirectHandle<v8::internal::Object>, v8::internal::DirectHandle<v8::internal::Object>, v8::base::Vector<v8::internal::DirectHandle<v8::internal::Object> const>) + 136
???  load address 0x106160000 + 0x4bf284
???  load address 0x106160000 + 0x88a5b8
???  load address 0x106160000 + 0x88a8c8
???  load address 0x106160000 + 0x88d04c
???  load address 0x106160000 + 0x88eb8c
tns::ModuleInternal::RequireCallback(v8::FunctionCallbackInfo<v8::Value> const&) + 2720  ModuleInternal.mm:308
tns::ModuleInternal::LoadImpl(v8::Isolate*, std::basic_string<char> const&, std::basic_string<char> const&, bool&) + 1808  ModuleInternal.mm:420
tns::ModuleInternal::LoadModule(v8::Isolate*, std::basic_string<char> const&, std::basic_string<char> const&) + 3752  ModuleInternal.mm:546
v8::Function::Call(v8::Isolate*, v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) + 452
v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::DirectHandle<v8::internal::Object>, v8::internal::DirectHandle<v8::internal::Object>, v8::base::Vector<v8::internal::DirectHandle<v8::internal::Object> const>) + 136
???  load address 0x106160000 + 0x4bf284
???  load address 0x106160000 + 0x88a5b8
???  load address 0x106160000 + 0x88a8c8
???  load address 0x106160000 + 0x88d04c
???  load address 0x106160000 + 0x88d04c
???  load address 0x106160000 + 0x88d04c
???  load address 0x106160000 + 0x88d04c
???  load address 0x106160000 + 0x88eb8c
tns::MetadataBuilder::MethodCallback(v8::FunctionCallbackInfo<v8::Value> const&) + 656  MetadataBuilder.mm:767
tns::MetadataBuilder::InvokeMethod(v8::Local<v8::Context>, tns::MethodMeta const*, v8::Local<v8::Object>, tns::V8Args&, std::basic_string<char>, bool) + 456  MetadataBuilder.mm:970
tns::ArgConverter::Invoke(v8::Local<v8::Context>, objc_class*, v8::Local<v8::Object>, tns::V8Args&, tns::MethodMeta const*, bool) + 3376  ArgConverter.mm:115
tns::Interop::CallFunction(tns::ObjCMethodCall&) + 24  Interop.mm:117
tns::Interop::CallFunctionInternal(tns::MethodCall&) + 972  Interop.mm:1653
ffi_call_int + 1520
ffi_call_SYSV + 80
_objc_msgSend_uncached + 64
lookUpImpOrForward + 208
initializeAndMaybeRelock(objc_class*, objc_object*, locker_mixin<lockdebug::lock_mixin<objc_lock_base_t>>&, bool) + 216
initializeNonMetaClass + 544
CALLING_SOME_+initialize_METHOD + 20
invocation function for block in tns::ClassBuilder::RegisterNativeTypeScriptExtendsFunction(v8::Local<v8::Context>)::$_0::operator()(v8::FunctionCallbackInfo<v8::Value> const&) const + 76  ClassBuilder.mm:272
v8::Locker::Locker(v8::Isolate*) + 36  v8-locker.h:114
v8::Locker::Locker(v8::Isolate*) + 36  v8-locker.h:114
v8::Locker::Initialize(v8::Isolate*) + 92
absl::Mutex::LockSlow(absl::MuHowS const*, absl::Condition const*, int) + 76
absl::Mutex::LockSlowWithDeadline(absl::MuHowS const*, absl::Condition const*, absl::synchronization_internal::KernelTimeout, int) + 208
absl::Mutex::LockSlowLoop(absl::SynchWaitParams*, int) + 668
absl::Mutex::Block(absl::base_internal::PerThreadSynch*) + 88
AbslInternalPerThreadSemWait + 76
absl::synchronization_internal::PthreadWaiter::Wait(absl::synchronization_internal::KernelTimeout) + 120
_pthread_cond_wait + 976
__psynch_cvwait + 8

Second worker thread — queued behind the class-init monitor
Thread_112392210   DispatchQueue_1636: NSOperationQueue 0x104605d00 (QOS: UNSPECIFIED)  (concurrent)
start_wqthread + 8
_pthread_wqthread + 228
_dispatch_worker_thread2 + 232
_dispatch_root_queue_drain + 336
_dispatch_async_redirect_invoke + 852
_dispatch_continuation_pop + 776
_dispatch_client_callout + 12
_dispatch_block_async_invoke2 + 104
__NSOQSchedule_f + 160
__NSOPERATIONQUEUE_IS_STARTING_AN_OPERATION__ + 12
-[NSOperation start] + 612
__NSOPERATION_IS_INVOKING_MAIN__ + 12
-[NSBlockOperation main] + 92
<deduplicated_symbol> + 16
invocation function for block in tns::WorkerWrapper::Start(std::shared_ptr<v8::Persistent<v8::Value, v8::NonCopyablePersistentTraits<v8::Value>>>, std::function<v8::Isolate* ()>, int) + 84  WorkerWrapper.mm:57
tns::WorkerWrapper::BackgroundLooper(std::function<v8::Isolate* ()>) + 112  WorkerWrapper.mm:109
std::function<v8::Isolate* ()>::operator()() const + 24  function.h:995
std::__function::__value_func<v8::Isolate* ()>::operator()[abi:de200100]() const + 60  function.h:436
std::__function::__func<tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0, v8::Isolate* ()>::operator()() + 28  function.h:319
std::__function::__alloc_func<tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0, v8::Isolate* ()>::operator()[abi:de200100]() + 24  function.h:167
std::__invoke_r[abi:de200100]<v8::Isolate*, tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&>(tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&) + 24  invoke.h:273
std::__invoke_void_return_wrapper<v8::Isolate*, false>::__call[abi:de200100]<tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&>(tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&) + 24  invoke.h:243
decltype(std::declval<tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&>()()) std::__invoke[abi:de200100]<tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&>(tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0&) + 24  invoke.h:179
tns::Worker::ConstructorCallback(v8::FunctionCallbackInfo<v8::Value> const&)::$_0::operator()() const + 1164  Worker.mm:228
tns::Runtime::RunModule(std::basic_string<char>) + 124  Runtime.mm:468
tns::ModuleInternal::RunModule(v8::Isolate*, std::basic_string<char>) + 1532  ModuleInternal.mm:185
v8::Function::Call(v8::Isolate*, v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) + 452
v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::DirectHandle<v8::internal::Object>, v8::internal::DirectHandle<v8::internal::Object>, v8::base::Vector<v8::internal::DirectHandle<v8::internal::Object> const>) + 136
???  load address 0x106160000 + 0x4bf284
???  load address 0x106160000 + 0x88a5b8
???  load address 0x106160000 + 0x88a8c8
???  load address 0x106160000 + 0x88d04c
???  load address 0x106160000 + 0x88eb8c
tns::ModuleInternal::RequireCallback(v8::FunctionCallbackInfo<v8::Value> const&) + 2720  ModuleInternal.mm:308
tns::ModuleInternal::LoadImpl(v8::Isolate*, std::basic_string<char> const&, std::basic_string<char> const&, bool&) + 1808  ModuleInternal.mm:420
tns::ModuleInternal::LoadModule(v8::Isolate*, std::basic_string<char> const&, std::basic_string<char> const&) + 3752  ModuleInternal.mm:546
v8::Function::Call(v8::Isolate*, v8::Local<v8::Context>, v8::Local<v8::Value>, int, v8::Local<v8::Value>*) + 452
v8::internal::Execution::Call(v8::internal::Isolate*, v8::internal::DirectHandle<v8::internal::Object>, v8::internal::DirectHandle<v8::internal::Object>, v8::base::Vector<v8::internal::DirectHandle<v8::internal::Object> const>) + 136
???  load address 0x106160000 + 0x4bf284
???  load address 0x106160000 + 0x88a5b8
???  load address 0x106160000 + 0x88a8c8
???  load address 0x106160000 + 0x88d04c
???  load address 0x106160000 + 0x88d04c
???  load address 0x106160000 + 0x88d04c
???  load address 0x106160000 + 0x88d04c
???  load address 0x106160000 + 0x88eb8c
tns::MetadataBuilder::MethodCallback(v8::FunctionCallbackInfo<v8::Value> const&) + 656  MetadataBuilder.mm:767
tns::MetadataBuilder::InvokeMethod(v8::Local<v8::Context>, tns::MethodMeta const*, v8::Local<v8::Object>, tns::V8Args&, std::basic_string<char>, bool) + 456  MetadataBuilder.mm:970
tns::ArgConverter::Invoke(v8::Local<v8::Context>, objc_class*, v8::Local<v8::Object>, tns::V8Args&, tns::MethodMeta const*, bool) + 3376  ArgConverter.mm:115
tns::Interop::CallFunction(tns::ObjCMethodCall&) + 24  Interop.mm:117
tns::Interop::CallFunctionInternal(tns::MethodCall&) + 972  Interop.mm:1653
ffi_call_int + 1520
ffi_call_SYSV + 80
_objc_msgSend_uncached + 64
lookUpImpOrForward + 208
initializeAndMaybeRelock(objc_class*, objc_object*, locker_mixin<lockdebug::lock_mixin<objc_lock_base_t>>&, bool) + 216
initializeNonMetaClass + 104
_os_unfair_lock_lock_slow + 172
__ulock_wait2 + 8

Possible fix directions

  1. Make the synthesized +initialize JS-free. The block only reads ObjCExposedMethods and prototype off the ctor function — data that is fully known at class-build time. Capturing it natively when the class is registered removes the v8::Locker from +initialize entirely, which breaks the cycle at its weakest link.
  2. Force +initialize eagerly at registration (e.g. [extendedClass self] right after objc_registerClassPair on the creating thread). Every later first-touch becomes a no-op, so no thread ever enters objc class-init while holding a foreign isolate's lock.
  3. More broadly, any ObjC→JS entry that takes a v8::Locker while the calling thread may hold another isolate's lock (notification observers, exposed methods invoked cross-thread) has this shape; 1 + 2 fix the concrete cycle observed.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions