CTS test aborts when used with VVL

[ziga-lunarg]

The test is dEQP-VK.binding_model.shader_access.primary_cmd_buf.bind.sampler_mutable.fragment.single_descriptor.1d

terminate called after throwing an instance of 'std::system_error'
  what():  Resource deadlock avoided

Thread 1 "deqp-vk" received signal SIGABRT, Aborted.
Download failed: Invalid argument.  Continuing without source file ./nptl/./nptl/pthread_kill.c.
__pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
warning: 44	./nptl/pthread_kill.c: No such file or directory
(gdb) where
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff764527e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff76288ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff7aa5ff5 in __gnu_cxx::__verbose_terminate_handler () at ../../../../src/libstdc++-v3/libsupc++/vterminate.cc:95
#6  0x00007ffff7abb0da in __cxxabiv1::__terminate (handler=<optimized out>) at ../../../../src/libstdc++-v3/libsupc++/eh_terminate.cc:48
#7  0x00007ffff7aa58e6 in __cxa_call_terminate (ue_header_in=0x55555bb46f20) at ../../../../src/libstdc++-v3/libsupc++/eh_call.cc:56
#8  0x00007ffff7aba8ba in __cxxabiv1::__gxx_personality_v0 (version=<optimized out>, actions=6, exception_class=5138137972254386944, ue_header=0x55555bb46f20, context=0x7fffffffd100)
    at ../../../../src/libstdc++-v3/libsupc++/eh_personality.cc:692
#9  0x00007ffff79f4b06 in _Unwind_RaiseException_Phase2 (exc=exc@entry=0x55555bb46f20, context=context@entry=0x7fffffffd100, frames_p=frames_p@entry=0x7fffffffd1f0)
    at ../../../src/libgcc/unwind.inc:64
#10 0x00007ffff79f51f1 in _Unwind_RaiseException (exc=0x55555bb46f20) at ../../../src/libgcc/unwind.inc:136
#11 0x00007ffff7abb384 in __cxxabiv1::__cxa_throw (obj=<optimized out>, tinfo=0x7ffff7c704f8 <typeinfo for std::system_error>, 
    dest=0x7ffff7aecb20 <std::system_error::~system_error()>) at ../../../../src/libstdc++-v3/libsupc++/eh_throw.cc:93
#12 0x00007ffff7aaa2a5 in std::__throw_system_error (__i=35) at ../../../../../src/libstdc++-v3/src/c++11/system_error.cc:595
#13 0x00007fffe0ac2343 in std::__shared_mutex_pthread::lock_shared (this=0x7fffe3621cc0 <vvl::dispatch::device_mutex>) at /usr/include/c++/13/shared_mutex:235
#14 0x00007fffe0ac23a0 in std::shared_mutex::lock_shared (this=0x7fffe3621cc0 <vvl::dispatch::device_mutex>) at /usr/include/c++/13/shared_mutex:429
#15 0x00007fffe0ac364d in std::shared_lock<std::shared_mutex>::shared_lock (this=0x7fffffffd460, __m=...) at /usr/include/c++/13/shared_mutex:741
#16 0x00007fffe0b6fcbc in vvl::dispatch::GetDeviceFromKey (key=0x55555bb2aea0) at /home/nnn/Projects/Vulkan-ValidationLayers/layers/chassis/dispatch_object_manual.cpp:568
#17 0x00007fffe0b6fdb0 in vvl::dispatch::GetData (device=0x55555c12db30) at /home/nnn/Projects/Vulkan-ValidationLayers/layers/chassis/dispatch_object_manual.cpp:590
#18 0x00007fffe16706c5 in DispatchFreeCommandBuffers (device=0x55555c12db30, commandPool=0x170000000017, commandBufferCount=1, pCommandBuffers=0x55555c095730)
    at /home/nnn/Projects/Vulkan-ValidationLayers/layers/vulkan/generated/dispatch_functions.h:528
#19 0x00007fffe16725e2 in gpuav::QueueSubState::~QueueSubState (this=0x55555c0956e0, __in_chrg=<optimized out>)
    at /home/nnn/Projects/Vulkan-ValidationLayers/layers/gpuav/resources/gpuav_state_trackers.cpp:376
#20 0x00007fffe16726ba in gpuav::QueueSubState::~QueueSubState (this=0x55555c0956e0, __in_chrg=<optimized out>)
    at /home/nnn/Projects/Vulkan-ValidationLayers/layers/gpuav/resources/gpuav_state_trackers.cpp:387
#21 0x00007fffe0f083e8 in std::default_delete<vvl::QueueSubState>::operator() (this=0x55555c08dfe8, __ptr=0x55555c0956e0) at /usr/include/c++/13/bits/unique_ptr.h:99
#22 0x00007fffe0f02d0a in std::unique_ptr<vvl::QueueSubState, std::default_delete<vvl::QueueSubState> >::~unique_ptr (this=0x55555c08dfe8, __in_chrg=<optimized out>)
    at /usr/include/c++/13/bits/unique_ptr.h:404
#23 0x00007fffe0f15ada in std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::QueueSubState, std::default_delete<vvl::QueueSubState> > >::~pair (this=0x55555c08dfe0, 
    __in_chrg=<optimized out>) at /usr/include/c++/13/bits/stl_pair.h:187
#24 0x00007fffe0f1050e in std::__new_allocator<std::_Rb_tree_node<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::QueueSubState, std::default_delete<vvl::QueueSubState> > > > >::destroy<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::QueueSubState, std::default_delete<vvl::QueueSubState> > > > (__p=0x55555c08dfe0, this=0x55555bbe58b8)
    at /usr/include/c++/13/bits/new_allocator.h:198
#25 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::QueueSubState, std::default_delete<vvl::QueueSubState> > > > > >::destroy<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::QueueSubState, std::default_delete<vvl::QueueSubState> > > > (__p=0x55555c08dfe0, __a=...)
    at /usr/include/c++/13/bits/alloc_traits.h:558
#26 std::_Rb_tree<LayerObjectTypeId, std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::QueueSubState, std::default_delete<vvl::QueueSubState> > >, std::_Select1st<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::QueueSubState, std::default_delete<vvl::QueueSubState> > > >, std::less<LayerObjectTypeId>, std::allocator<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::QueueSubState, std::default_delete<vvl::QueueSubState> > > > >::_M_destroy_node (this=0x55555bbe58b8, __p=0x55555c08dfc0)
    at /usr/include/c++/13/bits/stl_tree.h:625
#27 0x00007fffe0f0c7e1 in std::_Rb_tree<LayerObjectTypeId, std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::QueueSubState, std::default_delete<vvl::QueueSubState> > >, std::_Select1st<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::QueueSubState, std::default_delete<vvl::QueueSubState> > > >, std::less<LayerObjectTypeId>, std::allocator<std::pair<Laye--Type <RET> for more, q to quit, c to continue without paging--

[spencer-lunarg]

@ziga-lunarg what machine did you hit this with, it works with AMD and Intel Mesa for me right now

[spencer-lunarg]

unless this requires the latest CTS version to hit (I have slightly older)

[ziga-lunarg]

RADV, it shouldn't require latest CTS, there are multiple unrelated tests that hit this.

[pborsutzki]

I think I am observing a similar issue during __run_exit_handlers/vvl::dispatch::FreeAllData. Interestingly it does not happen for most of my tests, only for one that involes loading my lib from python (3.11.13 x86_64). I used vvl in version 1.3.296 before, which didn't show this problem.

To reproduce the issue I am using vvl's from the vulkan-sdk-1.4.321.0 tag compiled through gcc 11.2.1 on a RHEL8 on x86_64. GPU is an NVIDIA RTX A4000 with a 535.230.02 driver.

Callstack:

#0  0x00007f053ad6aaff in raise () from /lib64/libc.so.6
#1  0x00007f053ad3dea5 in abort () from /lib64/libc.so.6
#2  0x00007f0515ad709b in __gnu_cxx::__verbose_terminate_handler() [clone .cold.1] () from /lib64/libstdc++.so.6
#3  0x00007f0515add53c in __cxxabiv1::__terminate(void (*)()) () from /lib64/libstdc++.so.6
#4  0x00007f0515adc559 in __cxa_call_terminate () from /lib64/libstdc++.so.6
#5  0x00007f0515adced8 in __gxx_personality_v0 () from /lib64/libstdc++.so.6
#6  0x00007f051583fb03 in _Unwind_RaiseException_Phase2 () from /lib64/libgcc_s.so.1
#7  0x00007f0515840071 in _Unwind_RaiseException () from /lib64/libgcc_s.so.1
#8  0x00007f0515add7eb in __cxa_throw () from /lib64/libstdc++.so.6
#9  0x00007f0515ad923b in std::__throw_system_error(int) [clone .cold.28] () from /lib64/libstdc++.so.6
#10 0x00007f04568b3ea1 in std::__shared_mutex_pthread::lock_shared (this=0x7f0457d3ab60 <vvl::dispatch::device_mutex>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/shared_mutex:232
#11 std::shared_mutex::lock_shared (this=0x7f0457d3ab60 <vvl::dispatch::device_mutex>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/shared_mutex:426
#12 std::shared_lock<std::shared_mutex>::shared_lock (__m=..., this=<synthetic pointer>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/shared_mutex:727
#13 vvl::dispatch::GetDeviceFromKey (key=0x556a71d59ca0) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/chassis/dispatch_object_manual.cpp:564
#14 0x00007f0456e98c13 in DispatchDestroyBuffer (pAllocator=0x0, buffer=0xa000000000a, device=0x556a71f61590) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/vulkan/generated/dispatch_functions.h:306
#15 gpuav::gpuVkDestroyBuffer (device=0x556a71f61590, buffer=0xa000000000a, pAllocator=0x0) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/gpuav/core/gpuav_setup.cpp:135
#16 0x00007f0456b8d9bb in vmaDestroyBuffer (allocator=0x556a71f57430, buffer=<optimized out>, allocation=0x556a77101598) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/external/vma/vk_mem_alloc.h:16359
#17 0x00007f0456edaaca in gpuav::vko::Buffer::Destroy (this=0x556a762d1f00) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/gpuav/resources/gpuav_vulkan_objects.cpp:218
#18 gpuav::vko::GpuResourcesManager::BufferCache::DestroyBuffers (this=this@entry=0x556a71b74e08) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/gpuav/resources/gpuav_vulkan_objects.cpp:496
#19 0x00007f0456edab19 in gpuav::vko::GpuResourcesManager::BufferCache::~BufferCache (this=0x556a71b74e08, __in_chrg=<optimized out>) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/gpuav/resources/gpuav_vulkan_objects.cpp:413
#20 0x00007f0456ed5881 in gpuav::vko::GpuResourcesManager::~GpuResourcesManager (this=0x556a71b74de8, __in_chrg=<optimized out>) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/./gpuav/resources/gpuav_vulkan_objects.h:98
#21 gpuav::CommandBufferSubState::~CommandBufferSubState (this=0x556a71b74d00, __in_chrg=<optimized out>) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/gpuav/resources/gpuav_state_trackers.cpp:39
#22 0x00007f0456ed5a69 in gpuav::CommandBufferSubState::~CommandBufferSubState (this=0x556a71b74d00, __in_chrg=<optimized out>) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/gpuav/resources/gpuav_state_trackers.cpp:39
#23 0x00007f0456f96b92 in std::default_delete<vvl::CommandBufferSubState>::operator() (__ptr=<optimized out>, this=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:85
#24 std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> >::~unique_ptr (this=0x556a7627bbc8, __in_chrg=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:361
#25 std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > >::~pair (this=0x556a7627bbc0, __in_chrg=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_pair.h:211
#26 __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > > >::destroy<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > > (__p=0x556a7627bbc0, this=0x556a762781d0) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/ext/new_allocator.h:168
#27 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > > > >::destroy<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > > (__p=0x556a7627bbc0, __a=...) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/alloc_traits.h:535
#28 std::_Rb_tree<LayerObjectTypeId, std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > >, std::_Select1st<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > >, std::less<LayerObjectTypeId>, std::allocator<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > > >::_M_destroy_node (__p=0x556a7627bba0, this=0x556a762781d0) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_tree.h:623
#29 std::_Rb_tree<LayerObjectTypeId, std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > >, std::_Select1st<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > >, std::less<LayerObjectTypeId>, std::allocator<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > > >::_M_drop_node (__p=0x556a7627bba0, this=0x556a762781d0) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_tree.h:631
#30 std::_Rb_tree<LayerObjectTypeId, std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > >, std::_Select1st<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > >, std::less<LayerObjectTypeId>, std::allocator<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > > >::_M_erase_aux (__position=..., this=0x556a762781d0) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_tree.h:2487
#31 std::_Rb_tree<LayerObjectTypeId, std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > >, std::_Select1st<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > >, std::less<LayerObjectTypeId>, std::allocator<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > > >::_M_erase_aux (__last=Python Exception <class 'ValueError'> Unsupported implementation for unique_ptr: std::__uniq_ptr_data<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState>, true, true>:
{first = LayerObjectTypeObjectTracker, second = {_M_t = {<std::__uniq_ptr_impl<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> >> = {_M_t = std::tuple containing = {[1] = 0x28, [2] = {<No data fields>}}}, <No data fields>}}}, __first=Python Exception <class 'ValueError'> Unsupported implementation for unique_ptr: std::__uniq_ptr_data<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState>, true, true>:
{first = LayerObjectTypeObjectTracker, second = {_M_t = {<std::__uniq_ptr_impl<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> >> = {_M_t = std::tuple containing = {[1] = 0x28, [2] = {<No data fields>}}}, <No data fields>}}}, this=0x556a762781d0) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_tree.h:2501
#32 std::_Rb_tree<LayerObjectTypeId, std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > >, std::_Select1st<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > >, std::less<LayerObjectTypeId>, std::allocator<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > > >::erase (__x=<synthetic pointer>: LayerObjectTypeGpuAssisted, this=0x556a762781d0) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_tree.h:2512
#33 std::map<LayerObjectTypeId, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> >, std::less<LayerObjectTypeId>, std::allocator<std::pair<LayerObjectTypeId const, std::unique_ptr<vvl::CommandBufferSubState, std::default_delete<vvl::CommandBufferSubState> > > > >::erase (__x=<synthetic pointer>: LayerObjectTypeGpuAssisted, this=0x556a762781d0) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_map.h:1069
#34 vvl::SubStateManager<vvl::CommandBufferSubState>::RemoveSubState (id=LayerObjectTypeGpuAssisted, this=0x556a762781d0) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/./state_tracker/state_object.h:154
#35 operator() (state=..., __closure=<optimized out>) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/state_tracker/state_tracker.cpp:64
#36 vvl::DeviceState::ForEachShared<vvl::CommandBuffer, vvl::DeviceState::RemoveSubState(LayerObjectTypeId)::<lambda(std::shared_ptr<vvl::CommandBuffer>)> > (fn=..., this=0x556a71d6f3c0) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/./state_tracker/state_tracker.h:579
#37 vvl::DeviceState::RemoveSubState (this=0x556a71d6f3c0, id=LayerObjectTypeGpuAssisted) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/state_tracker/state_tracker.cpp:64
#38 0x00007f0456e9b1e1 in vvl::DeviceProxy::~DeviceProxy (this=0x556a71c323c0, __in_chrg=<optimized out>) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/./state_tracker/state_tracker.h:2062
#39 gpuav::GpuShaderInstrumentor::~GpuShaderInstrumentor (this=this@entry=0x556a71c323c0, __in_chrg=<optimized out>) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/./gpuav/instrumentation/gpuav_shader_instrumentor.h:69
#40 0x00007f0456e9b308 in gpuav::Validator::~Validator (this=0x556a71c323c0, __in_chrg=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/ext/new_allocator.h:89
#41 gpuav::Validator::~Validator (this=0x556a71c323c0, __in_chrg=<optimized out>) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/./gpuav/core/gpuav.h:61
#42 0x00007f04568af5a5 in std::default_delete<vvl::base::Device>::operator() (__ptr=<optimized out>, this=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:79
#43 std::unique_ptr<vvl::base::Device, std::default_delete<vvl::base::Device> >::~unique_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:361
#44 __gnu_cxx::new_allocator<std::unique_ptr<vvl::base::Device, std::default_delete<vvl::base::Device> > >::destroy<std::unique_ptr<vvl::base::Device, std::default_delete<vvl::base::Device> > > (__p=<optimized out>, this=0x556a71d4b398) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/ext/new_allocator.h:168
#45 std::allocator_traits<std::allocator<std::unique_ptr<vvl::base::Device, std::default_delete<vvl::base::Device> > > >::destroy<std::unique_ptr<vvl::base::Device, std::default_delete<vvl::base::Device> > > (__p=<optimized out>, __a=...) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/alloc_traits.h:535
#46 std::vector<std::unique_ptr<vvl::base::Device, std::default_delete<vvl::base::Device> >, std::allocator<std::unique_ptr<vvl::base::Device, std::default_delete<vvl::base::Device> > > >::pop_back (this=0x556a71d4b398) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_vector.h:1229
#47 vvl::dispatch::Device::~Device (this=0x556a71d48a80, __in_chrg=<optimized out>) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/chassis/dispatch_object_manual.cpp:812
#48 0x00007f04568afbd5 in std::default_delete<vvl::dispatch::Device>::operator() (this=<optimized out>, __ptr=0x556a71d48a80) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:79
#49 std::default_delete<vvl::dispatch::Device>::operator() (__ptr=0x556a71d48a80, this=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:79
#50 std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> >::~unique_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:361
#51 std::pair<void*, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > >::~pair (this=<optimized out>, __in_chrg=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/stl_pair.h:211
#52 __gnu_cxx::new_allocator<std::pair<void* const, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > > >::destroy<std::pair<void*, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > > > (__p=<optimized out>, this=<optimized out>) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/ext/new_allocator.h:168
#53 std::allocator_traits<std::allocator<std::pair<void* const, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > > > >::destroy<std::pair<void*, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > > > (__p=<optimized out>, __a=...) at /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/alloc_traits.h:535
#54 phmap::allocator_traits<std::allocator<std::pair<void* const, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > > > >::destroy_impl<std::allocator<std::pair<void* const, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > > >, std::pair<void*, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > > > (p=<optimized out>, a=...) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/external/parallel_hashmap/phmap_base.h:1394
#55 phmap::allocator_traits<std::allocator<std::pair<void* const, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > > > >::destroy<std::pair<void*, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > > > (p=<optimized out>, a=...) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/external/parallel_hashmap/phmap_base.h:1356
#56 phmap::priv::map_slot_policy<void*, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > >::destroy<std::allocator<std::pair<void* const, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > > > > (slot=<optimized out>, alloc=<optimized out>) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/external/parallel_hashmap/phmap_base.h:4081
#57 phmap::priv::FlatHashMapPolicy<void*, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > >::destroy<std::allocator<std::pair<void* const, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > > > > (slot=<optimized out>, alloc=<optimized out>) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/external/parallel_hashmap/phmap.h:3925
#58 phmap::priv::hash_policy_traits<phmap::priv::FlatHashMapPolicy<void*, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > >, void>::destroy<std::allocator<std::pair<void* const, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > > > > (slot=<optimized out>, alloc=<optimized out>) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/external/parallel_hashmap/phmap_base.h:445
#59 phmap::priv::raw_hash_set<phmap::priv::FlatHashMapPolicy<void*, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > >, phmap::Hash<void*>, std::equal_to<void*>, std::allocator<std::pair<void* const, std::unique_ptr<vvl::dispatch::Device, std::default_delete<vvl::dispatch::Device> > > > >::clear (this=0x7f0457d3ab20 <vvl::dispatch::device_data>) at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/external/parallel_hashmap/phmap.h:1168
#60 vvl::dispatch::FreeAllData () at /home/users/pbi1/artifact_builder/vulkan_validation/vulkan_validation-1.4.321.0/layers/chassis/dispatch_object_manual.cpp:594
#61 0x00007f053ad6d29c in __run_exit_handlers () from /lib64/libc.so.6
#62 0x00007f053ad6d3d0 in exit () from /lib64/libc.so.6
#63 0x00007f053ad56d8c in __libc_start_main () from /lib64/libc.so.6
#64 0x0000556a70ab474e in ?? ()
#65 0x00007ffccea4af88 in ?? ()
#66 0x00007f053c4a1100 in ?? () from /lib64/ld-linux-x86-64.so.2
#67 0x0000000000000004 in ?? ()
#68 0x00007ffccea4bd7d in ?? ()
#69 0x00007ffccea4be04 in ?? ()
#70 0x00007ffccea4be07 in ?? ()
#71 0x00007ffccea4be0e in ?? ()
#72 0x0000000000000000 in ?? ()

[ziga-lunarg]

This issue was caused by 86332ca

[spencer-lunarg]

This issue was caused by 86332ca

@ziga-lunarg you are saying if you revert that, it fixes the crashes you have? (only asking because the patch fixed a previous threading issues in ANGLE #9181)

[ziga-lunarg]

Yes

[pborsutzki]

I can confirm @ziga-lunarg's analysis, reverting 86332ca also fixed the issue for me.

[pborsutzki]

Not sure if it is a 100% related, but after reverting I am running into another issue, and to me it looks like the one that the reverted commit should have fixed. The callstack looks like this:

#0  0x00007feb91770525 in _Py_DumpExtensionModules () from /lib64/libpython3.11.so.1.0
#1  0x00007feb917707ce in faulthandler_fatal_error () from /lib64/libpython3.11.so.1.0
#2  <signal handler called>
#3  0x00007feaa282ad97 in vvl::dispatch::GetDeviceFromKey (key=0x56004d27abc0) at vulkan_validation-1.4.321.0/layers/chassis/dispatch_object_manual.cpp:567
#4  0x00007feaa281f5f9 in vulkan_layer_chassis::CreateShaderModule (device=0x56004db67140, pCreateInfo=0x7fe9e47f79e0, pAllocator=0x0, pShaderModule=0x7fe9e47f78a0) at vulkan_validation-1.4.321.0/layers/chassis/chassis_manual.cpp:844

At the time of failure, I can observe multiple threads sitting/failing in the same callstack. This is most likely because I am creating multiple VkShaderModule's at the same time in different threads.

It also looks like the new issue without 86332ca is happening sporadically, so I really suspect that I am now running into the ANGLE issue #9181.