Automated fix for refs/heads/master #5
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Build is failing, so temporarily removing dotnet from benchmarks CI.
paulosjca
pushed a commit
that referenced
this pull request
Nov 25, 2024
…haned (grpc#37683) Sample race - https://btx.cloud.google.com/invocations/0c4e65f2-3a38-4b4f-b67e-c53a4a4650ea/targets/%2F%2Ftest%2Fcore%2Fend2end:connectivity_test@poller%3Dpoll;config=2aed862ff4fd4384687d63aa95df415c7cb955355c2ab6dc6c6d7a9d123a76ec/log ``` WARNING: ThreadSanitizer: data race (pid=18) Write of size 8 at 0x72300000c318 by thread T29: #0 grpc_core::Chttp2ServerListener* std::__exchange(grpc_core::Chttp2ServerListener*&, grpc_core::Chttp2ServerListener*&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/move.h:152:13 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x68c85) #1 grpc_core::Chttp2ServerListener* std::exchange(grpc_core::Chttp2ServerListener*&, grpc_core::Chttp2ServerListener*&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/utility:287:14 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x68c05) #2 grpc_core::RefCountedPtr::reset(grpc_core::Chttp2ServerListener*) /proc/self/cwd/./src/core/lib/gprpp/ref_counted_ptr.h:126:20 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x68b32) #3 grpc_core::RefCountedPtr::operator=(grpc_core::RefCountedPtr&&) /proc/self/cwd/./src/core/lib/gprpp/ref_counted_ptr.h:66:5 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x54380) #4 grpc_core::Chttp2ServerListener::ActiveConnection::Start(grpc_core::RefCountedPtr, std::unique_ptr, grpc_core::ChannelArgs const&) /proc/self/cwd/src/core/ext/transport/chttp2/server/chttp2_server.cc:615:13 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x48914) #5 grpc_core::Chttp2ServerListener::OnAccept(void*, grpc_endpoint*, grpc_pollset*, grpc_tcp_server_acceptor*) /proc/self/cwd/src/core/ext/transport/chttp2/server/chttp2_server.cc:881:21 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x49ce2) #6 CreateEventEngineListener(grpc_tcp_server*, grpc_closure*, grpc_event_engine::experimental::EndpointConfig const&, grpc_tcp_server**)::$_2::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const /proc/self/cwd/src/core/lib/iomgr/tcp_server_posix.cc:228:11 (liblibiomgr.so+0xef627) #7 decltype(std::declval()(std::declval>>(), std::declval())) absl::lts_20240116::base_internal::Callable::Invoke>, grpc_event_engine::experimental::MemoryAllocator>(CreateEventEngineListener(grpc_tcp_server*, grpc_closure*, grpc_event_engine::experimental::EndpointConfig const&, grpc_tcp_server**)::$_2&, std::unique_ptr>&&, grpc_event_engine::experimental::MemoryAllocator&&) /proc/self/cwd/external/com_google_absl/absl/base/internal/invoke.h:185:12 (liblibiomgr.so+0xef3c2) grpc#8 decltype(Invoker>, grpc_event_engine::experimental::MemoryAllocator>::type::Invoke(std::declval(), std::declval>>(), std::declval())) absl::lts_20240116::base_internal::invoke>, grpc_event_engine::experimental::MemoryAllocator>(CreateEventEngineListener(grpc_tcp_server*, grpc_closure*, grpc_event_engine::experimental::EndpointConfig const&, grpc_tcp_server**)::$_2&, std::unique_ptr>&&, grpc_event_engine::experimental::MemoryAllocator&&) /proc/self/cwd/external/com_google_absl/absl/base/internal/invoke.h:212:10 (liblibiomgr.so+0xef325) grpc#9 void absl::lts_20240116::internal_any_invocable::InvokeR>, grpc_event_engine::experimental::MemoryAllocator, void>(CreateEventEngineListener(grpc_tcp_server*, grpc_closure*, grpc_event_engine::experimental::EndpointConfig const&, grpc_tcp_server**)::$_2&, std::unique_ptr>&&, grpc_event_engine::experimental::MemoryAllocator&&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (liblibiomgr.so+0xef2b5) grpc#10 void absl::lts_20240116::internal_any_invocable::LocalInvoker>, grpc_event_engine::experimental::MemoryAllocator>(absl::lts_20240116::internal_any_invocable::TypeErasedState*, absl::lts_20240116::internal_any_invocable::ForwardedParameter>>::type, absl::lts_20240116::internal_any_invocable::ForwardedParameter::type) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:310:10 (liblibiomgr.so+0xef1e2) grpc#11 absl::lts_20240116::internal_any_invocable::Impl>, grpc_event_engine::experimental::MemoryAllocator)>::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (libsrc_Score_Slibposix_Uevent_Uengine.so+0xa754f) grpc#12 grpc_event_engine::experimental::ThreadyEventEngine::CreateListener(absl::lts_20240116::AnyInvocable>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()::operator()() /proc/self/cwd/src/core/lib/event_engine/thready_event_engine/thready_event_engine.cc:61:15 (libsrc_Score_Slibthready_Uevent_Uengine.so+0x27cdb) grpc#13 decltype(std::declval>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&>()()) absl::lts_20240116::base_internal::Callable::Invoke>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&>(grpc_event_engine::experimental::ThreadyEventEngine::CreateListener(absl::lts_20240116::AnyInvocable>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&) /proc/self/cwd/external/com_google_absl/absl/base/internal/invoke.h:185:12 (libsrc_Score_Slibthready_Uevent_Uengine.so+0x27c45) grpc#14 decltype(Invoker>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&>::type::Invoke(std::declval>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&>())) absl::lts_20240116::base_internal::invoke>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&>(grpc_event_engine::experimental::ThreadyEventEngine::CreateListener(absl::lts_20240116::AnyInvocable>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&) /proc/self/cwd/external/com_google_absl/absl/base/internal/invoke.h:212:10 (libsrc_Score_Slibthready_Uevent_Uengine.so+0x27bf5) grpc#15 void absl::lts_20240116::internal_any_invocable::InvokeR>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&, void>(grpc_event_engine::experimental::ThreadyEventEngine::CreateListener(absl::lts_20240116::AnyInvocable>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libsrc_Score_Slibthready_Uevent_Uengine.so+0x27ba5) grpc#16 void absl::lts_20240116::internal_any_invocable::RemoteInvoker>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&>(absl::lts_20240116::internal_any_invocable::TypeErasedState*) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (libsrc_Score_Slibthready_Uevent_Uengine.so+0x279cd) grpc#17 absl::lts_20240116::internal_any_invocable::Impl::operator()() /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (libtest_Score_Send2end_Slibconnectivity_Ulibrary.so+0x337ff) grpc#18 grpc_core::Thread::Thread(char const*, absl::lts_20240116::AnyInvocable, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const /proc/self/cwd/./src/core/lib/gprpp/thd.h:108:15 (libsrc_Score_Slibthready_Uevent_Uengine.so+0x2e264) grpc#19 grpc_core::Thread::Thread(char const*, absl::lts_20240116::AnyInvocable, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) /proc/self/cwd/./src/core/lib/gprpp/thd.h:105:13 (libsrc_Score_Slibthready_Uevent_Uengine.so+0x2e1e9) grpc#20 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const /proc/self/cwd/src/core/lib/gprpp/posix/thd.cc:148:11 (liblibgpr.so+0x1d830) grpc#21 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) /proc/self/cwd/src/core/lib/gprpp/posix/thd.cc:118:9 (liblibgpr.so+0x1d659) Previous read of size 8 at 0x72300000c318 by main thread: #0 grpc_core::RefCountedPtr::operator!=(std::nullptr_t) const /proc/self/cwd/./src/core/lib/gprpp/ref_counted_ptr.h:192:50 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x52345) #1 grpc_core::Chttp2ServerListener::ActiveConnection::HandshakingState::~HandshakingState() /proc/self/cwd/src/core/ext/transport/chttp2/server/chttp2_server.cc:394:30 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x463ed) #2 std::enable_if::value, grpc_core::Chttp2ServerListener::ActiveConnection::HandshakingState*>::type grpc_event_engine::experimental::MemoryAllocator::New, grpc_pollset*&, std::unique_ptr, grpc_core::ChannelArgs const&>(grpc_core::RefCountedPtr&&, grpc_pollset*&, std::unique_ptr&&, grpc_core::ChannelArgs const&)::Wrapper::~Wrapper() /proc/self/cwd/include/grpc/event_engine/memory_allocator.h:117:65 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x67a1c) #3 std::enable_if::value, grpc_core::Chttp2ServerListener::ActiveConnection::HandshakingState*>::type grpc_event_engine::experimental::MemoryAllocator::New, grpc_pollset*&, std::unique_ptr, grpc_core::ChannelArgs const&>(grpc_core::RefCountedPtr&&, grpc_pollset*&, std::unique_ptr&&, grpc_core::ChannelArgs const&)::Wrapper::~Wrapper() /proc/self/cwd/include/grpc/event_engine/memory_allocator.h:117:27 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x67a59) #4 void grpc_core::UnrefDelete::operator()(grpc_core::Chttp2ServerListener::ActiveConnection::HandshakingState*) const /proc/self/cwd/./src/core/lib/gprpp/ref_counted.h:224:5 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x6449d) #5 grpc_core::InternallyRefCounted::Unref() /proc/self/cwd/./src/core/lib/gprpp/orphanable.h:132:7 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x52581) #6 grpc_core::Chttp2ServerListener::ActiveConnection::HandshakingState::Orphan() /proc/self/cwd/src/core/ext/transport/chttp2/server/chttp2_server.cc:407:3 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x466ac) #7 void grpc_core::OrphanableDelete::operator()(grpc_core::Chttp2ServerListener::ActiveConnection::HandshakingState*) /proc/self/cwd/./src/core/lib/gprpp/orphanable.h:60:8 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x653b1) grpc#8 std::unique_ptr::~unique_ptr() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:292:4 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x53ddf) grpc#9 grpc_core::Chttp2ServerListener::ActiveConnection::Orphan() /proc/self/cwd/src/core/ext/transport/chttp2/server/chttp2_server.cc:581:1 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x487db) grpc#10 void grpc_core::OrphanableDelete::operator()(grpc_core::Chttp2ServerListener::ActiveConnection*) /proc/self/cwd/./src/core/lib/gprpp/orphanable.h:60:8 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x66b01) grpc#11 std::unique_ptr::~unique_ptr() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:292:4 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x53d1f) grpc#12 std::pair>::~pair() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_pair.h:208:12 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x597f9) grpc#13 void __gnu_cxx::new_allocator>>>::destroy>>(std::pair>*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/ext/new_allocator.h:152:10 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x597c1) grpc#14 void std::allocator_traits>>>>::destroy>>(std::allocator>>>&, std::pair>*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/alloc_traits.h:496:8 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x59725) grpc#15 std::_Rb_tree>, std::_Select1st>>, std::less, std::allocator>>>::_M_destroy_node(std::_Rb_tree_node>>*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_tree.h:642:2 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x59674) grpc#16 std::_Rb_tree>, std::_Select1st>>, std::less, std::allocator>>>::_M_drop_node(std::_Rb_tree_node>>*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_tree.h:650:2 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x595f9) grpc#17 std::_Rb_tree>, std::_Select1st>>, std::less, std::allocator>>>::_M_erase(std::_Rb_tree_node>>*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_tree.h:1920:4 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x5945f) grpc#18 std::_Rb_tree>, std::_Select1st>>, std::less, std::allocator>>>::~_Rb_tree() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_tree.h:1000:9 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x593c5) grpc#19 std::map, std::less, std::allocator>>>::~map() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_map.h:300:22 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x51525) grpc#20 grpc_core::Chttp2ServerListener::Orphan() /proc/self/cwd/src/core/ext/transport/chttp2/server/chttp2_server.cc:923:1 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x4b5dd) grpc#21 void grpc_core::OrphanableDelete::operator()(grpc_core::Server::ListenerInterface*) /proc/self/cwd/./src/core/lib/gprpp/orphanable.h:60:8 (libsrc_Score_Slibchaotic_Ugood_Userver.so+0x1d0981) grpc#22 std::unique_ptr::reset(grpc_core::Server::ListenerInterface*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:402:4 (liblibserver.so+0x1d5c21) grpc#23 grpc_core::Server::StopListening() /proc/self/cwd/src/core/server/server.cc:1211:23 (liblibserver.so+0x1b9c62) grpc#24 grpc_core::Server::ShutdownAndNotify(grpc_completion_queue*, void*) /proc/self/cwd/src/core/server/server.cc:1195:3 (liblibserver.so+0x1b97da) grpc#25 grpc_server_shutdown_and_notify /proc/self/cwd/src/core/server/server.cc:1829:37 (liblibserver.so+0x1bf212) grpc#26 grpc_core::CoreEnd2endTest::ShutdownServerAndNotify(int) /proc/self/cwd/./test/core/end2end/end2end_tests.h:459:5 (libtest_Score_Send2end_Slibconnectivity_Ulibrary.so+0x33370) grpc#27 grpc_core::(anonymous namespace)::CoreEnd2endTest_RetryHttp2Test_ConnectivityWatch::RunTest() /proc/self/cwd/test/core/end2end/tests/connectivity.cc:74:3 (libtest_Score_Send2end_Slibconnectivity_Ulibrary.so+0x2ee8d) grpc#28 grpc_core::(anonymous namespace)::CoreEnd2endTest_RetryHttp2Test_ConnectivityWatch::TestBody() /proc/self/cwd/test/core/end2end/tests/connectivity.cc:32:1 (libtest_Score_Send2end_Slibconnectivity_Ulibrary.so+0x2dc96) grpc#29 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2612:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x16d2dc) grpc#30 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2648:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x14a51d) grpc#31 testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2687:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x120458) grpc#32 testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2836:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x1215f3) grpc#33 testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:3015:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x12230c) grpc#34 testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5921:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x138142) grpc#35 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2612:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x17508f) grpc#36 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2648:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x14e5b3) grpc#37 testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5485:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x13795b) grpc#38 RUN_ALL_TESTS() /proc/self/cwd/external/com_google_googletest/googletest/include/gtest/gtest.h:2316:73 (libtest_Score_Send2end_Slibend2end_Utest_Umain.so+0x8457) grpc#39 main /proc/self/cwd/test/core/end2end/end2end_test_main.cc:50:10 (libtest_Score_Send2end_Slibend2end_Utest_Umain.so+0x77b6) ``` We start the connection outside the critical region and that's where we supply the listener ref to the connection. There is a freak case where the connection can be orphaned due to the listener stopping to serve and the `Orphan()` would also be trying to access the listener ref resulting in a race. Closes grpc#37683 COPYBARA_INTEGRATE_REVIEW=grpc#37683 from yashykt:FixChttp2ServerRace e3c4529 PiperOrigin-RevId: 681552145
paulosjca
pushed a commit
that referenced
this pull request
Nov 25, 2024
…haned (grpc#37683) Sample race - https://btx.cloud.google.com/invocations/0c4e65f2-3a38-4b4f-b67e-c53a4a4650ea/targets/%2F%2Ftest%2Fcore%2Fend2end:connectivity_test@poller%3Dpoll;config=2aed862ff4fd4384687d63aa95df415c7cb955355c2ab6dc6c6d7a9d123a76ec/log ``` WARNING: ThreadSanitizer: data race (pid=18) Write of size 8 at 0x72300000c318 by thread T29: #0 grpc_core::Chttp2ServerListener* std::__exchange(grpc_core::Chttp2ServerListener*&, grpc_core::Chttp2ServerListener*&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/move.h:152:13 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x68c85) #1 grpc_core::Chttp2ServerListener* std::exchange(grpc_core::Chttp2ServerListener*&, grpc_core::Chttp2ServerListener*&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/utility:287:14 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x68c05) #2 grpc_core::RefCountedPtr::reset(grpc_core::Chttp2ServerListener*) /proc/self/cwd/./src/core/lib/gprpp/ref_counted_ptr.h:126:20 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x68b32) #3 grpc_core::RefCountedPtr::operator=(grpc_core::RefCountedPtr&&) /proc/self/cwd/./src/core/lib/gprpp/ref_counted_ptr.h:66:5 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x54380) #4 grpc_core::Chttp2ServerListener::ActiveConnection::Start(grpc_core::RefCountedPtr, std::unique_ptr, grpc_core::ChannelArgs const&) /proc/self/cwd/src/core/ext/transport/chttp2/server/chttp2_server.cc:615:13 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x48914) #5 grpc_core::Chttp2ServerListener::OnAccept(void*, grpc_endpoint*, grpc_pollset*, grpc_tcp_server_acceptor*) /proc/self/cwd/src/core/ext/transport/chttp2/server/chttp2_server.cc:881:21 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x49ce2) #6 CreateEventEngineListener(grpc_tcp_server*, grpc_closure*, grpc_event_engine::experimental::EndpointConfig const&, grpc_tcp_server**)::$_2::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const /proc/self/cwd/src/core/lib/iomgr/tcp_server_posix.cc:228:11 (liblibiomgr.so+0xef627) #7 decltype(std::declval()(std::declval>>(), std::declval())) absl::lts_20240116::base_internal::Callable::Invoke>, grpc_event_engine::experimental::MemoryAllocator>(CreateEventEngineListener(grpc_tcp_server*, grpc_closure*, grpc_event_engine::experimental::EndpointConfig const&, grpc_tcp_server**)::$_2&, std::unique_ptr>&&, grpc_event_engine::experimental::MemoryAllocator&&) /proc/self/cwd/external/com_google_absl/absl/base/internal/invoke.h:185:12 (liblibiomgr.so+0xef3c2) grpc#8 decltype(Invoker>, grpc_event_engine::experimental::MemoryAllocator>::type::Invoke(std::declval(), std::declval>>(), std::declval())) absl::lts_20240116::base_internal::invoke>, grpc_event_engine::experimental::MemoryAllocator>(CreateEventEngineListener(grpc_tcp_server*, grpc_closure*, grpc_event_engine::experimental::EndpointConfig const&, grpc_tcp_server**)::$_2&, std::unique_ptr>&&, grpc_event_engine::experimental::MemoryAllocator&&) /proc/self/cwd/external/com_google_absl/absl/base/internal/invoke.h:212:10 (liblibiomgr.so+0xef325) grpc#9 void absl::lts_20240116::internal_any_invocable::InvokeR>, grpc_event_engine::experimental::MemoryAllocator, void>(CreateEventEngineListener(grpc_tcp_server*, grpc_closure*, grpc_event_engine::experimental::EndpointConfig const&, grpc_tcp_server**)::$_2&, std::unique_ptr>&&, grpc_event_engine::experimental::MemoryAllocator&&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (liblibiomgr.so+0xef2b5) grpc#10 void absl::lts_20240116::internal_any_invocable::LocalInvoker>, grpc_event_engine::experimental::MemoryAllocator>(absl::lts_20240116::internal_any_invocable::TypeErasedState*, absl::lts_20240116::internal_any_invocable::ForwardedParameter>>::type, absl::lts_20240116::internal_any_invocable::ForwardedParameter::type) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:310:10 (liblibiomgr.so+0xef1e2) grpc#11 absl::lts_20240116::internal_any_invocable::Impl>, grpc_event_engine::experimental::MemoryAllocator)>::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (libsrc_Score_Slibposix_Uevent_Uengine.so+0xa754f) grpc#12 grpc_event_engine::experimental::ThreadyEventEngine::CreateListener(absl::lts_20240116::AnyInvocable>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()::operator()() /proc/self/cwd/src/core/lib/event_engine/thready_event_engine/thready_event_engine.cc:61:15 (libsrc_Score_Slibthready_Uevent_Uengine.so+0x27cdb) grpc#13 decltype(std::declval>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&>()()) absl::lts_20240116::base_internal::Callable::Invoke>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&>(grpc_event_engine::experimental::ThreadyEventEngine::CreateListener(absl::lts_20240116::AnyInvocable>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&) /proc/self/cwd/external/com_google_absl/absl/base/internal/invoke.h:185:12 (libsrc_Score_Slibthready_Uevent_Uengine.so+0x27c45) grpc#14 decltype(Invoker>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&>::type::Invoke(std::declval>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&>())) absl::lts_20240116::base_internal::invoke>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&>(grpc_event_engine::experimental::ThreadyEventEngine::CreateListener(absl::lts_20240116::AnyInvocable>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&) /proc/self/cwd/external/com_google_absl/absl/base/internal/invoke.h:212:10 (libsrc_Score_Slibthready_Uevent_Uengine.so+0x27bf5) grpc#15 void absl::lts_20240116::internal_any_invocable::InvokeR>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&, void>(grpc_event_engine::experimental::ThreadyEventEngine::CreateListener(absl::lts_20240116::AnyInvocable>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (libsrc_Score_Slibthready_Uevent_Uengine.so+0x27ba5) grpc#16 void absl::lts_20240116::internal_any_invocable::RemoteInvoker>, grpc_event_engine::experimental::MemoryAllocator)>, absl::lts_20240116::AnyInvocable, grpc_event_engine::experimental::EndpointConfig const&, std::unique_ptr>)::$_0::operator()(std::unique_ptr>, grpc_event_engine::experimental::MemoryAllocator) const::'lambda'()&>(absl::lts_20240116::internal_any_invocable::TypeErasedState*) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (libsrc_Score_Slibthready_Uevent_Uengine.so+0x279cd) grpc#17 absl::lts_20240116::internal_any_invocable::Impl::operator()() /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (libtest_Score_Send2end_Slibconnectivity_Ulibrary.so+0x337ff) grpc#18 grpc_core::Thread::Thread(char const*, absl::lts_20240116::AnyInvocable, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const /proc/self/cwd/./src/core/lib/gprpp/thd.h:108:15 (libsrc_Score_Slibthready_Uevent_Uengine.so+0x2e264) grpc#19 grpc_core::Thread::Thread(char const*, absl::lts_20240116::AnyInvocable, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) /proc/self/cwd/./src/core/lib/gprpp/thd.h:105:13 (libsrc_Score_Slibthready_Uevent_Uengine.so+0x2e1e9) grpc#20 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const /proc/self/cwd/src/core/lib/gprpp/posix/thd.cc:148:11 (liblibgpr.so+0x1d830) grpc#21 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) /proc/self/cwd/src/core/lib/gprpp/posix/thd.cc:118:9 (liblibgpr.so+0x1d659) Previous read of size 8 at 0x72300000c318 by main thread: #0 grpc_core::RefCountedPtr::operator!=(std::nullptr_t) const /proc/self/cwd/./src/core/lib/gprpp/ref_counted_ptr.h:192:50 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x52345) #1 grpc_core::Chttp2ServerListener::ActiveConnection::HandshakingState::~HandshakingState() /proc/self/cwd/src/core/ext/transport/chttp2/server/chttp2_server.cc:394:30 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x463ed) #2 std::enable_if::value, grpc_core::Chttp2ServerListener::ActiveConnection::HandshakingState*>::type grpc_event_engine::experimental::MemoryAllocator::New, grpc_pollset*&, std::unique_ptr, grpc_core::ChannelArgs const&>(grpc_core::RefCountedPtr&&, grpc_pollset*&, std::unique_ptr&&, grpc_core::ChannelArgs const&)::Wrapper::~Wrapper() /proc/self/cwd/include/grpc/event_engine/memory_allocator.h:117:65 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x67a1c) #3 std::enable_if::value, grpc_core::Chttp2ServerListener::ActiveConnection::HandshakingState*>::type grpc_event_engine::experimental::MemoryAllocator::New, grpc_pollset*&, std::unique_ptr, grpc_core::ChannelArgs const&>(grpc_core::RefCountedPtr&&, grpc_pollset*&, std::unique_ptr&&, grpc_core::ChannelArgs const&)::Wrapper::~Wrapper() /proc/self/cwd/include/grpc/event_engine/memory_allocator.h:117:27 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x67a59) #4 void grpc_core::UnrefDelete::operator()(grpc_core::Chttp2ServerListener::ActiveConnection::HandshakingState*) const /proc/self/cwd/./src/core/lib/gprpp/ref_counted.h:224:5 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x6449d) #5 grpc_core::InternallyRefCounted::Unref() /proc/self/cwd/./src/core/lib/gprpp/orphanable.h:132:7 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x52581) #6 grpc_core::Chttp2ServerListener::ActiveConnection::HandshakingState::Orphan() /proc/self/cwd/src/core/ext/transport/chttp2/server/chttp2_server.cc:407:3 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x466ac) #7 void grpc_core::OrphanableDelete::operator()(grpc_core::Chttp2ServerListener::ActiveConnection::HandshakingState*) /proc/self/cwd/./src/core/lib/gprpp/orphanable.h:60:8 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x653b1) grpc#8 std::unique_ptr::~unique_ptr() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:292:4 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x53ddf) grpc#9 grpc_core::Chttp2ServerListener::ActiveConnection::Orphan() /proc/self/cwd/src/core/ext/transport/chttp2/server/chttp2_server.cc:581:1 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x487db) grpc#10 void grpc_core::OrphanableDelete::operator()(grpc_core::Chttp2ServerListener::ActiveConnection*) /proc/self/cwd/./src/core/lib/gprpp/orphanable.h:60:8 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x66b01) grpc#11 std::unique_ptr::~unique_ptr() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:292:4 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x53d1f) grpc#12 std::pair>::~pair() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_pair.h:208:12 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x597f9) grpc#13 void __gnu_cxx::new_allocator>>>::destroy>>(std::pair>*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/ext/new_allocator.h:152:10 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x597c1) grpc#14 void std::allocator_traits>>>>::destroy>>(std::allocator>>>&, std::pair>*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/alloc_traits.h:496:8 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x59725) grpc#15 std::_Rb_tree>, std::_Select1st>>, std::less, std::allocator>>>::_M_destroy_node(std::_Rb_tree_node>>*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_tree.h:642:2 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x59674) grpc#16 std::_Rb_tree>, std::_Select1st>>, std::less, std::allocator>>>::_M_drop_node(std::_Rb_tree_node>>*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_tree.h:650:2 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x595f9) grpc#17 std::_Rb_tree>, std::_Select1st>>, std::less, std::allocator>>>::_M_erase(std::_Rb_tree_node>>*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_tree.h:1920:4 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x5945f) grpc#18 std::_Rb_tree>, std::_Select1st>>, std::less, std::allocator>>>::~_Rb_tree() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_tree.h:1000:9 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x593c5) grpc#19 std::map, std::less, std::allocator>>>::~map() /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_map.h:300:22 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x51525) grpc#20 grpc_core::Chttp2ServerListener::Orphan() /proc/self/cwd/src/core/ext/transport/chttp2/server/chttp2_server.cc:923:1 (libsrc_Score_Slibgrpc_Utransport_Uchttp2_Userver.so+0x4b5dd) grpc#21 void grpc_core::OrphanableDelete::operator()(grpc_core::Server::ListenerInterface*) /proc/self/cwd/./src/core/lib/gprpp/orphanable.h:60:8 (libsrc_Score_Slibchaotic_Ugood_Userver.so+0x1d0981) grpc#22 std::unique_ptr::reset(grpc_core::Server::ListenerInterface*) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/unique_ptr.h:402:4 (liblibserver.so+0x1d5c21) grpc#23 grpc_core::Server::StopListening() /proc/self/cwd/src/core/server/server.cc:1211:23 (liblibserver.so+0x1b9c62) grpc#24 grpc_core::Server::ShutdownAndNotify(grpc_completion_queue*, void*) /proc/self/cwd/src/core/server/server.cc:1195:3 (liblibserver.so+0x1b97da) grpc#25 grpc_server_shutdown_and_notify /proc/self/cwd/src/core/server/server.cc:1829:37 (liblibserver.so+0x1bf212) grpc#26 grpc_core::CoreEnd2endTest::ShutdownServerAndNotify(int) /proc/self/cwd/./test/core/end2end/end2end_tests.h:459:5 (libtest_Score_Send2end_Slibconnectivity_Ulibrary.so+0x33370) grpc#27 grpc_core::(anonymous namespace)::CoreEnd2endTest_RetryHttp2Test_ConnectivityWatch::RunTest() /proc/self/cwd/test/core/end2end/tests/connectivity.cc:74:3 (libtest_Score_Send2end_Slibconnectivity_Ulibrary.so+0x2ee8d) grpc#28 grpc_core::(anonymous namespace)::CoreEnd2endTest_RetryHttp2Test_ConnectivityWatch::TestBody() /proc/self/cwd/test/core/end2end/tests/connectivity.cc:32:1 (libtest_Score_Send2end_Slibconnectivity_Ulibrary.so+0x2dc96) grpc#29 void testing::internal::HandleSehExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2612:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x16d2dc) grpc#30 void testing::internal::HandleExceptionsInMethodIfSupported(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2648:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x14a51d) grpc#31 testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2687:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x120458) grpc#32 testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2836:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x1215f3) grpc#33 testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:3015:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x12230c) grpc#34 testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5921:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x138142) grpc#35 bool testing::internal::HandleSehExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2612:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x17508f) grpc#36 bool testing::internal::HandleExceptionsInMethodIfSupported(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2648:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x14e5b3) grpc#37 testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5485:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0x13795b) grpc#38 RUN_ALL_TESTS() /proc/self/cwd/external/com_google_googletest/googletest/include/gtest/gtest.h:2316:73 (libtest_Score_Send2end_Slibend2end_Utest_Umain.so+0x8457) grpc#39 main /proc/self/cwd/test/core/end2end/end2end_test_main.cc:50:10 (libtest_Score_Send2end_Slibend2end_Utest_Umain.so+0x77b6) ``` We start the connection outside the critical region and that's where we supply the listener ref to the connection. There is a freak case where the connection can be orphaned due to the listener stopping to serve and the `Orphan()` would also be trying to access the listener ref resulting in a race. Closes grpc#37683 COPYBARA_INTEGRATE_REVIEW=grpc#37683 from yashykt:FixChttp2ServerRace e3c4529 PiperOrigin-RevId: 681552145
paulosjca
pushed a commit
that referenced
this pull request
Jun 6, 2025
Remove tcp_tracer in chttp2_transport since it's unused and buggy. Fix for TSAN failure (similar to grpc#38728) - https://btx.cloud.google.com/invocations/97eb07c8-cf0c-4c26-a0c4-e7417adb1f23/targets/%2F%2Ftest%2Fcpp%2Fext%2Fotel:otel_tracing_test@poller%3Dpoll;config=1a7b8f82b5a4a85323c9dfbac159b038ad684a9dcca8ab0ce8fe99aa87dd2da2/log ``` WARNING: ThreadSanitizer: data race (pid=15) Read of size 8 at 0x726c00001588 by thread T44: #0 GetContext /proc/self/cwd/./src/core/lib/resource_quota/arena.h:296:9 (liblibgrpc_Utransport_Uchttp2.so+0x30af6e) #1 (anonymous namespace)::TcpTracerIfSampled(grpc_chttp2_stream*) /proc/self/cwd/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:243:17 (liblibgrpc_Utransport_Uchttp2.so+0x30af6e) #2 perform_stream_op_locked(void*, absl::lts_20240722::Status) /proc/self/cwd/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:1655:19 (liblibgrpc_Utransport_Uchttp2.so+0x3014aa) #3 grpc_combiner_continue_exec_ctx() /proc/self/cwd/src/core/lib/iomgr/combiner.cc:216:5 (liblibexec_Uctx.so+0x10d7e) #4 grpc_core::ExecCtx::Flush() /proc/self/cwd/src/core/lib/iomgr/exec_ctx.cc:77:17 (liblibexec_Uctx.so+0x16e45) #5 queue_offload(grpc_core::Combiner*)::$_0::operator()() const /proc/self/cwd/src/core/lib/iomgr/combiner.cc:165:14 (liblibexec_Uctx.so+0x13092) #6 void std::__invoke_impl(std::__invoke_other, queue_offload(grpc_core::Combiner*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/invoke.h:60:14 (liblibexec_Uctx.so+0x13005) #7 std::__invoke_result::type std::__invoke(queue_offload(grpc_core::Combiner*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/invoke.h:95:14 (liblibexec_Uctx.so+0x12fb5) grpc#8 std::invoke_result::type std::invoke(queue_offload(grpc_core::Combiner*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/functional:81:14 (liblibexec_Uctx.so+0x12f65) grpc#9 void absl::lts_20240722::internal_any_invocable::InvokeR(queue_offload(grpc_core::Combiner*)::$_0&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (liblibexec_Uctx.so+0x12f05) grpc#10 void absl::lts_20240722::internal_any_invocable::LocalInvoker(absl::lts_20240722::internal_any_invocable::TypeErasedState*) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:310:10 (liblibexec_Uctx.so+0x12e22) grpc#11 absl::lts_20240722::internal_any_invocable::Impl::operator()() /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (libsrc_Score_Slibping_Ucallbacks.so+0x260bf) grpc#12 grpc_event_engine::experimental::SelfDeletingClosure::Run() /proc/self/cwd/./src/core/lib/event_engine/common_closures.h:54:5 (libsrc_Score_Slibevent_Uengine_Uthread_Upool.so+0x501bd) grpc#13 grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::Step() /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:524:14 (libsrc_Score_Slibevent_Uengine_Uthread_Upool.so+0x4a949) grpc#14 grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::ThreadBody() /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:487:10 (libsrc_Score_Slibevent_Uengine_Uthread_Upool.so+0x4a013) grpc#15 grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::StartThread()::$_0::operator()(void*) const /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:257:17 (libsrc_Score_Slibevent_Uengine_Uthread_Upool.so+0x4b489) grpc#16 grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::StartThread()::$_0::__invoke(void*) /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:255:7 (libsrc_Score_Slibevent_Uengine_Uthread_Upool.so+0x4b429) grpc#17 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const /proc/self/cwd/src/core/util/posix/thd.cc:145:11 (liblibgpr.so+0x1c2e0) grpc#18 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) /proc/self/cwd/src/core/util/posix/thd.cc:115:9 (liblibgpr.so+0x1c109) Previous write of size 8 at 0x726c00001588 by thread T47: #0 void grpc_core::Arena::SetContext(grpc_core::CallTracerInterface*) /proc/self/cwd/./src/core/lib/resource_quota/arena.h:305:10 (liblibgrpc_Uclient_Uchannel.so+0x5ac294) #1 grpc_core::(anonymous namespace)::CreateCallAttemptTracer(grpc_core::Arena*, bool) /proc/self/cwd/src/core/client_channel/client_channel_filter.cc:2388:10 (liblibgrpc_Uclient_Uchannel.so+0x58403c) #2 grpc_core::ClientChannelFilter::LoadBalancedCall::LoadBalancedCall(grpc_core::ClientChannelFilter*, grpc_core::Arena*, absl::lts_20240722::AnyInvocable, bool) /proc/self/cwd/src/core/client_channel/client_channel_filter.cc:2402:11 (liblibgrpc_Uclient_Uchannel.so+0x583a98) #3 grpc_core::ClientChannelFilter::FilterBasedLoadBalancedCall::FilterBasedLoadBalancedCall(grpc_core::ClientChannelFilter*, grpc_call_element_args const&, grpc_polling_entity*, grpc_closure*, absl::lts_20240722::AnyInvocable, bool) /proc/self/cwd/src/core/client_channel/client_channel_filter.cc:2628:7 (liblibgrpc_Uclient_Uchannel.so+0x5864d4) #4 grpc_core::ClientChannelFilter::FilterBasedLoadBalancedCall* grpc_core::Arena::New, bool&>(grpc_core::ClientChannelFilter*&&, grpc_call_element_args const&, grpc_polling_entity*&, grpc_closure*&, absl::lts_20240722::AnyInvocable&&, bool&) /proc/self/cwd/./src/core/lib/resource_quota/arena.h:181:13 (liblibgrpc_Uclient_Uchannel.so+0x59459d) #5 grpc_core::ClientChannelFilter::CreateLoadBalancedCall(grpc_call_element_args const&, grpc_polling_entity*, grpc_closure*, absl::lts_20240722::AnyInvocable, bool) /proc/self/cwd/src/core/client_channel/client_channel_filter.cc:1142:19 (liblibgrpc_Uclient_Uchannel.so+0x578343) #6 grpc_core::RetryFilter::LegacyCallData::CreateLoadBalancedCall(absl::lts_20240722::AnyInvocable, bool) /proc/self/cwd/src/core/client_channel/retry_filter_legacy_call_data.cc:1635:36 (liblibgrpc_Uclient_Uchannel.so+0x5f3954) #7 grpc_core::RetryFilter::LegacyCallData::CallAttempt::CallAttempt(grpc_core::RetryFilter::LegacyCallData*, bool) /proc/self/cwd/src/core/client_channel/retry_filter_legacy_call_data.cc:129:21 (liblibgrpc_Uclient_Uchannel.so+0x5f2de1) grpc#8 grpc_core::RefCountedPtr grpc_core::MakeRefCounted(grpc_core::RetryFilter::LegacyCallData*&&, bool&) /proc/self/cwd/./src/core/util/ref_counted_ptr.h:369:31 (liblibgrpc_Uclient_Uchannel.so+0x60ad6a) grpc#9 grpc_core::RetryFilter::LegacyCallData::CreateCallAttempt(bool) /proc/self/cwd/src/core/client_channel/retry_filter_legacy_call_data.cc:1644:19 (liblibgrpc_Uclient_Uchannel.so+0x605a9b) grpc#10 grpc_core::RetryFilter::LegacyCallData::StartTransparentRetry(void*, absl::lts_20240722::Status) /proc/self/cwd/src/core/client_channel/retry_filter_legacy_call_data.cc:1937:12 (liblibgrpc_Uclient_Uchannel.so+0x60600b) grpc#11 exec_ctx_run(grpc_closure*) /proc/self/cwd/src/core/lib/iomgr/exec_ctx.cc:43:3 (liblibexec_Uctx.so+0x17536) grpc#12 grpc_core::ExecCtx::Flush() /proc/self/cwd/src/core/lib/iomgr/exec_ctx.cc:74:9 (liblibexec_Uctx.so+0x16e29) grpc#13 grpc_core::ExecCtx::~ExecCtx() /proc/self/cwd/./src/core/lib/iomgr/exec_ctx.h:137:5 (liblibgrpc++_Ubase.so+0x1647cc) grpc#14 grpc_core::WorkSerializer::WorkSerializerImpl::Run() /proc/self/cwd/src/core/util/work_serializer.cc:221:1 (liblibwork_Userializer.so+0x15947) grpc#15 non-virtual thunk to grpc_core::WorkSerializer::WorkSerializerImpl::Run() /proc/self/cwd/src/core/util/work_serializer.cc (liblibwork_Userializer.so+0x15b09) grpc#16 grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::Step() /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:524:14 (libsrc_Score_Slibevent_Uengine_Uthread_Upool.so+0x4a949) grpc#17 grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::ThreadBody() /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:487:10 (libsrc_Score_Slibevent_Uengine_Uthread_Upool.so+0x4a013) grpc#18 grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::StartThread()::$_0::operator()(void*) const /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:257:17 (libsrc_Score_Slibevent_Uengine_Uthread_Upool.so+0x4b489) grpc#19 grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::StartThread()::$_0::__invoke(void*) /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:255:7 (libsrc_Score_Slibevent_Uengine_Uthread_Upool.so+0x4b429) grpc#20 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const /proc/self/cwd/src/core/util/posix/thd.cc:145:11 (liblibgpr.so+0x1c2e0) grpc#21 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) /proc/self/cwd/src/core/util/posix/thd.cc:115:9 (liblibgpr.so+0x1c109) ``` Assumed sequence of events (based on stacktrace) - * The client application creates a new RPC. * A ClientChannelFilter::LoadBalancedCall is created for the first attempt. * A CallAttemptTracer is created for this first attempt and a pointer to this is saved on the arena context. * The first attempt fails without any bytes having been sent on the wire, making this RPC eligible for transparent retries. * A new ClientChannelFilter::LoadBalancedCall created for this new attempt. * A new CallAttemptTracer is created for this second (transparent) attempt and a pointer is saved on the arena context. * Around the same time as a new CallAttemptTracer is created, the chttp2 transport is performing some operations for the first stream (for example, a cancel op). Closes grpc#39015 COPYBARA_INTEGRATE_REVIEW=grpc#39015 from yashykt:FixTcpTracer 7e3746e PiperOrigin-RevId: 739343300
paulosjca
pushed a commit
that referenced
this pull request
Aug 23, 2025
This fixes a bug introduced in grpc#39736 whereby the LRS server may call `StartWrite()` after cancellation has already called `Finish()`, thus leading to a TSAN failure. This is arguably a deficiency in the callback API. The write should definitely fail, but it shouldn't cause a TSAN problem. But we can consider that as part of a separate effort. Example stack trace of the TSAN failure: ``` WARNING: ThreadSanitizer: data race (pid=17) Read of size 1 at 0x72680002dd80 by thread T5: #0 grpc::internal::CallbackBidiHandler::ServerCallbackReaderWriterImpl::Finish(grpc::Status) /proc/self/cwd/include/grpcpp/impl/server_callback_handlers.h:742:18 (xds_cluster_end2end_test@poller=epoll1+0x32896b2) #1 grpc::ServerBidiReactor::Finish(grpc::Status) /proc/self/cwd/include/grpcpp/support/server_callback.h:414:13 (xds_cluster_end2end_test@poller=epoll1+0x328dbb4) #2 grpc::testing::LrsServiceImpl::Reactor::OnCancel() /proc/self/cwd/test/cpp/end2end/xds/xds_server.cc:494:3 (xds_cluster_end2end_test@poller=epoll1+0x32c6ec1) #3 grpc::internal::ServerCallbackCall::CallOnCancel(grpc::internal::ServerReactor*)::$_0::operator()() const /proc/self/cwd/src/cpp/server/server_callback.cc:38:14 (xds_cluster_end2end_test@poller=epoll1+0x3b4755b) #4 void std::__invoke_impl(std::__invoke_other, grpc::internal::ServerCallbackCall::CallOnCancel(grpc::internal::ServerReactor*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/invoke.h:60:14 (xds_cluster_end2end_test@poller=epoll1+0x3b474d5) #5 std::__invoke_result::type std::__invoke(grpc::internal::ServerCallbackCall::CallOnCancel(grpc::internal::ServerReactor*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/invoke.h:95:14 (xds_cluster_end2end_test@poller=epoll1+0x3b47485) #6 std::invoke_result::type std::invoke(grpc::internal::ServerCallbackCall::CallOnCancel(grpc::internal::ServerReactor*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/functional:81:14 (xds_cluster_end2end_test@poller=epoll1+0x3b47435) #7 void absl::lts_20250127::internal_any_invocable::InvokeR(grpc::internal::ServerCallbackCall::CallOnCancel(grpc::internal::ServerReactor*)::$_0&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (xds_cluster_end2end_test@poller=epoll1+0x3b473d5) grpc#8 void absl::lts_20250127::internal_any_invocable::LocalInvoker(absl::lts_20250127::internal_any_invocable::TypeErasedState*) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:310:10 (xds_cluster_end2end_test@poller=epoll1+0x3b472f2) grpc#9 absl::lts_20250127::internal_any_invocable::Impl::operator()() /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (xds_cluster_end2end_test@poller=epoll1+0x4aa14ef) grpc#10 grpc_event_engine::experimental::SelfDeletingClosure::Run() /proc/self/cwd/./src/core/lib/event_engine/common_closures.h:54:5 (xds_cluster_end2end_test@poller=epoll1+0x56a78ad) grpc#11 grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::Step() /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:552:14 (xds_cluster_end2end_test@poller=epoll1+0x56a4663) grpc#12 grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::ThreadBody() /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:515:10 (xds_cluster_end2end_test@poller=epoll1+0x56a3d47) grpc#13 grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::StartThread()::$_0::operator()(void*) const /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:277:17 (xds_cluster_end2end_test@poller=epoll1+0x56a5049) grpc#14 grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::StartThread()::$_0::__invoke(void*) /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:275:7 (xds_cluster_end2end_test@poller=epoll1+0x56a4fe9) grpc#15 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const /proc/self/cwd/src/core/util/posix/thd.cc:145:11 (xds_cluster_end2end_test@poller=epoll1+0x58479f0) grpc#16 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) /proc/self/cwd/src/core/util/posix/thd.cc:115:9 (xds_cluster_end2end_test@poller=epoll1+0x5847819) Previous write of size 1 at 0x72680002dd80 by thread T4: #0 grpc::internal::CallbackBidiHandler::ServerCallbackReaderWriterImpl::Write(envoy::service::load_stats::v3::LoadStatsResponse const*, grpc::WriteOptions) /proc/self/cwd/include/grpcpp/impl/server_callback_handlers.h:790:38 (xds_cluster_end2end_test@poller=epoll1+0x3289e85) #1 grpc::ServerBidiReactor::StartWrite(envoy::service::load_stats::v3::LoadStatsResponse const*, grpc::WriteOptions) /proc/self/cwd/include/grpcpp/support/server_callback.h:350:13 (xds_cluster_end2end_test@poller=epoll1+0x32ef139) #2 grpc::ServerBidiReactor::StartWrite(envoy::service::load_stats::v3::LoadStatsResponse const*) /proc/self/cwd/include/grpcpp/support/server_callback.h:328:5 (xds_cluster_end2end_test@poller=epoll1+0x32cb6df) #3 grpc::testing::LrsServiceImpl::Reactor::OnReadDone(bool) /proc/self/cwd/test/cpp/end2end/xds/xds_server.cc:461:5 (xds_cluster_end2end_test@poller=epoll1+0x32c6535) #4 grpc::internal::CallbackBidiHandler::ServerCallbackReaderWriterImpl::SetupReactor(grpc::ServerBidiReactor*)::'lambda0'(bool)::operator()(bool) const /proc/self/cwd/include/grpcpp/impl/server_callback_handlers.h:839:22 (xds_cluster_end2end_test@poller=epoll1+0x328f196) #5 std::_Function_handler::ServerCallbackReaderWriterImpl::SetupReactor(grpc::ServerBidiReactor*)::'lambda0'(bool)>::_M_invoke(std::_Any_data const&, bool&&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/std_function.h:300:2 (xds_cluster_end2end_test@poller=epoll1+0x328ef1a) #6 std::function::operator()(bool) const /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/std_function.h:688:14 (xds_cluster_end2end_test@poller=epoll1+0x3279bb8) #7 grpc::internal::CallbackWithSuccessTag::Run(bool)::'lambda'()::operator()() const /proc/self/cwd/include/grpcpp/support/callback_common.h:230:11 (xds_cluster_end2end_test@poller=epoll1+0x3279ae5) grpc#8 void std::__invoke_impl(std::__invoke_other, grpc::internal::CallbackWithSuccessTag::Run(bool)::'lambda'() const&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/invoke.h:60:14 (xds_cluster_end2end_test@poller=epoll1+0x3279a65) grpc#9 std::__invoke_result::type std::__invoke(grpc::internal::CallbackWithSuccessTag::Run(bool)::'lambda'() const&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/invoke.h:95:14 (xds_cluster_end2end_test@poller=epoll1+0x3279a15) grpc#10 std::invoke_result::type std::invoke(grpc::internal::CallbackWithSuccessTag::Run(bool)::'lambda'() const&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/functional:81:14 (xds_cluster_end2end_test@poller=epoll1+0x32799c5) grpc#11 void absl::lts_20250127::functional_internal::InvokeObject(absl::lts_20250127::functional_internal::VoidPtr) /proc/self/cwd/external/com_google_absl/absl/functional/internal/function_ref.h:78:7 (xds_cluster_end2end_test@poller=epoll1+0x327996d) grpc#12 absl::lts_20250127::FunctionRef::operator()() const /proc/self/cwd/external/com_google_absl/absl/functional/function_ref.h:132:12 (xds_cluster_end2end_test@poller=epoll1+0x3bd0221) grpc#13 void grpc::GlobalCallbackHook::CatchingCallback&>(absl::lts_20250127::FunctionRef&) /proc/self/cwd/include/grpcpp/support/global_callback_hook.h:41:5 (xds_cluster_end2end_test@poller=epoll1+0x3bd01a1) grpc#14 grpc::DefaultGlobalCallbackHook::RunCallback(grpc_call*, absl::lts_20250127::FunctionRef) /proc/self/cwd/include/grpcpp/support/global_callback_hook.h:50:5 (xds_cluster_end2end_test@poller=epoll1+0x3bd011d) grpc#15 grpc::internal::CallbackWithSuccessTag::Run(bool) /proc/self/cwd/include/grpcpp/support/callback_common.h:227:32 (xds_cluster_end2end_test@poller=epoll1+0x3279715) grpc#16 grpc::internal::CallbackWithSuccessTag::StaticRun(grpc_completion_queue_functor*, int) /proc/self/cwd/include/grpcpp/support/callback_common.h:212:47 (xds_cluster_end2end_test@poller=epoll1+0x3279329) grpc#17 cq_end_op_for_callback(grpc_completion_queue*, void*, absl::lts_20250127::Status, void (*)(void*, grpc_cq_completion*), void*, grpc_cq_completion*, bool)::$_0::operator()() const /proc/self/cwd/src/core/lib/surface/completion_queue.cc:913:9 (xds_cluster_end2end_test@poller=epoll1+0x545e2e3) grpc#18 void std::__invoke_impl(std::__invoke_other, cq_end_op_for_callback(grpc_completion_queue*, void*, absl::lts_20250127::Status, void (*)(void*, grpc_cq_completion*), void*, grpc_cq_completion*, bool)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/invoke.h:60:14 (xds_cluster_end2end_test@poller=epoll1+0x545e205) grpc#19 std::__invoke_result::type std::__invoke(cq_end_op_for_callback(grpc_completion_queue*, void*, absl::lts_20250127::Status, void (*)(void*, grpc_cq_completion*), void*, grpc_cq_completion*, bool)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/invoke.h:95:14 (xds_cluster_end2end_test@poller=epoll1+0x545e1b5) grpc#20 std::invoke_result::type std::invoke(cq_end_op_for_callback(grpc_completion_queue*, void*, absl::lts_20250127::Status, void (*)(void*, grpc_cq_completion*), void*, grpc_cq_completion*, bool)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/functional:81:14 (xds_cluster_end2end_test@poller=epoll1+0x545e165) grpc#21 void absl::lts_20250127::internal_any_invocable::InvokeR(cq_end_op_for_callback(grpc_completion_queue*, void*, absl::lts_20250127::Status, void (*)(void*, grpc_cq_completion*), void*, grpc_cq_completion*, bool)::$_0&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (xds_cluster_end2end_test@poller=epoll1+0x545e115) grpc#22 void absl::lts_20250127::internal_any_invocable::RemoteInvoker(absl::lts_20250127::internal_any_invocable::TypeErasedState*) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (xds_cluster_end2end_test@poller=epoll1+0x545df6d) grpc#23 absl::lts_20250127::internal_any_invocable::Impl::operator()() /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (xds_cluster_end2end_test@poller=epoll1+0x4aa14ef) grpc#24 grpc_event_engine::experimental::SelfDeletingClosure::Run() /proc/self/cwd/./src/core/lib/event_engine/common_closures.h:54:5 (xds_cluster_end2end_test@poller=epoll1+0x56a78ad) grpc#25 grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::Step() /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:552:14 (xds_cluster_end2end_test@poller=epoll1+0x56a4663) grpc#26 grpc_event_engine::experimental::WorkStealingThreadPool::ThreadState::ThreadBody() /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:515:10 (xds_cluster_end2end_test@poller=epoll1+0x56a3d47) grpc#27 grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::StartThread()::$_0::operator()(void*) const /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:277:17 (xds_cluster_end2end_test@poller=epoll1+0x56a5049) grpc#28 grpc_event_engine::experimental::WorkStealingThreadPool::WorkStealingThreadPoolImpl::StartThread()::$_0::__invoke(void*) /proc/self/cwd/src/core/lib/event_engine/thread_pool/work_stealing_thread_pool.cc:275:7 (xds_cluster_end2end_test@poller=epoll1+0x56a4fe9) grpc#29 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const /proc/self/cwd/src/core/util/posix/thd.cc:145:11 (xds_cluster_end2end_test@poller=epoll1+0x58479f0) grpc#30 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) /proc/self/cwd/src/core/util/posix/thd.cc:115:9 (xds_cluster_end2end_test@poller=epoll1+0x5847819) ``` Closes grpc#39806 COPYBARA_INTEGRATE_REVIEW=grpc#39806 from markdroth:lrs_server_callback_fix e10ed85 PiperOrigin-RevId: 769289887
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PanCakes to the rescue!
We noticed that our 'sanity' test was going to fail, but we think we can fix that automatically, so we put together this PR to do just that!
If you'd like to opt-out of these PR's, add yourself to NO_AUTOFIX_USERS in .github/workflows/pr-auto-fix.yaml