Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: Intermittent test failures for clang & gcc-coverage #379

Closed
2 tasks done
dagardner-nv opened this issue Sep 6, 2023 · 2 comments · Fixed by #382 or #391
Closed
2 tasks done

[BUG]: Intermittent test failures for clang & gcc-coverage #379

dagardner-nv opened this issue Sep 6, 2023 · 2 comments · Fixed by #382 or #391
Labels
bug Something isn't working

Comments

@dagardner-nv
Copy link
Contributor

Version

23.11

Which installation method(s) does this occur on?

Source

Describe the bug.

Tests are intermittently failing with a SIGSEGV from RPC and control plane tests.

Examples:

clang

[ RUN      ] TestRPC.StreamingPingPong
*** Aborted at 1694016732 (unix time) try "date -d @1694016732" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 4166 (TID 0x7f84edca8000) from PID 0; stack trace: ***
    @     0x7f84fc542197 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f84fc8ce420 (unknown)
    @     0x7f84fc4f4f2e boost::fibers::wait_queue::notify_all()
    @     0x7f84fc4f2ee3 boost::fibers::condition_variable_any::notify_all()
    @     0x7f84fcbe6202 boost::fibers::promise<>::set_value()
    @     0x7f84fcbe50de _ZNK5rxcpp6detail17specific_observerIN3mrc3rpc13ProgressEventENS_8observerIS4_NS0_22stateless_observer_tagEZNS2_4node11GenericSinkIS4_NS2_8runnable7ContextEEC1EvEUlS4_E_NS0_12OnErrorEmptyEvEEvE7on_nextEOS4_
    @     0x7f84fcbe5b2a _ZNK5rxcpp6detail17specific_observerIN3mrc3rpc13ProgressEventENS_8observerIS4_NS0_22stateless_observer_tagEZNS2_4node6RxSinkIS4_NS2_8runnable7ContextEE12do_subscribeERNS_22composite_subscriptionEEUlS4_E_ZNSB_12do_subscribeESD_EUlNSt15__exception_ptr13exception_ptrEE_ZNSB_12do_subscribeESD_EUlvE_EEvE7on_nextEOS4_
    @     0x7f84fcbd7d3a rxcpp::subscriber<>::nextdetacher::operator()<>()
    @     0x7f84fcbe1b61 mrc::node::RxSinkBase<>::progress_engine()
    @     0x7f84fcbe12f7 _ZN5rxcpp12on_exceptionIZNKS_7sources6detail6createIN3mrc3rpc13ProgressEventEZNS4_4node10RxSinkBaseIS6_EC1EvEUlNS_10subscriberIS6_NS_8observerIS6_vvvvEEEEE_E12on_subscribeISD_EEvT_EUlvE_SD_EENSt9enable_ifIXsr13is_subscriberIT0_EE5valueENS_6detail17maybe_from_resultISH_E4typeEE4typeERKSH_RKSK_
    @     0x7f84fcbe11c5 _ZSt13__invoke_implIvRZN5rxcpp18dynamic_observableIN3mrc3rpc13ProgressEventEE9constructINS0_7sources6detail6createIS4_ZNS2_4node10RxSinkBaseIS4_EC1EvEUlNS0_10subscriberIS4_NS0_8observerIS4_vvvvEEEEE_EEEEvOT_ONS7_10tag_sourceEEUlSG_E_JSG_EESJ_St14__invoke_otherOT0_DpOT1_
    @     0x7f84fcbd8c6b rxcpp::detail::safe_subscriber<>::subscribe()
    @     0x7f84fcbdd43a rxcpp::schedulers::detail::action_tailrecurser::operator()()
    @     0x559f17344986 rxcpp::schedulers::current_thread::current_worker::schedule()
    @     0x7f84fcbd896b _ZNK5rxcpp10schedulers6worker8scheduleIRNS_6detail15safe_subscriberINS_18dynamic_observableIN3mrc3rpc13ProgressEventEEENS_10subscriberIS8_NS_8observerIS8_vvvvEEEEEEJEEENSt9enable_ifIXaaoosr6detail18is_action_functionIT_EE5valuesr15is_subscriptionISH_EE5valuentsr14is_schedulableISH_EE5valueEvE4typeEOSH_DpOT0_
    @     0x7f84fcbd84d7 rxcpp::observable<>::detail_subscribe<>()
    @     0x7f84fcbe5978 rxcpp::observable<>::subscribe<>()
    @     0x7f84fcbdef91 mrc::node::RxSink<>::do_subscribe()
    @     0x559f17337d04 mrc::node::RxRunnable<>::run()
    @     0x7f84fcbcfabe mrc::runnable::RunnableWithContext<>::main()
    @     0x7f84fcdcb9e5 std::_Function_handler<>::_M_invoke()
    @     0x7f84fccf5a76 boost::fibers::detail::task_object<>::run()
    @     0x7f84fccf5db2 _ZN5boost6fibers6detail11task_objectIZN3mrc4core14FiberTaskQueue7enqueueISt8functionIFvvEEJEEENS0_6futureINSt9result_ofIFT_DpT0_EE4typeEEEONS3_13FiberMetaDataEOSC_DpOSD_EUlvE_SaINS0_13packaged_taskIS8_EEEvJEE3runEv
    @     0x7f84fcd1409e boost::fibers::worker_context<>::run_()
    @     0x7f84fcd141bd boost::context::detail::fiber_entry<>()
    @     0x7f84fc4e511f make_fcontext

gcc-coverage:

[ RUN      ] TestControlPlane.DoubleClientConnectExchangeDisconnect
*** Aborted at 1694020510 (unix time) try "date -d @1694020510" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 26896 (TID 0x7f95767fc000) from PID 0; stack trace: ***
    @     0x7f963010b197 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f96304a0420 (unknown)
    @     0x7f96300bdf2e boost::fibers::wait_queue::notify_all()
    @     0x7f96300bbee3 boost::fibers::condition_variable_any::notify_all()
    @     0x7f963107bf20 boost::fibers::condition_variable::notify_all()
    @     0x7f963107c12f boost::fibers::detail::shared_state_base::mark_ready_and_notify_()
    @     0x7f96310b686b boost::fibers::detail::shared_state<>::set_value_()
    @     0x7f96310aa8fc boost::fibers::detail::shared_state<>::set_value()
    @     0x7f963109ca83 boost::fibers::promise<>::set_value()
    @     0x7f963108f5c3 mrc::rpc::PromiseHandler::on_data()
    @     0x7f96310ad02a _ZZN3mrc4node11GenericSinkINS_3rpc13ProgressEventENS_8runnable7ContextEEC4EvENKUlS3_E_clES3_
    @     0x7f9631115470 _ZNK5rxcpp8observerIN3mrc3rpc13ProgressEventENS_6detail22stateless_observer_tagEZNS1_4node11GenericSinkIS3_NS1_8runnable7ContextEEC4EvEUlS3_E_NS4_12OnErrorEmptyEvE7on_nextEOS3_
    @     0x7f9631107c92 _ZNK5rxcpp6detail17specific_observerIN3mrc3rpc13ProgressEventENS_8observerIS4_NS0_22stateless_observer_tagEZNS2_4node11GenericSinkIS4_NS2_8runnable7ContextEEC4EvEUlS4_E_NS0_12OnErrorEmptyEvEEvE7on_nextEOS4_
    @     0x7f96310db796 rxcpp::observer<>::on_next<>()
    @     0x7f963110e90c _ZZN3mrc4node6RxSinkINS_3rpc13ProgressEventENS_8runnable7ContextEE12do_subscribeERN5rxcpp22composite_subscriptionEENKUlS3_E_clES3_
    @     0x7f9631162662 _ZNK5rxcpp8observerIN3mrc3rpc13ProgressEventENS_6detail22stateless_observer_tagEZNS1_4node6RxSinkIS3_NS1_8runnable7ContextEE12do_subscribeERNS_22composite_subscriptionEEUlS3_E_ZNSA_12do_subscribeESC_EUlNSt15__exception_ptr13exception_ptrEE0_ZNSA_12do_subscribeESC_EUlvE1_E7on_nextEOS3_
    @     0x7f963115cbfc _ZNK5rxcpp6detail17specific_observerIN3mrc3rpc13ProgressEventENS_8observerIS4_NS0_22stateless_observer_tagEZNS2_4node6RxSinkIS4_NS2_8runnable7ContextEE12do_subscribeERNS_22composite_subscriptionEEUlS4_E_ZNSB_12do_subscribeESD_EUlNSt15__exception_ptr13exception_ptrEE0_ZNSB_12do_subscribeESD_EUlvE1_EEvE7on_nextEOS4_
    @     0x7f96310db796 rxcpp::observer<>::on_next<>()
    @     0x7f96310d249a rxcpp::subscriber<>::nextdetacher::operator()<>()
    @     0x7f96310c8d86 rxcpp::subscriber<>::on_next<>()
    @     0x7f96310c06c3 mrc::node::RxSinkBase<>::progress_engine()
    @     0x7f96310b75ee _ZZN3mrc4node10RxSinkBaseINS_3rpc13ProgressEventEEC4EvENKUlN5rxcpp10subscriberIS3_NS5_8observerIS3_vvvvEEEEE_clES9_
    @     0x7f96310dbc82 _ZZNK5rxcpp7sources6detail6createIN3mrc3rpc13ProgressEventEZNS3_4node10RxSinkBaseIS5_EC4EvEUlNS_10subscriberIS5_NS_8observerIS5_vvvvEEEEE_E12on_subscribeISC_EEvT_ENKUlvE_clEv
    @     0x7f96310e348f _ZN5rxcpp12on_exceptionIZNKS_7sources6detail6createIN3mrc3rpc13ProgressEventEZNS4_4node10RxSinkBaseIS6_EC4EvEUlNS_10subscriberIS6_NS_8observerIS6_vvvvEEEEE_E12on_subscribeISD_EEvT_EUlvE_SD_EENSt9enable_ifIXsrNS_13is_subscriberIT0_EE5valueENS_6detail17maybe_from_resultISH_E4typeEE4typeERKSH_RKSL_
    @     0x7f96310dbd68 _ZNK5rxcpp7sources6detail6createIN3mrc3rpc13ProgressEventEZNS3_4node10RxSinkBaseIS5_EC4EvEUlNS_10subscriberIS5_NS_8observerIS5_vvvvEEEEE_E12on_subscribeISC_EEvT_
    @     0x7f96310d2a34 _ZZN5rxcpp18dynamic_observableIN3mrc3rpc13ProgressEventEE9constructINS_7sources6detail6createIS3_ZNS1_4node10RxSinkBaseIS3_EC4EvEUlNS_10subscriberIS3_NS_8observerIS3_vvvvEEEEE_EEEEvOT_ONS6_10tag_sourceEENUlSF_E_clESF_
    @     0x7f96310f498f _ZSt13__invoke_implIvRZN5rxcpp18dynamic_observableIN3mrc3rpc13ProgressEventEE9constructINS0_7sources6detail6createIS4_ZNS2_4node10RxSinkBaseIS4_EC4EvEUlNS0_10subscriberIS4_NS0_8observerIS4_vvvvEEEEE_EEEEvOT_ONS7_10tag_sourceEEUlSG_E_JSG_EESJ_St14__invoke_otherOT0_DpOT1_
    @     0x7f96310ef937 _ZSt10__invoke_rIvRZN5rxcpp18dynamic_observableIN3mrc3rpc13ProgressEventEE9constructINS0_7sources6detail6createIS4_ZNS2_4node10RxSinkBaseIS4_EC4EvEUlNS0_10subscriberIS4_NS0_8observerIS4_vvvvEEEEE_EEEEvOT_ONS7_10tag_sourceEEUlSG_E_JSG_EENSt9enable_ifIX16is_invocable_r_vISJ_T0_DpT1_EESJ_E4typeEOSQ_DpOSR_
    @     0x7f96310ea198 _ZNSt17_Function_handlerIFvN5rxcpp10subscriberIN3mrc3rpc13ProgressEventENS0_8observerIS4_vvvvEEEEEZNS0_18dynamic_observableIS4_E9constructINS0_7sources6detail6createIS4_ZNS2_4node10RxSinkBaseIS4_EC4EvEUlS7_E_EEEEvOT_ONSC_10tag_sourceEEUlS7_E_E9_M_invokeERKSt9_Any_dataOS7_
    @     0x7f963113115a std::function<>::operator()()
    @     0x7f963112d1cb rxcpp::dynamic_observable<>::on_subscribe()
    @     0x7f9631126d2d rxcpp::detail::safe_subscriber<>::subscribe()

clang:

[ RUN      ] TestControlPlane.SingleClientConnectDisconnect
*** Aborted at 1694020471 (unix time) try "date -d @1694020471" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 4165 (TID 0x7fcce0ff9000) from PID 0; stack trace: ***
    @     0x7fcd8ebd6197 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7fcd8ef62420 (unknown)
    @     0x7fcd8eb88f2e boost::fibers::wait_queue::notify_all()
    @     0x7fcd8eb86ee3 boost::fibers::condition_variable_any::notify_all()
    @     0x7fcd8f27a202 boost::fibers::promise<>::set_value()
    @     0x7fcd8f2790de _ZNK5rxcpp6detail17specific_observerIN3mrc3rpc13ProgressEventENS_8observerIS4_NS0_22stateless_observer_tagEZNS2_4node11GenericSinkIS4_NS2_8runnable7ContextEEC1EvEUlS4_E_NS0_12OnErrorEmptyEvEEvE7on_nextEOS4_
    @     0x7fcd8f279b2a _ZNK5rxcpp6detail17specific_observerIN3mrc3rpc13ProgressEventENS_8observerIS4_NS0_22stateless_observer_tagEZNS2_4node6RxSinkIS4_NS2_8runnable7ContextEE12do_subscribeERNS_22composite_subscriptionEEUlS4_E_ZNSB_12do_subscribeESD_EUlNSt15__exception_ptr13exception_ptrEE_ZNSB_12do_subscribeESD_EUlvE_EEvE7on_nextEOS4_
    @     0x7fcd8f26bd3a rxcpp::subscriber<>::nextdetacher::operator()<>()
    @     0x7fcd8f275b61 mrc::node::RxSinkBase<>::progress_engine()
    @     0x7fcd8f2752f7 _ZN5rxcpp12on_exceptionIZNKS_7sources6detail6createIN3mrc3rpc13ProgressEventEZNS4_4node10RxSinkBaseIS6_EC1EvEUlNS_10subscriberIS6_NS_8observerIS6_vvvvEEEEE_E12on_subscribeISD_EEvT_EUlvE_SD_EENSt9enable_ifIXsr13is_subscriberIT0_EE5valueENS_6detail17maybe_from_resultISH_E4typeEE4typeERKSH_RKSK_
    @     0x7fcd8f2751c5 _ZSt13__invoke_implIvRZN5rxcpp18dynamic_observableIN3mrc3rpc13ProgressEventEE9constructINS0_7sources6detail6createIS4_ZNS2_4node10RxSinkBaseIS4_EC1EvEUlNS0_10subscriberIS4_NS0_8observerIS4_vvvvEEEEE_EEEEvOT_ONS7_10tag_sourceEEUlSG_E_JSG_EESJ_St14__invoke_otherOT0_DpOT1_
    @     0x7fcd8f26cc6b rxcpp::detail::safe_subscriber<>::subscribe()
    @     0x7fcd8f27143a rxcpp::schedulers::detail::action_tailrecurser::operator()()
    @     0x559f67bd5986 rxcpp::schedulers::current_thread::current_worker::schedule()
    @     0x7fcd8f26c96b _ZNK5rxcpp10schedulers6worker8scheduleIRNS_6detail15safe_subscriberINS_18dynamic_observableIN3mrc3rpc13ProgressEventEEENS_10subscriberIS8_NS_8observerIS8_vvvvEEEEEEJEEENSt9enable_ifIXaaoosr6detail18is_action_functionIT_EE5valuesr15is_subscriptionISH_EE5valuentsr14is_schedulableISH_EE5valueEvE4typeEOSH_DpOT0_
    @     0x7fcd8f26c4d7 rxcpp::observable<>::detail_subscribe<>()
    @     0x7fcd8f279978 rxcpp::observable<>::subscribe<>()
    @     0x7fcd8f272f91 mrc::node::RxSink<>::do_subscribe()
    @     0x559f67bc8d04 mrc::node::RxRunnable<>::run()
    @     0x7fcd8f263abe mrc::runnable::RunnableWithContext<>::main()
    @     0x7fcd8f45f9e5 std::_Function_handler<>::_M_invoke()
    @     0x7fcd8f389a76 boost::fibers::detail::task_object<>::run()
    @     0x7fcd8f389db2 _ZN5boost6fibers6detail11task_objectIZN3mrc4core14FiberTaskQueue7enqueueISt8functionIFvvEEJEEENS0_6futureINSt9result_ofIFT_DpT0_EE4typeEEEONS3_13FiberMetaDataEOSC_DpOSD_EUlvE_SaINS0_13packaged_taskIS8_EEEvJEE3runEv
    @     0x7fcd8f3a809e boost::fibers::worker_context<>::run_()
    @     0x7fcd8f3a81bd boost::context::detail::fiber_entry<>()
    @     0x7fcd8eb7911f make_fcontext

clang:

[ RUN      ] TestControlPlane.SingleClientConnectDisconnect
*** Aborted at 1694020205 (unix time) try "date -d @1694020205" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 347426 (TID 0x7f51a9ffb000) from PID 0; stack trace: ***
    @     0x7f5260053197 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f52603df420 (unknown)
    @     0x7f5260005f2e boost::fibers::wait_queue::notify_all()
    @     0x7f5260003ee3 boost::fibers::condition_variable_any::notify_all()
    @     0x7f52606f7202 boost::fibers::promise<>::set_value()
    @     0x7f52606f60de _ZNK5rxcpp6detail17specific_observerIN3mrc3rpc13ProgressEventENS_8observerIS4_NS0_22stateless_observer_tagEZNS2_4node11GenericSinkIS4_NS2_8runnable7ContextEEC1EvEUlS4_E_NS0_12OnErrorEmptyEvEEvE7on_nextEOS4_
    @     0x7f52606f6b2a _ZNK5rxcpp6detail17specific_observerIN3mrc3rpc13ProgressEventENS_8observerIS4_NS0_22stateless_observer_tagEZNS2_4node6RxSinkIS4_NS2_8runnable7ContextEE12do_subscribeERNS_22composite_subscriptionEEUlS4_E_ZNSB_12do_subscribeESD_EUlNSt15__exception_ptr13exception_ptrEE_ZNSB_12do_subscribeESD_EUlvE_EEvE7on_nextEOS4_
    @     0x7f52606e8d3a rxcpp::subscriber<>::nextdetacher::operator()<>()
    @     0x7f52606f2b61 mrc::node::RxSinkBase<>::progress_engine()
    @     0x7f52606f22f7 _ZN5rxcpp12on_exceptionIZNKS_7sources6detail6createIN3mrc3rpc13ProgressEventEZNS4_4node10RxSinkBaseIS6_EC1EvEUlNS_10subscriberIS6_NS_8observerIS6_vvvvEEEEE_E12on_subscribeISD_EEvT_EUlvE_SD_EENSt9enable_ifIXsr13is_subscriberIT0_EE5valueENS_6detail17maybe_from_resultISH_E4typeEE4typeERKSH_RKSK_
    @     0x7f52606f21c5 _ZSt13__invoke_implIvRZN5rxcpp18dynamic_observableIN3mrc3rpc13ProgressEventEE9constructINS0_7sources6detail6createIS4_ZNS2_4node10RxSinkBaseIS4_EC1EvEUlNS0_10subscriberIS4_NS0_8observerIS4_vvvvEEEEE_EEEEvOT_ONS7_10tag_sourceEEUlSG_E_JSG_EESJ_St14__invoke_otherOT0_DpOT1_
    @     0x7f52606e9c6b rxcpp::detail::safe_subscriber<>::subscribe()
    @     0x7f52606ee43a rxcpp::schedulers::detail::action_tailrecurser::operator()()
    @     0x55f060ea13d6 rxcpp::schedulers::current_thread::current_worker::schedule()
    @     0x7f52606e996b _ZNK5rxcpp10schedulers6worker8scheduleIRNS_6detail15safe_subscriberINS_18dynamic_observableIN3mrc3rpc13ProgressEventEEENS_10subscriberIS8_NS_8observerIS8_vvvvEEEEEEJEEENSt9enable_ifIXaaoosr6detail18is_action_functionIT_EE5valuesr15is_subscriptionISH_EE5valuentsr14is_schedulableISH_EE5valueEvE4typeEOSH_DpOT0_
    @     0x7f52606e94d7 rxcpp::observable<>::detail_subscribe<>()
    @     0x7f52606f6978 rxcpp::observable<>::subscribe<>()
    @     0x7f52606eff91 mrc::node::RxSink<>::do_subscribe()
    @     0x55f060e94754 mrc::node::RxRunnable<>::run()
    @     0x7f52606e0abe mrc::runnable::RunnableWithContext<>::main()
    @     0x7f52608dc9e5 std::_Function_handler<>::_M_invoke()
    @     0x7f5260806a76 boost::fibers::detail::task_object<>::run()
    @     0x7f5260806db2 _ZN5boost6fibers6detail11task_objectIZN3mrc4core14FiberTaskQueue7enqueueISt8functionIFvvEEJEEENS0_6futureINSt9result_ofIFT_DpT0_EE4typeEEEONS3_13FiberMetaDataEOSC_DpOSD_EUlvE_SaINS0_13packaged_taskIS8_EEEvJEE3runEv
    @     0x7f526082509e boost::fibers::worker_context<>::run_()
    @     0x7f52608251bd boost::context::detail::fiber_entry<>()
    @     0x7f525fff611f make_fcontext

Minimum reproducible example

Open a PR
https://github.com/nv-morpheus/MRC/pull/378

Relevant log output

No response

Full env printout

No response

Other/Misc.

No response

Code of Conduct

  • I agree to follow MRC's Code of Conduct
  • I have searched the open bugs and have found no duplicates for this bug report
@dagardner-nv dagardner-nv added the bug Something isn't working label Sep 6, 2023
@dagardner-nv
Copy link
Contributor Author

dagardner-nv commented Sep 6, 2023 •
edited
Loading

I got a failure locally with a clang debug build. I had to run the test repeatedly, and got a failure on the 112th iteration:

cd build/cpp/mrc/src/tests
CMD="./test_mrc_private.x"
RET=0
ITR=0
while [ ${RET} -eq 0 ]
do
    ${CMD}
    RET=$?
    ITR=$(expr ${ITR} + 1)

    echo "Iteration ${ITR} exited ${RET}"
done
[ RUN      ] TestControlPlane.SingleClientConnectDisconnect
*** Aborted at 1694023687 (unix time) try "date -d @1694023687" if you are using GNU date ***
PC: @                0x0 (unknown)
*** SIGSEGV (@0x0) received by PID 3442533 (TID 0x7f6ed7fff000) from PID 0; stack trace: ***
    @     0x7f6f9e7e6197 google::(anonymous namespace)::FailureSignalHandler()
    @     0x7f6f9d63daa0 (unknown)
    @     0x7f6f9f6b9f2e boost::fibers::wait_queue::notify_all()
    @     0x7f6f9f6b7ee3 boost::fibers::condition_variable_any::notify_all()
    @     0x7f6f9ef1c656 boost::fibers::condition_variable::notify_all()
    @     0x7f6f9ef1c596 boost::fibers::detail::shared_state_base::mark_ready_and_notify_()
    @     0x7f6f9ef1c3cc boost::fibers::detail::shared_state<>::set_value_()
    @     0x7f6f9ef1bfd7 boost::fibers::detail::shared_state<>::set_value()
    @     0x7f6f9ef1be96 boost::fibers::promise<>::set_value()
    @     0x7f6f9ef0e23c mrc::rpc::PromiseHandler::on_data()
    @     0x7f6f9ef19596 _ZZN3mrc4node11GenericSinkINS_3rpc13ProgressEventENS_8runnable7ContextEEC1EvENKUlS3_E_clES3_
    @     0x7f6f9ef195f8 _ZNK5rxcpp8observerIN3mrc3rpc13ProgressEventENS_6detail22stateless_observer_tagEZNS1_4node11GenericSinkIS3_NS1_8runnable7ContextEEC1EvEUlS3_E_NS4_12OnErrorEmptyEvE7on_nextEOS3_  
    @     0x7f6f9ef19441 _ZNK5rxcpp6detail17specific_observerIN3mrc3rpc13ProgressEventENS_8observerIS4_NS0_22stateless_observer_tagEZNS2_4node11GenericSinkIS4_NS2_8runnable7ContextEEC1EvEUlS4_E_NS0_12OnErrorEmptyEvEEvE7on_nextEOS4_
    @     0x7f6f9eefbc80 rxcpp::observer<>::on_next<>()
    @     0x7f6f9ef1b3f9 _ZZN3mrc4node6RxSinkINS_3rpc13ProgressEventENS_8runnable7ContextEE12do_subscribeERN5rxcpp22composite_subscriptionEENKUlS3_E_clES3_
    @     0x7f6f9ef1b458 _ZNK5rxcpp8observerIN3mrc3rpc13ProgressEventENS_6detail22stateless_observer_tagEZNS1_4node6RxSinkIS3_NS1_8runnable7ContextEE12do_subscribeERNS_22composite_subscriptionEEUlS3_E_ZNSA_12do_subscribeESC_EUlNSt15__exception_ptr13exception_ptrEE_ZNSA_12do_subscribeESC_EUlvE_E7on_nextEOS3_
    @     0x7f6f9ef1b2a1 _ZNK5rxcpp6detail17specific_observerIN3mrc3rpc13ProgressEventENS_8observerIS4_NS0_22stateless_observer_tagEZNS2_4node6RxSinkIS4_NS2_8runnable7ContextEE12do_subscribeERNS_22composite_subscriptionEEUlS4_E_ZNSB_12do_subscribeESD_EUlNSt15__exception_ptr13exception_ptrEE_ZNSB_12do_subscribeESD_EUlvE_EEvE7on_nextEOS4_
    @     0x7f6f9eefbc80 rxcpp::observer<>::on_next<>()
    @     0x7f6f9eefbb31 rxcpp::subscriber<>::nextdetacher::operator()<>()
    @     0x7f6f9eefba85 rxcpp::subscriber<>::on_next<>()
    @     0x7f6f9ef10c7e mrc::node::RxSinkBase<>::progress_engine()
    @     0x7f6f9ef10b68 _ZZN3mrc4node10RxSinkBaseINS_3rpc13ProgressEventEEC1EvENKUlN5rxcpp10subscriberIS3_NS5_8observerIS3_vvvvEEEEE_clES9_
    @     0x7f6f9ef10aa2 _ZZNK5rxcpp7sources6detail6createIN3mrc3rpc13ProgressEventEZNS3_4node10RxSinkBaseIS5_EC1EvEUlNS_10subscriberIS5_NS_8observerIS5_vvvvEEEEE_E12on_subscribeISC_EEvT_ENKUlvE_clEv
    @     0x7f6f9ef10955 _ZN5rxcpp12on_exceptionIZNKS_7sources6detail6createIN3mrc3rpc13ProgressEventEZNS4_4node10RxSinkBaseIS6_EC1EvEUlNS_10subscriberIS6_NS_8observerIS6_vvvvEEEEE_E12on_subscribeISD_EEvT_EUlvE_SD_EENSt9enable_ifIXsr13is_subscriberIT0_EE5valueENS_6detail17maybe_from_resultISH_E4typeEE4typeERKSH_RKSK_
    @     0x7f6f9ef108e9 _ZNK5rxcpp7sources6detail6createIN3mrc3rpc13ProgressEventEZNS3_4node10RxSinkBaseIS5_EC1EvEUlNS_10subscriberIS5_NS_8observerIS5_vvvvEEEEE_E12on_subscribeISC_EEvT_
    @     0x7f6f9ef1085b _ZZN5rxcpp18dynamic_observableIN3mrc3rpc13ProgressEventEE9constructINS_7sources6detail6createIS3_ZNS1_4node10RxSinkBaseIS3_EC1EvEUlNS_10subscriberIS3_NS_8observerIS3_vvvvEEEEE_EEEEvOT_ONS6_10tag_sourceEENUlSF_E_clESF_
    @     0x7f6f9ef107c6 _ZSt13__invoke_implIvRZN5rxcpp18dynamic_observableIN3mrc3rpc13ProgressEventEE9constructINS0_7sources6detail6createIS4_ZNS2_4node10RxSinkBaseIS4_EC1EvEUlNS0_10subscriberIS4_NS0_8observerIS4_vvvvEEEEE_EEEEvOT_ONS7_10tag_sourceEEUlSG_E_JSG_EESJ_St14__invoke_otherOT0_DpOT1_
    @     0x7f6f9ef1073d _ZSt10__invoke_rIvRZN5rxcpp18dynamic_observableIN3mrc3rpc13ProgressEventEE9constructINS0_7sources6detail6createIS4_ZNS2_4node10RxSinkBaseIS4_EC1EvEUlNS0_10subscriberIS4_NS0_8observerIS4_vvvvEEEEE_EEEEvOT_ONS7_10tag_sourceEEUlSG_E_JSG_EENSt9enable_ifIX16is_invocable_r_vISJ_T0_DpT1_EESJ_E4typeEOSQ_DpOSR_
    @     0x7f6f9ef10655 _ZNSt17_Function_handlerIFvN5rxcpp10subscriberIN3mrc3rpc13ProgressEventENS0_8observerIS4_vvvvEEEEEZNS0_18dynamic_observableIS4_E9constructINS0_7sources6detail6createIS4_ZNS2_4node10RxSinkBaseIS4_EC1EvEUlS7_E_EEEEvOT_ONSC_10tag_sourceEEUlS7_E_E9_M_invokeERKSt9_Any_dataOS7_
    @     0x7f6f9eef9e3e std::function<>::operator()()
    @     0x7f6f9ef0a51c rxcpp::dynamic_observable<>::on_subscribe()
    @     0x7f6f9eefd00a rxcpp::detail::safe_subscriber<>::subscribe()
/home/dagardner/scripts/dountil.sh: line 12: 3442533 Segmentation fault      (core dumped) ${CMD}
Iteration 112 exited 139

gdb backtrace from core file:

#0  0x00007f6f9f6b9f2e in boost::fibers::wait_queue::notify_all() () from /home/dagardner/work/conda/envs/mrc/lib/libboost_fiber.so.1.82.0
#1  0x00007f6f9f6b7ee3 in boost::fibers::condition_variable_any::notify_all() () from /home/dagardner/work/conda/envs/mrc/lib/libboost_fiber.so.1.82.0
#2  0x00007f6f9ef1c656 in boost::fibers::condition_variable::notify_all (this=0x7f6ec40413b0) at /home/dagardner/work/conda/envs/mrc/include/boost/fiber/condition_variable.hpp:159
#3  0x00007f6f9ef1c596 in boost::fibers::detail::shared_state_base::mark_ready_and_notify_ (this=0x7f6ec40413a0, lk=...) at /home/dagardner/work/conda/envs/mrc/include/boost/fiber/future/detail/shared_state.hpp:51
#4  0x00007f6f9ef1c3cc in boost::fibers::detail::shared_state<bool>::set_value_ (this=0x7f6ec40413a0, value=@0x7f6edc03f3a0: true, lk=...) at /home/dagardner/work/conda/envs/mrc/include/boost/fiber/future/detail/shared_state.hpp:168
#5  0x00007f6f9ef1bfd7 in boost::fibers::detail::shared_state<bool>::set_value (this=0x7f6ec40413a0, value=@0x7f6edc03f3a0: true) at /home/dagardner/work/conda/envs/mrc/include/boost/fiber/future/detail/shared_state.hpp:205
#6  0x00007f6f9ef1be96 in boost::fibers::promise<bool>::set_value (this=0x7f6ec403f610, value=@0x7f6edc03f3a0: true) at /home/dagardner/work/conda/envs/mrc/include/boost/fiber/future/promise.hpp:127
#7  0x00007f6f9ef0e23c in mrc::rpc::PromiseHandler::on_data (this=0x55e36d38de70, event=...) at /home/dagardner/work/mrc/cpp/mrc/src/internal/grpc/promise_handler.hpp:36
#8  0x00007f6f9ef19596 in mrc::node::GenericSink<mrc::rpc::ProgressEvent, mrc::runnable::Context>::GenericSink()::{lambda(mrc::rpc::ProgressEvent)#1}::operator()(mrc::rpc::ProgressEvent) const (this=0x55e36d300818, data=...) at /home/dagardner/work/mrc/cpp/mrc/include/mrc/node/generic_sink.hpp:58
#9  0x00007f6f9ef195f8 in rxcpp::observer<mrc::rpc::ProgressEvent, rxcpp::detail::stateless_observer_tag, mrc::node::GenericSink<mrc::rpc::ProgressEvent, mrc::runnable::Context>::GenericSink()::{lambda(mrc::rpc::ProgressEvent)#1}, rxcpp::detail::OnErrorEmpty, void>::on_next(mrc::rpc::ProgressEvent&&) const (this=0x55e36d300818, t=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-observer.hpp:335
#10 0x00007f6f9ef19441 in rxcpp::detail::specific_observer<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, rxcpp::detail::stateless_observer_tag, mrc::node::GenericSink<mrc::rpc::ProgressEvent, mrc::runnable::Context>::GenericSink()::{lambda(mrc::rpc::ProgressEvent)#1}, rxcpp::detail::OnErrorEmpty, void>, void>::on_next(mrc::rpc::ProgressEvent&&) const (this=0x55e36d300800, t=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-observer.hpp:383
#11 0x00007f6f9eefbc80 in rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void>::on_next<mrc::rpc::ProgressEvent> (this=0x55e36d38df18, v=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-observer.hpp:470
#12 0x00007f6f9ef1b3f9 in mrc::node::RxSink<mrc::rpc::ProgressEvent, mrc::runnable::Context>::do_subscribe(rxcpp::composite_subscription&)::{lambda(mrc::rpc::ProgressEvent)#1}::operator()(mrc::rpc::ProgressEvent) const (this=0x7f6edc04d828, data=...) at /home/dagardner/work/mrc/cpp/mrc/include/mrc/node/rx_sink.hpp:130
#13 0x00007f6f9ef1b458 in rxcpp::observer<mrc::rpc::ProgressEvent, rxcpp::detail::stateless_observer_tag, mrc::node::RxSink<mrc::rpc::ProgressEvent, mrc::runnable::Context>::do_subscribe(rxcpp::composite_subscription&)::{lambda(mrc::rpc::ProgressEvent)#1}, mrc::node::RxSink<mrc::rpc::ProgressEvent, mrc::runnable::Context>::do_subscribe(rxcpp::composite_subscription&)::{lambda(std::__exception_ptr::exception_ptr)#1}, mrc::node::RxSink<mrc::rpc::ProgressEvent, mrc::runnable::Context>::do_subscribe(rxcpp::composite_subscription&)::{lambda()#1}>::on_next(mrc::rpc::ProgressEvent&&) const (this=0x7f6edc04d828, t=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-observer.hpp:335
#14 0x00007f6f9ef1b2a1 in rxcpp::detail::specific_observer<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, rxcpp::detail::stateless_observer_tag, mrc::node::RxSink<mrc::rpc::ProgressEvent, mrc::runnable::Context>::do_subscribe(rxcpp::composite_subscription&)::{lambda(mrc::rpc::ProgressEvent)#1}, mrc::node::RxSink<mrc::rpc::ProgressEvent, mrc::runnable::Context>::do_subscribe(rxcpp::composite_subscription&)::{lambda(std::__exception_ptr::exception_ptr)#1}, mrc::node::RxSink<mrc::rpc::ProgressEvent, mrc::runnable::Context>::do_subscribe(rxcpp::composite_subscription&)::{lambda()#1}>, void>::on_next(mrc::rpc::ProgressEvent&&) const (this=0x7f6edc04d810, t=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-observer.hpp:383
#15 0x00007f6f9eefbc80 in rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void>::on_next<mrc::rpc::ProgressEvent> (this=0x7f6edc03f6d0, v=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-observer.hpp:470
#16 0x00007f6f9eefbb31 in rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >::nextdetacher::operator()<mrc::rpc::ProgressEvent> (this=0x7f6edc03f5b8, u=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-subscriber.hpp:54
#17 0x00007f6f9eefba85 in rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >::on_next<mrc::rpc::ProgressEvent> (this=0x7f6edc03f6b0, v=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-subscriber.hpp:181
#18 0x00007f6f9ef10c7e in mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::progress_engine (this=0x55e36d38de70, s=...) at /home/dagardner/work/mrc/cpp/mrc/include/mrc/node/rx_sink_base.hpp:88
#19 0x00007f6f9ef10b68 in mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}::operator()(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >) const (this=0x55e36d3274b0, s=<error reading variable: Cannot access memory at address 0x81>) at /home/dagardner/work/mrc/cpp/mrc/include/mrc/node/rx_sink_base.hpp:66
#20 0x00007f6f9ef10aa2 in rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}>::on_subscribe<rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > >(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >) const::{lambda()#1}::operator()() const (this=0x7f6edc03f780) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/sources/rx-create.hpp:70
#21 0x00007f6f9ef10955 in rxcpp::on_exception<rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}>::on_subscribe<rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > >(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >) const::{lambda()#1}, rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > >(rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}>::on_subscribe<rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > >(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >) const::{lambda()#1} const&, rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > const&) (f=..., s=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-observer.hpp:685
#22 0x00007f6f9ef108e9 in rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}>::on_subscribe<rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > >(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >) const (this=0x55e36d3274b0, o=<error reading variable: Cannot access memory at address 0x7fffffff>) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/sources/rx-create.hpp:68
#23 0x00007f6f9ef1085b in rxcpp::dynamic_observable<mrc::rpc::ProgressEvent>::construct<rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}> >(rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}>&&, rxcpp::sources::tag_source&&)::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}::operator()(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >) (this=0x55e36d3274b0, o=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-observable.hpp:70
#24 0x00007f6f9ef107c6 in std::__invoke_impl<void, rxcpp::dynamic_observable<mrc::rpc::ProgressEvent>::construct<rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}> >(rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}>&&, rxcpp::sources::tag_source&&)::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}&, rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > >(std::__invoke_other, rxcpp::dynamic_observable<mrc::rpc::ProgressEvent>::construct<rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}> >(rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}>&&, rxcpp::sources::tag_source&&)::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}&, rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >&&) (__f=..., __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:61
#25 0x00007f6f9ef1073d in std::__invoke_r<void, rxcpp::dynamic_observable<mrc::rpc::ProgressEvent>::construct<rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}> >(rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}>&&, rxcpp::sources::tag_source&&)::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}&, rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > >(rxcpp::dynamic_observable<mrc::rpc::ProgressEvent>::construct<rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}> >(rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}>&&, rxcpp::sources::tag_source&&)::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}&, rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >&&) (__fn=..., __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:111
#26 0x00007f6f9ef10655 in std::_Function_handler<void (rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >), rxcpp::dynamic_observable<mrc::rpc::ProgressEvent>::construct<rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}> >(rxcpp::sources::detail::create<mrc::rpc::ProgressEvent, mrc::node::RxSinkBase<mrc::rpc::ProgressEvent>::RxSinkBase()::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}>&&, rxcpp::sources::tag_source&&)::{lambda(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)#1}>::_M_invoke(std::_Any_data const&, rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >&&) (__functor=..., __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/std_function.h:291
#27 0x00007f6f9eef9e3e in std::function<void (rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >)>::operator()(rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> >) const (this=0x55e36d3274b0, __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/std_function.h:560
#28 0x00007f6f9ef0a51c in rxcpp::dynamic_observable<mrc::rpc::ProgressEvent>::on_subscribe (this=0x7f6edc0401b8, o=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-observable.hpp:97
#29 0x00007f6f9eefd00a in rxcpp::detail::safe_subscriber<rxcpp::dynamic_observable<mrc::rpc::ProgressEvent>, rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > >::subscribe (this=0x7f6edc068710) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-observable.hpp:450
#30 0x00007f6f9ef0a2c9 in rxcpp::detail::safe_subscriber<rxcpp::dynamic_observable<mrc::rpc::ProgressEvent>, rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > >::operator() (this=0x7f6edc068710) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-observable.hpp:461
#31 0x00007f6f9ef0a29d in std::__invoke_impl<void, rxcpp::detail::safe_subscriber<rxcpp::dynamic_observable<mrc::rpc::ProgressEvent>, rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > >&, rxcpp::schedulers::schedulable const&> (__f=..., __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:61
#32 0x00007f6f9ef0a23d in std::__invoke_r<void, rxcpp::detail::safe_subscriber<rxcpp::dynamic_observable<mrc::rpc::ProgressEvent>, rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > >&, rxcpp::schedulers::schedulable const&> (__fn=..., __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:111
#33 0x00007f6f9ef0a145 in std::_Function_handler<void (rxcpp::schedulers::schedulable const&), rxcpp::detail::safe_subscriber<rxcpp::dynamic_observable<mrc::rpc::ProgressEvent>, rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > > >::_M_invoke(std::_Any_data const&, rxcpp::schedulers::schedulable const&) (__functor=..., __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/std_function.h:291
#34 0x000055e36c0467ee in std::function<void (rxcpp::schedulers::schedulable const&)>::operator()(rxcpp::schedulers::schedulable const&) const (this=0x7f6edc068710, __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/std_function.h:560
#35 0x000055e36c046606 in rxcpp::schedulers::detail::action_tailrecurser::operator() (this=0x7f6edc068700, s=..., r=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-scheduler.hpp:712
#36 0x000055e36c046565 in std::__invoke_impl<void, rxcpp::schedulers::detail::action_tailrecurser&, rxcpp::schedulers::schedulable const&, rxcpp::schedulers::recurse const&> (__f=..., __args=..., __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:61
#37 0x000055e36c046515 in std::__invoke_r<void, rxcpp::schedulers::detail::action_tailrecurser&, rxcpp::schedulers::schedulable const&, rxcpp::schedulers::recurse const&> (__fn=..., __args=..., __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:111
#38 0x000055e36c0463dd in std::_Function_handler<void (rxcpp::schedulers::schedulable const&, rxcpp::schedulers::recurse const&), rxcpp::schedulers::detail::action_tailrecurser>::_M_invoke(std::_Any_data const&, rxcpp::schedulers::schedulable const&, rxcpp::schedulers::recurse const&) (__functor=..., __args=..., __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/std_function.h:291
#39 0x000055e36c042726 in std::function<void (rxcpp::schedulers::schedulable const&, rxcpp::schedulers::recurse const&)>::operator()(rxcpp::schedulers::schedulable const&, rxcpp::schedulers::recurse const&) const (this=0x7f6edc045e30, __args=..., __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/std_function.h:560
#40 0x000055e36c042667 in rxcpp::schedulers::detail::action_type::operator() (this=0x7f6edc045e20, s=..., r=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-scheduler.hpp:679
#41 0x000055e36c04258d in rxcpp::schedulers::action::operator() (this=0x7f6edc03ff68, s=..., r=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-scheduler.hpp:727
#42 0x000055e36c03e778 in rxcpp::schedulers::schedulable::operator() (this=0x7f6edc03ff18, r=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-scheduler.hpp:645
#43 0x000055e36c03e09d in rxcpp::schedulers::current_thread::current_worker::schedule (this=0x7f6f7c865fb0, when=..., scbl=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/schedulers/rx-currentthread.hpp:210
#44 0x000055e36c03df24 in rxcpp::schedulers::current_thread::current_worker::schedule (this=0x7f6f7c865fb0, scbl=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/schedulers/rx-currentthread.hpp:183
#45 0x00007f6f9eefced4 in rxcpp::schedulers::worker::schedule<rxcpp::detail::safe_subscriber<rxcpp::dynamic_observable<mrc::rpc::ProgressEvent>, rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > >&> (this=0x7f6edc040028, a0=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-scheduler.hpp:808
#46 0x00007f6f9eefcaab in rxcpp::observable<mrc::rpc::ProgressEvent, rxcpp::dynamic_observable<mrc::rpc::ProgressEvent> >::detail_subscribe<rxcpp::subscriber<mrc::rpc::ProgressEvent, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void> > > (this=0x7f6edc0401b8, o=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-observable.hpp:534
#47 0x00007f6f9ef1a4da in rxcpp::observable<mrc::rpc::ProgressEvent, rxcpp::dynamic_observable<mrc::rpc::ProgressEvent> >::subscribe<rxcpp::composite_subscription&, rxcpp::observer<mrc::rpc::ProgressEvent, void, void, void, void>&> (this=0x7f6edc0401b8, an=..., an=...) at /home/dagardner/work/mrc/.cache/cpm/rxcpp/b2d191b4c471273143046efec8f535a2cf41a228/Rx/v2/src/rxcpp/rx-observable.hpp:665
#48 0x00007f6f9ef0d7ca in mrc::node::RxSink<mrc::rpc::ProgressEvent, mrc::runnable::Context>::do_subscribe (this=0x55e36d38de70, subscription=...) at /home/dagardner/work/mrc/cpp/mrc/include/mrc/node/rx_sink.hpp:146
#49 0x000055e36c048d7e in mrc::node::RxSubscribable::subscribe (this=0x55e36d38ded8, subscription=...) at /home/dagardner/work/mrc/cpp/mrc/include/mrc/node/rx_subscribable.hpp:38
#50 0x000055e36c01de9d in mrc::node::RxRunnable<mrc::runnable::Context>::run (this=0x55e36d38dec8, ctx=...) at /home/dagardner/work/mrc/cpp/mrc/include/mrc/node/rx_runnable.hpp:70
#51 0x00007f6f9eee868d in mrc::runnable::RunnableWithContext<mrc::runnable::Context>::main (this=0x55e36d38dec8, context=...) at /home/dagardner/work/mrc/cpp/mrc/include/mrc/runnable/runnable.hpp:109
#52 0x00007f6f9f35c993 in mrc::runnable::Runner::enqueue(std::shared_ptr<mrc::runnable::IEngines>, std::vector<std::shared_ptr<mrc::runnable::Context>, std::allocator<std::shared_ptr<mrc::runnable::Context> > >&&)::$_0::operator()() const (this=0x55e36d3c4f90) at /home/dagardner/work/mrc/cpp/mrc/src/public/runnable/runner.cpp:136
#53 0x00007f6f9f35c8f5 in std::__invoke_impl<void, mrc::runnable::Runner::enqueue(std::shared_ptr<mrc::runnable::IEngines>, std::vector<std::shared_ptr<mrc::runnable::Context>, std::allocator<std::shared_ptr<mrc::runnable::Context> > >&&)::$_0&>(std::__invoke_other, mrc::runnable::Runner::enqueue(std::shared_ptr<mrc::runnable::IEngines>, std::vector<std::shared_ptr<mrc::runnable::Context>, std::allocator<std::shared_ptr<mrc::runnable::Context> > >&&)::$_0&) (__f=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:61
#54 0x00007f6f9f35c8b5 in std::__invoke_r<void, mrc::runnable::Runner::enqueue(std::shared_ptr<mrc::runnable::IEngines>, std::vector<std::shared_ptr<mrc::runnable::Context>, std::allocator<std::shared_ptr<mrc::runnable::Context> > >&&)::$_0&>(mrc::runnable::Runner::enqueue(std::shared_ptr<mrc::runnable::IEngines>, std::vector<std::shared_ptr<mrc::runnable::Context>, std::allocator<std::shared_ptr<mrc::runnable::Context> > >&&)::$_0&) (__fn=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:111
#55 0x00007f6f9f35c73d in std::_Function_handler<void (), mrc::runnable::Runner::enqueue(std::shared_ptr<mrc::runnable::IEngines>, std::vector<std::shared_ptr<mrc::runnable::Context>, std::allocator<std::shared_ptr<mrc::runnable::Context> > >&&)::$_0>::_M_invoke(std::_Any_data const&) (__functor=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/std_function.h:291
#56 0x00007f6f9eeef6d6 in std::function<void ()>::operator()() const (this=0x55e36d38f008) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/std_function.h:560
#57 0x00007f6f9f1a5035 in std::__invoke_impl<void, std::function<void ()>&>(std::__invoke_other, std::function<void ()>&) (__f=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:61
#58 0x00007f6f9f1a5015 in std::__invoke<std::function<void ()>&>(std::function<void ()>&) (__fn=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:96
#59 0x00007f6f9f1a4ff9 in std::_Bind<std::function<void ()> ()>::__call<void>(std::tuple<>&&, std::_Index_tuple<>) (this=0x55e36d38f008, __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/functional:420
#60 0x00007f6f9f1a4fb3 in std::_Bind<std::function<void ()> ()>::operator()<, void>() (this=0x55e36d38f008) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/functional:503
#61 0x00007f6f9f1a4f75 in std::__invoke_impl<void, std::_Bind<std::function<void ()> ()>&>(std::__invoke_other, std::_Bind<std::function<void ()> ()>&) (__f=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:61
#62 0x00007f6f9f1a4f55 in std::__invoke<std::_Bind<std::function<void ()> ()>&>(std::_Bind<std::function<void ()> ()>&) (__fn=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:96
#63 0x00007f6f9f1a4f39 in std::__apply_impl<std::_Bind<std::function<void ()> ()>&, std::tuple<>>(std::_Bind<std::function<void ()> ()>&, std::tuple<>&&, std::integer_sequence<unsigned long>) (__f=..., __t=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/tuple:1843
#64 0x00007f6f9f1a4f0d in std::apply<std::_Bind<std::function<void ()> ()>&, std::tuple<> >(std::_Bind<std::function<void ()> ()>&, std::tuple<>&&) (__f=..., __t=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/tuple:1854
#65 0x00007f6f9f1a4c10 in boost::fibers::detail::task_object<std::_Bind<std::function<void ()> ()>, std::allocator<boost::fibers::packaged_task<void ()> >, void>::run() (this=0x55e36d38efb0) at /home/dagardner/work/conda/envs/mrc/include/boost/fiber/future/detail/task_object.hpp:137
#66 0x00007f6f9f06dacd in boost::fibers::packaged_task<void ()>::operator()() (this=0x55e36d3273f0) at /home/dagardner/work/conda/envs/mrc/include/boost/fiber/future/packaged_task.hpp:121
#67 0x00007f6f9f1a5934 in mrc::core::FiberTaskQueue::enqueue<std::function<void ()>>(mrc::FiberMetaData&&, std::function<void ()>&&)::{lambda()#1}::operator()() (this=0x55e36d3273e8) at /home/dagardner/work/mrc/cpp/mrc/include/mrc/core/task_queue.hpp:79
#68 0x00007f6f9f1a58c5 in std::__invoke_impl<void, mrc::core::FiberTaskQueue::enqueue<std::function<void ()>>(mrc::FiberMetaData&&, std::function<void ()>&&)::{lambda()#1}&>(std::__invoke_other, mrc::core::FiberTaskQueue::enqueue<std::function<void ()>>(mrc::FiberMetaData&&, std::function<void ()>&&)::{lambda()#1}&) (__f=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:61
#69 0x00007f6f9f1a58a5 in std::__invoke<mrc::core::FiberTaskQueue::enqueue<std::function<void ()>>(mrc::FiberMetaData&&, std::function<void ()>&&)::{lambda()#1}&>(std::function<void ()>&&) (__fn=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:96
#70 0x00007f6f9f1a5889 in std::__apply_impl<mrc::core::FiberTaskQueue::enqueue<std::function<void ()>>(mrc::FiberMetaData&&, std::function<void ()>&&)::{lambda()#1}&, std::tuple<>>(std::function<void ()>&&, std::tuple<>&&, std::integer_sequence<unsigned long>) (__f=..., __t=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/tuple:1843
#71 0x00007f6f9f1a585d in std::apply<mrc::core::FiberTaskQueue::enqueue<std::function<void ()>>(mrc::FiberMetaData&&, std::function<void ()>&&)::{lambda()#1}&, std::tuple<> >(std::function<void ()>&&, std::tuple<>&&) (__f=..., __t=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/tuple:1854
#72 0x00007f6f9f1a555b in boost::fibers::detail::task_object<mrc::core::FiberTaskQueue::enqueue<std::function<void ()>>(mrc::FiberMetaData&&, std::function<void ()>&&)::{lambda()#1}, std::allocator<boost::fibers::packaged_task<void ()> >, void>::run() (this=0x55e36d327390) at /home/dagardner/work/conda/envs/mrc/include/boost/fiber/future/detail/task_object.hpp:137
#73 0x00007f6f9f06dacd in boost::fibers::packaged_task<void ()>::operator()() (this=0x7f6edc040b68) at /home/dagardner/work/conda/envs/mrc/include/boost/fiber/future/packaged_task.hpp:121
#74 0x00007f6f9f1f9ad5 in std::__invoke_impl<void, boost::fibers::packaged_task<void ()>>(std::__invoke_other, boost::fibers::packaged_task<void ()>&&) (__f=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:61
#75 0x00007f6f9f1f9ab5 in std::__invoke<boost::fibers::packaged_task<void ()>>(boost::fibers::packaged_task<void ()>&&) (__fn=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:96
#76 0x00007f6f9f1f9a99 in std::__apply_impl<boost::fibers::packaged_task<void ()>, std::tuple<>>(boost::fibers::packaged_task<void ()>&&, std::tuple<>&&, std::integer_sequence<unsigned long>) (__f=..., __t=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/tuple:1843
#77 0x00007f6f9f1f9a6d in std::apply<boost::fibers::packaged_task<void ()>, std::tuple<> >(boost::fibers::packaged_task<void ()>&&, std::tuple<>&&) (__f=..., __t=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/tuple:1854
#78 0x00007f6f9f1f95f9 in boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::run_(boost::context::fiber&&) (this=0x7f6edc041000, c=...) at /home/dagardner/work/conda/envs/mrc/include/boost/fiber/context.hpp:445
#79 0x00007f6f9f1fa464 in std::__invoke_impl<boost::context::fiber, boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*&)(boost::context::fiber&&), boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*&, boost::context::fiber>(std::__invoke_memfun_deref, boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*&)(boost::context::fiber&&), boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*&, boost::context::fiber&&) (__f=@0x7f6edc040f18: (boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()> >::*)(boost::fibers::worker_context<boost::fibers::packaged_task<void ()> > * const, boost::context::fiber &&)) 0x7f6f9f1f95a0 <boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::run_(boost::context::fiber&&)>, __t=@0x7f6edc040f28: 0x7f6edc041000, __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:74
#80 0x00007f6f9f1fa340 in std::__invoke<boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*&)(boost::context::fiber&&), boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*&, boost::context::fiber>(boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*&)(boost::context::fiber&&), boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*&, boost::context::fiber&&) (__fn=@0x7f6edc040f18: (boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()> >::*)(boost::fibers::worker_context<boost::fibers::packaged_task<void ()> > * const, boost::context::fiber &&)) 0x7f6f9f1f95a0 <boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::run_(boost::context::fiber&&)>, __args=..., __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:96
#81 0x00007f6f9f1fa2b2 in std::_Bind<boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*(boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*, std::_Placeholder<1>))(boost::context::fiber&&)>::__call<boost::context::fiber, boost::context::fiber&&, 0ul, 1ul>(std::tuple<boost::context::fiber&&>&&, std::_Index_tuple<0ul, 1ul>) (this=0x7f6edc040f18, __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/functional:420
#82 0x00007f6f9f1fa206 in std::_Bind<boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*(boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*, std::_Placeholder<1>))(boost::context::fiber&&)>::operator()<boost::context::fiber, boost::context::fiber>(boost::context::fiber&&) (this=0x7f6edc040f18, __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/functional:503
#83 0x00007f6f9f1fa198 in std::__invoke_impl<boost::context::fiber, std::_Bind<boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*(boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*, std::_Placeholder<1>))(boost::context::fiber&&)>&, boost::context::fiber>(std::__invoke_other, std::_Bind<boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*(boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*, std::_Placeholder<1>))(boost::context::fiber&&)>&, boost::context::fiber&&) (__f=..., __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:61
#84 0x00007f6f9f1fa158 in std::__invoke<std::_Bind<boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*(boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*, std::_Placeholder<1>))(boost::context::fiber&&)>&, boost::context::fiber>(std::_Bind<boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*(boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*, std::_Placeholder<1>))(boost::context::fiber&&)>&, boost::context::fiber&&) (__fn=..., __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/bits/invoke.h:96
#85 0x00007f6f9f1fa118 in std::invoke<std::_Bind<boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*(boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*, std::_Placeholder<1>))(boost::context::fiber&&)>&, boost::context::fiber>(std::_Bind<boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*(boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*, std::_Placeholder<1>))(boost::context::fiber&&)>&, boost::context::fiber&&) (__fn=..., __args=...) at /home/dagardner/work/conda/envs/mrc/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/include/c++/11.2.0/functional:97
#86 0x00007f6f9f1fa01b in boost::context::detail::fiber_record<boost::context::fiber, boost::context::basic_fixedsize_stack<boost::context::stack_traits>, std::_Bind<boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*(boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*, std::_Placeholder<1>))(boost::context::fiber&&)> >::run(void*) (this=0x7f6edc040f00, fctx=0x0) at /home/dagardner/work/conda/envs/mrc/include/boost/context/fiber_fcontext.hpp:157
#87 0x00007f6f9f1f9d40 in boost::context::detail::fiber_entry<boost::context::detail::fiber_record<boost::context::fiber, boost::context::basic_fixedsize_stack<boost::context::stack_traits>, std::_Bind<boost::context::fiber (boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>::*(boost::fibers::worker_context<boost::fibers::packaged_task<void ()>>*, std::_Placeholder<1>))(boost::context::fiber&&)> > >(boost::context::detail::transfer_t) (t=...) at /home/dagardner/work/conda/envs/mrc/include/boost/context/fiber_fcontext.hpp:97
#88 0x00007f6f9f6a811f in make_fcontext () from /home/dagardner/work/conda/envs/mrc/lib/libboost_context.so.1.82.0
#89 0x0000000000000000 in ?? ()

quit

@dagardner-nv
Copy link
Contributor Author

Running just TestControlPlane.SingleClientConnectDisconnect in isolation:

./test_mrc_private.x --gtest_filter=TestControlPlane.SingleClientConnectDisconnect --gtest_repeat=-1 --gtest_break_on_failure

This was referenced Sep 8, 2023
Closed
Merged
@cwharris cwharris mentioned this issue Sep 8, 2023
Open
2 tasks
@rapids-bot rapids-bot bot closed this as completed in #382 Sep 11, 2023
rapids-bot bot pushed a commit that referenced this issue Sep 11, 2023
@dagardner-nv
Revert boost upgrade, and update clang to v16 (#382)
5747320 
fixes #379 ?

Authors:
  - David Gardner (https://github.com/dagardner-nv)

Approvers:
  - Christopher Harris (https://github.com/cwharris)
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #382
This was referenced Sep 18, 2023
Closed
Closed
Merged
rapids-bot bot pushed a commit that referenced this issue Sep 23, 2023
@dagardner-nv
Safe handling of control plane promises & fix CI (#391)
40d20a3 
* Based on fixes from @mdemoret-nv : https://github.com/mdemoret-nv/MRC/tree/mdd_control-plane-promises & #381
* Adopts updated versions of boost, clang & IWYU
* Only run IWYU on files changed in PR
* Use clang for checks stage
* Allocate progress engine promises on the heap

fixes #379

Authors:
  - David Gardner (https://github.com/dagardner-nv)
  - Michael Demoret (https://github.com/mdemoret-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)
  - Christopher Harris (https://github.com/cwharris)

URL: #391
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
Morpheus Boards
Status: Done
Milestone
No milestone
1 participant
@dagardner-nv