Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

testSiStripHitResolution and testSiStripHitEfficiency fail when export MALLOC_CONF=junk:true is set #45084

Closed
mmusich opened this issue May 28, 2024 · 8 comments · Fixed by #46111

Comments

@mmusich
Copy link
Contributor

mmusich commented May 28, 2024

Just set export MALLOC_CONF=junk:true here:

function die { echo $1: status $2; exit $2; }

and here:

function die { echo $1: status $2; exit $2; }

crashes with:

A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Mon May 13 19:45:43 CEST 2024
Thread 2 (Thread 0x14862308c700 (LWP 3727752) "cmsRun"):
#0  0x000014864a6e7672 in waitpid () from /lib64/libpthread.so.0
#1  0x0000148645dff4d7 in edm::service::cmssw_stacktrace_fork() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000148645e023aa in edm::service::InitRootHandlers::stacktraceHelperThread() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000014864ad71a73 in std::execute_native_thread_routine (__p=0x148625ab2120) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000014864a6dd1ca in start_thread () from /lib64/libpthread.so.0
#5  0x000014864a349e73 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x148649869640 (LWP 3727550) "cmsRun"):
#0  0x000014864a434301 in poll () from /lib64/libc.so.6
#1  0x0000148645e4e1df in full_read.constprop () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000148645e0247c in edm::service::InitRootHandlers::stacktraceFromThread() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000148645e02de0 in sig_dostack_then_abort () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x000014864bf756b2 in TList::FindLink(TObject const*, int&) const () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libCore.so
#6  0x000014864bf7da64 in TList::Remove(TObject*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libCore.so
#7  0x000014864bf73546 in THashTable::Remove(TObject*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libCore.so
#8  0x000014864bf7286b in THashList::Delete(char const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libCore.so
#9  0x000014864c304de6 in TDirectoryFile::~TDirectoryFile() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libRIO.so
#10 0x000014864c304e49 in TDirectoryFile::~TDirectoryFile() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libRIO.so
#11 0x000014864bf72978 in THashList::Delete(char const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libCore.so
#12 0x000014864c304f4a in TDirectoryFile::Close(char const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libRIO.so
#13 0x000014864c322ad6 in TFile::Close(char const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libRIO.so
#14 0x0000148643e7fa7f in TFileService::~TFileService() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/libCommonToolsUtilAlgos.so
#15 0x0000148643ea4714 in edm::serviceregistry::ServiceWrapper<TFileService>::~ServiceWrapper() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/pluginCommonToolsToolsUtilAlgos_plugins.so
#16 0x000014864d1051db in edm::serviceregistry::ServicesManager::~ServicesManager() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#17 0x000014864d0fd4f7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#18 0x000014864d0fd4f7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#19 0x000014864cd59cb7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x000014864cd766e9 in edm::EventProcessor::~EventProcessor() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#21 0x0000000000405891 in (anonymous namespace)::EventProcessorWithSentry::~EventProcessorWithSentry() ()
#22 0x000000000040518f in main ()

and


A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Mon May 13 19:45:26 CEST 2024
Thread 16 (Thread 0x152dd1bf2700 (LWP 3725904) "cmsRun"):
#0  0x0000152e09b7eda6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000152e09b7ee98 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000152e05ab3706 in XrdCl::JobManager::RunJobs() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000152e05ab37b9 in RunRunnerThread () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x152dd1df3700 (LWP 3725903) "cmsRun"):
#0  0x0000152e09b7eda6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000152e09b7ee98 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000152e05ab3706 in XrdCl::JobManager::RunJobs() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000152e05ab37b9 in RunRunnerThread () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x152dd1ff4700 (LWP 3725902) "cmsRun"):
#0  0x0000152e09b7eda6 in do_futex_wait.constprop () from /lib64/libpthread.so.0
#1  0x0000152e09b7ee98 in __new_sem_wait_slow.constprop.0 () from /lib64/libpthread.so.0
#2  0x0000152e05ab3706 in XrdCl::JobManager::RunJobs() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000152e05ab37b9 in RunRunnerThread () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x152dd21f5700 (LWP 3725901) "cmsRun"):
#0  0x0000152e09b80180 in nanosleep () from /lib64/libpthread.so.0
#1  0x0000152e05ba2f08 in XrdSysTimer::Wait(int) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000152e05a2803c in XrdCl::TaskManager::RunTasks() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#3  0x0000152e05a28179 in RunRunnerThread () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdCl.so.3
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x152dd23f6700 (LWP 3725900) "cmsRun"):
#0  0x0000152e098d8247 in epoll_wait () from /lib64/libc.so.6
#1  0x0000152e05b9d282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000152e05b991ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000152e05ba2617 in XrdSysThread_Xeq () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x152dd25f7700 (LWP 3725899) "cmsRun"):
#0  0x0000152e098d8247 in epoll_wait () from /lib64/libc.so.6
#1  0x0000152e05b9d282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000152e05b991ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000152e05ba2617 in XrdSysThread_Xeq () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x152dd27f8700 (LWP 3725898) "cmsRun"):
#0  0x0000152e098d8247 in epoll_wait () from /lib64/libc.so.6
#1  0x0000152e05b9d282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000152e05b991ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000152e05ba2617 in XrdSysThread_Xeq () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x152dd29f9700 (LWP 3725897) "cmsRun"):
#0  0x0000152e098d8247 in epoll_wait () from /lib64/libc.so.6
#1  0x0000152e05b9d282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000152e05b991ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000152e05ba2617 in XrdSysThread_Xeq () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x152dd2bfa700 (LWP 3725896) "cmsRun"):
#0  0x0000152e098d8247 in epoll_wait () from /lib64/libc.so.6
#1  0x0000152e05b9d282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000152e05b991ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000152e05ba2617 in XrdSysThread_Xeq () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x152dd2dfb700 (LWP 3725895) "cmsRun"):
#0  0x0000152e098d8247 in epoll_wait () from /lib64/libc.so.6
#1  0x0000152e05b9d282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000152e05b991ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000152e05ba2617 in XrdSysThread_Xeq () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x152dd2ffc700 (LWP 3725894) "cmsRun"):
#0  0x0000152e098d8247 in epoll_wait () from /lib64/libc.so.6
#1  0x0000152e05b9d282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000152e05b991ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000152e05ba2617 in XrdSysThread_Xeq () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x152dd31fd700 (LWP 3725893) "cmsRun"):
#0  0x0000152e098d8247 in epoll_wait () from /lib64/libc.so.6
#1  0x0000152e05b9d282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000152e05b991ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000152e05ba2617 in XrdSysThread_Xeq () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x152dd33fe700 (LWP 3725892) "cmsRun"):
#0  0x0000152e098d8247 in epoll_wait () from /lib64/libc.so.6
#1  0x0000152e05b9d282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000152e05b991ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000152e05ba2617 in XrdSysThread_Xeq () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x152dd35ff700 (LWP 3725891) "cmsRun"):
#0  0x0000152e098d8247 in epoll_wait () from /lib64/libc.so.6
#1  0x0000152e05b9d282 in XrdSys::IOEvents::PollE::Begin(XrdSysSemaphore*, int&, char const**) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#2  0x0000152e05b991ed in XrdSys::IOEvents::BootStrap::Start(void*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#3  0x0000152e05ba2617 in XrdSysThread_Xeq () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libXrdUtils.so.3
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x152de2565700 (LWP 3725540) "cmsRun"):
#0  0x0000152e09b80672 in waitpid () from /lib64/libpthread.so.0
#1  0x0000152e052d54d7 in edm::service::cmssw_stacktrace_fork() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000152e052d83aa in edm::service::InitRootHandlers::stacktraceHelperThread() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000152e0a20aa73 in std::execute_native_thread_routine (__p=0x152de4f9a140) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x0000152e09b761ca in start_thread () from /lib64/libpthread.so.0
#5  0x0000152e097e2e73 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x152e08d02640 (LWP 3725262) "cmsRun"):
#0  0x0000152e098cd301 in poll () from /lib64/libc.so.6
#1  0x0000152e053241df in full_read.constprop () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x0000152e052d847c in edm::service::InitRootHandlers::stacktraceFromThread() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x0000152e052d8de0 in sig_dostack_then_abort () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#4  <signal handler called>
#5  0x0000152e0b40e6b2 in TList::FindLink(TObject const*, int&) const () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libCore.so
#6  0x0000152e0b416a64 in TList::Remove(TObject*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libCore.so
#7  0x0000152e0b40c546 in THashTable::Remove(TObject*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libCore.so
#8  0x0000152e0b40b86b in THashList::Delete(char const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libCore.so
#9  0x0000152e0b79dde6 in TDirectoryFile::~TDirectoryFile() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libRIO.so
#10 0x0000152e0b79de49 in TDirectoryFile::~TDirectoryFile() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libRIO.so
#11 0x0000152e0b40b978 in THashList::Delete(char const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libCore.so
#12 0x0000152e0b79df4a in TDirectoryFile::Close(char const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libRIO.so
#13 0x0000152e0b7bbad6 in TFile::Close(char const*) () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/external/el8_amd64_gcc12/lib/libRIO.so
#14 0x0000152e0335aa7f in TFileService::~TFileService() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/libCommonToolsUtilAlgos.so
#15 0x0000152e0337f714 in edm::serviceregistry::ServiceWrapper<TFileService>::~ServiceWrapper() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/pluginCommonToolsToolsUtilAlgos_plugins.so
#16 0x0000152e0c59e1db in edm::serviceregistry::ServicesManager::~ServicesManager() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#17 0x0000152e0c5964f7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#18 0x0000152e0c5964f7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#19 0x0000152e0c1f2cb7 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x0000152e0c20f6e9 in edm::EventProcessor::~EventProcessor() () from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-05-13-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#21 0x0000000000405891 in (anonymous namespace)::EventProcessorWithSentry::~EventProcessorWithSentry() ()
#22 0x000000000040518f in main ()

Current Modules:

Module: none (crashed)

also seen here: cms-sw/cms-bot#2228 (comment)

@cmsbuild
Copy link
Contributor

cmsbuild commented May 28, 2024

cms-bot internal usage

@cmsbuild
Copy link
Contributor

A new Issue was created by @mmusich.

@rappoccio, @sextonkennedy, @Dr15Jones, @smuzaffar, @makortel, @antoniovilela can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor

assign CalibTracker/SiStripHitEfficiency

@cmsbuild
Copy link
Contributor

New categories assigned: alca

@saumyaphor4252,@perrotta,@consuegs you have been requested to review this Pull request/Issue and eventually sign? Thanks

@makortel
Copy link
Contributor

@pcanal Do the stack traces (showing crashes in ROOT) give any hints of what could be going wrong?

@pcanal
Copy link
Contributor

pcanal commented May 28, 2024

One strong possibility is that one or more of the element of the TList (that belongs to the TFile or one of its TDirectory) has already been deleted.

@mmusich
Copy link
Contributor Author

mmusich commented May 29, 2024

Sharing some progress, with these changes:

diff --git a/CalibTracker/SiStripHitEfficiency/plugins/SiStripHitEffFromCalibTree.cc b/CalibTracker/SiStripHitEfficiency/plugins/SiStripHitEffFromCalibTree.cc
index c871bcb20c7..bcf19de5a44 100644
--- a/CalibTracker/SiStripHitEfficiency/plugins/SiStripHitEffFromCalibTree.cc
+++ b/CalibTracker/SiStripHitEfficiency/plugins/SiStripHitEffFromCalibTree.cc
@@ -692,8 +692,8 @@ void SiStripHitEffFromCalibTree::algoAnalyze(const edm::Event& e, const edm::Eve
   makeSQLite();
   totalStatistics();
   makeSummary();
-  makeSummaryVsBx();
-  makeSummaryVsLumi();
+  //makeSummaryVsBx();
+  //makeSummaryVsLumi();
   if (useCM_)
     makeSummaryVsCM();
 
diff --git a/CalibTracker/SiStripHitEfficiency/test/test_SiStripHitEfficiency.sh b/CalibTracker/SiStripHitEfficiency/test/test_SiStripHitEfficiency.sh
index 9397e50dcb3..c4551b3ee26 100755
--- a/CalibTracker/SiStripHitEfficiency/test/test_SiStripHitEfficiency.sh
+++ b/CalibTracker/SiStripHitEfficiency/test/test_SiStripHitEfficiency.sh
@@ -1,6 +1,8 @@
 #!/bin/bash
 function die { echo $1: status $2; exit $2; }
 
+export MALLOC_CONF=junk:true
+
 if [ "${SCRAM_TEST_NAME}" != "" ] ; then
   mkdir ${SCRAM_TEST_NAME}
   cd ${SCRAM_TEST_NAME}

scram b runtests_testSiStripHitEfficiency passes.
So the problem lies in these two methods:

void SiStripHitEffFromCalibTree::makeSummaryVsBx() {

and

void SiStripHitEffFromCalibTree::makeSummaryVsLumi() {

@mmusich
Copy link
Contributor Author

mmusich commented Jun 14, 2024

@jlagram @DenkMybu @cms-sw/trk-dpg-l2 FYI

mmusich added a commit to mmusich/cmssw that referenced this issue Aug 7, 2024
jyoti299 pushed a commit to jyoti299/cmssw that referenced this issue Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants