Set MALLOC_CONF=junk:true for PR tests #2228

smuzaffar · 2024-05-13T14:18:18Z

cmsbuild · 2024-05-13T14:18:44Z

A new Pull Request was created by @smuzaffar for branch master.

@smuzaffar, @iarspider, @aandvalenzuela, @cmsbuild can you please review it and eventually sign? Thanks.
@antoniovilela, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2024-05-13T14:18:44Z

cms-bot internal usage

smuzaffar · 2024-05-13T14:18:59Z

test parameters:

full_cmssw = true

smuzaffar · 2024-05-13T14:19:32Z

please test

lets build full cmssw to run all unit tests under MALLOC_CONF=junk:true

cmsbuild · 2024-05-13T19:08:31Z

-1

Failed Tests: UnitTests RelVals AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-414ef1/39361/summary.html
COMMIT: fe10837
CMSSW: CMSSW_14_1_X_2024-05-13-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2228/39361/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 2 errors in the following unit tests:

---> test testSiStripHitResolution had ERRORS
---> test testSiStripHitEfficiency had ERRORS

RelVals

141.042

----- Begin Fatal Exception 13-May-2024 19:56:58 CEST-----------------------
An exception of category 'PFEcalEndcapRecHitCreator' occurred while
   [0] Processing  Event run: 369978 lumi: 547 event: 625471048 stream: 0
   [1] Running path 'MC_PFScouting_v2'
   [2] Calling method for module PFRecHitProducer/'hltParticleFlowRecHitECALUnseeded'
Exception Message:
detid 2779096485not found in geometry
----- End Fatal Exception -------------------------------------------------

141.044A fatal system signal has occurred: segmentation violation
141.046A fatal system signal has occurred: segmentation violation

AddOn Tests

hlt_data_GRun

A fatal system signal has occurred: segmentation violation

hlt_data_GRun

A fatal system signal has occurred: segmentation violation

hlt_data_PRef

A fatal system signal has occurred: segmentation violation

Expand to see more addon errors ...

hlt_data_PRef

mmusich · 2024-06-14T06:27:00Z

@cmsbuild please test

Initialise the EcalElectronicsMappingHost with null detids cmssw#45209 should fix most of the failures

cmsbuild · 2024-06-14T12:48:57Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-414ef1/39883/summary.html
COMMIT: fe10837
CMSSW: CMSSW_14_1_X_2024-06-13-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2228/39883/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 2 errors in the following unit tests:

---> test testSiStripHitResolution had ERRORS
---> test testSiStripHitEfficiency had ERRORS

Comparison Summary

Summary:

You potentially removed 162 lines from the logs
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3345018
DQMHistoTests: Total failures: 54
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3344944
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

makortel · 2024-07-09T16:50:42Z

@cmsbuild, please test

cmsbuild · 2024-07-09T23:05:33Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-414ef1/40303/summary.html
COMMIT: fe10837
CMSSW: CMSSW_14_1_X_2024-07-09-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2228/40303/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found 2 errors in the following unit tests:

---> test testSiStripHitResolution had ERRORS
---> test testSiStripHitEfficiency had ERRORS

Comparison Summary

Summary:

You potentially removed 149 lines from the logs
Reco comparison results: 12 differences found in the comparisons
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3345094
DQMHistoTests: Total failures: 887
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3344187
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 202 log files, 165 edm output root files, 48 DQM output files
TriggerResults: no differences found

makortel · 2024-07-10T12:46:48Z

The testSiStripHitEfficiency test fails in

New IOV starting from run 325172 event 1 lumiBlock 1 time 1
-----------------

-----------------
Global Info
-----------------
BadComponent 		Modules 	Fibers 	Apvs	Strips
----------------------------------------------------------------
<cut>
----------------------------------------------------------------
		   Detid  	Modules Fibers Apvs
----------------------------------------------------------------
<cut>
TEC- Disk 9 :


A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Tue Jul  9 22:08:59 CEST 2024
Thread 2 (Thread 0x14cb4250b700 (LWP 226679) "cmsRun"):
#0  0x000014cb696706a2 in waitpid () from /lib64/libpthread.so.0
#1  0x000014cb65b7dd37 in edm::service::cmssw_stacktrace_fork() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014cb65b8043a in edm::service::InitRootHandlers::stacktraceHelperThread() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  0x000014cb69cd8a73 in std::execute_native_thread_routine (__p=0x14cb449c8240) at ../../../../../libstdc++-v3/src/c++11/thread.cc:82
#4  0x000014cb696661ca in start_thread () from /lib64/libpthread.so.0
#5  0x000014cb692c18d3 in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x14cb6b2f4680 (LWP 226379) "cmsRun"):
#0  0x000014cb693baac1 in poll () from /lib64/libc.so.6
#1  0x000014cb65b80657 in edm::service::InitRootHandlers::stacktraceFromThread() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#2  0x000014cb65b80854 in sig_dostack_then_abort () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x000014cb6b0b82a2 in TList::FindLink(TObject const*, int&) const () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libCore.so
#5  0x000014cb6b0c0654 in TList::Remove(TObject*) () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libCore.so
#6  0x000014cb6b0b6136 in THashTable::Remove(TObject*) () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libCore.so
#7  0x000014cb6b0b545b in THashList::Delete(char const*) () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libCore.so
#8  0x000014cb6b532426 in TDirectoryFile::~TDirectoryFile() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libRIO.so
#9  0x000014cb6b532489 in TDirectoryFile::~TDirectoryFile() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libRIO.so
#10 0x000014cb6b0b5568 in THashList::Delete(char const*) () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libCore.so
#11 0x000014cb6b53258a in TDirectoryFile::Close(char const*) () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libRIO.so
#12 0x000014cb6b550116 in TFile::Close(char const*) () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libRIO.so
#13 0x000014cb645e10af in TFileService::~TFileService() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/libCommonToolsUtilAlgos.so
#14 0x000014cb64607e14 in edm::serviceregistry::ServiceWrapper<TFileService>::~ServiceWrapper() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/pluginCommonToolsToolsUtilAlgos_plugins.so
#15 0x000014cb6c56a09f in edm::serviceregistry::ServicesManager::~ServicesManager() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#16 0x000014cb6c55ff37 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#17 0x000014cb6c55ff37 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#18 0x000014cb6c12dd87 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x000014cb6c147a55 in edm::EventProcessor::~EventProcessor() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x0000000000405711 in (anonymous namespace)::EventProcessorWithSentry::~EventProcessorWithSentry() ()
#21 0x00000000004050cd in main ()

Current Modules:

Module: none (crashed)

A fatal system signal has occurred: segmentation violation
/pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/src/CalibTracker/SiStripHitEfficiency/test/test_SiStripHitEfficiency.sh: line 16: 226379 Segmentation fault      (core dumped) cmsRun ${SCRAM_TEST_PATH}/testSiStripHitEffFromCalibTree_cfg.py inputFiles=HitEffTree.root runNumber=325172
failed running testSiStripHitEffFromCalibTree_cfg.py: status 139

The testSiStripHitResolution fails in

Testing SiStripHitResolutionFromCalibTree_cfg.py 
<cut>
Severity    # Occurrences   Total Occurrences
--------    -------------   -----------------
Info                 8986                8986
FwkInfo                 1                   1

dropped waiting message count 0


A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

Thread 1 (Thread 0x154ae3153680 (LWP 222707) "cmsRun"):
2  0x0000154adddc0854 in sig_dostack_then_abort () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/pluginFWCoreServicesPlugins.so
#3  <signal handler called>
#4  0x0000154ae34b82a2 in TList::FindLink(TObject const*, int&) const () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libCore.so
#5  0x0000154ae34c0654 in TList::Remove(TObject*) () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libCore.so
#6  0x0000154ae34b6136 in THashTable::Remove(TObject*) () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libCore.so
#7  0x0000154ae34b545b in THashList::Delete(char const*) () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libCore.so
#8  0x0000154ae3932426 in TDirectoryFile::~TDirectoryFile() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libRIO.so
#9  0x0000154ae3932489 in TDirectoryFile::~TDirectoryFile() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libRIO.so
#10 0x0000154ae34b5568 in THashList::Delete(char const*) () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libCore.so
#11 0x0000154ae393258a in TDirectoryFile::Close(char const*) () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libRIO.so
#12 0x0000154ae3950116 in TFile::Close(char const*) () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/external/el8_amd64_gcc12/lib/libRIO.so
#13 0x0000154adc8260af in TFileService::~TFileService() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/libCommonToolsUtilAlgos.so
#14 0x0000154adc84ce14 in edm::serviceregistry::ServiceWrapper<TFileService>::~ServiceWrapper() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/pluginCommonToolsToolsUtilAlgos_plugins.so
#15 0x0000154ae47b509f in edm::serviceregistry::ServicesManager::~ServicesManager() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#16 0x0000154ae47aaf37 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#17 0x0000154ae47aaf37 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/libFWCoreServiceRegistry.so
#18 0x0000154ae432dd87 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#19 0x0000154ae4347a55 in edm::EventProcessor::~EventProcessor() () from /pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/lib/el8_amd64_gcc12/libFWCoreFramework.so
#20 0x0000000000405711 in (anonymous namespace)::EventProcessorWithSentry::~EventProcessorWithSentry() ()
#21 0x00000000004050cd in main ()

Current Modules:

Module: none (crashed)

A fatal system signal has occurred: segmentation violation
/pool/condor/dir_2584642/jenkins/workspace/ib-run-pr-tests/CMSSW_14_1_X_2024-07-09-1100/src/CalibTracker/SiStripHitResolution/test/test_SiStripHitResolution.sh: line 16: 222707 Segmentation fault      (core dumped) cmsRun ${SCRAM_TEST_PATH}/SiStripHitResolutionFromCalibTree_cfg.py
failed running SiStripHitResolutionFromCalibTree_cfg.py: status 139

mmusich · 2024-07-10T13:34:52Z

The testSiStripHitEfficiency test fails in

just for the record there is a follow up issue for that since some time cms-sw/cmssw#45084

makortel · 2024-07-10T13:59:25Z

The testSiStripHitEfficiency test fails in

just for the record there is a follow up issue for that since some time cms-sw/cmssw#45084

Thanks for reminding!

cmsbuild · 2024-09-24T16:40:19Z

REMINDER @rappoccio, @mandrenguyen, @antoniovilela, @sextonkennedy: This PR was tested with cms-sw/cmssw#46111, please check if they should be merged together

smuzaffar · 2024-09-24T16:42:05Z

hold

Do not merge this, this is just for testing

cmsbuild · 2024-09-24T16:42:22Z

Pull request has been put on hold by @smuzaffar
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

antoniovilela · 2024-09-25T17:45:25Z

-orp

Set MALLOC_CONF=junk:true for PR tests

fe10837

as proposed in cms-sw/cmssw#44962

smuzaffar mentioned this pull request May 13, 2024

shall we set export MALLOC_CONF=junk:true for PR tests? cms-sw/cmssw#44962

Open

cmsbuild added orp-pending pending-signatures externals-pending tests-pending labels May 13, 2024

cmsbuild added tests-started and removed tests-pending labels May 13, 2024

mmusich mentioned this pull request May 13, 2024

HLT crashes when export MALLOC_CONF=junk:true is set. cms-sw/cmssw#44956

Closed

cmsbuild added tests-rejected and removed tests-started labels May 13, 2024

This was referenced May 28, 2024

improve SiStripHitEffFromCalibTree and SiStripHitResolFromCalibTree cms-sw/cmssw#45083

Open

testSiStripHitResolution and testSiStripHitEfficiency fail when export MALLOC_CONF=junk:true is set cms-sw/cmssw#45084

Closed

cmsbuild added tests-started and removed tests-rejected labels Jun 14, 2024

cmsbuild added tests-rejected and removed tests-started labels Jun 14, 2024

cmsbuild added tests-started and removed tests-rejected labels Jul 9, 2024

cmsbuild added tests-rejected and removed tests-started labels Jul 9, 2024

mmusich mentioned this pull request Sep 24, 2024

fix for cms-sw/cmssw#45084 cms-sw/cmssw#46111

Merged

cmsbuild added the hold label Sep 24, 2024

cmsbuild added orp-rejected and removed orp-pending labels Sep 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Set MALLOC_CONF=junk:true for PR tests #2228

Set MALLOC_CONF=junk:true for PR tests #2228

smuzaffar commented May 13, 2024

cmsbuild commented May 13, 2024

cmsbuild commented May 13, 2024 •

edited

Loading

smuzaffar commented May 13, 2024

smuzaffar commented May 13, 2024

cmsbuild commented May 13, 2024

mmusich commented Jun 14, 2024

cmsbuild commented Jun 14, 2024

makortel commented Jul 9, 2024

cmsbuild commented Jul 9, 2024

makortel commented Jul 10, 2024

mmusich commented Jul 10, 2024

makortel commented Jul 10, 2024

cmsbuild commented Sep 24, 2024

smuzaffar commented Sep 24, 2024

cmsbuild commented Sep 24, 2024

antoniovilela commented Sep 25, 2024

Set MALLOC_CONF=junk:true for PR tests #2228

Are you sure you want to change the base?

Set MALLOC_CONF=junk:true for PR tests #2228

Conversation

smuzaffar commented May 13, 2024

cmsbuild commented May 13, 2024

cmsbuild commented May 13, 2024 • edited Loading

smuzaffar commented May 13, 2024

smuzaffar commented May 13, 2024

cmsbuild commented May 13, 2024

Unit Tests

RelVals

AddOn Tests

mmusich commented Jun 14, 2024

cmsbuild commented Jun 14, 2024

Unit Tests

Comparison Summary

makortel commented Jul 9, 2024

cmsbuild commented Jul 9, 2024

Unit Tests

Comparison Summary

makortel commented Jul 10, 2024

mmusich commented Jul 10, 2024

makortel commented Jul 10, 2024

cmsbuild commented Sep 24, 2024

smuzaffar commented Sep 24, 2024

cmsbuild commented Sep 24, 2024

antoniovilela commented Sep 25, 2024

cmsbuild commented May 13, 2024 •

edited

Loading