For the record: cmsenv needed in the container

[katilp]

At the container start up in cmsopendata/cmssw_5_3_32_vnc:latest, CMS environment is set and updates are made to the environment e.g. for git clone https and curl to work.

However, these updates make certain parts of condition data reading fail with:

~/CMSSW_5_3_32/src/EventProductionExamplesTool/MCDrellYanProductionNoPU $ cmsRun python/recoDY.py

11-Nov-2021 15:44:01 CET  Initiating request to open file file:data/hltDY.root
11-Nov-2021 15:44:02 CET  Successfully opened file file:data/hltDY.root
----- Begin Fatal Exception 11-Nov-2021 15:44:13 CET-----------------------
An exception of category 'CaloTowers' occurred while
   [0] Processing run: 1
   [1] Running path 'reconstruction_step'
   [2] Calling beginRun for module PFRecHitProducerECAL/'particleFlowRecHitECAL'
   [3] Using EventSetup component CaloTowerConstituentsMapBuilder/'' to make data CaloTowerConstituentsMap/'' in record IdealGeometryRecord
Exception Message:
Cell with id 3400723c is already mapped to a CaloTower Tower (18,3)
----- End Fatal Exception -------------------------------------------------
Another exception was caught while trying to clean up files after the primary fatal exception.
11-Nov-2021 15:44:13 CET  Closed file file:data/hltDY.root

This can be solved by doing cmsenv again.

To be noted that this overwrites the updates and git clone https will fail. However, when one exits the container and starts it again, it opens with the updated env and git clone https will work again.

So, one needs to :

• git clone and curl in a container default env before cmsenv
• cmsRun after cmsenv

[katilp]

diff:

$ diff env_before env_after
34d33
< LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib64:/home/cmsusr/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/external/llvm/3.2-cms2/lib:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib64:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib
35a35
> LD_LIBRARY_PATH=/home/cmsusr/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/external/llvm/3.2-cms2/lib:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib64:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib:/usr/local/lib:/usr/local/lib64
44d43
< SRT_CLHEP_PARAM_PATH_SCRAMRTDEL=/opt/cms/slc6_amd64_gcc472/external/clhep/2.0.4.6
45a45
> SRT_CLHEP_PARAM_PATH_SCRAMRTDEL=/opt/cms/slc6_amd64_gcc472/external/clhep/2.0.4.6
51c51
< PATH=/usr/local/bin:/home/cmsusr/CMSSW_5_3_32/bin/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/external/slc6_amd64_gcc472/bin:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/bin/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/external/slc6_amd64_gcc472/bin:/opt/cms/slc6_amd64_gcc472/external/llvm/3.2-cms2/bin:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/bin:/opt/cms/common:/opt/cms/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
---
> PATH=/home/cmsusr/CMSSW_5_3_32/bin/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/external/slc6_amd64_gcc472/bin:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/bin/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/external/slc6_amd64_gcc472/bin:/opt/cms/slc6_amd64_gcc472/external/llvm/3.2-cms2/bin:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/bin:/usr/local/bin:/opt/cms/common:/opt/cms/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
54,55d53
< SRT_DBS_CLIENT_CONFIG_SCRAMRTDEL=/opt/cms/slc6_amd64_gcc472/cms/dbs-client/DBS_2_1_9-cms6/lib/DBSAPI/dbs.config
< SRT_CMSSW_SEARCH_PATH_SCRAMRTDEL=/home/cmsusr/CMSSW_5_3_32/src:/home/cmsusr/CMSSW_5_3_32/external/slc6_amd64_gcc472/data:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/src:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/external/slc6_amd64_gcc472/data
56a55,56
> SRT_CMSSW_SEARCH_PATH_SCRAMRTDEL=/home/cmsusr/CMSSW_5_3_32/src:/home/cmsusr/CMSSW_5_3_32/external/slc6_amd64_gcc472/data:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/src:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/external/slc6_amd64_gcc472/data
> SRT_DBS_CLIENT_CONFIG_SCRAMRTDEL=/opt/cms/slc6_amd64_gcc472/cms/dbs-client/DBS_2_1_9-cms6/lib/DBSAPI/dbs.config
85d84
< SRT_GENREFLEX_SCRAMRTDEL=/opt/cms/slc6_amd64_gcc472/lcg/root/5.32.00-cms/bin/genreflex
86a86
> SRT_GENREFLEX_SCRAMRTDEL=/opt/cms/slc6_amd64_gcc472/lcg/root/5.32.00-cms/bin/genreflex
90a91
> SRT_LD_LIBRARY_PATH_SCRAMRT=/home/cmsusr/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/external/llvm/3.2-cms2/lib:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib64:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib
103d103
< SRT_LD_LIBRARY_PATH_SCRAMRTDEL=/home/cmsusr/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/external/llvm/3.2-cms2/lib:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib64:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib
107d106
< SRT_XDAQ_OS_SCRAMRTDEL=linux
108a108
> SRT_XDAQ_OS_SCRAMRTDEL=linux

[katilp]

To reproduce:

Use cmsopendata/cmssw_5_3_32_vnc:latest

In the container (with the container default env):

git clone https://github.com/cms-opendata-analyses/EventProductionExamplesTool.git
cd EventProductionExamplesTool/
git checkout 2011
cd MCDrellYanProductionNoPU/

in python/hltDY.py:

• change file:gensimDY.root to file:data/gensimDY.root (or copy the file to . )
• Comment the line process.GlobalTag.connect = cms.string('sqlite_file ...

cmsRun python/hltDY.py

This fails, check env

Then

cmsenv
cmsRun python/hltDY.py

This is OK, check env

[tpmccauley]

In the entrypoint.sh script this line executes the first cmsenv: https://gitlab.cern.ch/cms-cloud/cmssw-docker/-/blob/master/standalone/entrypoint.sh#L8 . So right when you spin up the container you are ready-to-go cmsenv-wise

Note that cmsenv is just an alias

cmsenv: 	 aliased to eval `scramv1 runtime -csh`

As a cross-check I went to an already-installed version of CMSSW I had handy on lxplus.

What's strange is that scram runtime -csh gives me two different results before and after I run cmsenv.

Running scram runtime -csh the first time outputs all the usual you would expect. eval then sets it all in the env.

If I run it again I only get:

[mccauley@lxplus707 CMSSW_12_0_0]: scram runtime -csh
unsetenv  SRT_LD_LIBRARY_PATH_SCRAMRT;
setenv SRT_LD_LIBRARY_PATH_SCRAMRTDEL "/afs/cern.ch/work/m/mccauley/CMSSW_12_0_0/biglib/slc7_amd64_gcc900:/afs/cern.ch/work/m/mccauley/CMSSW_12_0_0/lib/slc7_amd64_gcc900:/afs/cern.ch/work/m/mccauley/CMSSW_12_0_0/external/slc7_amd64_gcc900/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0/biglib/slc7_amd64_gcc900:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0/lib/slc7_amd64_gcc900:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0/external/slc7_amd64_gcc900/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/llvm/12.0.0-llifpc/lib64:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/gcc/9.3.0/lib64:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/gcc/9.3.0/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/cuda/11.2.2-llifpc/drivers";
setenv PATH "/cvmfs/cms.cern.ch/share/overrides/bin:/afs/cern.ch/work/m/mccauley/CMSSW_12_0_0/bin/slc7_amd64_gcc900:/afs/cern.ch/work/m/mccauley/CMSSW_12_0_0/external/slc7_amd64_gcc900/bin:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0/bin/slc7_amd64_gcc900:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0/external/slc7_amd64_gcc900/bin:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/llvm/12.0.0-llifpc/bin:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/gcc/9.3.0/bin:/afs/cern.ch/user/m/mccauley/.local/bin:/afs/cern.ch/cms/lumi/brilconda-1.1.7/bin:/afs/cern.ch/cms/caf/scripts:/cvmfs/cms.cern.ch/common:/usr/sue/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin";

Where the diff of my output from env is:

[mccauley@lxplus707 CMSSW_12_0_0]: diff my_env_after_cmsenv my_env_after_2cmsenv 
74d73
< SRT_LD_LIBRARY_PATH_SCRAMRT=/afs/cern.ch/work/m/mccauley/CMSSW_12_0_0/biglib/slc7_amd64_gcc900:/afs/cern.ch/work/m/mccauley/CMSSW_12_0_0/lib/slc7_amd64_gcc900:/afs/cern.ch/work/m/mccauley/CMSSW_12_0_0/external/slc7_amd64_gcc900/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0/biglib/slc7_amd64_gcc900:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0/lib/slc7_amd64_gcc900:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0/external/slc7_amd64_gcc900/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/llvm/12.0.0-llifpc/lib64:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/gcc/9.3.0/lib64:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/gcc/9.3.0/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/cuda/11.2.2-llifpc/drivers
192a192
> SRT_LD_LIBRARY_PATH_SCRAMRTDEL=/afs/cern.ch/work/m/mccauley/CMSSW_12_0_0/biglib/slc7_amd64_gcc900:/afs/cern.ch/work/m/mccauley/CMSSW_12_0_0/lib/slc7_amd64_gcc900:/afs/cern.ch/work/m/mccauley/CMSSW_12_0_0/external/slc7_amd64_gcc900/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0/biglib/slc7_amd64_gcc900:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0/lib/slc7_amd64_gcc900:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0/external/slc7_amd64_gcc900/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/llvm/12.0.0-llifpc/lib64:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/gcc/9.3.0/lib64:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/gcc/9.3.0/lib:/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/cuda/11.2.2-llifpc/drivers

[katilp]

Note that this is likely related to the updated paths needed for git and curl, in /opt/cms/entrypoint.sh of the container

export LD_LIBRARY_PATH=${UPDATE_PATH}/lib:${LD_LIBRARY_PATH}
export PATH=${UPDATE_PATH}/bin:${PATH}

[tpmccauley]

Original LD_LIBRARY_PATH:

/usr/local/lib:/usr/local/lib64:/home/cmsusr/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/external/llvm/3.2-cms2/lib:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib64:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib

If I set the LD_LIBRARY_PATH to just this:

/home/cmsusr/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/external/llvm/3.2-cms2/lib:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib64:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib

it works. Works as well for this (i.e. after 2nd cmsenv):

/home/cmsusr/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/external/llvm/3.2-cms2/lib:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib64:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib:/usr/local/lib:/usr/local/lib64

If I remove the last path it works. So /usr/local/lib64 contains the culprit:

[17:09:43] cmsusr@b9539c87f473 ~/CMSSW_5_3_32/src/EventProductionExamplesTool/MCDrellYanProductionNoPU $ echo ${LD_LIBRARY_PATH}ARY_PATH}
/home/cmsusr/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/home/cmsusr/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/biglib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/lib/slc6_amd64_gcc472:/opt/cms/slc6_amd64_gcc472/cms/cmssw/CMSSW_5_3_32/external/slc6_amd64_gcc472/lib:/opt/cms/slc6_amd64_gcc472/external/llvm/3.2-cms2/lib:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib64:/opt/cms/slc6_amd64_gcc472/external/gcc/4.7.2-cms/lib:/usr/local/lib
[17:10:04] cmsusr@b9539c87f473 ~/CMSSW_5_3_32/src/EventProductionExamplesTool/MCDrellYanProductionNoPU $ ls /usr/local/lib64    
engines-1.1  libcrypto.a  libcrypto.so	libcrypto.so.1.1  libssl.a  libssl.so  libssl.so.1.1  pkgconfig