Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

read smaps_rollup instead of parsing smaps using "awk" #11667

Closed
VinInn opened this issue Jul 23, 2023 · 7 comments · Fixed by #11676 · May be fixed by #11677
Closed

read smaps_rollup instead of parsing smaps using "awk" #11667

VinInn opened this issue Jul 23, 2023 · 7 comments · Fixed by #11676 · May be fixed by #11677
Assignees
@todor-ivanov
Labels
BUG High Priority New Feature Operations QPrio: High quarter priority

Comments

@VinInn
Copy link

VinInn commented Jul 23, 2023 •
edited
Loading

Impact of the new feature
safer

Is your feature request related to a problem? Please describe.
inconsistent report of PSS vs RSS (see cms-sw/cmssw#40437)

Describe the solution you'd like
read in /proc/pid/smaps_rollup (direclty in python) instead of

WMCore/src/python/WMCore/WMRuntime/Monitors/PerformanceMonitor.py

Line 226 in 762bae9

smaps_cmd = self.pssMemoryCommand % (stepPID)

Describe alternatives you've considered
this is the simplest that come to my mind

Additional context
judge yourself

[innocent@worker6304 tauCrash]$ cat memDump
#!/usr/bin/csh
while (1)
set pid = `ps -af | grep $USER | grep cmsRun | grep -v grep | awk '{printf $2" "}'`
grep Anon /proc/meminfo;
awk '/^Rss/ {pss += $2} END {print pss}' /proc/$pid/smaps ; awk '/^Pss/ {pss += $2} END {print pss}' /proc/$pid/smaps ; ps -v $pid
sleep 10
end

[innocent@worker6304 tauCrash]$ cat /proc/438416/smaps_rollup
00400000-7ffffffff000 ---p 00000000 00:00 0                              [rollup]
Rss:            10110556 kB
Pss:            10104477 kB
Pss_Anon:        9949060 kB
Pss_File:         155417 kB
Pss_Shmem:             0 kB
Shared_Clean:       6524 kB
Shared_Dirty:          0 kB
Private_Clean:    154972 kB
Private_Dirty:   9949060 kB
Referenced:     10110556 kB
Anonymous:       9949060 kB
LazyFree:              0 kB
AnonHugePages:   5785600 kB
ShmemPmdMapped:        0 kB
FilePmdMapped:         0 kB
Shared_Hugetlb:        0 kB
Private_Hugetlb:       0 kB
Swap:                  0 kB
SwapPss:               0 kB
Locked:                0 kB
[innocent@worker6304 tauCrash]$ ./memDump 438416
AnonPages:      19360932 kB
AnonHugePages:  14716928 kB
10054684
10068043
    PID TTY      STAT   TIME  MAJFL   TRS   DRS   RSS %MEM COMMAND
 438416 pts/1    Rl   231:52   6285    50 17245433 10055624  0.8 cmsRun config.py
@VinInn
Copy link
Author

VinInn commented Jul 24, 2023 •
edited
Loading

there seems to be a python utility (available in cmssw distro) that can be used to get pss rss etc

>>> import psutil
>>> p = psutil.Process(pid)
>>> p.memory_full_info()
pfullmem(rss=4886528, vms=34488320, shared=3092480, text=405504, lib=0, data=1941504, dirty=0, uss=1802240, pss=1955840, swap=0)

@VinInn
Copy link
Author

VinInn commented Jul 24, 2023

One question remains: given the large combinations of kernel and os this tool need to support HOW it is supposed to be tested?

@sextonkennedy
Copy link

Yes, a text file with key value pairs is closer to a real API that can be kept stable with time, while parsing a text file that makes assumptions about format is unsafe. How far back in versions of kernel is smaps_rollup supported?

@VinInn
Copy link
Author

VinInn commented Jul 24, 2023

smaps_rollup is not supported by 3.10 (at least I did not find it on lxplus7)
python psutil seems to work on 3.x 4.x and 5.x (I do not have access to any 6.x)

@jthiltges
Copy link

The psutil memory_full_info() function looks like a fine choice.

Looks like smaps_rollup was merged in kernel v4.14, and is only available in EL8 and beyond. psutil uses the more efficient smaps_rollup if available and falls back to smaps, so should work on a wide range of kernel versions.

@todor-ivanov
Copy link
Contributor

Hi @VinInn , thanks for creating the issue and for researching the problem.

The psutil package seems to be a quite neat solution. It is a package that our code depends in general, listed in our dependency list and seems to be a mainstream package, not depending on a specific cmssw version. Which is good.

I am about to test if it works for kernels bellow 3.x, And if not we may think of a combined solution, between psutils for newer kernels and /proc/<pid>/smaps parsing as currently for older ones - this for sure would be a last resort.

@todor-ivanov todor-ivanov self-assigned this Jul 24, 2023
@VinInn
Copy link
Author

VinInn commented Jul 24, 2023 •
edited
Loading

According to the information gathered by @mmascher no site of interest for CMS is running a kernel lower than 3.10.
btw where did you find a slc6 machine?

This was referenced Jul 25, 2023
Merged
Closed
Merged
Open
@jamesletts jamesletts mentioned this issue Aug 2, 2023
Open
@VinInn VinInn mentioned this issue Aug 8, 2023
Closed
@todor-ivanov todor-ivanov mentioned this issue Aug 11, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@todor-ivanov todor-ivanov

Labels
BUG High Priority New Feature Operations QPrio: High quarter priority
Projects
WMCore quarterly developments
Status: Done
Milestone
No milestone
4 participants
@jthiltges @sextonkennedy @VinInn @todor-ivanov