Unpack error

[wpoely86]

I'm trying this on our Slurm 21.08.8 cluster with python 3.10 and I get:

$ python3.10 straw.py --auth munge -v master02.hydra.os 21.08
Using authentication method: munge
Trying master02.hydra.os:6817...
Traceback (most recent call last):
  File "/vscmnt/brussel_pixiu_data/_data_brussel/100/vsc10042/straw/straw.py", line 425, in <module>
    main()
  File "/vscmnt/brussel_pixiu_data/_data_brussel/100/vsc10042/straw/straw.py", line 422, in main
    fetch_config(args.server, args.auth, args.output_dir)
  File "/vscmnt/brussel_pixiu_data/_data_brussel/100/vsc10042/straw/straw.py", line 350, in fetch_config
    response_msg = send_recv(server, payload)
  File "/vscmnt/brussel_pixiu_data/_data_brussel/100/vsc10042/straw/straw.py", line 324, in send_recv
    resp_len = int(unpack('!I', recv_len)[0])
struct.error: unpack requires a buffer of 4 bytes

Any clue?

[pllopis]

Can you please rerun using -vv and post the result? The increased verbosity will enable debug mode.

[wpoely86]

Namespace(server=['master02.hydra.os'], version='21.08', auth='munge', output_dir='./', verbose=2, list=False)
Using protocol version: 9472
Using authentication method: munge
Header (18):
00000000  25 00 00 00 00 00 07 df 00 00 00 04 00 00 00 00  %...............
00000010  00 00                                            ..
custom str:
00000000  07 df                                            ..
hash input:
00000000  00 00 00 01                                      ....
raw hash:
00000000  02 fb 94 bf 65 9a ba 27 ad 7b 1c 7c 87 34 0d ed  ....e..'.{.|.4..
00000010  52 16 2f 6a 55 3e cc 68 1e 4d f8 9f 34 34 27 35  R./jU>.h.M..44'5
00000020  03                                               .
Auth: (196)
00000000  00 00 00 65 00 00 00 bc 4d 55 4e 47 45 3a 41 77  ...e....MUNGE:Aw
00000010  51 46 41 41 43 74 34 48 66 6f 67 6b 74 5a 44 51  QFAACt4HfogktZDQ
00000020  73 78 78 63 70 74 68 52 73 30 63 35 76 30 6b 72  sxxcpthRs0c5v0kr
00000030  65 48 38 43 6b 49 66 79 41 6f 7a 4a 42 72 53 6f  eH8CkIfyAozJBrSo
00000040  31 70 30 4a 42 70 72 6c 72 66 74 59 67 34 54 41  1p0JBprlrftYg4TA
00000050  4b 48 5a 34 56 4f 69 41 61 64 79 77 6c 50 73 65  KHZ4VOiAadywlPse
00000060  66 51 50 52 4d 78 52 64 66 51 5a 6f 33 33 4d 30  fQPRMxRdfQZo33M0
00000070  37 70 58 72 45 57 7a 33 78 44 48 55 6e 62 31 79  7pXrEWz3xDHUnb1y
00000080  4c 52 75 4d 4b 76 4e 70 66 4a 32 31 59 77 77 45  LRuMKvNpfJ21YwwE
00000090  4d 35 76 45 33 2b 36 49 42 5a 52 30 32 5a 35 51  M5vE3+6IBZR02Z5Q
000000a0  6e 6a 49 35 56 77 74 69 32 33 44 6a 38 50 6b 6a  njI5Vwti23Dj8Pkj
000000b0  6d 66 62 43 64 58 39 72 69 67 48 4f 37 44 71 67  mfbCdX9rigHO7Dqg
000000c0  3d 3d 3a 00                                      ==:.
Body: (4)
00000000  00 00 00 01                                      ....
Full message: (218)
00000000  25 00 00 00 00 00 07 df 00 00 00 04 00 00 00 00  %...............
00000010  00 00 00 00 00 65 00 00 00 bc 4d 55 4e 47 45 3a  .....e....MUNGE:
00000020  41 77 51 46 41 41 43 74 34 48 66 6f 67 6b 74 5a  AwQFAACt4HfogktZ
00000030  44 51 73 78 78 63 70 74 68 52 73 30 63 35 76 30  DQsxxcpthRs0c5v0
00000040  6b 72 65 48 38 43 6b 49 66 79 41 6f 7a 4a 42 72  kreH8CkIfyAozJBr
00000050  53 6f 31 70 30 4a 42 70 72 6c 72 66 74 59 67 34  So1p0JBprlrftYg4
00000060  54 41 4b 48 5a 34 56 4f 69 41 61 64 79 77 6c 50  TAKHZ4VOiAadywlP
00000070  73 65 66 51 50 52 4d 78 52 64 66 51 5a 6f 33 33  sefQPRMxRdfQZo33
00000080  4d 30 37 70 58 72 45 57 7a 33 78 44 48 55 6e 62  M07pXrEWz3xDHUnb
00000090  31 79 4c 52 75 4d 4b 76 4e 70 66 4a 32 31 59 77  1yLRuMKvNpfJ21Yw
000000a0  77 45 4d 35 76 45 33 2b 36 49 42 5a 52 30 32 5a  wEM5vE3+6IBZR02Z
000000b0  35 51 6e 6a 49 35 56 77 74 69 32 33 44 6a 38 50  5QnjI5Vwti23Dj8P
000000c0  6b 6a 6d 66 62 43 64 58 39 72 69 67 48 4f 37 44  kjmfbCdX9rigHO7D
000000d0  71 67 3d 3d 3a 00 00 00 00 01                    qg==:.....
Trying master02.hydra.os:6817...
recvd 0 bytes
Traceback (most recent call last):
  File "/vscmnt/brussel_pixiu_data/_data_brussel/100/vsc10042/straw/straw.py", line 425, in <module>
    main()
  File "/vscmnt/brussel_pixiu_data/_data_brussel/100/vsc10042/straw/straw.py", line 422, in main
    fetch_config(args.server, args.auth, args.output_dir)
  File "/vscmnt/brussel_pixiu_data/_data_brussel/100/vsc10042/straw/straw.py", line 350, in fetch_config
    response_msg = send_recv(server, payload)
  File "/vscmnt/brussel_pixiu_data/_data_brussel/100/vsc10042/straw/straw.py", line 324, in send_recv
    resp_len = int(unpack('!I', recv_len)[0])
struct.error: unpack requires a buffer of 4 bytes

[wpoely86]

I've just tried it with a JWT token and that works fine.

[wpoely86]

I've also tried it on a Slurm 22.05.6 cluster:

$ python straw.py --auth munge -v -v master35.doduo.os 22.05
Namespace(server=['master35.doduo.os'], version='22.05', auth='munge', output_dir='./', verbose=2, list=False)
Using protocol version: 9728
Using authentication method: munge
Header (16):
00000000  26 00 00 00 07 df 00 00 00 04 00 00 00 00 00 00  &...............
custom str:
00000000  07 df                                            ..
hash input:
00000000  00 00 00 01                                      ....
raw hash:
00000000  02 fb 94 bf 65 9a ba 27 ad 7b 1c 7c 87 34 0d ed  ....e..'.{.|.4..
00000010  52 16 2f 6a 55 3e cc 68 1e 4d f8 9f 34 34 27 35  R./jU>.h.M..44'5
00000020  03                                               .
Auth: (196)
00000000  00 00 00 65 00 00 00 bc 4d 55 4e 47 45 3a 41 77  ...e....MUNGE:Aw
00000010  51 46 41 41 41 64 52 6d 50 6c 72 49 49 68 55 39  QFAAAdRmPlrIIhU9
00000020  4d 30 51 45 45 2b 7a 49 32 37 59 6c 41 4f 4a 6a  M0QEE+zI27YlAOJj
00000030  53 74 4c 50 32 54 50 71 37 66 67 37 45 41 45 56  StLP2TPq7fg7EAEV
00000040  48 45 74 72 66 4b 37 77 58 2f 56 61 65 30 59 47  HEtrfK7wX/Vae0YG
00000050  52 5a 35 6a 55 4f 47 50 58 41 54 6b 61 62 50 4c  RZ5jUOGPXATkabPL
00000060  2f 7a 57 64 5a 35 32 69 70 6b 6e 75 73 41 78 48  /zWdZ52ipknusAxH
00000070  47 6e 6c 32 78 52 41 51 76 37 42 6a 59 6c 43 2f  Gnl2xRAQv7BjYlC/
00000080  6c 53 70 47 49 4b 34 6a 49 65 47 58 50 39 2b 41  lSpGIK4jIeGXP9+A
00000090  35 42 55 5a 63 78 55 46 48 43 51 51 4b 70 76 59  5BUZcxUFHCQQKpvY
000000a0  59 36 68 46 33 7a 7a 61 2b 51 33 58 4f 6d 36 6e  Y6hF3zza+Q3XOm6n
000000b0  72 43 64 41 6f 4d 33 6c 71 76 79 69 4c 36 35 41  rCdAoM3lqvyiL65A
000000c0  3d 3d 3a 00                                      ==:.
Body: (4)
00000000  00 00 00 01                                      ....
Full message: (216)
00000000  26 00 00 00 07 df 00 00 00 04 00 00 00 00 00 00  &...............
00000010  00 00 00 65 00 00 00 bc 4d 55 4e 47 45 3a 41 77  ...e....MUNGE:Aw
00000020  51 46 41 41 41 64 52 6d 50 6c 72 49 49 68 55 39  QFAAAdRmPlrIIhU9
00000030  4d 30 51 45 45 2b 7a 49 32 37 59 6c 41 4f 4a 6a  M0QEE+zI27YlAOJj
00000040  53 74 4c 50 32 54 50 71 37 66 67 37 45 41 45 56  StLP2TPq7fg7EAEV
00000050  48 45 74 72 66 4b 37 77 58 2f 56 61 65 30 59 47  HEtrfK7wX/Vae0YG
00000060  52 5a 35 6a 55 4f 47 50 58 41 54 6b 61 62 50 4c  RZ5jUOGPXATkabPL
00000070  2f 7a 57 64 5a 35 32 69 70 6b 6e 75 73 41 78 48  /zWdZ52ipknusAxH
00000080  47 6e 6c 32 78 52 41 51 76 37 42 6a 59 6c 43 2f  Gnl2xRAQv7BjYlC/
00000090  6c 53 70 47 49 4b 34 6a 49 65 47 58 50 39 2b 41  lSpGIK4jIeGXP9+A
000000a0  35 42 55 5a 63 78 55 46 48 43 51 51 4b 70 76 59  5BUZcxUFHCQQKpvY
000000b0  59 36 68 46 33 7a 7a 61 2b 51 33 58 4f 6d 36 6e  Y6hF3zza+Q3XOm6n
000000c0  72 43 64 41 6f 4d 33 6c 71 76 79 69 4c 36 35 41  rCdAoM3lqvyiL65A
000000d0  3d 3d 3a 00 00 00 00 01                          ==:.....
Trying master35.doduo.os:6817...
recvd 4 bytes
Read a message of length 216
00000000  26 00 00 00 1f 41 00 00 00 04 00 00 00 00 00 00  &....A..........
00000010  00 00 00 65 00 00 00 bc 4d 55 4e 47 45 3a 41 77  ...e....MUNGE:Aw
00000020  51 46 41 41 42 50 31 4b 51 69 63 37 6b 70 4c 73  QFAABP1KQic7kpLs
00000030  58 77 35 7a 4e 74 62 6f 32 61 4a 7a 79 49 6c 6e  Xw5zNtbo2aJzyIln
00000040  36 4c 32 76 4d 32 34 42 63 7a 62 4d 54 59 6f 4f  6L2vM24BczbMTYoO
00000050  77 79 71 54 32 6a 2f 33 6b 6a 66 33 58 70 32 41  wyqT2j/3kjf3Xp2A
00000060  64 58 6b 45 32 54 32 78 68 47 39 36 2f 68 65 61  dXkE2T2xhG96/hea
00000070  31 51 43 72 68 73 37 53 66 58 56 36 38 46 6e 49  1QCrhs7SfXV68FnI
00000080  4b 6e 46 77 42 71 7a 76 55 39 55 73 72 6f 65 68  KnFwBqzvU9Usroeh
00000090  43 69 6d 70 6b 39 41 73 6f 2f 76 74 42 44 4a 2f  Cimpk9Aso/vtBDJ/
000000a0  49 52 4f 41 6e 55 32 6e 49 32 69 7a 52 70 6b 6c  IROAnU2nI2izRpkl
000000b0  79 64 4d 36 39 49 4f 71 65 53 62 76 79 6b 43 72  ydM69IOqeSbvykCr
000000c0  73 42 6a 41 71 38 6a 51 65 78 74 4c 41 68 69 51  sBjAq8jQextLAhiQ
000000d0  3d 3d 3a 00 00 00 08 4e                          ==:....N
Munge cred: b'MUNGE:AwQFAABP1KQic7kpLsXw5zNtbo2aJzyIln6L2vM24BczbMTYoOwyqT2j/3kjf3Xp2AdXkE2T2xhG96/hea1QCrhs7SfXV68FnIKnFwBqzvU9UsroehCimpk9Aso/vtBDJ/IROAnU2nI2izRpklydM69IOqeSbvykCrsBjAq8jQextLAhiQ==:'
We got a reply with errno: 2126

[pllopis]

Second one on 22.05 should be easy: it corresponds to ESLURM_CONFIGLESS_DISABLED

i.e. your slurmctld does not seem to have configless enabled I guess?

Need to do some better errno management :) thanks for reporting these bugs! I need to have a more detailed look at the munge issue with 21.08

[wpoely86]

Right, my bad. The 22.05 is indeed not using a configless setup.

What's the proper way to use this now with munge inside a container? Pass through the munge socket with a volume mount?