Bug: expired and obsolete rse counters aren't being updated since June 7th

[haozturk]

Bug Description

$ rucio list-rse-usage T2_CH_CERN | grep -A 5 expired
  source: expired
  used: 5.257 PB
  updated_at: 2024-06-07 19:30:26
  rse: T2_CH_CERN
------
$ rucio list-rse-usage T2_CH_CERN | grep -A 5 obsolete
  source: obsolete
  used: 5.112 TB
  updated_at: 2024-06-07 19:30:21
  rse: T2_CH_CERN

Other rse counters (rucio, unavailable) are being updated fine.

@ericvaandering I was looking at abacus-rse daemon. Looks like it's updating the rse counter rucio[1], not expired or unavailable. So, looks like this is not the relevant daemon. Before I further investigate, I saw that you had some commits in production config in flux [2] on the same day this issue started. Can it be related somehow?

[1] https://github.com/rucio/rucio/blob/48ba197bda5eed8107cd7bcb3e396b0c126ba00a/lib/rucio/core/rse_counter.py#L132
[2] https://github.com/dmwm/rucio-flux/commits/main/

Reproduction Steps

Run the cli commands above

Expected Behavior

These counters should be updated regularly.

Possible Solution

No response

Related Issues

https://its.cern.ch/jira/browse/CMSTRANSF-932

[dynamic-entropy]

Something else that is broken since around the same time is the Request status dashboard:
https://monit-grafana.cern.ch/d/-j0KiY0Mk/rucio-overview?from=now-7d&orgId=11&refresh=5m&to=now

[ericvaandering]

Which day's commits should I look at?

[haozturk]

June 7th

[ericvaandering]

I fixed the dashboard. Just needed to find the equivalent with rucio_probes_ at the start.

I'll check into the rse counters. I suspect it is related.

[ericvaandering]

OK, the probe doing this is /probes/cms/check_used_space and it seems to be running. So there must be some issue with the proxy perhaps?

[ericvaandering]

Issue is solved. #812 is the cleanup