Ignore Duplicate Scrobbles from Last.fm #173
Labels
bug
Something isn't working
Comments
|
Let me make sure I understand your MS configuration: flowchart TB
rn1[Roon] --> rnlfm[Last.fm]
subgraph multi-scrobbler
rnlfm[Last.fm Source] --> lzc[Listenbrainz Client]
lz[Listenbrainz Source] --> ml[Maloja]
end
Is this correct?
MS should already be detecting small differences in scrobbles with the same timestamp as duplicates, I have a whole test suite for it :) Let's try to figure out why this isn't working for your use case. If you are not already using it please switch to using File or AIO File for your listenbrainz client configuration. Then, add the following (example of [
{
"name": "brainz",
"enable": true,
"configureAs": "client",
"data": {
"token": "029b081ba-9156-4pe7-88e5-3be671f5ea2b",
"username": "MyUSer"
},
+ "options": {
+ "verbose": {
+ "match": {
+ "onNoMatch": true,
+ "confidenceBreakdown": true
+ }
+ }
+ }
}
]This will make multi-scrobbler log details to the DEBUG log level when a scrobble client does not find duplicate scrobbles, like this: Make sure MS is logging to DEBUG -- if you have not modified env Please turn on these settings and restart MS. Then wait for a duplicate scrobble to occur and copy-paste the detailed logs here so we can figure out what is going on. |
|
Pretty much spot on, and I've added that extra config now, so will monitor. I've attached my config files to make it even more clear. Once again thank you :) |
|
Ignoring visyn configs, it seems you have a circular reference in your configs: flowchart TB
subgraph verbose
lfms[src-hidyn-lastfm] --> lzc[cli-hidyn-listenbrainz]
lzs[src-hidyn-listenbrainz] --> ml[Maloja]
lzs --> lfmc[cli-hidyn-lastfm]
%% lfmc <-.-> lfms
%% lzs <-.-> lzc
end
subgraph simplified
lfm[hidyn-lastfm] --> lz[hidyn-listenbrainz]
lz --> mls[Maloja]
lz --> lfm
end
Is this on purpose? MS isn't really designed/tested for handing this kind of feedback loop. Ideally, all data should flow in one direction. |
|
Yeah that's on purpose and seems to be working okay so far. I'm pretty sure it's not the cause of this issues as I've had Roon randomly posting duplicate scrobbles before I used MS. I scrobble from my phone with Pano Scrobbler to ListenBrainz and as the whole setup is new wasn't ready to ditch Last.fm fully so setup the loop and controlled it by limiting the clients each source scrobbles too. If worst comes to the worst and that is the cause I can always remove it. |
|
I suppose I can always just make Pano scrobble to Last.fm too as a backup if that will play nicely and I won't get duplicate imports into ListenBrainz. The reason I didn't go this route initially is because Last.fm is awful at handling multi-artist tracks compared to ListenBrainz and Maloja so wasn't sure how to handle it. |
|
We finally have a duplicate. Last.fm ListenBrainz |
True, LFM has the least data-rich scrobbling. Why do you need to scrobble back to LFM from LZ? Couldn't it be: Or is LFM for recommendations? P.S. Roon has an API and I've been wanting to implement it as a source but I've had a hard time getting the company to communicate. I don't want to pay for it as I won't really be using it, just developing for it. |
|
If I didn't have a brand new 4 day old to look after I'd look to spring for a licence for you just to add Roon as a source. Honestly Roon is amazing and I struggle to find anything as good for playlists and auto-radio. The fact it can cast to pretty much any speaker in the house flawlessly too (AirPlay/Chromecast etc) I tried replacing it with MusicAssistant but it's not there yet. |
|
Thanks for the logs. Do you have any logs from when the original scrobble was made?
Congratulations! Totally get it I appreciate the donations already. I'll get to Roon eventually... |
FoxxMD
added a commit
that referenced
this issue
Jul 23, 2024
|
Sorry was just trying to keep it to what you might need so you don't have to read loads you might not need. Only Roon log I could find So it seems Roon is only logging the (Album Version) I don't get what's happening on Last.fm's side to remove that and duplicate the log. Here's a more expanded log from MS If you think the feedback loop is the cause I'll ditch it and setup how you suggest. As poor as Last.fm is now compared to ListenBrainz and Maloja it's hard to let go of an old friend isn't it ha but if this is the cause I'll bite the bullet. P.S. You know Roon does a 14 day trial? Granted not a great amount of time to work in but maybe you can use multiple different e-mails to sign up. |
Yeah...and it's probably enough time to implement a Source but then I need to be able to support users going forward which may require me to reproduce behavior from Roon as well as provide fixes and update the api when they cut new releases. I'd be paying them $15/mo so that I can provide free software to make their product better. It doesn't sit well with me.
Ok so here's what I'm parsing from the log (putting here for you but more for future me): LFM source discovers track from history LZ client refreshes existing scrobbles (history) LZ client existing scrobbles does not have track, scrobbles (correctly) LZ source discovers track from history Maloja client existing scrobbles does not have track, scrobbles LFM client existing scrobbles (incorrectly) does not have track, scrobbles (May be fixed by forced history refresh) LFM source discovers duplicate track LZ client (incorrectly) does not find duplicate in history (May be fixed by forced history refresh) LZ client (incorrectly) scrobbles dup track So there's two things going wrong here, other than the underlying feedback loop:
This would definitely fix the issue but I don't think your scenario is unfixable. The mechanisms listed above should be working to preventing dups even with your set up. I will try to see if I can reproduce in a test suite and determine if these are actually bugs. In the meantime you can brute-force the refresh history bug by using docker image Additionally, you should add |
|
I've added both to help test the PR in the hope it's helpful for others. I'll be taking your advice and removing the loopback altogether afterwards though. What you suggested makes more sense. Just to confirm where is best to place "refreshForce": true in my instance. On the source or client side? Currently have it added to the LB client and LFM client. |
|
Added to clients in the same |
|
Great testing as we speak. This is what I have. Hopefully it's correct. |
|
Yes that is correct. P.S. I've made a post on the roon forums as a final effort to try to get an account for dev. |
|
Okay got another duplicate in Last.fm but it didn't get submitted to ListenBrainz.
|
|
Great! So forcing a refresh of the client's upstream scrobbles is preventing dup scrobbles, as it should, when the existing scrobble is in the history. Just need to make sure that refresh happens in time for the comparison without the need for force refresh... We can see the reason the LFM duplicate happened is because MS didn't think the track title's were similar enough:
With a weighted score of 0.91 it almost considered it a dupe (needs to be Ironically, LFM actually also provides metabrainz IDs for historic scrobbles but only for artist and album -- no release id. So it's not possible to programmatically determine it's the same track without using the track title at face value. The score weights are pretty well balanced from my testing so I'm reluctant to change those. I also don't want to hardcode an exception for My first impression for fixing this, then, would be to allow user-configrable terms to be removed from a play so that edge cases like this can be taken care of by the user in |
|
@HidYn please update to the latest If you find it is still doing duplicates that aren't related to the track title being different you can use the new option
|
|
I took your advice and just removed the loopback due to the sightly different titles causing an issue. More than happy to put my Last.fm source config back in place if you'd like me to help test though and report back the results? |
|
Nah it's all good I've got a test written for it already anyways. Thanks again for the detailed writeup, logging, and testing. |
|
@HidYn if you'd like to try this out and go back to your original setup I have a way to modify scrobble data using configs. Switch to The docs are here until it's cut for a release and for your particular scenario the config would look like this for [
{
"name": "myLastFm",
"configureAs": "client"
// ...
"options": {
"refreshStaleAfter": 3,
// ...
"playTransform": {
"compare": {
"existing": {
"title": [
"(Album Version)" // ignore '(Album Version)' in existing scrobble titles during duplicate comparison
]
},
},
}
}
}
] |
|
Scrobble modification and refresh forcing configuration ( |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Firstly, thank you for such a wonderful project. MS has allowed me to focus on ListenBrainz and Maloja instead of Last.fm
I've got what more like a feature request. I use an application called Roon which only supports Last.fm scrobbling and it appears to have a long-standing bug that never gets addressed that involves duplicate scrobbles. https://community.roonlabs.com/t/play-count-not-updating-as-expected/218686/67
The track names appear to be slightly different, but the scrobble time is exactly the same down to the second.
I've got Last.fm setup as a source only for Roon and I would love the ability to have those duplicate scrobbles ignored when pulled through to ListenBrainz as at the moment the duplicates get pushed there. It appears to be already ignoring them when pushed through to Maloja to which I have ListenBrainz as the source so no duplicates get posted there.
Keep up the great work.
The text was updated successfully, but these errors were encountered: