lbry-sdk not making progress during initial address sync when connecting to herald.go

[moodyjon]

This data was gathered with some local changes plus my published branch herald.go/blockchain_tx_rpc1:

failure.txt

It's not completing, even though some messages are emitted like these:

2022-11-18 17:12:30,189 - lbry.wallet.ledger - INFO - subscribed to 15/15 addresses on 127.0.0.1:50002
2022-11-18 17:12:30,189 - lbry.wallet.ledger - INFO - finished subscribing to 15 addresses on 127.0.0.1:50002

It just keeps spinning in some loop alternating between address.get_history and transaction.get_batch. It's also only mentioning a couple of specific transactions -- one at height=0 and one at height=1

I will continue investigating this, but maybe @eukreign, @jackrobison, or @shyba could point me in the right direction.

Herald.go is not generating address notifications and header notifications at this time AFAICT. That's one difference in behavior between herald.go and python herald that may be relevant.

[moodyjon]

Lbry-sdk <-> Herald RPCs sent & responses:
Baseline: a.txt
Herald.go: b.txt

It's starting to diverge at the point of blockchain.address.subcribe (Python hub returns None a lot of the time) and later at blockchain.address.get_history (Herald.go returns 2 different transactions at height 0 and height 1, Python returns same transaction at height -1 and later height 201).

[moodyjon]

Found a problem with address.get_history. The interpretation of IncludeStop iteration option was confusing me. Hence I was finding an extra record following the desired hashX when no matching record existed.

Secondly, I have narrowed down where it's stuck. Test setup sequence is now at the stage (send_to_address_and_wait) where mempool TXs must generate notifications and be reportable. The existing mechanism detectChanges() is only monitoring height in DBStateValue. But there can't be a change in height until send_to_address_and_wait moves on to generate blocks.