Background maintenance tasks disabled when running with --ifile option

[jketterl]

Context: We integrated dump1090 into OpenWebRX (a web-based SDR solution) and it is working fine for the most part. Since OpenWebRX supports a number of different SDR devices and handles them directly, we feed dump1090 with IQ data using the --ifile option (on STDIN).

One problem that we noticed is that after prolonged operation, dump1090 will start printing a decent amount of these error messages:

ICAO hash table full, increase ICAO_FILTER_SIZE

On my test setup this happened somewhere around the second day of continuous operation, but as far as my understanding goes, this would probably depend on the amount of traffic received.

I should probably also mention that the decoder still seemed to be working, at least to the extent that I could not detect any degradation, so the main concern here is probably log spam.

Additional research on the issue:

Some quick research shows that this is related to the logic in icao_filter.c. I'm not really sure what purpose it serves at this point, but the error message shows that icaoFilterAdd() is being called.

In the same file there is a method icaoFilterExpire() with the comment "call this periodically". I found the code that seems to be responsible for calling it in backgroundTasks() in dump1090.c. The problem here is that this particular piece of code is disabled when the input is a file (as is the case in our integration).

With my limited understanding, I went in and removed the condition, and as far as I can tell, the error messages are gone.

One of the questions that eludes me is why these maintenance tasks are disabled in the first place. I do understand that they are probably based on timestamps, so my guess is that they are disabled because they would probably not work correctly when replaying an actual file (at a higher-than-realtime rate). I am not sure if there is any harm in simply removing the condition, but if there's not that would certainly make for an easy solution.

[mutability]

So there are a few things going on here..

--ifile is mostly used for demodulator testing, as it lets you reproducibly run the same data with tweaks to the code and compare output. Originally, it was an entirely separate code path; later, it got plumbed into the common SDR framework. It runs as fast as it can (unless --throttle, which was added later so interactive paths could be tested). I never really envisioned someone using it to process live data!

The exclusions in backgroundTasks() are mostly there so that you can repeatedly run the same input file through and get the same results (since the background task logic is running against system time, not the time implied by how many samples we processed).

The hash table error message means that the hash table used to look up aircraft is full and new unique aircraft will not be tracked. So it'll have a real effect if you just ignore it; the expire function does need to be called if you're running for any real length of time.

Does OpenWebRX have a way to provide data via SoapySDR (perhaps via the remote driver)? The dev branch has soapysdr support merged, so that might be the simplest fix for you.

[jketterl]

No, we don't have a SoapySDR driver for this. It would involve a lot of "faking" stuff because many of the SDR control APIs would basically not work. And there's also the problem of what to report for all the query APIs (frequency range, sample rates etc.).

Feeding data through STDIN/STDOUT is basically the go-to method when integrating third-party decoders, there is a number of other modes that work that way (for example, Packet / APRS, FreeDV, DRM, dumphfdl, dumpvdl2). It's not necessarily the only option (for example, during development I was using --net-ro-port until I was told that STDOUT works, too), but other implementations require additional scaffolding and code.

Up until recently, the whole DSP pipeline in OpenWebRX was based on individual components being connected via shell piping, this has been recently reworked to use a more efficient method and a standardized API. To make this work for external applications however there needs to be a library with some kind of data in and out functions. I don't think this is the case for dump1090.

So in short, feeding data via STDIN / STDOUT is probably the best option, at least for now.

I do understand that simply removing the condition would mess with testing because it would introduce the variable "time" and would make it unpredictable.

I guess the ultimate solution would be to reimplement the code in a way that makes it independent of the system clock, and using calculation based on sample count + sample rate to replace any time-dependent functions. I suppose this will raise some other issues though.

Would you consider adding a simple flag in order to simply enable the background tasks for this case when applicable?

[mutability]

A flag would be fine. Might be worth looking at rearranging that whole family of flags to make a little more sense.

Re SoapySDR I was thinking of a driver that supports exactly one frequency / sampling rate / etc only (and returns that from the query APIs); you already must know what these settings should be for the stdin case. But if it's not there currently, maybe that's a thought for the future.