Adding new srng field to grid and map files + range/CPID filtering with grid_filter

[egthomas]

This pull request adds a new srng field to grid and map files that contains the (average) slant range of the velocity vector recorded by a radar in a given grid cell. All C and IDL read/write routines have been updated to handle files both with and without this new field.

Note this is a preliminary step towards more flexible filtering of line-of-sight data based on the measured slant range, which can possibly be contaminated by E-region echoes at near ranges or incorrectly mapped at very far ranges. Ultimately, it should be faster to apply a post-processing range filter on this new srng field than using the current -minsrng and -maxsrng options with make_grid (since those require slant range calculations for all ranges of every input scan).

To test, try creating grid and map files with your favorite options and see if anything breaks. The contents of the srng field can be viewed with either dmapdump -d or your favorite IDL plotting software (at least until pydarnio is updated).

[pasha-ponomarenko]

@egthomas, I guess this is designed to work with elevation-based range otherwise I don't see how it will work with the standard or Chisham models with fixed relationship between the group range and distance to the ground projection of the scatter point. Am I correct?

[egthomas]

@pasha-ponomarenko thanks for the question. This new field only stores the slant range (or group path) to the line-of-sight measurement, which can vary depending on radar operating parameters (eg, frang and rsep) and is independent of any geolocation method. So here a grid file is produced as normal using either the standard or Chisham virtual height models to calculate latitude/longitude locations and assign them to a grid cell; this just adds one extra field allowing you to possibly exclude data afterwards based on the original slant range.

[pasha-ponomarenko]

OK, so it just memorises the slant range for each contributing radar which is not stored otherwise, right? So that it enables the user to post-filter the data without re-running the whole gridding/mapping/fitting chain.

[egthomas]

@pasha-ponomarenko yes, exactly! Originally no slant range information was retained once a grid vector was created (note for the TS18 model I modified the code to store the slant range in the grid files' power field, which is not a portable or long-term solution).

Here is an example plot using the new -rng option with grid_plot (note it can take the average, min, or max in a cell depending on the command line options if data from more than one radar is present):

[mtwalach]

Hi, I really welcome these changes and it looks great but I am afraid I can't test right now.
I don't have a working copy of RST-dev at the moment and due to all the install issues I have been having (#609). I don't really want to mess with the little bits of RST that I have got right now because it took me two days to get working what I have got. Until I have a set-up that allows me to switch branches and reinstall RST quickly, I can't do any testing I am afraid.

[egthomas]

@mtwalach no problem! Even if you are unable to test, any feedback or ideas based on your own extensive work with map files will be greatly appreciated as this pull request is inevitably discussed.

[egthomas]

I've just modified the grid_filter tool (which previously only supersampled a grid file in time, which seems useless?) to allow removal of grid vectors from a file based on user-defined slant range criteria. Note that by default, nothing will be removed (default limits are 0 km and 10,000 km).

With the -vb option, some diagnostics are printed about the number of grid vectors in each record before and after filtering, eg

grid_filter -minrng 800 -maxrng 2000 -vb 20240101.north.grdmap > out.grdmap
2024-01-01 00:00:00  530 > 349 pts (65.85%)
2024-01-01 00:02:00  527 > 340 pts (64.52%)
2024-01-01 00:04:00  549 > 357 pts (65.03%)
2024-01-01 00:06:00  553 > 379 pts (68.54%)

The idea is to eventually support a -cpid option with this routine as well that can apply some further exclusion based on radar operating mode (ie CPID), but that requires further discussion by the DPTF and others.

[egthomas]

I've just made one further change to grid_filter: with the new -cpid option, one can now also exclude velocity vectors from a grid file whose CPIDs do not fall within the ranges listed in a new cpid.dat file, eg something like

0000 1000
3300 3310
3500 3503

Obviously these values will need further iteration, so feel free to modify for your own testing purposes.