Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GPSU Invalid Timestamp information #162

Open
time4tea opened this issue Jun 30, 2022 · 18 comments
Open

GPSU Invalid Timestamp information #162

time4tea opened this issue Jun 30, 2022 · 18 comments

Comments

@time4tea
Copy link

time4tea commented Jun 30, 2022 •
edited
Loading

I am trying to parse data stream from HERO7 Black.

However, in the data stream there appears to be a corrupt data item:

GPSU type 'U' size 16 data: "2225525525425525",

It seems unlikely that this is due to storage device data corruption, as the rest of the stream is totally fine. I guess I'm asking if there is a known issue with GPSU timestamps being emitted incorrectly from the device, or if this is a known value to indicate some kind of error condition.

I do know that in some cases the first GPS5 with a GPSF of 2 or 3 is actually invalid - seems that if the lock is acquired part way through a packet, the entrie packet is marked with a lock, but some samples correspond to a non-locked period, but here the GPSF is 0

The previous and subsequent timestamps in the data stream are just fine, however the previous GPS5 sample looks odd:

    STRM nest size 264 
      EMPT type 'L' size 4 data: 4,
      TSMP type 'L' size 4 data: 58,
      STNM type 'c' size 44 data: "GPS (Lat., Long., Alt., 2D speed, 3D speed)"
      GPSF type 'L' size 4 data: 3,
      GPSU type 'U' size 16 data: "220617220042.050",
      GPSP type 'S' size 2 data: 128,
      UNIT type 'c' samplesize 3 repeat 5 data: "deg", "deg", "m", ...
      SCAL type 'l' samplesize 4 repeat 5 data: 10000000, 10000000, 1000, 1000, 100,
      GPS5 type 'l' samplesize 20 repeat 4 data:  (redacted valid looking data)

    STRM nest size 184 
      EMPT type 'L' size 4 data: 5,
      TSMP type 'L' size 4 data: 58,
      STNM type 'c' size 44 data: "GPS (Lat., Long., Alt., 2D speed, 3D speed)"
      GPSF type 'L' size 4 data: 3,
      GPSU type 'U' size 16 data: "220617220042.050",
      GPSP type 'S' size 2 data: 128,
      UNIT type 'c' samplesize 3 repeat 5 data: "deg", "deg", "m", ...
      SCAL type 'l' samplesize 4 repeat 5 data: 10000000, 10000000, 1000, 1000, 100,
      GPS5 type 'l' samplesize 20 repeat 0 

STRM nest size 244 
      EMPT type 'L' size 4 data: 5,
      TSMP type 'L' size 4 data: 61,
      STNM type 'c' size 44 data: "GPS (Lat., Long., Alt., 2D speed, 3D speed)"
      GPSF type 'L' size 4 data: 0,
      GPSU type 'U' size 16 data: "2225525525425525",
      GPSP type 'S' size 2 data: 0,
      UNIT type 'c' samplesize 3 repeat 5 data: "deg", "deg", "m", ...
      SCAL type 'l' samplesize 4 repeat 5 data: 10000000, 10000000, 1000, 1000, 100,
      GPS5 type 'l' samplesize 20 repeat 3 data: 13,17788664,629855,-134217724,81, (redacted valid looking data)

    STRM nest size 284 
      EMPT type 'L' size 4 data: 5,
      TSMP type 'L' size 4 data: 66,
      STNM type 'c' size 44 data: "GPS (Lat., Long., Alt., 2D speed, 3D speed)"
      GPSF type 'L' size 4 data: 3,
      GPSU type 'U' size 16 data: "220617220044.415",
      GPSP type 'S' size 2 data: 125,
      UNIT type 'c' samplesize 3 repeat 5 data: "deg", "deg", "m", ...
      SCAL type 'l' samplesize 4 repeat 5 data: 10000000, 10000000, 1000, 1000, 100,
      GPS5 type 'l' samplesize 20 repeat 5 data:  (redacted valid looking data)

Really appreciate any insights! - Thank you!
@time4tea time4tea mentioned this issue Jun 30, 2022
Closed
@time4tea
Copy link
Author

Is there any significance that the string (spacing for clarity) is "22 255 255 254 255 25" - looks like some kind of uninitialised buffer write?

@dnewman-gpsw
Copy link
Collaborator

I have never seen this error. I expect you are correct in that there was some invalid data delivered from the GPS chip, or empty payload. We do have any error signalling for this, as we didn't know it could happen. Have you seen this error more than once?

@time4tea
Copy link
Author

Hiya - Yes - this is the second time that it has been reported to me (although it is a defect reported against my parsing software, its not my actual device) - The end user has supplied the MP4 file in the linked issue above if you'd like the whole thing.
Interestingly, the invalid GPSU data was the same in both cases.

@dnewman-gpsw
Copy link
Collaborator

Yes please to the sample clip.

@time4tea
Copy link
Author

Sure. The MP4 is in the linked issue above, along with a bit of narrative, but I'll duplicate it here: https://s3.amazonaws.com/s3.voneicken.com/GH010359.MP4

@tve
Copy link

tve commented Jun 30, 2022

This seems to happen when I use my Hero 7 black without having the date set, i.e., it's set to some date in 2016. This in turn happens 'cause I run the camera with a USB-C power source and no internal battery: it looses the date after some hours without having any power when I remove the USB-C power to recharge the power pack.

Basically: remove battery from Hero 7 black, wait long enough for it to loose date&time (a few hours?), plug in USB-C power, switch to prefs and see that date is in 2016, start recording, observe the issue described here.

@dnewman-gpsw
Copy link
Collaborator

That is surprising as that time and date field is not from real time clock, it is the value reported from GPS.

@time4tea
Copy link
Author

It's not unbelievable though. This capture is from right at the start of the trace. It could be that after a few packets with a GPS time after a hard boot with loss of RTC, the firmware decides to do some extra work, which misses a deadline, so the output isn't captured for that timeslice. I have no idea how the gopro firmware is written, so this is pure conjecture!!

@dnewman-gpsw
Copy link
Collaborator

In the look at the number of samples per payload of GPS5, the GPS chip is behaving eratically. Instead of 17-19 samples per payload (averaging around 18.1Hz), the stream (linked above) has payloads of 13, 9, 0, 4, 5, 0, 3.., averaging 4.6Hz. The GPS chip is not behaving ideally. This camera likely needs a factory reset.

@tve
Copy link

tve commented Jul 3, 2022

I'm not sure the problem is reproducible. I just took a look at the recordings I have since the first one without date&time set:

  • 2022/4/16 - date&time not set, GPS track is fine
  • 2022/6/17 - date&time not set, GPS track messed-up, sample linked 4 comments up
  • 2022/6/18 - date&time sync'd before recording, GPS track is fine
  • 2022/6/23 - date&time not set, GPS track is fine
  • 2022/6/24 - date&time not set, GPS track has issues with data points out of order (by <1sec)
  • 2022/6/25 - date&time sync'd, GPS track is fine

I'm concluding that if the date&time are lost there is a chance that the GPS track ends up messed-up, but it's not systematic. Presumably some race condition. I'm not sure this can be pursued productively at this point.

@dnewman-gpsw
Copy link
Collaborator

We also don't have any firmware releases planned for HERO7. This should not come up on more recent cameras.

@time4tea time4tea mentioned this issue Aug 6, 2023
Closed
@time4tea
Copy link
Author

time4tea commented Aug 6, 2023

Hi @dnewman-gpsw - Just FYI - I had another user report of invalid data in GPSU. time4tea/gopro-dashboard-overlay#141

@dnewman-gpsw
Copy link
Collaborator

I look forward to knowing the camera model this was.

@time4tea
Copy link
Author

time4tea commented Sep 2, 2023

Hi - The Camera was Hero 9 Black - HD9.01.01.52.00

I can see the same sort of behaviour as before, where the number of samples from the GPS chip is erratic for the whole file.
Here is some debug from my software (the GPS5 values are redacted)

  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 31 35 37 2e 37 32 39] [b'230625112157.729']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 1, Length Bytes=28/20 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 31 35 37 2e 37 32 39] [b'230625112157.729']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 4, Length Bytes=88/80 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 30 2e 35 33 34] [b'230625112200.534']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 30 2e 35 33 34] [b'230625112200.534']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 5, Length Bytes=108/100 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 31 2e 36 33 34] [b'230625112201.634']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 32 2e 35 36 39] [b'230625112202.569']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 3, Length Bytes=68/60 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 32 2e 35 36 39] [b'230625112202.569']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 3, Length Bytes=68/60 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 34 2e 30 35 34] [b'230625112204.054']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 34 2e 30 35 34] [b'230625112204.054']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 2, Length Bytes=48/40 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 34 2e 30 35 34] [b'230625112204.054']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 37 2e 30 32 34] [b'230625112207.024']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 37 2e 37 33 39] [b'230625112207.739']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 1, Length Bytes=28/20 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 30 38 2e 35 36 34] [b'230625112208.564']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 2, Length Bytes=48/40 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 30 2e 33 37 39] [b'230625112210.379']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 9, Length Bytes=188/180 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 30 2e 33 37 39] [b'230625112210.379']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 3, Length Bytes=68/60 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 31 2e 38 30 39] [b'230625112211.809']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 31 2e 38 30 39] [b'230625112211.809']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 9, Length Bytes=188/180 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 33 2e 37 33 34] [b'230625112213.734']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 33 2e 37 33 34] [b'230625112213.734']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 3, Length Bytes=68/60 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 36 2e 31 35 34] [b'230625112216.154']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 37 2e 37 34 39] [b'230625112217.749']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 1, Length Bytes=28/20 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 37 2e 37 34 39] [b'230625112217.749']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 8, Length Bytes=168/160 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 39 2e 38 33 39] [b'230625112219.839']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 31 39 2e 38 33 39] [b'230625112219.839']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 1, Length Bytes=28/20 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 32 2e 32 35 39] [b'230625112222.259']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 32 2e 32 35 39] [b'230625112222.259']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 3, Length Bytes=68/60 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 34 2e 31 38 35] [b'230625112224.185']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 2, Length Bytes=48/40 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 34 2e 31 38 35] [b'230625112224.185']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 6, Length Bytes=128/120 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 35 2e 33 39 35] [b'230625112225.395']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [35 35 30 39 31 30 30 30 38 32 32 34 30 34 30 30] [b'5509100082240400']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [35 35 30 39 31 30 30 30 38 32 32 34 30 34 30 30] [b'5509100082240400']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 1, Length Bytes=28/20 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 39 2e 30 32 35] [b'230625112229.025']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 10, Length Bytes=208/200 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 39 2e 30 32 35] [b'230625112229.025']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 6, Length Bytes=128/120 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 39 2e 36 38 35] [b'230625112229.685']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 32 39 2e 36 38 35] [b'230625112229.685']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 4, Length Bytes=88/80 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 31 2e 36 36 35] [b'230625112231.665']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 33 2e 37 30 30] [b'230625112233.700']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 3, Length Bytes=68/60 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 33 2e 37 30 30] [b'230625112233.700']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 6, Length Bytes=128/120 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 35 2e 36 38 30] [b'230625112235.680']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 36 2e 35 30 35] [b'230625112236.505']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 3, Length Bytes=68/60 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 36 2e 35 30 35] [b'230625112236.505']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 6, Length Bytes=128/120 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 38 2e 37 30 35] [b'230625112238.705']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 33 38 2e 37 30 35] [b'230625112238.705']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 2, Length Bytes=48/40 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 34 30 2e 32 34 35] [b'230625112240.245']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 34 31 2e 35 36 35] [b'230625112241.565']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 2, Length Bytes=48/40 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [32 33 30 36 32 35 31 31 32 32 34 31 2e 35 36 35] [b'230625112241.565']

As you can see, the GPSU values are incorrect for a few samples:

GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [35 35 30 39 31 30 30 30 38 32 32 34 30 34 30 30] [b'5509100082240400']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 0, Length Bytes=8/0 
  GPMDItem: GPSU, Type=U, Size=16, Repeat: 1, Length Bytes=24/16 [35 35 30 39 31 30 30 30 38 32 32 34 30 34 30 30] [b'5509100082240400']
  GPMDItem: GPS5, Type=l, Size=20, Repeat: 1, Length Bytes=28/20

I don't have permission to share the file - is there any more information that would be useful for you? I'm guessing that this is probably happening quite a lot, as my software doesn't have that many users, and we have seen a couple of incidences now.

Thanks!

@time4tea
Copy link
Author

time4tea commented Sep 2, 2023

I hadn't previously noticed, but the timings of the samples are also very erratic, sometimes multiple packets at exact same time (hence 0 size, i guess)

@dnewman-gpsw
Copy link
Collaborator

Filtering is likely the only solution for these older camera sources. HERO11 deprecated GPS5 and GSPU, adding GSP9 as cleaner way to sync time and location.

@time4tea
Copy link
Author

time4tea commented Sep 2, 2023

I can filter the invalid GPSU, but incorrect values are correlated with loss of sync with GPS chip or something, as can be seen by the erratic readings.
I take it that no further firmware will be created for these devices, so it is what it is...

@dnewman-gpsw
Copy link
Collaborator

Correct, new firmware is not likely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tve @time4tea @dnewman-gpsw