Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NFT tracking jittery #97

Open
Vasheer26 opened this issue Apr 30, 2020 · 30 comments
Open

NFT tracking jittery #97

Vasheer26 opened this issue Apr 30, 2020 · 30 comments

Comments

@Vasheer26
Copy link

Hi all,
I have been creating my tests around the examples/nft_threejs_wasm.html.
So far I have found this to be the best NFT tracking example since it works on the iPhone, Pixel, and Samsung however, I noticed it is very jittery when I placed a plane/video over the NFT image.

The jittery is also there when the 3D sphere is used but it is not easily noticeable.
I would like to spend some time and effort to see if I can help smooth or reduce the jittery.

I assume I would need to look into artoolkit_wasm.js.

Following the instructions- I have started with installing Docker and slowly finding my way through various Youtube videos on how to set up the environments and pulling the Git repo. I would greatly appreciate connecting with anyone here who's more knowledgeable for assistance and direction.
I know I will need to work with the c file and eventually use emscripten.
@ThorstenBux
Copy link

ThorstenBux commented Apr 30, 2020 via email

@kalwalt
Copy link

kalwalt commented Apr 30, 2020

Hi @Vasheer26, i will share here with you my work. Not on Jsartoolki5 but i'm working on my lighter version JsartoolkitNFT you can try here https://github.com/kalwalt/jsartoolkitNFT/tree/improving-lerp/examples i'm trying to use the one euro filter (see the oneEuroFilter.js) i think you can use the same code for jsartoolkit5. I'm not sure of my code, nor of my implementation. I hope that his will help you...

@Vasheer26
Copy link
Author

@ThorstenBux Thank you for heads up.
@kalwalt Thank you. I will spend time on implementing and testing this on a handful of devices and share my results in the coming days/weeks.

@kalwalt
Copy link

kalwalt commented Apr 30, 2020

@ThorstenBux Thank you for heads up.
@kalwalt Thank you. I will spend time on implementing and testing this on a handful of devices and share my results in the coming days/weeks.

That's magic! Ask if you need other infos, help, support 🙂

@kalwalt
Copy link

kalwalt commented Apr 30, 2020

Also, this is the oneEuroFilter that @ThorstenBux point me to

@Vasheer26
Copy link
Author

@ThorstenBux @kalwalt

After spending some times testing the provided codes, these are my findings:-

  1. Visually, the 'interpolationFactor' (IF) approached looked good but it was flawed.
    The gentle updating of position matrix (interpolationFactor=24) masked the jitterness.
    The lower the interpolationFaction (e.g 2 or less) makes rendering object follow the marker/image snappier but brought back the jitterness. We want the tracking to be snappier :)

  2. I implemented 3 stages of delta displacement so I could apply different levels of interplationFactor

  • "small" displacement delta <= 0.003 ( using IF = 10 )
  • "medium" displacement where delta > 0.003 ( using IF = 1.5 )
  • "big" displacement where delta > 0.005 ( using IF = 1 )

Again, remember the lower the IF, the snappier for the rendered object to match the marker.
When the displacement is small, meaning it is near the correct position, we have to use IF 4 or greater to smooth out the jitterness. This approach was okay but not snappy enough.

  1. I also discovered the root cause of the jitterness is in the data being return in matrixGL_RH
  • With the webcam facing the stationary image, you can see the returning dataset bouncing all over the place. I tried to use every 5th, 10th, 20th and 50th dataset to see if jitterness could be lessen but unfortunately no. If anything you can clearly see the wide range of returning datasets.

I was thinking next

  • I might try to smooth out the data sets or removed wild data sets.
  • I might try and see how the datasets are being generated and then why there is such a wide variance.

Just sharing my finding and thoughts. Happy to hear any of your thoughts or suggestions.

@ThorstenBux
Copy link

ThorstenBux commented May 4, 2020 via email

@kalwalt
Copy link

kalwalt commented May 4, 2020

@Vasheer26 I don't understand the relation betwenn the jtterness and MatrixGL_RH, Do uou think it is a difference between matrix and matrixGL_RH? that is, one with less jtterness than the other?
and it's not clear to me what do you mean with:
I tried to use every 5th, 10th, 20th and 50th dataset
Does we have only 16 elements in the matrix?

@Vasheer26
Copy link
Author

@kalwalt
Re: jitterness and MatrixGL_RH

  • I tested this by fixing a webcam and pointing it to the NFT image.
  • This is to eliminate any hand movements.
  • world = JSON.parse( msg.matrixGL_RH ); <-- is returned continuously
  • if the camera and image is stationary then you would expect the data in world object to be the same in every update.
  • if you output the data from each cycle, you will see indeed it does have 16 elements.
  • if you compare the data sets between the cycles, you will see that the numbers jump up and down.
    => To me, it shows that function that does the recognition and returning the matrixGL_RH not consistent with each pass.
    For this reason I tried test using every 5th, 10th data set ( Not 5th, 10th element in the matrix)

I hope my explanation that make sense. Please let me know if it doesn't.

Anyway, next I will need look deeper into artoolkitNFT.worker.js
=> markerResult = {type: "found", matrixGL_RH: JSON.stringify(ev.data.matrixGL_RH)

@Vasheer26
Copy link
Author

Also, is there anyway I can review this file build/artoolkitNFT_wasm.js with the formatting in place.
Currently, there's no tabbing or line breaks- very difficult to read :)

@Vasheer26
Copy link
Author

My next attempt was to smooth out the data sets by averaging out 5 dataset coming back from matrixGL_RH.

I did find that the jitterness was significantly reduced.
(Usable but not 100% stable)

If I continue this approach and take the averaging of 10 data sets, the jitterness decreased again but at the expense of slower mapping of the video over NFT image. For now averaging over 5 data sets is relatively a good number to hold at.

Ideally, I need to get a better understanding why the data coming back from matrixGL_RH varies wildly with each return cycle.

@kalwalt
Copy link

kalwalt commented May 7, 2020

Ideally, I need to get a better understanding why the data coming back from matrixGL_RH varies wildly with each return cycle.

Could be the cause is the fact that detection and tracking are not on a separate level as point out @hiukim #95 (comment) ?

@ThorstenBux
Copy link

ThorstenBux commented May 7, 2020 •
edited
Loading

Thanks for sharing your findings. As said I don’t think oneeurofilter for smoothing and elimination of off values would be a good approach. Kind regards Thorsten Bux

@Vasheer26
Hi sorry this was written wrong. I meant oneeurofilter is a good pick. It will eliminate these off values and also handle fast movements of the marker.

The average approach is good but doesn't eliminate values that are far off. OneEuroFilter would.

If you want to understand the values from the matrix you need to dive into the c code.

@Vasheer26
Copy link
Author

@ThorstenBux - cool thank you. Yes, unfortunately I did read you saying oneeurofilter was not good so I did not bother to look at that approach. At least now I will have the oneeurofilter to play with. Thank you for updating that.

@Vasheer26
Copy link
Author

Ideally, I need to get a better understanding why the data coming back from matrixGL_RH varies wildly with each return cycle.

Could be the cause is the fact that detection and tracking are not on a separate level as point out @hiukim #95 (comment) ?

I think no. According to Hiukim, he mentioned there are 2 phases, that being detection and tracking. The cycle that returns the matrixGL_RH, I believe lives with the detection phase.

@ThorstenBux
Copy link

ThorstenBux commented May 8, 2020 via email

@Vasheer26
Copy link
Author

@ThorstenBux
haha.. and it turns out I got my wording wrong also.. You are correct.. The matrix comes from the tracking... not in the detection phase.

@kalwalt kalwalt mentioned this issue May 15, 2020
Open
@evaristoc
Copy link

evaristoc commented May 16, 2020 •
edited
Loading

Hi @kalwalt @ThorstenBux @Vasheer26

Some observations?

  1. It seems you are reaching similar conclusions to the ones I have also reached when studying a script that does more or less the same thing as the artoolkit5 for NFT. I invite you to go through the original demo on which my project was based, or check my changes to the original one, with additional explicative notes. The library used for that was https://inspirit.github.io/jsfeat/, which I found well documented. Any other material regarding this kind of procedures could help. Eg. there are plenty of examples with opencv.

I like this line in the original demo...

  1. I am not an expert, but I am having the impression that there are many changes in this project that are following a naïve approach. Please don't get me wrong (especially you, @kalwalt ): I fully admire you all, as I am also coming from a similar route but you are working with C/C++ (Wauw!). However I personally think the simple trial-and-error route is not the most appropriate. Why?...

  2. As far as I understand, NFT procedures are mature and well established. I don't think the artoolkit5 NFT implementation is far from what is already known. My invitation is to further understand the procedure even before trying to change things. It is possible that there are things that this algorithm is unable to do as it is likely outdated and not maintained. You might be trying to reinvent the wheel.

  3. The people who originally worked on the artoolkit NFT algorithm didn't use a variable naming that helped to reveal what the algorithm is about. I think they kept it general in case of having to change some aspects of the procedure. I think if we can unveil the naming and relate it to the steps of the formal procedures for NFT would help substantially in understanding what can be done and what not.

The route that you follow after this is your choice though. This is a personal suggestion.

@evaristoc
Copy link

evaristoc commented May 16, 2020 •
edited
Loading

@Vasheer26 with my humble understanding of the implemented procedure, I think the matrixGL_RH you are working on is what it is known as the homographic matrix?

@evaristoc
Copy link

evaristoc commented May 16, 2020 •
edited
Loading

@kalwalt @ThorstenBux @Vasheer26

I will prepare a youtube video with some empirical results of implementing moving averages (what @Vasheer26 appears to implement here: #97 (comment), as well as a linear interpolation for alpha == .5, sample size = 2 ), Kalman filters and what I think is called stacking, by itsefl and combined, all on the javascript side. I made already a demo to show something about the AR.js NFT project to a prospective client. I will work the demo a bit further.

I haven't test the oneeuro filter.

My approach is likely NOT the right one but it might be good for some comparisons.

@Vasheer26
Copy link
Author

@evaristoc - thank you for letting me know the name of the matrix is homographic. This will at least give me a handle to grab onto when I am doing further research. I admit what I know and understand is very surface level and from that level is where I try to tackle what I can see.

While I have not yet had a chance to apply the oneeuro filter, I believe it is a bandage approach as it would be used to patch a problem and not fully understanding the original cause to prevent the root problem. A massive thank you for inviting me to go through the original demo and your changes with the additional notes. Greatly appreciate it.

@ThorstenBux
Copy link

Hi,
I highly appreciate that you want to look into the depths of artoolkit and find and fix the problem there. For what it is worth, my knowledge about the topic.
Nft tracking was implemented quite some time ago by PhD students. I assume thay had quite some knowledge about CV and the algorithms, it can well be that the algorithms improved since then.
Now, from my perspective a oneeurofilter approach would show quick, easy and fast results which would improve the project and the motivation. Low hanging fruit 😊.
And then I would look into the rabbit hole 🕳.
Now as said that is my view and you are free to choose the way you want to go. Simply thought I’d share.

@evaristoc
Copy link

evaristoc commented May 18, 2020 •
edited
Loading

@ThorstenBux

Now, from my perspective a oneeurofilter approach would show quick, easy and fast results which would improve the project and the motivation. Low hanging fruit .
And then I would look into the rabbit hole .
Now as said that is my view and you are free to choose the way you want to go. Simply thought I’d share.

Thanks! No, it is actually more your choice, I think. I was more like helping out 😄 . If we (your target group) are happy with the results, then why not 👍 !

@evaristoc
Copy link

evaristoc commented May 18, 2020 •
edited
Loading

@Vasheer26

@evaristoc - thank you for letting me know the name of the matrix is homographic. This will at least give me a handle to grab onto when I am doing further research. I admit what I know and understand is very surface level and from that level is where I try to tackle what I can see.

Yes. I understand your position. It has taken me some time to get the idea of how the whole stuff more or less works. As @ThorstenBux well mentioned, the project needs results soon and it is ok to try at least something.

not fully understanding the original cause to prevent the root problem.

I think the problem with the algorithm is that it might be matching many points that are false positives - they lay away from its corresponding position within the actual target image as seen in the camera. Those false positives, which are outliers, are randomly targeted per frame, so that is the variability.

That is enough to provide wrong measures for point positioning and orientation, which is based on the total points considered as "good" match. One parameter affected would be the homographic matrix, which I think uses all those points for its estimation.

The false positives problem is HARD to solve and it might be inherent to probably all algorithms for image detection.

Because it is hard to evaluate if a point is an outlier or not after detection, one way to solve the issue is post-processing. I believe there are procedures that are very good in matching correct points overall. There are one which I heard is very good but it is licensed.

While I have not yet had a chance to apply the oneeuro filter, I believe it is a bandage approach as it would be used to patch a problem

It is a variation, more advanced filter that uses low pass filter (https://en.wikipedia.org/wiki/Low-pass_filter) at several stages. It samples points at two moments of the signal. For what I can get from the script:

  1. It creates a weighting value alpha, which is based on the freq argument; assumes circular error (gaussian?).
  2. It samples two values for comparison: the current one and another happening at timespan before. Timespan is argument and fixed.
  3. It estimates a expected "distance" between those two values (dx) on frequency (eg. by frame).
  4. It then runs a low pass filter over that difference having also a dcutoff as argument: this part is smoothing the change in the range between two different positions along a fixed time.
  5. It uses that value to adjust the cutoff value eventually used to assign the final new entry point by calculating another low pass, this time over the points, not the range.

It seems powerful: it takes in consideration an adjustment of the estimation of the error change, likely the variance (in the filter that refers to the adjustment of the range).

For the values of the parameter to use, I would guess as first values to try (but NOT sure...):

  1. The freq value calculated by the algo appears to be close to number of frames, perhaps something no more than fps.
  2. If the jumping in values occurs substantially frame by frame (ie. the dx is large frame to frame), I think you want to sample very close values, so timespan should be small but NOT too small (otherwise there will be no sampling at all). I think it would be okey to keep it within a second but depends how far the error occurs.
  3. With a high dCutoff value, I think you are like less concerned by how large the error. However you might want a dCutoff relatively small if the error is high and frequent.
  4. The minCutoff depends on how far you are expected to control for a visible movement from A to B based on your reference system.

A massive thank you for inviting me to go through the original demo and your changes with the additional notes. Greatly appreciate it.

Sorry I can't help more. Please use it cautiously. I am also not a specialist.

@evaristoc
Copy link

evaristoc commented May 18, 2020 •
edited
Loading

@kalwalt maybe the message above I just sent to @Vasheer26 can work for you? I wrote to him my impressions about the oneEuroFilter. I am not absolutely sure my impressions are correct but well... it is something.

@albjeremias
Copy link

I think the problem with the algorithm is that it might be matching many points that are false positives - they lay away from its corresponding position within the actual target image as seen in the camera. Those false positives, which are outliers, are randomly targeted per frame, so that is the variability.

That is enough to provide wrong measures for point positioning and orientation, which is based on the total points considered as "good" match. One parameter affected would be the homographic matrix, which I think uses all those points for its estimation.

The false positives problem is HARD to solve and it might be inherent to probably all algorithms for image detection.

Hi, I've been looking into how to implement this algorithms. My first approach was using opencvjs with ORB or SIFT. I wonder which algorithm artoolkit5 uses, but the important is that on most of opencv implementations detection and tracking is different algorithms, and already someone mention that happening with artoolkit.. anyway in the most classical approaches i've seen tracking smoothing using a kalman filter, or optical flow estimation.. oneEuroFilter or LowPassFilter does not seem the most adequate at least in most approaches for a computer vision software.

@hiukim
Copy link

hiukim commented May 19, 2020

I think there are two separate algorithms for detection and tracking. The one behind detection is FREAK. I'm still digesting the tracking part.

@albjeremias
Copy link

I think there are two separate algorithms for detection and tracking. The one behind detection is FREAK. I'm still digesting the tracking part.

@hiukim can you please post some references on the code lines? id like to digg a bit into it.

@hiukim
Copy link

hiukim commented May 19, 2020

@albjeremias
For the detection part, you may start with these entry points:

To generate dataset: https://github.com/artoolkit/ARToolKit5/blob/5bf0b671ff16ead527b9b892e6aeb1a2771f97be/lib/SRC/KPM/kpmRefDataSet.cpp#L287

To do matching:
https://github.com/artoolkit/ARToolKit5/blob/5bf0b671ff16ead527b9b892e6aeb1a2771f97be/lib/SRC/KPM/kpmMatching.cpp#L383

These are some good entries point for the feature detection and matching algorithm. Before that there are still a few processing steps.

Also, these parts is only for the detection. Once a marker is detected, there is another algorithm for tracking.

@dandingol03
Copy link

Hi @Vasheer26, i will share here with you my work. Not on Jsartoolki5 but i'm working on my lighter version JsartoolkitNFT you can try here https://github.com/kalwalt/jsartoolkitNFT/tree/improving-lerp/examples i'm trying to use the one euro filter (see the oneEuroFilter.js) i think you can use the same code for jsartoolkit5. I'm not sure of my code, nor of my implementation. I hope that his will help you...

this link https://github.com/kalwalt/jsartoolkitNFT/tree/improving-lerp/examples is invalid now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@hiukim @kalwalt @evaristoc @dandingol03 @ThorstenBux @albjeremias @Vasheer26