Floating Point operations on iOS / Safari

[jamesmfriedman]

Your library is my last hope in solving my problems for this app.

Basically, I'm trying to do an extraction of a card using hough lines and finding the intersection of the points. A few things.

• I'm still very new to computer vision, so some of the more intensive topics are lost on me.
• The best I can deduce is that there is a floating point issue with WebGL on safari.
• You can see from my screenshots, chrome is "mostly" detecting the appropriate lines. I am able to filter the bad ones out to get to the result I need.
• Safari is all over the place and doesn't seem to work consistently.
• The same behavior can be seen on examples page if you open chrome vs safari. https://gammacv.com/examples/pc_lines

Good Example (Chrome), I'm able to extract the license using the pclines.

Bad Example (Safari). Same exact code, wildly different result. You'll notice the lines (in the top right image) don't seem to correspond to anything.

Any help you could possibly give would be amazing. I wasted several weeks trying to get openCV js working before finally giving up. Found your library after, was able to get to the same point I was at with openCV in half a day, and the performance is incredible.

[apilguk]

Thanks for the feedback
Yes you absolutely right the issue with a floating point on Safari and we will fix it ASAP

[jamesmfriedman]

Thank you so much for the quick response! A few more things that may or may not be related.

• the pcLines operation seems greedy as compared to something like houghLines from openCV. I’ve reduced the input image as much as possible but it still finds lines everywhere! The picture only contains the reduced set I’m using.
• the parameters for pcLines might account for this, but I don’t actually know what they do. I was able to get to this point by reverse engineering the example code for pcLines.
• the docs could benefit from some more robust written examples. For instance, I didn’t know you could chain operations in a pipeline until I found it in the example src code.
• a nice to have for my particular use case would be a 4 point affine transform operation. I was able to get something working with a code snippet I found on the web, but my linear algebra is terrible to say the least :). I divided the image in two triangles and implemented two separate 3 point transforms instead.

This library deserves some serious traction. I haven’t come across another library that’s as screaming fast as this, and trust me I’ve looked hard ;).

[apilguk]

• pcLines is pretty similar to houghLines in Open CV the difference paLines utilize parallel coordinate system is and the PC line is a greedy algorithm and it looking for N maximums in Hough Space.
• Good idea to add more practical examples, we also would like to improve documentation, feel free to contribute 🙂
• this approach could work and you also able to use builtin operation from gammacv

The reason we created this library is we weren't able to find an alternative, nice to see you it works for you

[rmhrisk]

I am curious can you tell us more about your project?

[jamesmfriedman]

For sure. Working on an app to help law enforcement officers document vehicle collisions. Currently, they either use paper and pencil or really terrible software. There is a lot of information they have to record, and the drivers licenses are the most time consuming part.

I’ve looked into many different ways to scan the drivers license but I have the following limitations.

• must be a web app
• must be done in device and not rely on the network.
• must work as a progressive web app on iOS and Android.

It’s actually been quite challenging, and of all of the methods I’ve tried, I’m placing my bets on a high quality still photo of the front and back of the license that is properly extracted and transformed so that I can then do feature extraction of the PDF417 barcode to accurately decode it. I can then compare that with an ocr result of the front for accuracy and probably throw in a bit of machine learning for good measure.

[rmhrisk]

Thanks for the description. We could provide a reusable component that does this (and more). Ping me at [email protected] or rmhrisk on Skype.

[apilguk]

@jamesmfriedman issue should be fixed in #26, please check it out using your code

[jamesmfriedman]

Hi! Finally had a chance to test this out.

This appears to fix the issue on Desktop Safari, but iOS still suffers from some strange issues that look like the whole coordinate plane is off. Thanks for the quick turnaround on the last fix!

UPDATE: A little more context, I'm running pcLines on this as my input image. I don't think I can feasibly clean it up any more than this :). It's almost to the point I could write a dumb algorithm to extract it.

[apilguk]

@WorldThirteen found a couple possible reasons which make line detection broken on mobile Safari and we will try to solve it ASAP

Look like you have a kind of morphologic filtering, do you use some post-processing after Canny Edges?

[jamesmfriedman]

I am :). Got to this result by playing around, but here is my pipeline. I think I know what close does (erodes black towards center), but either way it seems almost like magic that I get that clean of an image. The dilation was an attempt for pcLines to see the lines nice and clear although it doesn't seem to help much.

let pipeline = input;
      pipeline = gm.grayscale(pipeline);
      pipeline = gm.norm(pipeline, 'l2');
      pipeline = gm.gaussianBlur(pipeline, 3, 3);
      pipeline = gm.morphologyEx(pipeline, 'close', [51, 51]);
      pipeline = gm.sobelOperator(pipeline);
      pipeline = gm.cannyEdges(pipeline, 1, 1);
      pipeline = gm.dilate(pipeline, [5, 5]);
      pipeline = gm.downsample(pipeline, 2);

[WorldThirteen]

@jamesmfriedman, in the latest release (0.3.2) we enhanced the precision of floats, so now lines detection works fine on iOS, please check.

[jamesmfriedman]

Everyone, thank you for the prompt responses and fixes.

There are still minor abnormalities likely dealing with Floating point math on iOS, but i will emphasize MINOR. I'm able to hack around the small differences now and am officially unblocked! :)

Thanks you so so much for your help on this and the quick turn around. Really an amazing library, loving using it.

[rmhrisk]

Please file bug regarding abnormalities

[jamesmfriedman]

I can just reopen then.

Sorry I don't have more info on this, I can post a screenshot tomorrow morning. Basically, iOS finds most of the lines properly, and still occasionally sees a line that isn't there. I want to say that the issue is present on Desktop Safari as well. Standby...

[WorldThirteen]

@jamesmfriedman, please give more description, screenshots, environment details (device, OS and browser versions).

[jamesmfriedman]

Sorry for the late reply, I've currently abandoned trying to do automatic document extraction via pcLines. I know when I was doing this with HoughLines and openCV, there were minimum thresholds / distances I could set that would allow it to filter out a lot of the noise.

Given my example image above, I don't think I can get any clearer of an image, and still Chrome and Safari both detect things that are barely there.

Looking at the OpenCV api, it was a combination of rho and theta that would allow me to achieve this. I guess this bug is turning into more of a feature request?
https://docs.opencv.org/3.4/dd/d1a/group__imgproc__feature.html#ga46b4e588934f6c8dfd509cc6e0e4545a