Occlusion with multiple objects of the same type?

[pgaston1]

I currently have linemod working well detecting one of my custom objects. Thank you!

I now would like to extend this to support multiple objects of the same type using the occlusion model. However, your approach seems hard-coded to one model type per image. i.e., one 'drill' per image. This seems to manifest itself in creating the mask_all/merged_masks images - I need to identify each specific instance aligning the masked area to a specific object.

How would you recommend I implement multiple objects per type? (the default idea is to create new 'pseudo types', e.g., drill1, drill2, drill3 - but that fails as the objects are identical and would conflict.)

Here's an example of a single pallet being found in a simulated warehouse... And following that is an example of what I would like to detect, i.e., detect the pose of each pallet in a stack of 5 pallets.

and

[alishan2040]

Dear @pgaston1, can you please tell how did you generated dataset for your custom object? I'm also working on some similar problem but facing issues in dataset generation in required linemod format. I'm particularly interested in how to calculate object parameters such as diameter, min_x, min_y, min_z, size_x, size_y, size_z from .ply file for example. I've tried MeshLab/Blender to set object origin to (0, 0, 0) to get these params but they seem incorrect. I would like to hear your approach on this.

Any sort of help is appreciated.
Thanks

[pgaston1]

Let's see - short version - as mentioned this would make a good Medium article. I start with a simulated image, provided by Isaac Sim (warehouse) and I found a CAD pallet on the web in obj format. Using Meshlab I then created a ply file of the pallet with the conversions: * rotated appropriately, noting the Y/Z change between Isaac Sim and the model * centered on 0,0,0 * converted to MM * saved as ply file - non-Binary, though not sure it matters. These are all easily done in Meshlab. On the Isaac Sim side I had to modify mGenerator.py to spit out the data in a reasonable format. e.g., remove the camera rotation, find the right objects (pallet), ignore some fatal bugs, and more. I then wrote some Python code which converted the output from there over to what EfficientPose needs. Clearly one could use to support other formats as well. I've included the code below to answer your specific question (basically turn bbox3d into the needed numbers.) Good luck! This basically took me most of the summer to get working! I'm now playing tricks to get occlusion working when there are multiple objects of the same type - not directly supported by this model - but that's a story for another time! p # Model info - for our only model at this point def dist(pta,ptb): xs = (pta[0]-ptb[0])*(pta[0]-ptb[0]) ys = (pta[1]-ptb[1])*(pta[1]-ptb[1]) zs = (pta[2]-ptb[2])*(pta[2]-ptb[2]) dis = math.sqrt(xs+ys+zs) return dis # need to get base size of object - this comes from bbox3 # do a sample read bbox = np.zeros([9,3]) def getBaseInfo(): global uniqueId0,width,height,bbox fpSampleBBox3 = os.path.join(fpSrcBBox3D,"0.npy") data3d = np.load(fpSampleBBox3,allow_pickle=True) np1 = data3d[0] uniqueId0 = np1[0] # model's unique id # now width, height fpRGB = os.path.join(fpSrcRGB,"0.png") assert(os.path.isfile(fpRGB)) rgb = cv2.imread(fpRGB) assert(rgb is not None) height = rgb.shape[0] width = rgb.shape[1] #print("height/width",height,width) # now, builds models_info.yml # values at 6-11 are base object it seems min_x = np1[6] min_y = np1[7] min_z = np1[8] max_x = np1[9] max_y = np1[10] max_z = np1[11] ## Swap Y and Z min_x = np1[6] min_y = np1[8] min_z = np1[7] max_x = np1[9] max_y = np1[11] max_z = np1[10] if toMM: # to MM min_x *= 10.0 min_y *= 10.0 min_z *= 10.0 max_x *= 10.0 max_y *= 10.0 max_z *= 10.0 size_x = max_x-min_x size_y = max_y-min_y size_z = max_z-min_z # build bounding box bbox = np.zeros([8,3]) #untere ebende bbox[0, :] = np.array([min_x, min_y, min_z]) bbox[1, :] = np.array([min_x + size_x, min_y, min_z]) bbox[2, :] = np.array([min_x + size_x, min_y + size_y, min_z]) bbox[3, :] = np.array([min_x, min_y + size_y, min_z]) #obere ebene bbox[4, :] = np.array([min_x, min_y, min_z + size_z]) bbox[5, :] = np.array([min_x + size_x, min_y, min_z + size_z]) bbox[6, :] = np.array([min_x + size_x, min_y + size_y, min_z + size_z]) bbox[7, :] = np.array([min_x, min_y + size_y, min_z + size_z]) # find diameter - largest distance between points dis = np.zeros([28]) k = 0 for i in range(7): for j in range(i+1,8): dis[k] = dist(bbox[i],bbox[j]) #print("compare",i,j,k,dis[k]) k = k+1 diameter = np.amax(dis) dict_file = {1: { 'diameter': float(diameter), 'min_x': float(min_x), 'min_y': float(min_y), 'min_z': float(min_z), 'size_x': float(size_x), 'size_y': float(size_y), 'size_z': float(size_z) }} print("dict_file",dict_file) with open(fpModelInfo, 'w') as file: documents = yaml.dump(dict_file, file)

…

________________________________ From: Shan Ali ***@***.***> Sent: Saturday, October 2, 2021 5:31 AM To: ybkscht/EfficientPose ***@***.***> Cc: Peter Gaston ***@***.***>; Mention ***@***.***> Subject: Re: [ybkscht/EfficientPose] Occlusion with multiple objects of the same type? (#34) CAUTION - EXTERNAL EMAIL: Do not open attachments or links unless you recognize the sender and the content is safe. Dear @pgaston1 [github.com]<https://urldefense.com/v3/__https://github.com/pgaston1__;!!GenOTVeOfQ!CcDuRXHhhtyyZJ1N20Y1c6MX0B0LZ5xrfsEFm2DDMjK3zCQ19CMz6DOuSTwvSpAde5xP$>, can you please tell how did you generated dataset for your custom object? I'm also working on some similar problem but facing issues in dataset generation in required linemod format. I'm particularly interested in how to calculate object parameters such as diameter, min_x, min_y, min_z, size_x, size_y, size_z from .ply file for example. I've tried MeshLab/Blender to set object origin to (0, 0, 0) to get these params but they seem incorrect. I would like to hear your approach on this. Any sort of help is appreciated. Thanks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [github.com]<https://urldefense.com/v3/__https://github.com/ybkscht/EfficientPose/issues/34*issuecomment-932722228__;Iw!!GenOTVeOfQ!CcDuRXHhhtyyZJ1N20Y1c6MX0B0LZ5xrfsEFm2DDMjK3zCQ19CMz6DOuSTwvSubMhfEW$>, or unsubscribe [github.com]<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AR7FMT4JCWTCL2T4I2HD2QLUE3GQ5ANCNFSM5E3HKFEQ__;!!GenOTVeOfQ!CcDuRXHhhtyyZJ1N20Y1c6MX0B0LZ5xrfsEFm2DDMjK3zCQ19CMz6DOuSTwvSkreVJWY$>. Triage notifications on the go with GitHub Mobile for iOS [apps.apple.com]<https://urldefense.com/v3/__https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!GenOTVeOfQ!CcDuRXHhhtyyZJ1N20Y1c6MX0B0LZ5xrfsEFm2DDMjK3zCQ19CMz6DOuSTwvSgEOf20m$> or Android [play.google.com]<https://urldefense.com/v3/__https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!GenOTVeOfQ!CcDuRXHhhtyyZJ1N20Y1c6MX0B0LZ5xrfsEFm2DDMjK3zCQ19CMz6DOuSTwvSsNE5L7n$>.

[alishan2040]

Hi @pgaston1 Thanks for sharing the code and explanation! very helpful.
Can you please tell what should be the content of 0.npy file in getBaseInfo() method. How did you created this file?
Generally we do not have such files in efficientpose format?
Thanks again!

[pgaston1]

That file is generated by mGenerator.py. The screenshot what folders it creates. There are multiple 0.npy files - probably one under each folder - though rgb would have a 0.png. [cid:75fe7c84-0158-44d0-ae3d-7cb1934cb48b]

…

________________________________ From: Shan Ali ***@***.***> Sent: Tuesday, October 5, 2021 1:16 AM To: ybkscht/EfficientPose ***@***.***> Cc: Peter Gaston ***@***.***>; Mention ***@***.***> Subject: Re: [ybkscht/EfficientPose] Occlusion with multiple objects of the same type? (#34) CAUTION - EXTERNAL EMAIL: Do not open attachments or links unless you recognize the sender and the content is safe. Hi @pgaston1 [github.com]<https://urldefense.com/v3/__https://github.com/pgaston1__;!!GenOTVeOfQ!ED41W6cGUxyJjjYX9-ledpET_t_71CPCkGwyqWYIMv3BZxfruIaeb41IRbQzU33oBUn3$> Thanks for sharing the code and explanation! this is really helpful. Can you please tell what should be the content of 0.npy file in getBaseInfo() method. How did you created this file? Generally we do not have such files in efficientpose format? Thanks again! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub [github.com]<https://urldefense.com/v3/__https://github.com/ybkscht/EfficientPose/issues/34*issuecomment-934072250__;Iw!!GenOTVeOfQ!ED41W6cGUxyJjjYX9-ledpET_t_71CPCkGwyqWYIMv3BZxfruIaeb41IRbQzUwvhfL8I$>, or unsubscribe [github.com]<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AR7FMT2JJHE3FI5UQJQRVA3UFKCZRANCNFSM5E3HKFEQ__;!!GenOTVeOfQ!ED41W6cGUxyJjjYX9-ledpET_t_71CPCkGwyqWYIMv3BZxfruIaeb41IRbQzU6O9jg8N$>. Triage notifications on the go with GitHub Mobile for iOS [apps.apple.com]<https://urldefense.com/v3/__https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675__;!!GenOTVeOfQ!ED41W6cGUxyJjjYX9-ledpET_t_71CPCkGwyqWYIMv3BZxfruIaeb41IRbQzUzV5Myvr$> or Android [play.google.com]<https://urldefense.com/v3/__https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign*3Dnotification-email*26utm_medium*3Demail*26utm_source*3Dgithub__;JSUlJSU!!GenOTVeOfQ!ED41W6cGUxyJjjYX9-ledpET_t_71CPCkGwyqWYIMv3BZxfruIaeb41IRbQzU1bkdgXC$>.

[ybkscht]

Hi @pgaston1,

very nice results so far, thanks for sharing them!

You are right that the currently provided Linemod and Occlusion generators do not support multiple instances of the object category.
The simple reason for this is because these datasets do not contain multiple instances per image.
But nevertheless, EfficientPose itself is fully capable of handling multiple instances of the same object category.
The problem is just how the dataset is stored and the generator is implemented.

I personally would recommend the following solution:

• Creating an instance segmentation mask in which each object instance got an unique value (possibly also RGB)
• Add this unique id to the corresponding object in gt.yml so that every annotated object got it's own mask value.
• Add an additional mask_values array to the annotations dictionary (see the convert_gt function in generators/occlusion.py) and fill in the unique mask values for every object when loading the dataset.
• Instead of using the mask values from the hardcoded name_to_mask_value dictionary in the generator, change these code lines to use the mask values of the annotations dict. Also check generators/common.py.

Then you should be fine with multiple object categories as well as multiple instances per object category.

Sincerely,
Yannick

[nikonnext]

Hi @pgaston1, i'm wondering how many images you had to train on one object and if you have any overtraining. The fact is that I am training on 10k images with a heavily loaded scene, and the images are synthetic. But it seems that this is not enough for good results.

Did you manage to implement the MiSo or MiMo task for this model? I have just started to deal with a similar task, can you give me hints or instructions?

[satpalsr]

Hey @pgaston1, how did it turn out? Were you able to make it identify multiple instances of the same object?

[pgaston1]

Have to admit, I gave up on that effort. (Company wanted to use LIDAR - ended up with a patent on that though...) Currently working on panoptic segmentation - which would identify the broad outlines of an object. Actually, planning to re-engage in a month or so now on this. I can't believe it would've been too hard though. Just follow through with the idea presented. btw, I've been coached that using a CAD model is probably overkill for my needs, i.e., pallets - where a simple cuboid shape should work. (And be more easily customizable to variants, e.g., length/width/height.)

…

________________________________ From: Satpal Singh Rathore ***@***.***> Sent: Monday, July 18, 2022 12:46 PM To: ybkscht/EfficientPose ***@***.***> Cc: Peter Gaston ***@***.***>; Mention ***@***.***> Subject: Re: [ybkscht/EfficientPose] Occlusion with multiple objects of the same type? (#34) CAUTION - EXTERNAL EMAIL: Do not open attachments or links unless you recognize the sender and the content is safe. Hey @pgaston1 [github.com]<https://urldefense.com/v3/__https://github.com/pgaston1__;!!GenOTVeOfQ!XnG8k_E8ZX61GKqcRZ-VGxvw9iyE0Mw5eIhH92z7niBFECsqu0NHIw3Opgf-u-34DgjD0Z9kHh7Tsyurb-ThezwYtcWmJQ$>, how did it turn out? Were you able to make it identify multiple instances of the same object? — Reply to this email directly, view it on GitHub [github.com]<https://urldefense.com/v3/__https://github.com/ybkscht/EfficientPose/issues/34*issuecomment-1187733516__;Iw!!GenOTVeOfQ!XnG8k_E8ZX61GKqcRZ-VGxvw9iyE0Mw5eIhH92z7niBFECsqu0NHIw3Opgf-u-34DgjD0Z9kHh7Tsyurb-ThezzLLva9_Q$>, or unsubscribe [github.com]<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AR7FMT2TJU4IOYEIUX5F3KDVUWC7HANCNFSM5E3HKFEQ__;!!GenOTVeOfQ!XnG8k_E8ZX61GKqcRZ-VGxvw9iyE0Mw5eIhH92z7niBFECsqu0NHIw3Opgf-u-34DgjD0Z9kHh7Tsyurb-ThezybXV2u8Q$>. You are receiving this because you were mentioned.Message ID: ***@***.***> Please note that this message may contain confidential information. If you have received this message by mistake, please inform the sender of the mistake, then delete the message from your system without making, distributing or retaining any copies of it. Although we believe that the message and any attachments are free from viruses and other errors that might affect the computer or IT system where it is received and read, the recipient opens the message at his or her own risk. We assume no responsibility for any loss or damage arising from the receipt or use of this message. When we process personal data relating to physical persons, such processing will meet the requirements of applicable data protection legislation and will be in accordance with our Privacy Policy https://www.agmobilerobots.com/privacy-policy/

[satpalsr]

Thanks @pgaston1 , I'll try it out.

[monajalal]

@alishan2040 were you able to figure how to calculate min_x, min_y, min_z, size_x, size_y, size_z for a given object for

For example,
"1": {"diameter": 102.099, "min_x": -37.9343, "min_y": -38.7996, "min_z": -45.8845, "size_x": 75.8686, "size_y": 77.5992, "size_z": 91.769},
is object 1 in linemod dataset in (base) mona@ada:/data2/data/BOP$ vi ./lm/models/models_info.json

I have some guesses about min_x/y/z and I assume it is the lowest corner on 3D bbox but not sure how to calculate.

What about size_x? Is there an easy way for these calculations or an automated way?

[monajalal]

I got it figured here is my answer https://gist.github.com/monajalal/ca0eb02bae787bc556a2b17656c7e58e