Joint Learning of Instance and Semantic Segmentation for Robotic Pick-and-Place with Heavy Occlusions in Clutter (ICRA2019)
Paper | Video | Poster | Supplimentary
Instance Segmentation of Visible and Occluded Regions for Finding and Picking Target from a Pile of Objects (IROS2018)
Paper | Video | Slides | Poster
# create catkin workspace
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws
git clone https://github.com/start-jsk/jsk_apc.git
cd src
rosdep install --from-path . -i -y -r
sudo -H pip install cupy-cuda101 # CUDA10.1, cupy-cuda92 for 9.2
cd ~/catkin_ws
source /opt/ros/kinetic/setup.zsh
catkin build instance_occlsegm --no-deps# create catkin workspace
mkdir -p ~/catkin_ws/src
cd ~/catkin_ws
git clone https://github.com/start-jsk/jsk_apc.git
cd src
wstool init
cat jsk_apc/.travis.rosinstall >> .rosinstall
cat jsk_apc/.travis.rosinstall.kinetic >> .rosinstall
wstool update -j -1
rosdep install --from-path . -i -y -r
sudo -H pip install cupy-cuda101 # CUDA10.1, cupy-cuda92 for 9.2
cd ~/catkin_ws
source /opt/ros/kinetic/setup.zsh
catkin build instance_occlsegm# Dataset for instance (roi-level) occlusion segmentation
cd examples/instance_occlsegm/instance_occlusion_segmentation
./view_dataset_occlusion.py
# Dataset for semantic (image-level) occlusion segmentation
cd examples/instance_occlsegm/occlusion_segmentation
./view_dataset.py
# Dataset for joint learning
cd examples/instance_occlsegm/panoptic_occlusion_segmentation
./view_dataset.py
# Training script for instance-only vs. joint-learning
cd examples/instance_occlsegm/panoptic_occlusion_segmentation
./train.py --gpu 0 --notrain pix # instnce-only
./train.py --gpu 0 # joint-learning
# multi-gpu training for faster training with larger dataset
# mpirun -n 4 ./train.py --multinode --notrain pix --dataset occlusion+synthetic
# mpirun -n 4 ./train.py --multinode --pix-loss-scale 0.25 --dataset occlusion+synthetic
./demo.py logs/<log_dir>
Comparison: instance-only vs. joint-learning (included in the supplimentary)
| Backbone | Model | Dataset | Lambda | mPQ |
|---|---|---|---|---|
| ResNet50 | instance-only | occlusion | - | 41.0 |
| joint-learning | 0.25 | 42.2 | ||
| instance-only | occlusion+synthetic | - | 47.3 | |
| joint-learning | 0.25 | 48.9 | ||
| ResNet101 | instance-only | occlusion | - | 43.5 |
| joint-learning | 0.25 | 44.5 | ||
| instance-only | occlusion+synthetic | - | 50.0 | |
| joint-learning | 0.25 | 50.9 |
The real-world dataset annotated by human can be downloaded from following links:
# Find occluded target and plan the next target
roslaunch instance_occlsegm sample_panoptic_segmentation.launch
Figure: Picking Order Planning for the White Binder (From left: RGB, visible regions, occluded regions, next target)
# Pick-and-Place demo
roslaunch instance_occlsegm baxter.launch
roslaunch instance_occlsegm setup.launch
# for target picking, change the ~context and ~target params in setup.lauch
roscd instance_occlsegm/euslisp
> (upick-upick) # random picking
> (pick-pick) # target picking
@inproceedings{Wada:etal:ICRA2019,
title={Joint Learning of Instance and Semantic Segmentation for Robotic Pick-and-Place with Heavy Occlusions in Clutter},
author={Kentaro Wada, Kei Okada, Masayuki Inaba},
booktitle={{Proceedings of the IEEE International Conference on Robotics and Automation (ICRA)}},
year={2019},
}
@inproceedings{Wada:etal:IROS2018,
title={Instance Segmentation of Visible and Occluded Regions for Finding and Picking Target from a Pile of Objects},
author={Kentaro Wada, Shingo Kitagawa, Kei Okada, Masayuki Inaba},
booktitle={{Proceedings of the IEEE/RSJ International Conference on Robotics and Intelligent Systems (IROS)}},
year={2018},
}
make install # Python3
# make install2 # Python2source .anaconda/bin/activate
python -c 'import instance_occlsegm_lib'make lint
make test