Point-set registration

While MPL has put strong emphasis on image-based registration techniques, we have also investigated point set registration methods (i.e. 3D-3D registration) in the context of our work on RGB-D camera tracking solutions. While it is of course possible to extract descriptive information about the points from the registered images, we have approached the 3D-3D point-set registration in its most general and fundamentally important form, which is correspondence-less:

• Point-set registration with semantic labels:

  
  

  L. Hu, J. Wei, Z. Ouyang, and L. Kneip. Point Set Registration With Semantic Region Association Using Cascaded Expectation Maximization. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2021

  
  
  

  The work considers the specific case in which the point sets are segmented into semantically annotated parts. Such information may for example come from object detection or instance-level semantic segmentation in the registered RGB image. Prior methods incorporate the additional information to restrict or re-weight the point-pair associations occurring throughout the registration process. The present method introduces a novel hierarchical association framework for a simultaneous inference of semantic region association likelihoods. The formulation is elegantly solved using cascaded expectation-maximization, and demonstrates a substantial improvement over existing alternatives on open RGBD datasets.

  
  
  

• Point-set registration for symmetric objects:

  
  

  L. Hu and L. Kneip. Globally optimal point set registration by joint symmetry plane fitting. Journal of Mathematical Imaging and Vision (JMIV), 63:689–707, February 2021. Open-access [pdf]

  
  
  

  In another work we propose a solution to the challenging problem of registering two partial point sets of the same object with very limited overlap. We leverage the fact that most objects found in man-made environments contain a plane of symmetry. By reflecting the points of each set with respect to the plane of symmetry, we can largely increase the overlap between the sets and therefore boost the registration process. However, prior knowledge about the plane of symmetry is generally unavailable or at least very hard to find, especially with limited partial views. Finding this plane could strongly benefit from a prior alignment of the partial point sets. We solve this chicken-and-egg problem by jointly optimizing the relative pose and symmetry plane parameters. We present a globally optimal solver by employing the branch-and-bound (BnB) paradigm, and thereby demonstrate that joint symmetry plane fitting leads to a great improvement over the current state-of-the-art in globally optimal point set registration for common objects. As demonstrated in the figure, the method can be used to align minimal overlap scans of even just approximately symmetric objects. It can furthermore be used to perform scene completion if the same object is present multiple times in the same scan.

  
  

• Very fast point-set registration in Manhattan worlds:

  
  

  Y Zhou, L Kneip, C Rodriguez, and H Li. Divide and conquer: Efficient density-based tracking of 3d sensors in manhattan worlds. In Proceedings of the Asian Conference on Computer Vision (ACCV), Taipei, Taiwan, November 2016b. Oral presentation [pdf] [code]

  
  

  This work aims at a highly efficient solution to the point set registration problem in the particular case where the point sets have been collected in Manhattan worlds. Manhattan worlds are characterised by the fact that we may have a primarily piece-wise planar structure where the various plane pieces have a normal vector that is aligned with one of three primary orthogonal directions. Our efficient idea for point set registration is then given by a two step approach in which we first identify rotation and then translation. In order to identify the rotation, we extract normal vectors for each point in the point cloud and try to fit the Manhattan reference frame. This procedure even gives us absolute orientation information for each point set. We then rotate the point sets into the Manhattan frame and project them onto the basis axes. This gives characteristic density signals that can be aligned individually for each translation dimension. The approach is used for RGB-D tracking in Manhattan worlds and achieves frame-rates over 100Hz on regular CPUs.

  
  
  

• Very fast point-set registration in piece-wise planar worlds:

  
  

  Y Zhou, L Kneip, and H Li. Real-Time Rotation Estimation for Dense Depth Sensors in Piece-wise Planar Environments. In Proceedings of the IEEE/RSJ Conference on Intelligent Robots and Systems (IROS), Deajeon, Korea, October 2016 [pdf] [video]

  
  

  The final work in this list simply extends the previous approach from our Manhattan world assumption-based point-set registration to more general piece-wise planar environments.