Complete 3D Scene Scanning with D455
Hello Team,
My research group has a D455 and wants to create the complete 3D scans of indoor scenes like those seen in the S3DIS or ScanNet datasets. We do not have a tracking module like T265. We have looked around and only found proprietary solutions like Dot3D, RecFusion, etc. The scanning does not necessarily need to take place in real time, but we would like the scans of the rooms to be as complete as possible (similar to S3DIS or ScanNet).
We are wondering:
1) if this feature is implemented in the open source RealSense API, or can be easily implemented using existing code the API
2) if there are any open source implementations of this feature
3) since there exists proprietary solutions, as well as 3D scanning apps for the smartphones, what are the modern approaches to this problem (say, if we were to contribute this feature to the RealSense API). Out of curiosity, do they require directly locating and tracking the camera position?
4) I know the D455 also comes with an accelerometer and gyroscope. Was this one of the uses for these modules? If so, are there any localization features built into the pre-existing RealSense API.
-
Hi Jkchen2. Thanks very much for your questions.
1 and 2. The open-source RealSense SDK example program rs-kinfu (KinectFusion) that is based on C++ and OpenCV may meet your needs. A single camera can be moved around a scene, progressively building up a pointcloud scan of the scene by fusing frames. Once you are satisfied with the amount of detail on the scan, you can export it as a .ply format pointcloud file that can then be imported into 3D modelling software for conversion to a solid model.
https://github.com/IntelRealSense/librealsense/tree/master/wrappers/opencv/kinfu
An example of an open-source 3D software that can be used with RealSense .ply files is MeshLab.
https://www.andreasjakl.com/capturing-3d-point-cloud-intel-realsense-converting-mesh-meshlab/
3. In my experience, the number of people who create their own approach of recording a ply and then converting it to a solid mesh with open-source solutions is greater than the number who use commercial software to do so. Neither approach is incorrect though and simply depends on your preferences and project requirements / budget.
Tracking the camera position is usually not required for scene scanning applications.
4. A type of large scale scene modelling that can benefit from an IMU (gyro and accelerometer) is SLAM. A moving camera, typically attached to a mobile robot can navigate a patrol route through rooms, recording a point cloud map as it travels. When you are satisfied with the amount of detail on the scan, the recording can be stopped and saved.
Intel provide a SLAM map recording tutorial for ROS that utilizes the IMU. It was designed for the D435i camera model but should work with D455 too.
https://github.com/IntelRealSense/realsense-ros/wiki/SLAM-with-D435i
Whilst it does not feature relocalization, a RealSense user created their own version of the guide that adds relocalization and relative camera position tracking and plotting.
https://wshinkansan.github.io/2019-UGRP-DPoom/SLAM.html
Other open source SLAM solutions are compatible with RealSense cameras. A couple of examples are ORB-SLAM (such as ORB-SLAM2 and ORB-SLAM3) and Kimera.
Please sign in to leave a comment.
Comments
1 comment