Guidance on merging point clouds from two RealSense D455f cameras for telepresence
Dear RealSense Technical Support Team,
I am currently working on a telepresence application using two RealSense D455f cameras.
My goal is to capture depth data from both cameras, generate point clouds, and merge the two point clouds into a single unified 3D representation in real time.
Could you please advise on the recommended approach for this setup?
In particular, I would appreciate guidance on the following:
- The recommended method for extrinsic calibration and spatial registration between two D455f cameras.
- Whether hardware synchronization is recommended or required for reliable real-time point cloud fusion.
- Whether there are any concerns regarding interference or depth quality when operating two D455f cameras simultaneously.
- Recommended RealSense SDK 2.0 tools, examples, or reference implementations for merging point clouds from multiple cameras.
- Best practices for minimizing latency and improving stability in a real-time telepresence application.
If there are any relevant documentation pages, sample code, or supported workflows for multi-camera point cloud fusion, I would be grateful if you could share them.
Thank you for your support.
Best regards,
Moamen Ali
-
Hi Moamen Ali Thanks very much for your questions.
1. There is more than one approach available for calibrating cameras together. It could be done with program code, or with ROS.
2. Hardware sync is not required for creating a combined pointcloud.
3. Multiple RealSense 400 Series cameras do not interfere with each other.
4. RealSense provide an official example in the C++ programming language at the link below for using two devices to "stitch" their viewpoints together in order to generate a combined pointcloud. The method uses a tool called MATLAB to calibrate the cameras together.
Alternatively, another RealSense example, that is in the Python language, is box_dimensioner_multicam, which can calibrate multiple cameras together by calculating the spatial relationship between them and creating a point cloud from the depth frames of all the cameras. It does not actually visibly output that cloud though, so you may have to modify the program if your application requires a visible on-screen cloud.
Another RealSense example, in C++ language, is rs-kinfu. Instead of using two fixed position cameras though, it uses a single camera to build up a pointcloud in real-time by moving the camera around.
https://github.com/realsenseai/librealsense/tree/master/wrappers/opencv/kinfu
-
5. In regard to minimizing latency and optimizing stability, you should use a computer that can handle two RealSense cameras simultaneously. Most PCs can cope with this, but low-end development boards (single-board computing devices) may struggle. If you plan to use a development board, an Nvidia Jetson board such as Jetson Orin Nano is recommendable.
Depth stability can be affected by a range of environmental factors. Try to avoid pointing the camera at dark grey / black colored surfaces which may not be picked up by the camera and also avoid reflective surfaces. Also do not use the camera in an office room with fluorescent ceiling strip lights if possible as it can result in noise on the depth image.
-
Dear MartyX Grover,
Thank you very much for your detailed response and for sharing the RealSense examples.
I would like to clarify that my current implementation is being developed in Unity. I am using two Intel RealSense D455f cameras, and my goal is to generate a point cloud from each camera and merge both point clouds into a single unified 3D representation inside Unity in real time.
Could you please advise whether the calibration matrix generated using the
rs-pointcloud-stitchingworkflow can be used directly in Unity to transform the point cloud from the second camera into the coordinate system of the first camera?Also, if there is a recommended workflow for applying the extrinsic calibration result inside Unity, I would really appreciate your guidance. For example, how to correctly convert the rotation and translation values into a Unity
Matrix4x4, and whether any coordinate system conversion is needed between RealSense and Unity.Any advice, sample code, or recommended approach for implementing this multi-camera point cloud merge specifically in Unity would be very helpful.
Thank you again for your time and support.
-
The Unity references at the links below may be helpful.
https://github.com/realsenseai/librealsense/issues/6147
-
Hi Moamen Ali Do you require further assistance with this case, please? Thanks!
Please sign in to leave a comment.
Comments
5 comments