Align to color stream using D405 using Java on Android
Hi,
I use RealSense Java API to align the depth to color. Both the depth and color streams are 640 x 480. The FOV of depth stream is 79.79 x 64.18 and the FOV of the color stream is 73.1 x 58.2.
I generate the point cloud with the following code snippet:
mFilters.add(new Align(StreamType.COLOR));
mFilters.add(new Pointcloud(StreamType.COLOR));
Then call the following method provided by SDK example:
private FrameSet applyFilters(FrameSet frameSet, List<Filter> filters)
Finally, I visualize both color video and the point cloud side by side on the screen. However, the object represented by the point cloud looks bigger than that in the video and it looks the FOV of the point cloud is smaller. I use the intrinsic of the color stream to visualize the point cloud.
Here I have two questions:
1) why object represented by the point cloud looks bigger than that in the video?
2) I check the extrinsic from the infrared1 to the color sensor and found the extrinsic transform is very close to an Identity matrix. As I know there is no a physical RGB sensor right?
-
Hi Yixun Liu Thanks very much for your questions.
1. When depth to color alignment is performed, the depth field of view resizes to match the color field of view size. If the color FOV is smaller than the depth FOV then the edges of the aligned image will be cut off. So there may be some resizing of the display happening as a consequence but it would not mean that the aligned image is inaccurate.
2. You are correct, the D405 does not have an RGB sensor, It instead obtains an RGB image from the depth sensor. This image is passed through an Image Signal Processor (ISP) chip in the camera to generate the final RGB image. Although the D405's depth and RGB share the same viewpoint, this RGB image tends to not be perfectly aligned with the depth image.
A better-aligned image can be obtained by instead using the 'RGB from left infrared' mode of the D405. To do so, you define an Infrared 1 stream instead of an RGB stream and set the stream's format to RGB8 or BGR8 instead of the default Y8 format of the IR stream. This alternate RGB from IR mode will be perfectly aligned with depth.
-
On Android with Java the following code generate exception: Failed to start pipeline. Couldn't resolve requests.
config.enableStream(StreamType.INFRARED, 1, 640, 480, StreamFormat.RGB8, 30);
try (PipelineProfile pp = mPipe.start(config)) {} catch (Exception e) {
Log.d(TAG, "Failed to start pipeline. " + e.getMessage());
}
Please sign in to leave a comment.
Comments
6 comments