Align color and depth images
Hi, I'm using D435 and python wrapper.
I want to align depth to color image in 60fps, but when I use the align function(5th and 8th line), the fps drops to about 20, if I don't align them, the fps can reach 60.
- pipeline = rs.pipeline()
- config = rs.config()
- config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 60)
- config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 60)
- align = rs.align(rs.stream.color)
- profile = pipeline.start(config)
- frames = pipeline.wait_for_frames()
- frames = align.process(frames)
- color_frame = frames.get_color_frame()
- depth_frame = frames.get_depth_frame()
- color_image = color_frame.get_data()
- depth_image = depth_frame.get_data()
To solve this problem, I want to save color and depth image to .png separately, and align them afterwards, rather than align depth and color when they are in a frame set.
Is there any solution to my problem?
-
The best way to do an alignment of depth and color data is to record the streams into a type of file called a 'bag'. Once you have a bag file, you could use Intel's Python tutorial for aligning depth and color from a recorded bag file.
https://github.com/IntelRealSense/librealsense/blob/jupyter/notebooks/distance_to_object.ipynb
If you wanted to go the less efficient route of recording color and depth to an image file, the color should be a PNG but the depth should ideally be an image format that can store depth data, such as a .raw file. This is because saving depth to a PNG causes depth information to be lost.
-
Hello. MartyG,
Thank you for your reply,
but the size of bag file is too big, so it's not in my consideration.
opencv can save depth image into PNG now, and use
cv2.imread('xxx.png', cv2.IMREAD_UNCHANGED)
can get the original 16-bit depth image array.
Is there any api that can align two image?
-
A RealSense user posted an align script for Python a couple of weeks ago. Their approach was to save color as a PNG and depth as an array of scaled matrices saved as an .npy file.
https://github.com/IntelRealSense/librealsense/issues/4934#issuecomment-537705225
If you are an OpenCV user, you may also be interested in this OpenCV align program.
https://github.com/UnaNancyOwen/RealSense2Sample/tree/master/sample/Align
-
Thanks for your suggestion of the format to save depth data.
I've read the OpenCV align program, but it still seems to align color and depth when they're in a frame set, and then gets depth and color image from the frame set.
My target is to align color and depth images, in other words, to align depth.npy to color.png.
-
Python language in general, and npy in particular, is outside of my knowledge. The RealSense GitHub may have somebody who can provide a good answer, whether an Intel RealSense team member or another RealSense Python user.
https://github.com/IntelRealSense/librealsense/issues
If you are seeking to align a depth image file with a color image file, I am guessing that you are trying to align an individual frame and not a long stream, yes? If that is the case, an SDK instruction called Keep() might help with your FPS problems. Instead of doing processing on the frames as they come in, it waits until the stream is ended and then applies processing on the frames all at the same time and saves the data.
Because the frames are stored in memory instead of written to disk, it is suited to short streaming sequences of up to 10 seconds (because the computer's memory gets consumed otherwise). If you are just needing to align one frame though then it might be suitable.
https://github.com/IntelRealSense/librealsense/issues/3164#issuecomment-458959674
Alternatively, you could try the SDK's own Python stream alignment example program instead of your own if you have not tried it already, to see if you get better FPS performance.
Please sign in to leave a comment.
Comments
5 comments