correct depth frame intrinsics after filter decimation and alignment
Hi,
I am reading frames from a ROSbag file and perform the following operations for estimating real world coordinates from a set of identified pixel values. The data was collected from a D415.
- Get intrinsics from depth stream during initialization.
- Read frameset from file
- Alight depth to color
- Run decimation and spatial filters
- Update intrinsics from depth frame
- Process color frame to identify set of pixels of interest.
- Scale these pixel location values down by the ratio of decimated depth frame width & height to color frame width and height.
- Use these scaled pixel locations to estimated depth and corresponding deprojection to real world coordinates.
So I noticed that aligning depth changes the intrinsic parameters. My question is, if it is OK to use the decimated depth frame intrinsics for deprojection as I have? Is this the correct way to estimate real world coordinates after identifying pixel locations of interest after aligning frames?
Thanks!
-
Hi Shiva Keshavan Apologies for the delay in responding. To begin with, it is recommended that align is done after the post-processing is applied and not before. This helps to avoid distortions such as aliasing (jagged lines).
Could you try putting align after the post-processing filters in your application please and see what difference it makes to your results? Thanks!
-
Thank you for the response MartyG, I actually saw you say this in another post but considering that I run the post-processing only the depth frame, would I need to create a new frameset for aligning?
Also after decimation, the depth frame is downsampled to a lower res. How would alignment work in this case when depth and color are of different resolutions? FYI I could also downsample the color frame but I don't want to because I am passing that to a deep learning module where I believe higher resolutions provide better accuracy.
Thanks!
Shiva
-
I checked my notes on how the SDK handles frame creation. They say that a created frame is never modified, and that if processing such as post-processing is done on the frame then a new frame is created automatically and the original version of the frame remains unchanged. Every frame has a counter (how many copies of the same frame are being held), and a frameset is just a collection of frame objects..
When a new frame enters the frame queue, old frames get pushed out.
When aligning depth to color, the depth is aligned to the field of view size of the color sensor. This is because in the D435 models, the depth has a larger field of view than the color.
In the new D455 camera model though, the color and depth both have the same FOV size and both have a fast global shutter to improve the correspondence between the depth and RGB images.
https://www.intelrealsense.com/depth-camera-d455/
Dorodnic the RealSense SDK Manager has said that although depth and IR should be equal, there is not a dependency between depth and RGB resolutions.
https://github.com/IntelRealSense/librealsense/issues/3376#issuecomment-469345072
-
Thanks for the information! So if I understand correctly after post-processing the depth frame, I need to create a new frameset with the original color frame and the filtered depth frame and then pass this to align right? How do I correctly re-insert frames into a frameset?
Edit: Also just to confirm, aligning can still happen when the color and depth frame images are at different pixel resolutions?
Thanks!
-
The RealSense SDK has systems called Processing Blocks to simplify and automate certain operations. Alignment can be processed by the Align Processing Block. So you do not have to write everything yourself, as the processing block can handle the bulk of the operation.
A basic example program for alignment can be found here:
https://github.com/IntelRealSense/librealsense/tree/master/examples/align
On the 400 Series cameras, sometimes you cannot help but have different depth and color resolutions, since color has a maximum available resolution of 1920 x 1080, whilst the maximum depth resolution is 1280 x 720. There should not be not consequences to alignment for using a different color resolution.
-
Yes, this is in fact how I am aligning the frames. I think my question here is much simpler. I post-process the depth frame after extracting it as indicated in: https://github.com/IntelRealSense/librealsense/tree/master/examples/post-processing.
How would I align the depth_frame and video_frame (color frame) after this? Align is directly applicable to the frameset but postprocessing is only on the depth frame. How can I insert the post-processed depth frame into a frameset with the color frame for alignment?
Thanks!
-
I hope the example script by Dorodnic in the link below may provide useful guidance.
After the post-processing filters have been set up, the align processing block operation block is carried out with:
frames = pipe.wait_for_frames();
aligned_frames = align.process(frames);https://github.com/IntelRealSense/librealsense/issues/1658#issuecomment-387356272
-
Thanks but the alignment itself was never the issue. The issue was re-inserting the processed depthframe into a frameset before alignment. I think this actually shows how to do that:
-
"Re-inserting" processed frames into the frameset may be over-thinking the problem. As far as I am aware, the handling of the post-processed frames should be taken care of automatically when the Align processing block is applied to the data. I hope that the link that you found will enable you to create the application that you have in mind though.
-
In the example you provided: https://github.com/IntelRealSense/librealsense/issues/1658#issuecomment-387356272
Dorodnic, seems to actually apply the filters AFTER the alignment process. He only sets up the filter parameters before aligning. But the alignment is done BEFORE the actual filtering.
Please sign in to leave a comment.
Comments
11 comments