View my account

Skewed alignment depth and visual images taken from moving vehicle D435i

Follow
Derek

Hi,

I am trying to align depth and visual images taken from a slowly moving vehicle usnig the D435i. I am reading ROS stored data using code similar to https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/align-depth2color.py

There seems to be a non-constant deviation between the visual and depth images. Could you provide support on how I may resolve this? The framerate of the depth captures is 90 so there should always be an image that aligns well. 

Thank you for your support. 

my code:
try:
    # Create pipeline
    pipeline = rs.pipeline()

    # Create a config object
    config = rs.config()

    # Tell config that we will use a recorded device from file to be used by the pipeline through playback.
    rs.config.enable_device_from_file(config, args.input)

    # Configure the pipeline to stream the depth stream
    # Change this parameters according to the recorded bag file resolution
    # config.enable_stream(rs.stream.depth, rs.format.z16, 6)
    # config.enable_stream(rs.stream.visual, rs.format.z16, 6)

            # cam_config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, self.stream_framerate)
    config.enable_stream(rs.stream.depth, 848, 480, rs.format.z16, 90)
    config.enable_stream(rs.stream.color, 1920, 1080, rs.format.bgr8, 6)

    # Start streaming from file
    pipeline.start(config)
    align_to = rs.stream.color
    align = rs.align(align_to)

    # # Create opencv window to render image in
    # cv2.namedWindow("Depth Stream", cv2.WINDOW_AUTOSIZE)
    
    # Create colorizer object
    colorizer = rs.colorizer()

    # Streaming loop
    while True:
        # Get frameset of depth
        frames = pipeline.wait_for_frames()
        aligned_frames = align.process(frames)

        # Get aligned frames
        color_frame = aligned_frames.get_color_frame()
        aligned_depth_frame = aligned_frames.get_depth_frame() 

        # Validate that both frames are valid
        if not aligned_depth_frame or not color_frame:
            continue

        aligned_depth_color_frame = colorizer.colorize(aligned_depth_frame)
        depth_image = np.asanyarray(aligned_depth_color_frame.get_data())
        color_image = np.asanyarray(color_frame.get_data())

        depth_colormap_dim = depth_image.shape
        color_colormap_dim = color_image.shape
        print(depth_colormap_dim, color_colormap_dim)
        start = time.time()
        # If depth and color resolutions are different, resize color image to match depth image for display
        if depth_colormap_dim != color_colormap_dim:
            resized_color_image = cv2.resize(color_image, dsize=(depth_colormap_dim[1], depth_colormap_dim[0]), interpolation=cv2.INTER_AREA)
            images = np.hstack((resized_color_image, depth_image))
        else:
            images = np.hstack((color_image, depth_image))

        print(time.time() - start)
        # Show images

        cv2.imwrite(f"data/realsense/a_{time.localtime().tm_min}_{time.localtime().tm_sec}_colour.jpg", color_image)
        cv2.imwrite(f"data/realsense/a_{time.localtime().tm_min}_{time.localtime().tm_sec}_depth.jpg", depth_image)

 

 

0

Comments

8 comments

Date Votes
  • MartyX Grover
    • Edited

    Hi Derek  There is a large difference between the frames per second (FPS) of depth and color.  If it is a slow moving vehicle, I would therefore suggest returning to 6 FPS for depth (which your commenting-out indicates that you have already tested).  

    If you find that there is a varying of frame rate when using depth and color together, if you have Auto-Exposure enabled then you can enforce a constant FPS by disabling an RGB option called Auto-Exposure Priority.  A simple code snippet for disabling it in Python is provided in the link below.

    https://github.com/IntelRealSense/librealsense/issues/5885#issuecomment-590502775

     

    A more complex example of disabling Auto-Exposure Priority in a Python script is here:

    https://support.intelrealsense.com/hc/en-us/community/posts/360037286634-Strange-FPS-when-enable-both-color-and-depth-streams

    0
    Comment actions Permalink
  • Derek

    Hi MartyG, thanks for the quick response.

    I have indeed tried to set the framerate lower for depth. This didn't resolve the issue. In fact, the discrepancy appears to be higher when the framerates of both images are set lower.

    I also tried disabling the RGB option using the code snippet  you provided, but a lag between the depth and visual remains, although it seems slightly lesser.

    Is it possible that the camera is not ideally suited to deal well with movement in the images? Since only the depth shutter is global and the rgb is not..

    0
    Comment actions Permalink
  • MartyX Grover

    The depth sensing of the RealSense models that are equipped with a global-shutter depth sensor is well suited to capturing during motion when mounted to a vehicle even when it is travelling at high speed, as illustrated in the YouTube video link below.

    https://www.youtube.com/watch?v=OwJmCyAn3JQ

    Whilst the RGB sensor of the D415, D435 and D435i does have a slower rolling shutter (the D455 has a global shutter on both RGB and depth), matching up depth and RGB should not be a significant problem if the vehicle is travelling at a slow speed.  How fast is your test vehicle travelling though, please?  I could see RGB rolling shutter speed becoming an issue if the vehicle speed was faster than human walking pace.

    How does it perform if you set both depth and RGB to 60 FPS at 848x480 resolution?

    0
    Comment actions Permalink
  • Derek

    We have tried different speeds, ranging from 1-8 km/h. Ideally we would like to move to a situation where we can go 8 km/h. We looked at the d455 but found the same issue here as with https://github.com/IntelRealSense/librealsense/issues/7155.

    Cool video! I don't see a discrepancy in my live either though, I only see it when I store the images. The trees in our videofeed are also quite close to the camera and therefore move quite fast.

    By setting both to 60fps and 848x480 I get the best results. I have not yet tried this from the vehicle, only by walking in front of the camera. We are trying to run multiple camera's on an Intel Nuc, so I'm not sure if such a high fps for rgb and depth is feasible. Can you explain to me why setting rgb 6fps and depth 60fps does not work as well as setting both to 60?

     

    0
    Comment actions Permalink
  • MartyX Grover
    • Edited

    My understanding is that as the FPS of the RGB stream is increased, the amount of rolling shutter delay compared to the depth stream progressively decreases.  A RealSense team member comments on this in the link below.

    https://github.com/IntelRealSense/librealsense/issues/3554#issuecomment-478712289

    In another case, it was advised that setting RGB to 6 FPS and setting a manual RGB exposure value of '70' could help (there are separate exposure settings for depth and RGB sensors).

    0
    Comment actions Permalink
  • Derek

    Ahh yes my findings coincide more or less with this issue. Is the manual RGB exposure value 'exposure' in https://intelrealsense.github.io/librealsense/python_docs/_generated/pyrealsense2.option.html?

     

    0
    Comment actions Permalink
  • MartyX Grover
    • Edited

    RGB exposure can be manually defined by specifying a reference for identifying the RGB sensor, either by using first_color_sensor or by providing an index number ('0' for the depth sensor and '1' for the RGB sensor).  The link below provides Python scripting for setting the RGB manual exposure with an index number of '1' (change '156.000' in the script to '70.000').

    https://support.intelrealsense.com/hc/en-us/community/posts/360052346893/comments/360014794613

    0
    Comment actions Permalink
  • Derek

    I don't really see a difference by changing that parameter. I think I got a lot more insight into what is going on though. Thanks for the support!

     

    0
    Comment actions Permalink

Please sign in to leave a comment.

Didn't find what you were looking for?

New post