View my account

[solved] D455 HDR merge C++ code example needed

Follow
Huawatuam
  • Edited

I got a D455 and want to use HDR in my own C++ application.

I got my C++ code so far setup that i get the flickering images due to different exposure settings.

However i do not know how to do the post processing in my C++ code to merge the two images into one HDR result.

Has anybody got a code example on how to do that in C++?

0

Comments

9 comments

Date Votes
  • MartyX Grover
    • Edited

    Hi Huawatuam  The High Dynamic Range (HDR) toggle instruction RS2_OPTION_HDR_ENABLED was introduced in SDK 2.39.0 and requires at least firmware version 5.12.8.200.  So you should be using SDK and firmware with those versions or newer in order to access the feature.

    https://github.com/IntelRealSense/librealsense/wiki/API-Changes#version-2390

    Intel have published a white-paper document about HDR.

    https://dev.intelrealsense.com/docs/high-dynamic-range-with-stereoscopic-depth-cameras

    This paper only provides HDR toggle code in the Python language (rs.option.hdr_enabled).  It should be straightforward to convert to C++ for RS2_OPTION_HDR_ENABLED though.

    You could use the example code in the link below for configuring the IR emitter and swap in the RS2_OPTION instruction for HDR instead.

    https://github.com/IntelRealSense/librealsense/wiki/API-How-To#controlling-the-laser

     

    *******************

     

    rs2::pipeline pipe;
    rs2::pipeline_profile selection = pipe.start();
    rs2::device selected_device = selection.get_device();
    auto depth_sensor = selected_device.first<rs2::depth_sensor>();
    depth_sensor.set_option(RS2_OPTION_HDR_ENABLED , 1.f); // Enable HDR

     

    ********************

    The above adaptation roughly follows the structure of the Python example toggle code in the HDR white-paper.

    0
    Comment actions Permalink
  • Huawatuam
    • Edited

    Hi MartyG

    thank you for the info.

    What i am still not able to do is the merging process itself.

    Based on the code example below, how would i correctly insert the "rs2::hdr_merge" filter?

    I tried something like this but the depth image keeps on flickering:

     

    ...
    rs2::hdr_merge myHDRmergeFilter;

    rs2::frameset data = pipe.wait_for_frames();

    data = myHDRmergeFilter.process(data).as<rs2::frameset>();
    rs2::frame depth_frame = data.get_depth_frame();
    ...

     

     

    // Streaming initialization
    rs2::pipeline pipe;
    ...
    // Declare filters
    rs2::decimation_filter dec_filter;
    rs2::spatial_filter spat_filter;

    // Configure filter parameters
    decimation_filter.set_option(RS2_OPTION_FILTER_MAGNITUDE, 3);
    ...
    spatial_filter.set_option(RS2_OPTION_FILTER_SMOOTH_ALPHA, 0.55f);
    ...


    // Main Loop
    while (true) {
    rs2::frameset data = pipe.wait_for_frames();
    rs2::frame depth_frame = data.get_depth_frame();
    ...
    rs2::frame filtered = depth_frame;
    // Note the concatenation of output/input frame to build up a chain
    filtered = dec_filter.process(filtered);
    filtered = spatial_filter.process(filtered);

    }
    0
    Comment actions Permalink
  • MartyX Grover
    • Edited

    There is a recommended order for filter types to be applied in, as described in the link below. 

    https://github.com/IntelRealSense/librealsense/blob/master/doc/post-processing-filters.md#using-filters-in-application-code

    Whilst this documentation does not currently mention the HDR Merge filter, given that the order of the list of post-processing filters in the Viewer roughly matches the suggested order of filters in the above link then it may be worth applying the HDR filter in your code directly after the Decimation Filter and before the Spatial Filter.

     

    0
    Comment actions Permalink
  • Huawatuam
    • Edited

    I just made another observation on my Jetson nano system compared to my Laptop setup (where HDR works flawlessly).

    When using the realsense-viewer on the Jetson nano with the exact same HDR settings as on my Laptop, i can not get it work flickerfree/merged.

    It seems that on the Jetson nano there is a general issue with the merge process - no matter if i use it in the realsense-viewer or in my code.

    Maybe the warning in theimage frames of the Jetson nano saying the following might be the cause?

    "Timestamp Domain: System Time. Hardware timestaps unavailable! Please refer to frame_metadata.md for more information"

     

    Here are the versions installed:

    D455 FW-version: 05.12.11.00

    SDK version: 2.41.0

     

    0
    Comment actions Permalink
  • MartyX Grover

    It could be related to the problem.  The white paper states that per-frame metadata is necessary for HDR merging.

    0
    Comment actions Permalink
  • Huawatuam

    Ok, so i guess i have to patch the kernel. :/

    0
    Comment actions Permalink
  • MartyX Grover
    • Edited

    Alternatively, metadata support should be included in the librealsense build if it is built on Nano from source code with CMake with the RSUSB method using the -DFORCE_RSUSB_BACKEND=ON flag.

    https://github.com/IntelRealSense/librealsense/issues/7905#issuecomment-737544294

    1
    Comment actions Permalink
  • Huawatuam
    • Edited

    Thank you for the hint!

    After building the SDK using -DFORCE_RSUSB_BACKEND=ON the HDR mode is now working flawlessly in the realsense-viewer as well as my own C++ code on the Jetson nano.

    0
    Comment actions Permalink
  • MartyG

    Great news Huawatuam - thanks very much for the update about your success. :)

    0
    Comment actions Permalink

Please sign in to leave a comment.

Didn't find what you were looking for?

New post