Get Disparity Map of D455 by RealSense SDK 2.0

rs2::disparity_transform disparity_transform(true);
rs2::disparity_frame disparity_frame = disparity_transform.process(depth_frame);
Mat mDisparity = Mat(Size(1280, 720), CV_16UC1, (void*)disparity_frame.get_data());
ofstream outFile(filename, ofstream::binary);
outFile.write((char*)mDisparity.data, mDisparity.elemSize() * mDisparity.total());

• MartyG

  • October 13, 2020 09:35
  • Edited

  I looked at your case carefully.  If you are using the disparity transform block (as indicated by your use of disparity_transform.process) then the correct disparity to depth conversion procedure may be to set depth_to_disparity to False and set disparity_to_depth to True instead of using an equation to convert the disparity map to depth.  For example:

   

  rs2::disparity_transform depth_to_disparity(false);
  rs2::disparity_transform disparity_to_depth(true);

   

  With these two bools, at least one of them should be set to False, and they should not be both True at the same time.

  Intel's post-processing documentation states that depth-to-disparity is equal to '1' in the Disparity Mode and disparity-to-depth is equal to '0'.

  https://dev.intelrealsense.com/docs/depth-post-processing#section-implementing-post-processing-with-intel-real-sense-sdk-api 

  0

  Comment actions Permalink
• Temeiwang

  • October 18, 2020 14:01

  Thanks to MartyG's comment. My purpose is to save the native disparity map directly from D455, not the depth map. The reason to convert the disparity map to a depth map is to verify whether the disparity map is correct or not. I revised my code as below, but the disparity map is still incorrect. Could you help me to revise the code to save native disparity map directly from D455?

  auto depth_frame = frameset.get_depth_frame(); // depth data
  rs2::disparity_transform depth_to_disparity(true);
  rs2::disparity_transform disparity_to_depth(false);

  rs2::disparity_transform disparity_transform(true);

  rs2::disparity_frame disparity_frame = disparity_transform.process(depth_frame);

  Mat mDisparity = Mat(Size(imageW, imageH), CV_16UC1, (void*)disparity_frame.get_data());

  ofstream outFile("Disparity.raw", ofstream::binary);
  outFile.write((char*)mDisparity.data, mDisparity.elemSize() * mDisparity.total());

  0

  Comment actions Permalink
• MartyG

  • October 18, 2020 15:30
  • Edited

  Are you requesting how to access the raw depth data directly from the D455 hardware before it undergoes adjustment and is sent through the USB cable to the computer, please?

  0

  Comment actions Permalink
• Temeiwang

  • October 19, 2020 02:11

  I would like to use my code to access the raw disparity data directly from the D455 hardware without post-processing. Please provide example code. Thank you!

  0

  Comment actions Permalink
• MartyG

  • October 19, 2020 06:16

  Typically, RealSense SDK programs are processed in a High-Level API.  When accessing the camera hardware directly though, it is done through the Low-Level API via 'callback' code.

  https://dev.intelrealsense.com/docs/api-architecture 

  Low-Level sensor callbacks have the following traits:

  -  Minimum SDK-imposed overhead == minimum latency

  -  Each frame is handled with a separate callback - “first come – first serve”

  -  All the synchronization stuff is left for the user side to implement.

  It is worth looking at the sensor-control example program, which "is designed to offer low-level access to the hardware, mapping more closely to underlying OS APIs".

  https://github.com/IntelRealSense/librealsense/tree/master/examples/sensor-control 

  I realize that this does not directly answer your question of how to achieve disparity from direct hardware access but it will hopefully provide a starting point towards that goal.

  0

  Comment actions Permalink
• Temeiwang

  • October 19, 2020 07:06

  Thanks a lot for providing the reference links.

  I can control the emitters on/off by sensor-control (the code is shown below), but the disparity map cannot be accessed by sensor-control.

  rs2::config config;
  rs2::pipeline pipe;
  
  rs2::pipeline_profile pipe_profile = pipe.start(config);
  rs2::device dev = pipe_profile.get_device();
  auto depth_sensor = dev.first<rs2::depth_sensor>();
  
  if (depth_sensor.supports(RS2_OPTION_EMITTER_ENABLED))
      depth_sensor.set_option(RS2_OPTION_EMITTER_ENABLED, 0.f); // emitter OFF

  I'm trying to test the code provided by yck011522 in https://github.com/IntelRealSense/librealsense/issues/2229 to see whether it can work in SDK 2.0 or not.

  yck011522 loaded a JSON file to set the settings as below. But, I'm looking for the code that is able to set the "disparity mode" to 1 directly.

  #include <librealsense2/rs.hpp> // Include RealSense Cross Platform API
  #include <librealsense2/rs_advanced_mode.hpp>
  rs2::pipeline pipe;
  rs2::config cfg;
  cfg.enable_stream(RS2_STREAM_DEPTH, 1280, 720, RS2_FORMAT_Z16, 30);
  rs2::pipeline_profile profile = pipe.start(cfg);
  rs2::device selected_device = profile.get_device();
  if (selected_device.is<rs400::advanced_mode>())
  {
      auto advanced_mode_dev = selected_device.as<rs400::advanced_mode>();
      // Check if advanced-mode is enabled
      if (!advanced_mode_dev.is_enabled())
      	advanced_mode_dev.toggle_advanced_mode(true); // Enable advanced-mode
  
      std::ifstream t(arg_jsonName);
      std::string str((std::istreambuf_iterator<char>(t)),
      std::istreambuf_iterator<char>());
      std::string json_content = str;
      cout << "Advanced Mode JSON File Loaded: " << arg_jsonName << " Char Count: " << json_content.length();
      advanced_mode_dev.load_json(json_content);
  }
  else
      cout << "Current device doesn't support advanced-mode!\n";
  

  0

  Comment actions Permalink
• MartyG

  • October 19, 2020 07:39

  My understanding is that Disparity Mode is accessible through the Advanced Mode attribute disparityMode in STDepthTableControl

  https://intelrealsense.github.io/librealsense/doxygen/struct_s_t_depth_table_control.html#a494128c21676ae254740585d20a6420a 

  However, using a json is the more typical way to set its value.  A RealSense team member offers advice in the link below how one might go about setting the effect with C++ if that was the method that they really needed to use:

  https://github.com/IntelRealSense/librealsense/issues/3344#issuecomment-468171910 

  0

  Comment actions Permalink
• Temeiwang

  • October 19, 2020 08:00

  Thanks for the comments. I'm revising my code to do the tests and will share the results later.

  0

  Comment actions Permalink
• Temeiwang

  • October 22, 2020 09:00

  I successfully get the raw disparity map directly from D455. My code is shown below:

  #include <librealsense2\rs.hpp>
  #include <librealsense2\rs_advanced_mode.hpp>
  
  void GetDisparityMap()
  {
      int imageW = 1280;
      int imageH = 720;
      int fps = 30;
  
      rs2::config config;
      config.enable_stream(RS2_STREAM_DEPTH, imageW, imageH, RS2_FORMAT_Z16, fps);
  
      rs2::pipeline pipe;
      rs2::pipeline_profile pipe_profile = pipe.start(config);
      rs2::device dev = pipe_profile.get_device();
  
      auto depth_sensor = dev.first<rs2::depth_sensor>();
      depth_sensor.set_option(RS2_OPTION_EMITTER_ENABLED, 1.f);	// emitter ON
  
      auto advanced_mode = dev.as<rs400::advanced_mode>();
  //  advanced_mode.toggle_advanced_mode(true);	    // don't need, cause error
      STDepthTableControl depth_table = advanced_mode.get_depth_table();
      depth_table.disparityMode = 1.f;	// enable disparity mode
      advanced_mode.set_depth_table(depth_table);     // write changes
  //  Need to wait and confirm the settings; otherwise, the disparity map is incorrect
      depth_table = advanced_mode.get_depth_table();  // get depth table again to confirm the settings
      cout << "Disparity Mode: " << depth_table.disparityMode << endl;
  
      rs2::frameset frameset = pipe.wait_for_frames();
      auto disparity_frame = frameset.get_depth_frame();
      Mat mDisparity = Mat(Size(imageW, imageH), CV_16UC1, (void*)disparity_frame.get_data());
  
      ofstream outFile("Disparity.raw", ofstream::binary);
      outFile.write((char*)mDisparity.data, mDisparity.elemSize() * mDisparity.total());
      outFile.close();
  }

  0

  Comment actions Permalink
• Temeiwang

  • October 22, 2020 09:08

  Update the title to "Get Disparity Map of D455 by RealSense SDK 2.0" for better description.

  0

  Comment actions Permalink
• MartyX Grover

  • October 22, 2020 09:13
  • Edited

  Great news that you developed a solution - thanks for the update!

  0

  Comment actions Permalink

Please sign in to leave a comment.