Converting Depth Frame to Color Frame With RealSense Depth Cameras

• Sandesh Kumar S

  • April 01, 2021 10:10

  No that method of conversion mentioned in issue 2634 requires the rs2 depth frame to be sent to the processing block as the rs2 frame source.

  I do not have access to the rs2 depth frame to serve as rs2 frame source.

  0

  Comment actions Permalink
• MartyX Grover

  • April 01, 2021 12:14
  • Edited

  software_device is going to be your main option for feeding OpenCV data to the librealsense SDK.  It should not matter if you do not have access to rs2 depth frames, as the purpose of software_device is to enable simulation of a RealSense device without actually having a physical camera attached.

  https://github.com/IntelRealSense/librealsense/tree/master/examples/software-device

  0

  Comment actions Permalink
• Sandesh Kumar S

  • April 05, 2021 11:52

  Hi MartyG while trying to declare the depth_sensor variable I am getting the following error. 

  error: no matching function for call to ‘rs2::software_sensor::software_sensor()’
  rs2::software_sensor depth_sensor;

  May I know what is the type of the depth_sensor used in the rs-software-device.cpp line138. In the code "auto" keyword is used (https://github.com/IntelRealSense/librealsense/blob/5ff27fca5aaeec4736d6bb3bfb958fee82ee198b/examples/software-device/rs-software-device.cpp#L138)

   

  0

  Comment actions Permalink
• Sandesh Kumar S

  • April 23, 2021 06:39

  MartyG Thanks a lot for your help I was able to convert the cv mat frame to rs2::frame using the software device and syncer class.

  Now in my application I am exploring the python wrapper for realsense and wanted to check if conversion from numpy frame to python realsense frame is supported or not.

  I mean can we use the same software device APIs in python to convert from numpy to python rs2 frames ?

  0

  Comment actions Permalink
• MartyX Grover

  • April 23, 2021 06:53

  Great news about your success with the conversion!

  Conversion of numpy to rs2::frame is something that other RealSense users have explored.  As far as I know though, a definitive solution for doing so has not been found yet.

  https://support.intelrealsense.com/hc/en-us/community/posts/360047819294-Conversion-of-Frame-to-Numpy-Array-and-Vice-Versa

   

  0

  Comment actions Permalink
• Sandesh Kumar S

  • April 26, 2021 05:55
  • Edited

  MartyG Is software_device supported in python(pyrealsense2) ? 

  I do not see software_device in the documentation for pyrealsense2 in the below link.

  https://intelrealsense.github.io/librealsense/python_docs/_generated/pyrealsense2.html

  Without software_device how can the depth and color frame be simulated ?

  0

  Comment actions Permalink
• MartyG

  • April 26, 2021 06:30

  software_device is referenced in the official pyrealsense2 documentation as being supported for Python.

  https://intelrealsense.github.io/librealsense/python_docs/_generated/pyrealsense2.extension.html?highlight=software_device#pyrealsense2.extension.software_device

  There is not much information available regarding how to implement it though, with a partially working Python script for software_device in the link below being the best resource that I am aware of.

  https://github.com/IntelRealSense/librealsense/issues/7057

  0

  Comment actions Permalink
• Sandesh Kumar S

  • April 28, 2021 16:54

  MartyX Grover Will the rotation and translation attributes of rs2_extrinsics Struct be the same for all RealSense depth camera models ?

  I see that in rs-software-device.cpp example they are hardcoded as {{ 1,0,0,0,1,0,0,0,1 },{ 0,0,0 }} in Line 161

  https://github.com/IntelRealSense/librealsense/blob/5ff27fca5aaeec4736d6bb3bfb958fee82ee198b/examples/software-device/rs-software-device.cpp#L161

  Similarly will the coeffs attribute of rs2_instrinsics Struct be the same for color and depth stream of all the RealSense depth camera models ? They too are hardcoded as { 0,0,0,0,0 } for color and depth intrinsics

  https://github.com/IntelRealSense/librealsense/blob/5ff27fca5aaeec4736d6bb3bfb958fee82ee198b/examples/software-device/rs-software-device.cpp#L91

  0

  Comment actions Permalink
• MartyX Grover

  • April 28, 2021 17:37
  • Edited

  I can understand why the rotation and translation extrinsic values would be hardcoded values in rs-software-device since the software-device system is used to simulate camera hardware, and so there may not be a physical RealSense camera attached to read values from.

  The rs-enumerate-devices tool can be used to read the rotation and translation extrinsics from a live camera.  Launching it with the command rs-enumerate-devices -c  lists the full calibration information of that particular camera, including the rotation and translation extrinsics for all combinations of streams (e.g the extrinsics between color and depth).  The values below are from a D435i camera.

   

  

   

  

   

  In regard to the coefficients, they are deliberately all set to zero on the 400 Series cameras for reasons described in the link below.

  https://github.com/IntelRealSense/librealsense/issues/1430#issuecomment-375945916

  An exception to this is the coefficients of the color profiles on the D455 model, which may be reported as non-zero.

  https://github.com/IntelRealSense/realsense-ros/issues/1580#issuecomment-748038697

  0

  Comment actions Permalink
• Sandesh Kumar S

  • April 29, 2021 05:21

  Ok so will hardcoding the rotation and translation attributes of rs2_extrinsics to {{ 1,0,0,0,1,0,0,0,1 },{ 0,0,0 }} work for all the cameras?

  Note my application requires reading rs2 frames from realsense camera, convert it to cv mat and then again convert back from cv mat to rs2 frame. Hence I am using the software-device and syncer class for simulating rs2 frames. 

  0

  Comment actions Permalink
• MartyX Grover

  • April 29, 2021 14:04
  • Edited

  I would expect there to be differences between models regarding the extrinsics between sensors.  This is demonstrated by the three images below, from a D435i (top), D455 (middle) and D415 (bottom).

   

  

  

  

  0

  Comment actions Permalink
• Sandesh Kumar S

  • April 30, 2021 05:04

  Thanks a lot MartyG for providing these details.

  Now if I use software-device APIs and hardcoded the rs2 extrinsics values to {{ 1,0,0,0,1,0,0,0,1 },{ 0,0,0 }} with these camera models will it work in converting from cv mat to rs2 frame ? 

  0

  Comment actions Permalink
• MartyG

  • April 30, 2021 06:13
  • Edited

  The link below may be helpful in developing such an application.

  https://stackoverflow.com/questions/52585921/convert-opencv-mat-to-frame-type-of-realsense-in-librealsense-sdk

  0

  Comment actions Permalink
• Sandesh Kumar S

  • July 28, 2021 08:24

  Hi MartyG I had used rs2 stream_profile class to read the rs2 instrinsics and extrinsics values and used them to convert the cv mat depth and color frames to rs2 frame using rs2 software_device APIs

  Now I am working with T265 camera and similarly want to convert the cv mat fisheye frames to rs2 fisheye frame using rs2 software_device APIs. 

  Is it possible to simulate rs2 fisheye frame from T265 using rs2 software device APIs ?

  0

  Comment actions Permalink
• MartyX Grover

  • July 28, 2021 08:32

  Hi Sandesh Kumar S  Please create a new case on this forum for your T265 question so that the appropriate Intel RealSense support team member can assist you with that question.  Thanks!

  0

  Comment actions Permalink
• Sandesh Kumar S

  • August 09, 2021 10:29

  $ realsense-viewer
  09/08 15:55:18,965 WARNING [140293605148416] (backend-v4l2.cpp:1357) Pixel format 36315752-1a66-a242-9065-d01814a likely requires patch for fourcc code RW16!
  09/08 15:55:19,002 WARNING [140293605148416] (sensor.cpp:1086) RGB Camera: skipping Gamma control. descriptor: [min/max/step/default]= [0/0/0/0]
  09/08 15:55:19,002 WARNING [140293605148416] (sensor.cpp:1086) RGB Camera: skipping Hue control. descriptor: [min/max/step/default]= [0/0/0/0]
  09/08 15:57:27,851 WARNING [140296208190400] (ds5-motion.cpp:477) IMU Calibration is not available, default intrinsic and extrinsic will be used.
  09/08 15:57:27,856 WARNING [140293217187584] (backend-hid.cpp:715) HID set_power 1 failed for /sys/devices/pci0000:00/0000:00:14.0/usb2/2-1/2-1:1.5/0003:8086:0B3A.0008/HID-SENSOR-200076.7.auto/iio:device3/buffer/enable

  MartyG what is the reason for the above failure while trying to use Motion module? There seems to be an issue with the IMU sensor.

  Device details:

  

  0

  Comment actions Permalink
• MartyX Grover

  • August 09, 2021 11:31

  Did the problem occur when trying to enable the Motion Module in the Viewer?

  Also, did you enable three stream types - Depth, RGB and Motion Module - or just the Motion Module on its own?

  0

  Comment actions Permalink
• Sandesh Kumar S

  • August 09, 2021 12:40

  Yes it occurred  when trying to enable Motion Module in realsense-viewer.

  I tried both enabling all three streams and just Motion Modules. Both ways I see this failure.

  0

  Comment actions Permalink
• Sandesh Kumar S

  • August 09, 2021 13:08

  Please note T265 device is also connected in the system(ubuntu 20) and Motion Module issue is not observed with T265

  0

  Comment actions Permalink
• MartyX Grover

  • August 09, 2021 16:55

  A RealSense team member provides information about this HID set_power 1 failed error in the link below, advising that if you are able to receive IMU data despite the message occurring then it can be disregarded.

  https://github.com/IntelRealSense/librealsense/issues/5901#issuecomment-591405292

  0

  Comment actions Permalink

Please sign in to leave a comment.