D435 librealsense2 how to get frames efficiently?
Hi,
We are developing a project that forces us to work at 6FPS. After several tests, we saw that the D435 at 6FPS sometimes does not synchronize correctly the color and infrared/depth images (not the issue of this topic).
We want to make the camera run at it's highest frequency for the desired profiles and when the process is ready to get more data from the camera, "poll" the last frames available. Namely, if the camera is working at 60FPS and the system at 6FPS, we want to drop the 54 other frames.
Is there a way with librealsense2 (c++) to NOT copy all the frames over the USB, but just those we are interested in?
What would be the best code structure to just get the last frame and letting the other frames drop?
The buffer created by (for instance) `frame_queue` is physically on the d435 module or in the pc?
Thanks in advanced!
-
On the RealSense 400 Series cameras, frame buffering management is handled on the computing device and not in the camera hardware.
https://github.com/IntelRealSense/librealsense/wiki/Frame-Buffering-Management-in-RealSense-SDK-2.0
I would recommend a minimum frame rate of 15 FPS if possible, as errors can occur at 6 FPS that disappear at 15 FPS upwards.
If you only wanted 6 frames out of a set of 60, it is possible to set a program to only handle every "nth" frame (for example, every 5th frame at 30 FPS to get a frame rate of 6 FPS, or every 10th frame to get a frame rate of 6 FPS from 60 FPS). The subject is discussed in the link below. The script in it is in the Python language but may give you some guidance about how to structure a C++ equivalent.
-
I was afraid of that answer...
The post you suggest has been helpful but it does not solve our case.
The main problem we have is that we have 3x D435 connected to a Jetson AGX Xavier through the USB-C ports with the camera set to stream Color@1280x720@6FPS, Infra1@640x480@6FPS and Depth@640x480@6FPS and we are using ROS.
If we try to increase the FPS, we face an Overflow error of the usb driver, hence, losing one of the cameras at least.
From your answer I'm guessing that is not possible to just poll the frames of interest from the camera without buffering them on the pc then?
-
You may be able to reduce the strain put on Xavier by setting enable_infra2:=false in ROS to disable the Infra2 channel. It saves on USB transportation and kernel work, according to Doronhi the RealSense ROS wrapper developer. Doronhi adds "It will have no effect on the depth quality. It only disables the infra2 images' transmission via the USB port. They are still taken and the depth is calculated using them inside the device.".
In regard to accessing the camera hardware directly, the SDK provides a Low Level API.
https://github.com/IntelRealSense/librealsense/blob/master/doc/api_arch.md#low-level-device-api
-
Indeed, we are already using the enable_infra2:=false, the problem explained above is with this setting already.
That is also
I had seen the low-level-device-api, however, I'm unsure on where to find it's documentation, could you please provide it?
Additionally, you commented taht at 6FPS, the camera shows some errors, could you be a little more specific on what these errors are?
-
There is not much information available about the Low Level API. A few years ago, a RealSense user offered the following commentary on it though:
I think in this case the "medium" level interface (stated as low level on the Low Level API document) is literally just using the device class directly:
I also found this information:
Low-level “sensor” callbacks API calls:
- Minimum SDK-imposed overhead == minimum latency.- Each frame is handled with a separate callback - “first come – first serve”.
- All the synchronization stuff is left for the user side to implement.
- Check rs-data-collect tool of the SDK that uses this method
https://github.com/IntelRealSense/librealsense/tree/master/tools/data-collect
Another information source says about the rs-data-collect tool: "Store and serialize IMU and Tracking (pose) data in Excel-friendly csv format. The tool uses low-level sensor API to minimize software-imposed latencies. Useful for performance profiling".
In regard to errors at 6 FPS, the kind of errors that you are most likely to receive are ones related to timeout when waiting for frames.
Please sign in to leave a comment.
Comments
5 comments