Technical question about RealSense D455 camera-side latency for low-latency application
Dear RealSense Support Team,
We are using an Intel RealSense D455 for a low-latency eye-tracking related project.
Our current setup is:
- Camera: RealSense D455
- Host OS: Windows
- Development: C++ with NVIDIA GPU processing
- Stream mode: Infrared Y8, 8-bit grayscale
- Resolution / FPS: 848×480 @ 90 fps
- IR projector: disabled
- Main concern: camera-side latency before the image reaches the host OS
We would like to understand the latency inside the camera, excluding Windows OS scheduling, USB driver buffering, and application-side processing.
Could you help clarify the following questions?
- For the D455 infrared Y8 stream at 848×480 @ 90 fps, what is the typical latency from the sensor exposure midpoint to the start of frame readout / USB transmission?
- In librealsense metadata, is
SENSOR_TIMESTAMPthe best representation of the image exposure time? Does it represent the start, middle, or end of exposure for D455 infrared Y8 frames? - Is
FRAME_TIMESTAMPgenerated when frame readout starts, when USB transmission starts, or when the full frame is ready to be transferred? - Can we estimate the camera-side latency by calculating:
FRAME_TIMESTAMP - SENSOR_TIMESTAMP
for the infrared Y8 stream? - Does the D455 firmware or UVC device side introduce any internal frame buffering before USB transmission? If yes, what is the minimum buffer depth or expected delay?
- If we also enable the depth stream, what is the additional on-camera processing latency for generating the Z16 depth frame from the left/right stereo images?
- Is the depth frame timestamp associated with the raw stereo image exposure time, the D4 ASIC depth computation completion time, or the USB transmission time?
- Does disabling the IR projector affect exposure timing, frame timing, or depth generation latency?
- For lowest and most deterministic latency, do you recommend fixed manual exposure/gain instead of auto exposure?
- Is
rs-data-collectthe recommended tool to profile low-level camera-side timing for this use case? If so, could you suggest the correct command or procedure for D455 Y8 848×480 @ 90 fps?
Thank you for your help.
Best regards,
Charlie
-
Hi ChenLin Yeh Thanks very much for your questions.
1. Readout begins when exposure is completed. So the length of the exposure time should influence when readout starts. I do not have a specific value available for the latency at 848×480 @ 90 FPS though.2. I agree that SENSOR_TIMESTAMP would likely be the best representation of the image exposure time as it marks the middle of sensor exposure.
3. FRAME_TIMESTAMP marks the beginning of UVC frame transmission (the first USB chunk sent towards the host).
SENSOR_TIMESTAMP and FRAME_TIMESTAMP are both generated by the firmware driver and both use the same clock. Their relation is that SENSOR_TIMESTAMP << FRAME_TIMESTAMP. More information about how different types of timestamp behave can be found here:
https://github.com/realsenseai/librealsense/issues/2188#issuecomment-409985803
4. Your suggestion of FRAME_TIMESTAMP - SENSOR_TIMESTAMP seems valid. Though a D455 user at the link below who was exploring latency took the alternative approach of calculating latency with TIME_OF_ARRIVAL - FRAME_TIMESTAMP
https://github.com/realsenseai/librealsense/issues/11667
5. The SDK's frame buffering documentation at the link below describes how "latency in the system will come from system level buffering done in drivers or software".
By default the frame pipeline can hold a maximum of 16 frames of each type of stream at a time, and the oldest frames get pushed out of the frame queue by newly arriving frames. The pipeline is already configured by default for minimum latency. Introducing latency into the queue can improve stability and reduce the number of dropped frames if more than one stream is being enabled.
6. For this question I recommend to you Section 4.5 (Latency) of RealSense's 'High-Speed Capture Mode' white-paper document, which provides detailed technical information about latency.
https://dev.realsenseai.com/docs/high-speed-capture-mode-of-intel-realsense-depth-camera-d435/
This section states that "the depth camera itself introduces very little latency because it has no full frame buffer, which means that images by necessity flow through the ASIC with ~100 lines of latency".
7. As mentioned in question 3, the FRAME_TIMESTAMP reflects the beginning of UVC frame transmission.
8. If the emitter is disabled then the IR image could become under-exposed unless the lighting in the scene is strong.
9. The default settings of RealSense 400 Series cameras are configured for optimal performance and minimal latency. So I would recommend using the default of auto-exposure being enabled.
10. rs-data-collect is an excellent tool for profiling low-level camera performance.
To configure the infrared stream to 848×480 @ 90 fps, you should edit the stream width, height and FPS parameters in the rs-data-collect.h file of the project and then build the source code of the program so that it can use the updated parameters.
https://github.com/realsenseai/librealsense/blob/master/tools/data-collect/rs-data-collect.h#L13The Low-Level API can produce the best possible latency, but there is not much documentation for it and the rs-data-collect program is the only example program I am aware of that demonstrates code for accessing this API.
https://github.com/realsenseai/librealsense/blob/master/doc/api_arch.md#low-level-device-api
You could also consider trying the rs-latency-tool example program.
https://github.com/realsenseai/librealsense/tree/master/wrappers/opencv/latency-tool
-
Hi ChenLin Yeh Do you require further assistance with this case, please? Thanks!
Please sign in to leave a comment.
Comments
2 comments