Incorrect depth measurement at high speeds
So I have a computer vision program which uses the intel Realsense D435 camera for the depth measurements and for the RGB-image. The program is being developed on a PC with a strong CPU (i7-10700K), but will eventually be run on a SBC (Nvidia Jetson AGX Xavier) with a much less powerful CPU. The program is made with opencv in c++.
At the moment the program works perfectly on the PC, all the measurements and data seem to be correct. But when I run the program on the SBC the depth measurements return the max value.
I tracked the problem down to the program being too fast (??). Which makes no sense when looking at the structure of the program:
-
First start thread to get frame (RGB and depth)
-
Execute Program (measurements etc.)
-
wait for the 'get frame' thread to finish
-
start at beginning
when I insert a wait in the program OR in the thread, there seems to be no problem. This is weird as the program waits for the 'get frame' thread to finish, so everything should be loaded in before further execution.
If someone can help me figure out what exactly is the problem here, that would be great!
-
May I begin by first asking if your program uses wait_for_frames() so that the program will block until a complete frame is received? In the link below, a RealSense team member explains the difference between wait_for_frames() and other types of '_for_frames' instructions, such as poll_for_frames() that returns frames instantly without waiting.
-
Thanks very much for the tag correction, Lschreurs25 - I have adjusted the comment above!
I note that you are using rs2::align. If you built the librealsense SDK on Jetson with CUDA support included then the program will be likely taking advantage of the automatic acceleration for align operations that is provided by the Jetson's Nvidia graphics GPU chip. This can considerably reduce the processing burden on the CPU by offloading the processing work onto the GPU.
CUDA support will be included if librealsense was built on your Jetson from packages or if you built from source code with CMake and included the build term -DBUILD_WITH_CUDA=true
-
You would still need to include an instruction to perform alignment. What CUDA support can do is significantly reduce the percentage usage of the CPU when performing alignment (it also accelerates pointcloud and color conversion operations). Alignment is a CPU-intensive processing operation.
The link below provides rough examples of CPU percentage usage values with and without CUDA support when performing pointcloud generation and color conversion (align support was added a little later, so figures for % usage reduction for alignment are not included in that link).
-
Thanks very much. I look forward to your next report.
The link below has CUDA performance figures relating to align acceleration.
-
Hey MartyG,
So I got the Jetson working again, only the problem remains.
On top of that, the PC is having a similar problem. Where the Jetson gets incorrect depth measurements with little to no time between getting the frame and measuring depth, the PC gets more incorrect depth measurements with more time between getting the frame and measuring depth. This is all with the same c++ program.
I really am getting confused, as I thought maybe the CPU of the Jetson was too slow or something. But that doesn't explain the problem with the PC.
-
It may be worth testing the RealSense SDK's rs-grabcuts OpenCV C++ example, which also aligns depth and color, to see whether the same problems occur when running that program on your PC and Jetson. If it works normally then that would suggest that there is a problem in your script code.
https://github.com/IntelRealSense/librealsense/tree/master/wrappers/opencv/grabcuts
Please sign in to leave a comment.
Comments
11 comments