Realsense D435i: Motion data freezes after sometime
Hi,
I use the Intel Realsense D435i camera module with the Raspberry pi 4 running Ubuntu 18.04. In the realsense viewer, I enable all the 3 streams - Stereo, RGB and motion module. When I enable the motion module, at first a popup appears that says "Motion Module Failure", however the motion module continues to work after this. After a few minutes or an hour the motion module windows show a flashing text reading "No frames received". What could be the possible reasons for this error?
Could this be a resource availibility problem? If yes what could I do to improve it?
This happens only when all the 3 streams are activated together.
Thanks in advance for the help. Please let me know if more information is required.
Kany.
-
Hi K Tchikaya As you observed, the Motion Module keeps working normally after a Motion Module Failure pop-up occurs. So it is not a serious problem.
It is a known issue though that problems can occur when streaming RGB, depth and IMU simultaneously. There are methods in C++ and Python programming that can work around this problem to help avoid it. I do not know of a solution for the RealSense Viewer program though.
If you use the rs-multicam example program with your D435i (which works with one camera despite the program's name) then it usually displays depth, RGB and IMU panels without the problems that occur in the RealSense Viewer when having all three stream types active.
-
Hi MartyG,
Thank you for the reply.
Just to give a summary as to what I am trying to achieve here. I want the aligned color depth image and the IMU data together for my application. So I looked into the rs-align code and was able to get the color and depth frame aligned. I use ROS for communication and publish this image on a topic: I tried to get the IMU data as well and publish it on another topic, the data comes in for a few frames but then it stops and the topic freezes. I know that the rate of IMU data is greater than the color-depth frame, hence I use a callback to get this data and use poll_for_frames for the color-depth data. So I tried to check if this is an intrinsic problem and hence checked on the realsense viewer and found the problem I reported.
I tried the rs-multicam example as you mentioned, but even this gets stuck after 30 mins or so. You mentioned that there are other methods. Can you please let me know what these are? Or any other way that can help?
Thank you in advance for the help.
Kany.
-
In Python programming, separating color / depth and IMU into two separate pipelines (one for color and depth and the other dedicated to IMU) usually helps.
https://github.com/IntelRealSense/librealsense/issues/5628#issuecomment-575943238
A RealSense user with the same depth-color-IMU problem who tried the multiple pipeline technique in Python found that it did not work for them and they used callbacks instead.
https://github.com/IntelRealSense/librealsense/issues/6370
My experience with this subject is that Python provides the most reliable results in regard to depth-color-imu simultaneously streaming.
If you are using ROS and the RealSense ROS wrapper, you could align all images to the depth image by adding this term to your roslaunch statement:
align_depth:=true
The RealSense ROS wrapper documentation provides an example use of this flag:
https://github.com/IntelRealSense/realsense-ros#aligned-depth-frames
You can also unite the ROS accel and gyro messages together into a single 'imu' topic using this roslaunch flag:
unite_imu_method:=linear_interpolation
To quote from the RealSense ROS documentation:
**************
unite_imu_method: The D435i and T265 cameras have built in IMU components which produce 2 unrelated streams: gyro - which shows angular velocity and accel which shows linear acceleration. Each with it's own frequency. By default, 2 corresponding topics are available, each with only the relevant fields of the message sensor_msgs::Imu are filled out. Setting unite_imu_method creates a new topic, imu, that replaces the default gyro and accel topics. The imu topic is published at the rate of the gyro. All the fields of the IMU message under the imu topic are filled out.
linear_interpolation: Every gyro message is attached by the an accel message interpolated to the gyro's timestamp.
copy: Every gyro message is attached by the last accel message.
-
Hi MartyG,
Okay so i tried to use the parameter align_depth:= true with the roslaunch and I manage to retrieve color and depth aligned, but then the rest of my application doesn’t work anymore:
I’m trying to generate a segmentation of the color stream based on a person detection algorithm, I retrieve the depth data of the center of the bounding box of the person detected in the depth topic and look at the neighboring pixels to generate the blop of the just the person detected from the color topic, but this part doesn’t work with the those topics, I receive the flag “ process has died “ right from the start with no explanation of the why and the log file concerning the crash doesn’t appear in the /.ros folder.
Also after trying to set the parameter unite_imu_method I received the topic camera_imu but not data coming from the IMU, not sure what cause the issue.
On another hand I also manage to retrieve an alignment of color and depth by taking inspiration from the rs-align code, with this method the rest of my application work and the segmentation doesn’t crash, I use a callback in my pipeline to manage to retrieve the frames and the IMU data at the same time, which work fine, then I get angles information from IMU with a function inspired by rs-motion; after around 30 minutes of running the angles information get stuck with no error and I just receive the same values continuously, would there be a way to “reset” only the IMU data without having to reset everything ?
Or do you have any other suggestions that could lead me to something
Thank you
Tchikaya Kany
-
If ROS was being used to access the IMU data, you could try unsubscribing from the IMU topic and then re-subscribing to it.
If librealsense was being used, you could put IMU on a separate pipeline from the pipeline that depth / color is on.
In regard to ROS IMU messages when using unite_imu, the link below is a good reference source:
-
Hi MartyG,
Thank you for the reply.
I tried the two separate pipelines suggestion but it did not give me good results. I could get the different streamq together but the rate of the IMU data was slow. I want the frames for color and depth at their own rate and the IMU frame at its own rate. Do you have any idea on this?
I tried the two pipelines in CPP code and in the link you provided the code is in python. Do you have any resources for this in CPP?
Please let me know. Thank you for the help in advance.
Kany.
-
There was a C++ case in the link below that featured a slow IMU rate when using callbacks. A fix for the problem was turning off global time for the motion sensor.
motionModule.set_option(RS2_OPTION_GLOBAL_TIME_ENABLED, 0);
-
Hi K Tchikaya,
Did you find a solution for this problem?
It would be really helpful if you could share your experience.
Thanks -
Hi Aribeiro When problems are encountered when IMU, depth and color are all enabled at the same time, the two main methods for coding a workaround are to create 2 separate pipelines or to use callbacks. The majority of references for doing so are in Python language, and the small number of references about this particular subject that are available for C++ focus primarily on the callbacks method.
The link below contains a C++ script shared recently by a RealSense user for setting up IMU, depth, infrared and color streams using callbacks.
https://support.intelrealsense.com/hc/en-us/community/posts/1500001145522/comments/1500000962462
-
Hi MartyG,
Thank you for the fast reply. However I am not using IMU. I use the Intel Realsense D435 camera module with the Odroid-XU4 running Ubuntu 18.04.
The acquisiton of depth and color gets stuck after 30 mins or so, as K Tchikaya mentioned. Can you help me with this problem?
Thank you in advance,
Alexandrine Ribeiro
-
The issue that you are experiencing may be different to the one that K Tchikaya had if the IMU is not involved. I will handle your question at the RealSense GitHub case that you have posted. Thanks!
Please sign in to leave a comment.
Comments
11 comments