View my account

D435/D435i on robot: how should camera angles be measured in CAD, and what could cause repeated false near points?

Follow
Jorge Casatejada Santamarta
  • Edited

Hello,

I am working with multiple Intel RealSense cameras mounted on a mobile robot and I would like to ask for guidance on two related topics:

  1. how the camera angles should be measured theoretically in CAD

  2. possible causes of repeated false near points in the depth image / point cloud

Setup

  • Cameras installed on the robot: 3x D435 and 1x D435i

  • Two cameras mounted at the front and two at the rear

  • Main robot reference frame: base_link, located at the center of the robot

  • The camera I am mainly analyzing is the front-left camera

  • Software:

    • librealsense 2.56.5

    • RealSense Viewer 2.57.6

    • ROS2 Humble

    • ros-humble-librealsense2 2.56.4

  • Firmware updated to the latest available version through RealSense Viewer

Frame convention used

For the CAD analysis, I use:

Robot frame

  • X: forward

  • Y: left

  • Z: up

Camera frame

  • X: forward, in the camera viewing direction

  • Y: left

  • Z: up

For the RealSense reference frame in ROS, I am using camera_link, following the RealSense ROS wrapper frame convention:
https://github.com/realsenseai/realsense-ros?tab=readme-ov-file#ros2robot-vs-opticalcamera-coordination-systems

1) Theoretical orientation from CAD vs real orientation from IMU

For the front-left camera, I computed the theoretical orientation from CAD and compared it against the real orientation measured from the IMU.

Theoretical orientation from CAD

  • roll = 97.7°

  • pitch = -16.64°

  • yaw = -26.09°

Real orientation measured from IMU

  • roll = 96.6°

  • pitch = -14.49°

  • yaw = -27.5°

The main discrepancy is in pitch.

At the moment, I am deriving the CAD angles using plane comparisons:

  • roll: robot XY plane vs camera XY plane

  • pitch: robot XY plane vs camera XZ plane

  • yaw: robot XZ plane vs camera XY plane

To validate the real orientation, I do not rely only on IMU values:

  • I also check in RViz that overlapping point clouds from different cameras align correctly in their common area

  • I compare the resulting point clouds against LiDAR information as an external reference

My main question here is:

What is the correct way to measure roll, pitch and yaw in CAD for a RealSense camera mounted on a robot?

More specifically:

  • Should these angles be measured directly from the camera axes relative to the robot axes?

  • Is my current plane-comparison method valid?

  • If not, how should the angles be measured correctly in CAD so that they are consistent with the real pose obtained from the IMU?

I would especially like to understand whether I may be measuring pitch incorrectly in CAD.

2) Repeated false near points in depth image and point cloud

I am observing a repeated issue in one specific area of the environment:

  • points that should be approximately 4 to 5 meters away

  • always appear at around 2 meters from the camera

Important observations:

  • the false near points appear repeatedly in the same area

  • I can see the issue both in RealSense Viewer without post-processing and in RViz / ROS2

  • therefore, I currently do not think this is caused by post-processing filters

  • I believe the effect appears in both the raw depth image and the point cloud

  • when I set the camera pitch to 0°, I do not observe those false near points

Scene description

  • the camera is looking toward a door

  • there are metal bars on the ceiling

  • we tried covering different possible interference sources, but the noise could not be eliminated

Because the problem depends strongly on orientation, I am unsure whether the cause is:

  • an incorrect rotation / transform

  • wrong frame selection

  • reflective or shiny surfaces

  • multipath / false depth

  • some scene-specific stereo matching artifact

  • or another known D435 / D435i limitation

3) Large mounting rotation

The approximate real orientation of this front-left camera is:

  • roll = 96.6°

  • pitch = -14.49°

  • yaw = -27.5°

Is this kind of mounting expected to work normally for depth and point cloud generation, assuming the transforms are correct?

Or is there any known degradation in D435 / D435i depth quality when the camera is mounted with relatively large roll / pitch / yaw values?

Material I can provide

If useful, I can provide:

  • CAD screenshots

  • TF tree

  • camera mounting images

  • raw depth captures

  • point cloud screenshots

  • exact transforms and frame definitions

Any guidance on:

  • how the angles should be measured correctly in CAD

  • whether my CAD method is valid

  • and the likely explanation for these repeated false near points

would be very helpful.

Thank you.

0

Comments

4 comments

Date Votes
  • MartyX Grover

    Hi Jorge Casatejada Santamarta I am not familiar with calculating angles theoretically in CAD.  However, the RealSense SDK makes use of a plane-fit algorithm to work out the angle between the camera and the ground without requiring an IMU, which is important if three of your four cameras are the IMU-less D435 model. The plane-fit algorithm RealSense uses is based on one at the link below.

    https://www.ilikebigbits.com/2015_03_04_plane_from_points.html

     

    This algorithm is also built into the SDK's Depth Quality Tool program for analyzing the depth quality of a RealSense camera.

     

    In regard to the camera frame axes, RealSense uses a different convention to the usual one, where X is left, Y is up and Z is forward.  These are described here:

    https://github.com/realsenseai/librealsense/blob/master/doc/d435i.md#sensor-origin-and-coordinate-system

     

    In the RealSense ROS wrapper, when the data is outputted these axes are automatically converted to the ROS standard (X-forward, Y-left, Z-up).

     

    In regard to roll-pitch-yaw, only roll and pitch can be extracted from a RealSense camera's IMU.

    https://github.com/realsenseai/librealsense/issues/4391#issuecomment-510578991

     

    As long as your camera's tilt pitch is not greater than +- 30 degrees then the camera's depth calculations should not be negatively affected, so it should be okay if the retrieved pitch is only a couple of degrees different from the theoretical one.  Unforeseen issues in the image may be introduced if the tilt pitch exceeds 30 degrees.

     

    In regard to the 96 degree roll, which would put the camera oriented vertically on its side I believe, RealSense camera work fine on a 90 degree roll orientation, so 96 should be okay.  However, it is important that the image is not rotated 96 degrees in the opposite direction to put it in the normal orientation, as all the calibration values would be incorrect.  You can though rotate the image orientation safely using the post-processing rotation filter, which will put the image in the straight-ahead orientation whilst also correctly updating the calibration parameters and other data to take account of the rotation.

     

    If there are a repeating series of similar looking metal bars on the ceiling then this could introduce a repetitive pattern and confuse the depth sensing.  There is a guide below about mitigating repetitive patterns.

    https://dev.realsenseai.com/docs/mitigate-repetitive-pattern-effect-stereo-depth-cameras

     

    Reflectivity of surfaces can also make those surfaces more difficult to accurately read.  Fitting a thin-film inexpensive linear polarizing filter over the lenses on the front of the cameras should greatly reduce the negative effect of glare from reflections and make those surfaces much easier for the camera to analyze.  You can find polarizing filters cheaply in stores such as Amazon by searching for linear polarizing filter sheet.

     

    I would add that with the D435 / D435i camera models, it is expected that depth accuracy will start to noticeably drift beyond 3 meters from the camera.  This is a phenomenon called RMS Error, where error increases linearly as distance from the camera increases.  The D455 type cameras provide 2x the accuracy over distance of the D435 type cameras, though I appreciate that it may not be practical to change the cameras.

    0
    Comment actions Permalink
  • MartyX Grover

    I especially emphasize that you should be using Z as your depth-forward value instead of X.

    0
    Comment actions Permalink
  • Jorge Casatejada Santamarta
    • Edited

    Hi MartyX Grover ,

    Thank you very much for the detailed reply.

    Regarding the plane-fit algorithm, thank you for mentioning it. I did not know about that approach, so I will definitely keep it in mind, especially because three of my four cameras are D435 models without IMU.

    Regarding the camera frame axes, thank you as well. When I have the chance, I will try to repeat the measurements using those RealSense axes and check whether that is part of the discrepancy I am seeing.

    Regarding roll, pitch and yaw from the IMU, thank you. Yes, I was already aware that only roll and pitch can be obtained directly from the IMU. In my case, the yaw value was estimated manually.

    Thank you also for the information about pitch and roll limits. I did not know that this was considered acceptable, so it is very useful for me in order to better judge whether my issue could be related to the camera mounting angles or not.

    Regarding the metal bars and possible reflectivity issues, I will try both suggestions when I can:

    • checking the repetitive pattern effect from the ceiling bars

    • testing linear polarizing filters

    Once I do that, I will report back with the results.

    Regarding the distance-related accuracy drift of the D435 / D435i, I also have a D455 available for testing, and it seems that this specific issue:

    • points that should be approximately 4 to 5 meters away

    • appearing at around 2 meters from the camera

    either disappears or is significantly reduced when using the D455.

    So that may also be an important clue.

    Thank you again for all the information. It is very helpful.

    0
    Comment actions Permalink
  • MartyX Grover

    You are very welcome.  I'm pleased that I could help.

     

    Please do update me with your test results and let me know about any further questions that you have.  Thanks again!

    0
    Comment actions Permalink

Please sign in to leave a comment.

Didn't find what you were looking for?

New post