D415 Projection Matrix
I am currently using a D415 camera and the pyrealsense2 library for development. For my current project, I need to display the orientation of an external object on the 2D color image, in a manner very similar to the co-ordinate system displayed in SOLIDWORKS or any other 3D cad software. My current implementation does not take the camera projection into consideration. My question is, if I know the orientation of an object with respect to the camera's coordinate system, is there a simple method of determining the projection matrix to calculate that object's orientation with respect to the color image coordinate system? Any advice is greatly appreciated, thank you.
-
Hi Gfrank RealSense-compatible solutions for obtaining the pose (angle and position) using ROS are available, like the examples of Python pose projects below.
Obtaining pose by object color
http://by-the-w3i.github.io/2019/10/06/ColorBlockTracking
Obtaining pose by object recognition
https://github.com/avinashsen707/AUBOi5-D435-ROS-DOPE
https://github.com/pauloabelha/Deep_Object_Pose
If you would prefer not to use ROS to obtain pose, an alternative is to use a SolvePNP algorithm in OpenCV.
https://docs.opencv.org/master/d7/d53/tutorial_py_pose.html
And if you simply require the angle of a pixel, you may be able to accomplish that with the method below:
https://github.com/IntelRealSense/librealsense/issues/5553#issue-543234625
-
Hi Marty, Thanks so much for your quick response and all the great references. In my application, I am actually trying to do the inverse of this. In my project I have an object that is currently outside of the field of view, but will enter the image at a known position and orientation, and I want to display this pose to the operator before the movement. The pose is known with respect to the camera's coordinate system. In other words, I want to transform the known pose in 3D camera coordinates to the 2D color image coordinates.
-
I considered your question carefully. Rather than track the offscreen pose of the object in real-time, I wonder if it would be easier just to print the known pose as a plain text overlay on the image at the area where the object will first enter if the entry location is always the same. You may be able to do so with the cv2.PutText instruction. An example of its use in a librealsense Python script to overlay text on a color image is highlighted in the link below, on the bottom two lines of the script.
-
The object is a tool that the operator is controlling remotely. I want to be able to display the target pose (which is constantly changing) of the tool such that the operator gets visual feedback that the correct motion has been achieved. I can display the target position easily, but due to the 2D nature of the color image, I am having trouble displaying the known orientation, so am looking for away to 'project' the known 3D orientation into the image coordinate system. Essentially, I want to draw something like a line or shape on the image that will line up with the tool if the target pose has been achieved. I am currently able to track the target pose in the 3D camera coordinate system. So the flow would be something like this:
1. Tool is in a 'home' position. Determine target tool pose to accomplish a task.
2. Display a line, or other shape, on the 2D color image with the target pose position and orientation.
3. Tool moves to target position. Operator gets visual feedback that the correct motion took place since the line/shape lines up with the tool on the image.
4. Tool does task then returns to home position. Repeat process for new target pose.
In the program flow, I am only struggling with task 2, all other elements are working as intended.
-
Perhaps something similar to the 'virtual object' drawn on screen in the rs-ar-basic example program for the RealSense T265 tracking camera.
-
You are very welcome. It's great to hear that the example is a useful reference for your project. Please do feel free to ask further questions if you have any.
C++ source code for this example is provided at the link below. As you currently have a D415, I should emphasize of course that the example is for the RealSense T265 model only, though you should be able to gain some useful information about rendering colored lines to represent the pose as a virtual object.
Good luck!
https://github.com/IntelRealSense/librealsense/tree/master/examples/ar-basic
Please sign in to leave a comment.
Comments
7 comments