Matching the Depth and RGB Frames in D435 in Matlab
I have extracted the RGB and depth Camera data from a D435 camera. I am tracking some points on the RGB frames and would like to do track the corresponding points in the Depth frames. As I tried to find the mathematical equations, I got there is no linear relationship between them and the depth camera has a wider lense. Is there any way I align/match them in Matlab?
-
Hi Pc Mehdi Heidarian The links below provide examples of scripting for performing depth-color alignment in the RealSense MATLAB wrapper.
RealSense MATLAB wrapper
https://github.com/IntelRealSense/librealsense/tree/master/wrappers/matlab
Depth-color alignment
https://github.com/IntelRealSense/librealsense/issues/9773#issuecomment-925553900
https://github.com/IntelRealSense/librealsense/issues/3735
Depth-color alignment and generating a pointcloud
https://github.com/IntelRealSense/librealsense/issues/6763#issuecomment-657217234
-
Hi MartyG,
I am sorry but I am a little bit lost, I can not find any help on how to start running/working with Matlab Wrapper!! The guide in the following page is not clear enough!!
https://github.com/IntelRealSense/librealsense/tree/master/wrappers/matlab
-
I hope that the guidance at the link below will be helpful to you.
https://github.com/IntelRealSense/librealsense/issues/6014#issuecomment-1003577992
The instructions in this link depend upon having first installed the full RealSense SDK on Windows using the automated installer file on the SDK 'Releases' page, as described here:
https://github.com/IntelRealSense/librealsense/issues/6014#issuecomment-1003575399
-
Once you have copied the +realsense folder to a location where MATLAB can find it or have added the folder to the MATLAB search path then you can start testing the MATLAB wrapper's example programs. These example scripts, which are available at the main MATLAB wrapper page, have 'example' in their name.
https://github.com/IntelRealSense/librealsense/tree/master/wrappers/matlab
advanced_mode_example.m
capture_example.m
depth_example.m
depth_view_example.m
pointcloud_example.m
rosbag_example.m
RealSense MATLAB scripts are almost 1:1 with the SDK's C++ language, with some differences. So the example programs are a very good place to start in learning RealSense MATLAB wrapper scripting.
Once you feel ready to write your own wrapper script then I recommend posting another question on this forum about which particular function you need information about, as information about MATLAB wrapper scripting is harder to find than with other SDK programming languages such as C++ and Python.
https://github.com/IntelRealSense/librealsense/issues/6014#issuecomment-1004138679
-
Thank you MartyG,
Now, I've got the SDK Matlab wrapper is not like the LabView wrapper which has a graphic feature too. Actually, the Matlab wrapper consists of some scripts which help in the coding.
I am sorry about my first question. In my case, I need to find the corresponding pixels in the Depth and RGB frames. For example, what is the coordination (X, Y) in the depth camera which is corresponding to a special pixel's coordination (x,y) in the RGB camera? I have checked the align.m script and the Depth-Color alignment codes you referenced. It seems they are mostly trying to synchronize the Depth and RGB frames if I am not wrong.
Please, let me know if I can use the provided scripts in my work or if there is any similar topic.
Many thanks
-
Don't worry about the first question, it is no trouble at all. :)
Yes, the align operation aligns depth data to color data. It automatically resizes the field of view (FOV) size of the depth stream to match the FOV size of the color stream. As the D435 and D435i models have a smaller color sensor FOV than the depth sensor FOV, this can cause the outer regions of the depth information to be excluded from the aligned image.
I researched your question carefully about retrieving XY coordinates but could not find much information for doing so with MATLAB. Instead of using align, you could consider generating a 3D RGB pointcloud and then retrieving its vertices with the instruction get_vertices. You can then read the stored vertices to retrieve values for the individual axes (X, Y and Z).
There is an example of a MATLAB script in the link below for generating an aligned RGB point cloud and storing its vertices with get_vertices().
https://github.com/IntelRealSense/librealsense/issues/6763#issuecomment-657217234
-
Thanks for the links. I think you mean I should use projection/deprojection.
In my project, I have developed a tracking algorithm that will process the RGB frames. Meanwhile, I need to have the Depth feature of tracked pixels. So, I am not sure if the (x,y) in the RGB frame is the same as (X,Y) in the generated 3D RGB point cloud. Do you think they are same?
By the way, I am gonna give it a shot. Any help in generating the 3D RGB pointcloud from a .bag file?
-
Separate stream types have their own 3D coordinate system with physical sensor positions that differ from each other. You can align depth coordinates with other types of coordinate such as RGB though. The positional and rotational relationships between the physical sensors are described by the camera's extrinsics. The documentation in the link below provides detailed information about this.
https://dev.intelrealsense.com/docs/projection-in-intel-realsense-sdk-20
In regard to obtaining xy from xyz, I was suggesting generating a 3D pointcloud (for example, by deprojecting 2D pixels to 3D points) and then retrieving the x and y real-world values from those 3D coordinates. If you prefer 2D pixel coordinates instead of 3D world coordinates then it is possible to project 3D world coordinates of a point cloud back to 2D pixel coordinates though.
The two most common ways of generating a 3D pointcloud in the RealSense SDK and obtaining xyz coordinates are:
1. By using the pc.map_to() SDK instruction to map between depth and color to generate a textured 3D point cloud and then use the points.get_vertices() instruction to obtain the xyz values of the pointcloud. This method does not involve deprojection and can be more accurate when projecting 3D back to 2D pixels, as using depth-color alignment may introduce distortion into results.
2. Perform depth-color alignment with the align_to() instruction and then use deprojection to produce the 3D cloud with the instruction rs2_deproject_pixel_to_point
If you plan to use the MATLAB wrapper then the wrapper's pointcloud example can be adapted to use a recorded bag as the data source instead of a live camera. Details of how to do this are in the link below.
https://github.com/IntelRealSense/librealsense/issues/3241#issuecomment-462496753
-
Hello MartyG,
I checked the link you sent in your last email. I believe it is the same way as mine but it seems two below links which are referred in the issue list are not available anymore in the intel forums.
https://forums.intel.com/s/question/0D70P0000069ICXSA2
https://forums.intel.com/s/question/0D70P000006ANoxSAG
I tried to do write my script for them but did not go well, just wondering if there is any other link to follow it?
Also, can you let me know which one of Matlab wrappers can help me to record RGB, Depth and point cloud?
-
It is not possible to locate the new addresses of these particular old cases using only those links. I researched a list of MATLAB point cloud related cases. The most relevant link was one that you may already have seen.
https://community.intel.com/t5/Items-with-no-label/MATLAB-RGB-point-cloud/td-p/584717
These links may also be helpful:
https://github.com/IntelRealSense/librealsense/issues/10185
https://github.com/IntelRealSense/librealsense/issues/8368#issuecomment-779849034
A point cloud cannot be recorded into a bag file that was recorded in the RealSense SDK. It can store the individual depth and RGB streams. You then load the bag file and generate the point cloud from the bag data in real-time.
-
Hello MartyG,
I am going to run the pointcloud_example.m function by a .bag file instead of using D435 camera. According to the post below, I edited the function but I face errors.
https://github.com/IntelRealSense/librealsense/edit/master/doc/sample-data.md
The post has said to replace pipe.start() with
rs2::config cfg; cfg.enable_device_from_file(<filename>); pipe.start(cfg); // Load from file
and my current code is:
function pointcloud_example()
% Make Pipeline object to manage streaming
pipe = realsense.pipeline();
pointcloud = realsense.pointcloud();% Start streaming on an arbitrary camera with default settings
%profile = pipe.start();
rs2::config cfg;
cfg.enable_device_from_file('20220224_145101.bag');
pipe.start(cfg); % Load from filefigure('visible','on'); hold on;
figure('units','normalized','outerposition',[0 0 1 1])% Main loop
for i = 1:10000% Obtain frames from a streaming device
fs = pipe.wait_for_frames();
% Select depth frame
depth = fs.get_depth_frame();
%color = fs.get_color_frame();% Produce pointcloud
if (depth.logical())% && color.logical())%pointcloud.map_to(color);
points = pointcloud.calculate(depth);
% Adjust frame CS to matlab CS
vertices = points.get_vertices();
X = vertices(:,1,1);
x{i} = X;
Y = vertices(:,2,1);
y{i} = Y;
Z = vertices(:,3,1);
z{i} = Z;
plot3(X,Z,-Y,'.');
grid on
hold off;
view([45 30]);xlim([-0.5 0.5])
ylim([0.3 1])
zlim([-0.5 0.5])xlabel('X');
ylabel('Z');
zlabel('Y');
pause(0.01);
end
%pcshow(vertices); Toolbox required
end% Stop streaming
pipe.stop();
endI see the below error:
Error using realsense.pointcloud_example
Error: File: pointcloud_example.m Line: 10 Column: 9
Invalid use of operator.My question is:
1) How can I replace a bag file instead of the camera,
Also,
2) I am using Matlab2020b. is there any issue regarding it? I am asking this because I have faced errors for some simple functions too. Look at these 7 line codes that give me errors;
cfg = realsense.config();
% set resolution for RGB & DEPTH
cfg.enable_stream(realsense.stream.color,1920,1080,realsense.format.rgb8,30);
cfg.enable_stream(realsense.stream.depth,1024,768,realsense.format.z16,30);pipe = realsense.pipeline();
colorizer = realsense.colorizer();
pcl_obj = realsense.pointcloud();
profile = pipe.start(cfg);error is as below:
Error using librealsense_mex
Couldn't resolve requests
Error in realsense.pipeline/start (line 31)
out = realsense.librealsense_mex('rs2::pipeline', 'start', this.objectHandle, varargin{1}.objectHandle); -
The link below has scripting for modifying the pointcloud.m example in the MATLAB wrapper to access a bag instead of live data.
https://github.com/IntelRealSense/librealsense/issues/3241#issuecomment-462496753
-
Thank you MartyG,
I am trying to try two different ways to solve my issue. Currently, I have two different questions. Can you please reply my questions separately?
1) I have now the pointcloud from the bag file. I have also extracted the X,Y,Z by get_vertices but the values need to be interpreted. The obtained values are some numbers that are not in the range of real length, width, and depth. Is there any way I can get more realistic info from the points clouds?
2) I have tried to align the Depth and Color frames using align function. I have followed the links below but have faced errors in alignment. Can you please, let me know what the issue is?!
https://github.com/IntelRealSense/librealsense/issues/9773#issuecomment-925553900
https://github.com/IntelRealSense/librealsense/issues/3735
My code is as below:
cfg = realsense.config();
cfg.enable_device_from_file('20220225_154744.bag');
% set resolution for RGB & DEPTH
cfg.enable_stream(realsense.stream.color,1280,720,realsense.format.rgb8,30);
cfg.enable_stream(realsense.stream.depth,1280,720,realsense.format.z16,30);
% Make Pipeline object to manage streaming
pipe = realsense.pipeline();
pointcloud = realsense.pointcloud();
%Start streaming on an arbitrary camera with default settings %pipe.start();
pipe.start(cfg); % Load from file
fs = pipe.wait_for_frames();
% Select depth frame
depth = fs.get_depth_frame();
color = fs.get_color_frame();
align_to = color;alignedFs = realsense.align(align_to);
and the error is:
Error using realsense.align (line 7)
Expected input to be one of these types:realsense.stream, double, single, uint8, uint16, uint32, uint64, int8, int16, int32, int64
Instead its type was realsense.video_frame.
Error in realsense.alignkardan (line 26)
alignedFs = realsense.align(align_to); -
1. The get_vertices() method is an accurate way of generating a pointcloud. It is possible that there are factors in the real-world scene that may be affecting accuracy. Would it be possible to provide an RGB image of the scene that the camera is observing, as this can be helpful in identifying elements in the scene that may be causing disruption. This could include the following:
- Fluorescent lights such as ceiling strip lights, which flicker at frequencies that are hard to see with the human eye and can create noise on the image. Setting an RGB option called Power Line Frequency can help with this - typically '60' for North America and '50' for European regions.
- Objects / surfaces that are colored dark grey or black. It is a general physics principle that these colors absorb light, making them difficult for depth cameras to read depth information from. Projecting a stong light source onto such colors can help to bring out depth detail.
- Surfaces that have low texture or no texture, such as doors, walls and desks. Using the camera's in-built projector or an external projector to cast a pattern of dots onto objects in the scene that can be analyzed for depth detail can be helpful.
- Horizontal and / or vertical rows of repeating patterns in the scene, such as a horizontal row of fence posts or tree-tops, or a vertical stack of window blind slats. This is known as a repetitive pattern that can confuse the camera's depth sensing. A RealSense team member provides advice about reducing the effects of such patterns in the link below.
https://github.com/IntelRealSense/librealsense/issues/6713#issuecomment-651114720
2. Comparing your code against the two script links, I note that you are using align_to = color whilst the linked-to scripts both make their align_to instructions equal to realsense.stream.color;
You could therefore try changing that line in your script to:
align_to = realsense.stream.color;
Please sign in to leave a comment.
Comments
15 comments