Filters for depth images in D435 camera
I have some inquiries regarding the filters:
1- I need to know if I make a streaming from the camera without applying filters or specify any filters in the code. The saved depth images will be filtered with the default values or should I specify it in the code?
2- When using visual preset, is this mean that there is a predefined filters applied to my depth image or should I use with it some filters also to enhance the quality?
3- Is there a threshold filter for RGB as in the depth or not?
4- I need to apply some filters to my depth; should I filtered the frames first then align them to the color frame or what should I do?
-
Hi Jomanaashraf8 Thanks very much for your questions.
1. Whilst the RealSense Viewer tool applies a range of filters by default, when creating your own program script no filters will be applied by default. Filtering must be deliberately programmed into your script.
2. A visual preset applies specific values to a range of camera settings to affect the image in a particular way depending on how the various settings in the preset are balanced against each other. For example, the High Accuracy preset prioritizes depth accuracy at the cost of a reduced amount of detail on the depth image, whilst Medium Density is a good balance between accuracy and the amount of depth detail on the image. You can use both a visual preset and filtering if you wish.
3. There is not a threshold filter for RGB, though if you align depth and color then defining a depth threshold will cause the RGB information to also reduce or increase. The RealSense SDK's rs-align-advanced C++ example program provides a demonstration of this principle.
https://github.com/IntelRealSense/librealsense/tree/master/examples/align-advanced
4. It is recommended to apply post-processing filters first and alignment afterwards.
-
If I have this code, I put the post processing first and then the alignments:
#Apply the filters to depth frames
filtered_depth=decimation.process(depth_frame)
filtered_depth=depth_to_disparity.process(filtered_depth)
filtered_depth=spatial.process(filtered_depth)
filtered_depth=temporal.process(filtered_depth)
filtered_depth=disparity_to_depth.process(filtered_depth)
filtered_depth=hole_filling.process(filtered_depth)
filtered_depth=threshold_filter.process(filtered_depth)
aligned_frames = align.process(frames)
aligned_depth_frame = aligned_frames.get_depth_frame()
color_frame = aligned_frames.get_color_frame()
depth_image=np.asanyarray(filtered_depth.get_data())
color_image=np.asanyarray(color_frame.get_data())Is the aligned_frames will process the filtered_frames or will take the frame of the depth from the camera without any filters?
Can you tell me if this code is right or the depth frames are not being filtered?
-
Yes, you can apply the preset with code. The link below has an example of Python code for applying the High Accuracy preset.
https://github.com/IntelRealSense/librealsense/issues/2577#issuecomment-432137634
-
I am a little bit confused about the threshold so, if I made the simple rs.align with setting the threshold of the depth frame to remove the background as in the following code; Is the background of the color frame will be removed also?
threshold_filter = rs.threshold_filter()
threshold_filter.set_option(rs.option.max_distance,max_threshold)
threshold_filter.set_option(rs.option.min_distance,min_threshold)
try:
whileTrue:
# Get frameset of color and depth
frames=pipeline.wait_for_frames()
#Apply the filters to depth frames
filtered_depth=decimation.process(frames.get_depth_frame())
filtered_depth=depth_to_disparity.process(filtered_depth)
filtered_depth=spatial.process(filtered_depth)
filtered_depth=temporal.process(filtered_depth)
filtered_depth=disparity_to_depth.process(filtered_depth)
filtered_depth=hole_filling.process(filtered_depth)
filtered_depth=threshold_filter.process(filtered_depth)
# Align the depth frame to color frame
aligned_frames=align.process(frames)
# Get aligned frames
aligned_depth_frame=aligned_frames.get_depth_frame()
color_frame=aligned_frames.get_color_frame()If the answer of the previous question is no; Is there a python code for rs-align-advanced which is demonstrated in the following link to remove the background for both RGB and Depth?
https://github.com/IntelRealSense/librealsense/tree/master/examples/align-advanced
Also I found this code but it is in c++ is there any example of it in python?
https://github.com/IntelRealSense/librealsense/tree/master/wrappers/opencv/grabcuts
-
rs-align-advanced does not use the Threshold Filter. Instead, it uses a depth_clipping_distance value to strip away the background, removing both the color and depth detail.
I believe though that using a Threshold Filter in rs-align should have the same background removing effect as the clipping method used in rs-align-advanced
The RealSense SDK Python alignment example program align_depth2color.py also has a background clipping mechanism.
I do not know of a direct equivalent in Python to rs-grabcuts that cuts out the background whilst leaving a particular part of the image remaining.
-
The threshold filter's max distance refers to the distance in real-world meters beyond which depth information will be excluded from the depth image. So if the max distance is set to 16 then no depth information will be removed, as it is saying that the camera can render depth detail for as far as it is able to see. A better test value would be '4' meters, which is the threshold filter's default value. When the max distance is 4, depth detail that is more than 4 meters from the camera will be excluded from the depth image.
If you are using a program with a clipping distance mechanism, like the one in align_depth2color.py, then you should not need to have a threshold filter in the program too. So please comment out the line below to prevent the threshold filter from being applied in order to see what effect it has.
filtered_depth = threshold_filter.process(filtered_depth)
-
Thanks a lot I deleted this part as you mentioned.
but I need to ask; Now, in my code as shown here, I saved the both images which is callled "bg_removed_color" and "bg_removed_depth" .
I made to remove the background to put the pixel that is far from the clipped distance to be white in both color and depth; My question is the depth image will be affected if i put it white or should I put it as the example grey color?
# Create a pipeline pipeline = rs.pipeline() #Create a config and configure the pipeline to stream # different resolutions of color and depth streams config = rs.config() color_profiles, depth_profiles = get_profiles() if args.record_imgs or args.record_rosbag: # note: using 640 x 480 depth resolution produces smooth depth boundaries # using rs.format.bgr8 for color image format for OpenCV based image visualization print('Using the default profiles: \n color:{}, depth:{}'.format( color_profiles[0], depth_profiles[0])) w, h, fps, fmt = depth_profiles[0] config.enable_stream(rs.stream.depth, w, h, fmt, fps) w, h, fps, fmt = color_profiles[0] config.enable_stream(rs.stream.color, w, h, fmt, fps) if args.record_rosbag: config.enable_record_to_file(path_bag) if args.playback_rosbag: config.enable_device_from_file(path_bag, repeat_playback=True) # Start streaming profile = pipeline.start(config) depth_sensor = profile.get_device().first_depth_sensor() color_sensor = profile.get_device().query_sensors()[1] color_sensor.set_option(rs.option.white_balance, wb_value) # Using preset HighAccuracy for recording if args.record_rosbag or args.record_imgs: depth_sensor.set_option(rs.option.visual_preset, preset) # Getting the depth sensor's depth scale (see rs-align example for explanation) depth_scale = depth_sensor.get_depth_scale() # We will not display the background of objects more than # clipping_distance_in_meters meters away clipping_distance_in_meters = clipped_distance # 1 meter clipping_distance = clipping_distance_in_meters / depth_scale # Create an align object # rs.align allows us to perform alignment of depth frames to others frames # The "align_to" is the stream type to which we plan to align depth frames. align_to = rs.stream.color align = rs.align(align_to) #Define filters to depth farmes decimation = rs.decimation_filter() decimation.set_option(rs.option.filter_magnitude, 2) depth_to_disparity = rs.disparity_transform(True) spatial = rs.spatial_filter() spatial.set_option(rs.option.filter_magnitude, 2) spatial.set_option(rs.option.filter_smooth_alpha, 0.5) spatial.set_option(rs.option.filter_smooth_delta, 20) temporal = rs.temporal_filter() temporal.set_option(rs.option.filter_smooth_alpha,0.4) temporal.set_option(rs.option.filter_smooth_delta,20) disparity_to_depth = rs.disparity_transform(False) hole_filling = rs.hole_filling_filter() # Streaming loop frame_count = 0 try: while True: # Get frameset of color and depth frames = pipeline.wait_for_frames() #Apply the filters to depth frames filtered_depth = decimation.process(frames.get_depth_frame()) filtered_depth = depth_to_disparity.process(filtered_depth) filtered_depth = spatial.process(filtered_depth) filtered_depth = temporal.process(filtered_depth) filtered_depth = disparity_to_depth.process(filtered_depth) filtered_depth = hole_filling.process(filtered_depth) # filtered_depth = threshold_filter.process(filtered_depth) # Align the depth frame to color frame aligned_frames = align.process(frames) # Get aligned frames aligned_depth_frame = aligned_frames.get_depth_frame() color_frame = aligned_frames.get_color_frame() # Validate that both frames are valid if not aligned_depth_frame or not color_frame: continue depth_image = np.asanyarray(aligned_depth_frame.get_data()) color_image = np.asanyarray(color_frame.get_data()) # Remove background - Set pixels further than clipping_distance to white white_color = 255 #depth image is 1 channel, color is 3 channels depth_image_3d = np.dstack((depth_image, depth_image, depth_image)) bg_removed_color = np.where((depth_image_3d > clipping_distance)|\ (depth_image_3d <= 0), white_color, color_image) bg_removed_depth = np.where((depth_image > clipping_distance) | (depth_image <= 0), white_color, depth_image) if args.record_imgs: if frame_count == 0: save_intrinsic_as_json( join(args.output_folder, "camera_intrinsic.json"), color_frame) cv2.imwrite("%s/%06d.png" % \ (path_depth, frame_count), bg_removed_depth) cv2.imwrite("%s/%06d.jpg" % \ (path_color, frame_count), bg_removed_color) print("Saved color + depth image %06d" % frame_count) frame_count += 1 # Render images depth_colormap = cv2.applyColorMap( cv2.convertScaleAbs(bg_removed_depth, alpha=0.09), cv2.COLORMAP_JET) images = np.hstack((bg_removed_color, depth_colormap)) cv2.namedWindow('Recorder Realsense', cv2.WINDOW_AUTOSIZE) cv2.imshow('Recorder Realsense', images) key = cv2.waitKey(1) # if 'esc' button pressed, escape loop and exit program if key == 27: cv2.destroyAllWindows() break finally: pipeline.stop()Also, The saved images are right or not from what I understood that this two line of codes will remove the background as the clipping distance so it was implemented on the color image and I added the other line to have also the depth background removed.
So this is right or there is something I am missing?
bg_removed_color = np.where((depth_image_3d > clipping_distance)|(depth_image_3d <= 0), white_color, color_image)
bg_removed_depth = np.where((depth_image > clipping_distance) | (depth_image <= 0), white_color, depth_image) -
It will be no problem to color the furthest depth and color pixels as white.
align_depth2color.py only has a removal line for the depth array.
bg_removed = np.where((depth_image_3d > clipping_distance) | (depth_image_3d <= 0), grey_color, color_image)
It looks as though the color image array (named color_image) is accessed in that instruction as part of the grey_color, color_image commands on the end of the line. So you may not need to have a bg_removed_color line.
Please sign in to leave a comment.
Comments
11 comments