Modify RGB frame pixels within .bag files and synchronize different bag streams
Hi,
Apologies for the wall of text. I tried to keep it short while providing all the relevant details. Also, big thanks to the whole RealSense community. The docs and support threads were very helpful so far.
Context: we recorded a large sample of Activities of Daily Living (ADL) with 5 RealSense D435 cameras (using C++ API on Windows 10), resulting in a bag file from each camera for each activity (10-30s long each). About 2.5TB in total. We are planning to publish this dataset, but we are required to blur the faces before that.
Problem 1: is it possible within the RealSense SDK to edit a recorded bag file’s RGB frames while keeping the rest of the bag streams & metadata intact?
I have searched around only to discover that editing a recorded bag file is not a commonly discussed topic in RealSense communities. Among other things I tried to open a pipe in playback and save to file mode, which of course resulted in errors.
In retrospect, we should have blurred the faces frame by frame as they were read from the sensor, and then record them to a bag file, but recording the whole dataset again is not an option.
We could export each bag's RGB frames with the rs-convert tool (and optionally stitch the frames onto video files), and then do the blurring, but we would lose all the metadata within the bag files, which we need for synchronizing all sensor streams (problem 2 of this post).
I’ve searched the RealSense SDK C++ API, GitHub issues and other forums for ways to read/open a bag, then edit a frame, and then overwriting just that frame in the bag file, but it looks like this feature is not supported. If I'm wrong, I'd appreciate some hints.
I will now try to use either the rosbag API or realsense-ROS Wrapper, but my ROS is rusty, and I have some concerns before fully committing. This is the approach I thought:
1) replay the "RealSense SDK bag" into several topics from node A;
2) listen to the RGB streams from node B, edit each frame (blur the faces), and republish the edited frames under a new topic;
3) from node C, record into a (new) "ROS bag file" all topics except the original RGB stream, while keeping the original timestamps.
Would this be the optimal approach, or do the rosbag API or realsense-ROS wrapper support directly modifying the underlying data (i.e., RGB frames)?
Also, IIRC the "ROS bag" format is not entirely compatible with the "RealSense SDK bag format" (as described at GitHub issue #3020 (comment) https://github.com/IntelRealSense/librealsense/issues/3020#issuecomment-452321766 or https://github.com/IntelRealSense/realsense-ros/issues/836 , so we would lose the ability to replay the "ROS bag" files from the RealSense Viwer, among other things. Is this correct?
Problem 2 (please advise if I should open a different issue): after blurring the faces, for each activity we want to synchronize the 5 RGB-D streams (plus two additional RGB streams recorded by two webcams, recorded in one MP4 file each), and merge them all in a single bag file, ideally synchronizing every frame from every stream to a single stream-frame-timestamp nearest to all. Since there is a lot of data involved, I thought of doing the blurring and merging on one go, without having to needlessly process the 2.5TB twice. I foresee that this feature is not supported in the SDK either, so I would again need to resort to the rosbag API andr RealSense-ROS wrapper. Does this make sense? If not, what would be the best approach for “merging” or synchronizing all streams?
-
Hi Bindeglied The contents of a bag file cannot be edited. So I agree that your best course of action would be to load the bag file into memory, edit the RGB images in real-time and then re-save the data to a new bag file.
You are also correct that bags recorded in ROS with rosbag record have some incompatibilities with playback in RealSense SDK applications such as the RealSense Viewer. Bags recorded in ROS can be played back in ROS with the rosbag play command.
The RealSense SDK cannot record multiple cameras to the same bag, so publishing the RGB of all RealSense and webcam cameras in ROS and then using rosbag record --all to record all currently active topics to the bag would seem to be the optimal option.
Alternatively, synching of recorded videos from RealSense and non-RealSense cameras has previously been achieved using the TIME_OF_ARRIVAL metadata timestamp, as described at the link below.
-
Hi MartyG Thanks a lot for your insights and hints. I'll certainly switch over to rosbag then.
But I still have one concern... Am I wrong to assume that doing rosbag play > [real-time editing of frames] > rosbag record would on its default configuration modify the "general" frame timestamps (not the TIME_OF_ARRIVAL)? My ROS knowledge is quite rusty now, so I'm just conjecturing.
-
I believe that the timestamps will be based on global time if the RealSense SDK's metadata support is enabled, otherwise they will be based on the computer's system time. More information about this can be found at the link below.
-
Hello Bindeglied !
I'm working on a similar project and I also need to blur faces (patients in a hospital) from already recorded .bag files.
Could you please explain to me how did you do this?:1) replay the "RealSense SDK bag" into several topics from node A;
2) listen to the RGB streams from node B, edit each frame (blur the faces), and republish the edited frames under a new topic;
3) from node C, record into a (new) "ROS bag file" all topics except the original RGB stream, while keeping the original timestamps.
I'm also new to ROS and RealSense and I'm having a hard time trying to create a new bag from an already existing one.
Thank you!
-
Hi Antoniogrimalt To create a new bag file from an existing one, you could try using the original bag as the data source instead of a live camera with the RealSense SDK's enable_device_from_file() instruction. A Python example for doing so can be found at the link below.
Then in the script, save the adjusted data to a new bag file using the enable_record_to_file() instruction.
In regard to detecting and blurring faces, this could be done with OpenCV code using a face detection algorithm such as a Haar Cascade and then blurring the detected faces with a gaussian filter. You can find relevant information about this approach by googling for the search term opencv face blur gaussian
-
Hi MartyG !
Thank you very much for the reply.
I had previously managed to read from a .bag and display its contents with opencv. What I have not been able to do yet is manage the pipelines so that what is being read can also be recorded.
Do you think I can do that without having to create nodes with ROS through another library (rospy or rosbag)? It would be great if I could do it all from pyrealsense2.
-
If you want to read and record at the same time then multi-threading may be what you need. The link below has resources about doing this with Python.
https://support.intelrealsense.com/hc/en-us/community/posts/4405523307027/comments/4405823606547
-
Hello MartyG
Thank you for the suggestion!
I have tried to understand, through the discussions you shared with me, how multithreading could help me achieve what I want, but I have not succeeded. If you could guide me on how to use it or share a working example with me, I would greatly appreciate it.
A question I have that could simplify things is: Do I need to use the RealSense library if in the end I decide to simply cut the sections in which the faces appear? Is it possible to create a trimmed version of an SDK .bag without RealSense?
-
Multithreading is rarely used in RealSense projects and so there are not RealSense examples of it for Python available except for the links above, unfortunately.
There is a bag cutting tool for ROS2 that does not require RealSense. Bag files created in the RealSense SDK have some differences to ones recorded in ROS though, so I cannot guarantee that SDK bags would be compatible with this tool.
https://github.com/AIT-Assistive-Autonomous-Systems/ros2bag_tools
-
Hi MartyG,
I realized that the bags are not exactly compatible when I tried to copy a segment of the original SDK .bag using rosbag and then the RealSense Viewer couldn't play it, white frames appeared for Color and Depth streams, and I got the console error:
20:30:09 [Error] C:\jenkins_sys_rsbuild\workspace\LRS_windows_compile_pipeline\src\media\playback\playback_device.cpp:537 - Failed to read next frame from file: Buffer Overrun
I've been considering using the ROS1 (I'm using Noetic on Windows) wrapper for RealSense, but I'm not sure I'll be able to achieve anything if the .bag I want to manipulate already exists. Is the wrapper intended only to display a camera in real time or could it also be used to manipulate SDK .bags?
Anyway, if I can't manipulate an SDK .bag with common ROS tools, I'm not sure if ROS will do me any good.
I'll try to use the cropping tool you shared with me, but I think I'll encounter the same compatibility issue.
-
Hi MartyG,
I also tried to do something like this to make it work just with pyrealsense2:
# Playback
playback_config = rs.config()
playback_pipeline = rs.pipeline()
playback_config.enable_device_from_file(file_name=str(input_bag_path), repeat_playback=False)
playback_config.enable_all_streams()
playback_profile = playback_pipeline.start(config=playback_config)
playback_device = playback_profile.get_device()
playback = playback_device.as_playback()
playback.set_real_time(False)# Recording
recorder_config = rs.config()
recorder_pipeline = rs.pipeline()
recorder_config.enable_record_to_file(file_name=str(output_bag_path))
recorder_profile = recorder_pipeline.start(config=recorder_config)
recorder_device = recorder_profile.get_device()
recorder = recorder_device.as_recorder()But I get the following error in the console:
recorder_profile = recorder_pipeline.start(config=recorder_config)
RuntimeError: No device connectedI found in the documentation for the config class: https://intelrealsense.github.io/librealsense/doxygen/classrs2_1_1config.html
enable_record_to_file()
This request cannot be used if
enable_device_from_file()is called for the current config, and vice versa. as available.And given that I can't use the same device as the playback, I have no idea how I could use these methods if the source of what is going to the recorded is another .bag file, and not a camera. Do you know if enable_record_to_file() could have another file as the device?
Another thing I would like to ask, since I haven't used GitHub much, is where do you think I should ask these types of code questions in the project's page? -
Are you attempting to use a bag file as the data source instead of a live camera and then re-record the data to a new bag file, please? If you are then you would not need to playback the bag with playback_device.as_playback(). Instead, just set the bag as the data source for the script with playback_config.enable_device_from_file() to access the streams stored in the bag and then use enable_record_to_file() to record amended data to a new bag.
-
Yes!, I'm attempting that.
I'm sorry but I didn't understand the solution. How can I use these two methods if I can't request enable_record_to_file() and enable_device_from_file() for the same config (as the documentation says)?
Besides getting rid of these two lines, do you know what other changes I need to make for this to work?
playback = playback_device.as_playback()
playback.set_real_time(False)
Thank you!
Please sign in to leave a comment.
Comments
16 comments