efficient way to retrieve 3d mapping for several 2d pixel points
I have several points that I want to retrieve 3d mapping. I have a bounding box of some object(s) in the scene. I am listing all the pixel points that lie within this bounding box. My idea is to check the depth value ( or 3d mapping) for all. Some of these will not have any depth associated, i.e., when such points do not lie on the object. From this information, I can retrieve the points that belong to the box, and extrapolate some more information.
A set of 3 methods was suggested in another post, that return a 3d value corresponding to a 2d pixel point. My question is, is there a way that such mapping could be done and stored for later reference. Because to check each point one by one could be in-efficient, as these methods, enable streams (depth and rgb) and aligning and then calculate the 3d value.
thanks,
-
Hi Zahid Iqbal If you export depth to a .csv format file then all of the depth coordinates will be preserved in a text format that can later be imported into tools such as spreadsheets and databases for viewing the values.
-
We have a window that displays frames on loop, captured through realsense camera D455 model. For this, we have the pipeline started, i.e. pipe.start().
Can, we execute another function that also needs to issue pipe.start(), for example, the function that gives us the 3d mapping for a 2d pixel point. I need to call this function, while my main loop is running capturing frames.
What effect does this have, can we start two pipes simultaneously?
thanks,
-
Yes, you can create two separate pipelines without a significant impact on a program's performance. The link below has an example of doing so in Python code.
https://github.com/IntelRealSense/librealsense/issues/5628#issuecomment-575943238
Whilst the link below is a C++ example:
https://github.com/IntelRealSense/librealsense/issues/11292
-
thanks MartyX Grover
I have two pipelines, but both of these work on the same streams (aligned color and depth stream with the same characteristics). Can, I still start these two pipelines concurrently. Since, I am receiving an error or "resource or Device busy". The first pipeline, let's call it "pipe_1" is running inside a thread and continuously streaming aligned (rgb & depth) stream into an openCV window. Then, another thread is processing the images from this stream to create object bounding boxes. Now, I want to print out the 3d coordinates of the vertices for these bounding boxes. This is done using one of the functions that were suggested in other posts. However, That function, also enables streams, creates a pipeline and then finds the mapped coordinates. When, I want to run my program, it aborts with the error of resource or device busy.
void realSenseWrapper :: get3dCoordinates(int r_idx, int c_idx, float point_in_color_coordinates[3], float point_in_depth_coordinates[3]) {
rs2::align align_to(RS2_STREAM_COLOR);
rs2::config _cfg;
rs2::pipeline _pipe; // Declare RealSense pipeline, encapsulating the actual device and sensors
rs2::pipeline_profile _selection; // Start streaming with default recommended configuration
_cfg.enable_stream(RS2_STREAM_DEPTH, 848, 480, RS2_FORMAT_Z16, 30);
_cfg.enable_stream(RS2_STREAM_COLOR, 1280, 720, RS2_FORMAT_BGR8, 30);
_selection = _pipe.start(_cfg);
auto sensor = _selection.get_device().query_sensors()[0];
if (sensor.get_option(RS2_OPTION_EMITTER_ENABLED) == 0) {
sensor.set_option(RS2_OPTION_EMITTER_ENABLED,1);
}
while (true) {
// Wait for the next set of frames from the camera
auto frames = _pipe.wait_for_frames();
auto aligned_frameset = align_to.process(frames);
rs2::video_frame aligned_color_stream = aligned_frameset.get_color_frame();
rs2::depth_frame aligned_depth_stream = aligned_frameset.get_depth_frame();
rs2::video_stream_profile depth_stream_profile = aligned_depth_stream.get_profile().as<rs2::video_stream_profile>();
const auto depth_intrinsics = depth_stream_profile.get_intrinsics();
rs2::video_stream_profile color_stream_profile = aligned_color_stream.get_profile().as<rs2::video_stream_profile>();
const auto color_intrinsics = color_stream_profile.get_intrinsics();
const int w = aligned_color_stream.as<rs2::video_frame>().get_width();
const int h = aligned_color_stream.as<rs2::video_frame>().get_height();
cv::Mat color_frame(cv::Size(w, h), CV_8UC3, (void*)aligned_color_stream.get_data(), cv::Mat::AUTO_STEP);
rs2_extrinsics color_extrin_to_depth = aligned_color_stream.get_profile().as<rs2::video_stream_profile>().get_extrinsics_to(aligned_depth_stream.get_profile().as<rs2::video_stream_profile>());
// assume depth and color same size after alignment, find depth pixel in color coordinate system
float pixel[2];
pixel[0] = r_idx;
pixel[1] = c_idx;
cout << "pixel[0]: " << c_idx <<", pixel[1]: " << r_idx << endl;
float dist = aligned_depth_stream.get_distance(static_cast<int>(pixel[0]), static_cast<int>(pixel[1]));
cout << "dist: " << dist << endl;
rs2_deproject_pixel_to_point(point_in_color_coordinates, &color_intrinsics, pixel, dist);
rs2_transform_point_to_point(point_in_depth_coordinates, &color_extrin_to_depth, point_in_color_coordinates);
break;
}
_pipe.stop();
}And, the other thread that continuously captures frames is given as follows:
// captureFrame is first thread, that should start the process
signed char realSenseWrapper :: captureFrames(rs2::pipeline& pipe_) {
rs2::frameset frames;
rs2::align align_to(RS2_STREAM_COLOR);
try{
while (true)
{
frames = pipe_.wait_for_frames();
frames = frames.apply_filter(align_to);
auto color_aligned = frames.get_color_frame();
// color frame to opencv matrix of size (w,h) from the rgb data
Mat rgb_cv_img(Size(color_aligned.get_width(), color_aligned.get_height()), CV_8UC3, (void*)color_aligned.get_data(), Mat::AUTO_STEP);
if (rgb_cv_img.empty()) {
cerr << "Error: Could not read the image." << endl;
return -1;
}
cout << "able to capture an image ... " << endl;
// Lock the mutex before updating the latest frame
{
std::unique_lock<std::mutex> ul(frame_mutex);
rgb_cv_img.copyTo(latest_frame);
latestFrameArrived = true;
latestFrameProcessed = false;
frame_counter += 1;
cout << "(captureFrames) got the lock to the frame: " << frame_counter << endl;
g_cv.notify_one();
}
}
} catch (const rs2::error& e) {
std::cerr << "RealSense error in captureFrames: " << e.what() << std::endl;
} catch (...) {
std::cerr << "Caught unknown exception" << std::endl;
}
}Do I need to access the pipeline inside a mutex.
thanks,
-
Each stream can only be accessed by one pipeline, so you cannot start a stream on a second pipeline that has already been enabled on the first pipeline. For example, pipeline 1 could have depth on it and pipeline 2 have RGB, but you could not have depth + RGB on both pipeline 1 and pipeline 2.
The rules for enabling streams are governed by the RealSense SDK's Multi-Streaming Model at the link below.
In instructions that refer to the pipeline name, if you have more than one pipeline then you can choose which specific pipeline to draw data from. For example: frames = pipeline_2.wait_for_frames();
A C++ discussion about multi-threading at the link below may be a helpful reference.
Please sign in to leave a comment.
Comments
5 comments