Construct Frameset Object

• MartyG

  • October 17, 2019 07:32
  • Edited

  When capture is activated, if you have auto-exposure enabled then the first several frames are spent settling down the auto-exposure values.  You can either disable auto-exposure and use your own exposure values, or use code to tell the camera to skip the first frames by using a 'for' instruction to wait until there is a complete frameset available.

  https://github.com/IntelRealSense/librealsense/issues/2269#issuecomment-414211209 

  0

  Comment actions Permalink
• Asfrga

  • June 04, 2020 15:01

  Hi everyone,

  I am also having this kind of problem. The solution provided by MartyG was not very helpful (rs2::software_sensor does not support auto exposure as an option), so I wonder if anyone can provide a better answer.

  Have you found a valid solution to this problem, Frank?

  I also wrote a program that is based on the software_device example mentioned by Frank (https://github.com/IntelRealSense/librealsense/tree/master/examples/software-device), and my program's structure is very similar to the main function of that example. What I am trying to do is read two images (a depth image encoded as Z16 and a color image encoded as BGR8) from the file system, feed them into their respective sensors, then call sync.wait_for_frames(). What perplexes me is that no matter how many loops I run the frameset I received always only contain 1 frame, and that is the depth frame. The color frame just would not show up. Here is my code:

  #include <iostream>
  #include <opencv2/opencv.hpp>
  #include <librealsense2/rs.hpp>
  #include <librealsense2/hpp/rs_internal.hpp>
  
  double W = 640;
  double H = 480;
  
  
  int main() {
      
      rs2::software_device dev;
      
      auto depth_sensor = dev.add_sensor("Depth");
      auto color_sensor = dev.add_sensor("Color");
      
      rs2_intrinsics depth_intrinsics{W, H, W / 2, H / 2, 383.859, 383.859, RS2_DISTORTION_BROWN_CONRADY,
                                      {0, 0, 0, 0, 0}};
      rs2_intrinsics color_intrinsics{W, H, W / 2, H / 2, 618.212, 618.463, RS2_DISTORTION_INVERSE_BROWN_CONRADY,
                                      {0, 0, 0, 0, 0}};
      
      auto depth_stream = depth_sensor.add_video_stream(
              {RS2_STREAM_DEPTH, 0, 0, W, H, 60, 2, RS2_FORMAT_Z16, depth_intrinsics});
      auto color_stream = color_sensor.add_video_stream(
              {RS2_STREAM_COLOR, 0, 1, W, H, 60, 3, RS2_FORMAT_BGR8, color_intrinsics});
      
      dev.create_matcher(RS2_MATCHER_DEFAULT);
      rs2::syncer sync;
      
      depth_sensor.open(depth_stream);
      color_sensor.open(color_stream);
      
      depth_sensor.start(sync);
      color_sensor.start(sync);
      
      depth_stream.register_extrinsics_to(color_stream, {{1, 0, 0, 0, 1, 0, 0, 0, 1},
                                                         {0, 0, 0}});
      
      rs2::frameset fs;
      
      cv::Mat color_image;
      cv::Mat depth_image;
      
      //stream in the images
      for (int idx = 0; idx < 120; ++idx) {
          color_image = cv::imread("/home/username/Desktop/images/color.jpg", cv::IMREAD_COLOR);
          depth_image = cv::imread("/home/username/Desktop/images/depth.png", cv::IMREAD_UNCHANGED);
          
          color_sensor.on_video_frame({(void*) color_image.data, // Frame pixels from capture API
                                       [](void*) {}, // Custom deleter (if required)
                                       3 * 640, 3, // Stride and Bytes-per-pixel
                                       (double) (idx) / 60, RS2_TIMESTAMP_DOMAIN_COUNT,
                                       idx, // Timestamp, Frame# for potential sync services
                                       color_stream});
          depth_sensor.on_video_frame({(void*) depth_image.data, // Frame pixels from capture API
                                       [](void*) {}, // Custom deleter (if required)
                                       2 * 640, 2, // Stride and Bytes-per-pixel
                                       (double) (idx) / 60, RS2_TIMESTAMP_DOMAIN_COUNT,
                                       idx, // Timestamp, Frame# for potential sync services
                                       depth_stream});
          
          fs = sync.wait_for_frames();
      }
      
      return 0;
  }

  Can anyone PLEASE give me some explanation? 

  1

  Comment actions Permalink
• Nicholasmcdonald40

  • March 16, 2021 17:43

  Hi everybody,

  This problem still persists in 2021, but I can propose a simple workaround.

  I am also trying to load frames (rgb + depth) from disk and then process them using the standard pipeline by creating frames in a software device. Sadly, the rs-software-device example is not fully functional and fails because the syncer fails to consistently extract the constructed frames.

  I do this particularly because I want to do expensive processing after recording at a separate time and am loading frames from raw byte data.

  The syncer likes to throw a tantrum particularly when the simulated timestamps don't match the FPS of the software sensor. @Asfrga this is a problem in the code you posted. That's why the example multiplies `ind` with 16, which is roughly 1000/60 (unit milliseconds).

  I am not sure under the hood what is preventing the syncer from extracting the frames correctly, I think it has to do with timestamps.

  Here is my original implementation:

  #include <librealsense2/rs.hpp>
  #include <librealsense2/hpp/rs_internal.hpp>
  #include <iostream>
  #include <vector>
  #include <csignal>
  #include <unistd.h>
  
  struct synthframe{
  int W, H, B;
  std::vector<uint8_t> dat;
  };
  
  const int CW = 960; //Color Frame
  const int CH = 540;
  const int CB = 4;
  
  const int DW = 640; //Depth Frame
  const int DH = 480;
  const int DB = 2;
  
  const rs2_intrinsics intrDepth = { DW, DH,
  324.83203125, 254.3828125,
  456.63671875 , 457.02734375 ,
  RS2_DISTORTION_NONE ,
  { 0,0,0,0,0 } };
  
  const rs2_intrinsics intrColor = { CW, CH,
  488.421417236328, 268.80029296875,
  673.101379394531, 673.328247070312,
  RS2_DISTORTION_INVERSE_BROWN_CONRADY ,
  { 0.142735555768013, -0.466687560081482, -0.000759649323299527, 0.000336397089995444, 0.420803397893906 } };
  
  synthframe cframe; //Color-Frame
  synthframe dframe; //Depth-Frame
  
  synthframe& getColor(){
    //...
    return cframe;
  }
  
  synthframe& getDepth(){
    //...
    return dframe;
  }
  
  bool isalive = true;
  void sighandler(int signal){
  isalive = false;
  }
  
  int main(int argc, char * argv[]) try {
  
  signal(SIGINT, &sighandler);
  
  rs2::software_device dev; // Create software-only device
  dev.create_matcher(RS2_MATCHER_DEFAULT);
  
  auto depth_sensor = dev.add_sensor("Depth"); // Define single sensor
  auto color_sensor = dev.add_sensor("Color"); // Define single sensor
  
  auto depth_stream = depth_sensor.add_video_stream({RS2_STREAM_DEPTH, 0, 0,
  DW, DH, 100, DB, RS2_FORMAT_Z16, intrDepth });
  
  depth_sensor.add_read_only_option(RS2_OPTION_DEPTH_UNITS, 0.00025f);
  
  auto color_stream = color_sensor.add_video_stream({RS2_STREAM_COLOR, 1, 1,
  CW, CH, 100, CB, RS2_FORMAT_RGBA8, intrColor });
  
  dev.create_matcher(RS2_MATCHER_COUNT);
  
  color_stream.register_extrinsics_to(depth_stream, {
  { 0.999966, -0.00700524, 0.00429877,
  0.00689519, 0.999661, 0.0251015,
  -0.00447315, -0.025071, 0.999676 },
  {-0.000823423033580184, -0.0139116607606411, 0.00651831179857254 } });
  
  depth_stream.register_extrinsics_to(color_stream, {
  { 0.999966 , 0.00689519 , -0.00447315,
  -0.00700524 , 0.999661 , -0.025071,
  0.00429877 , 0.0251015 , 0.999676 },
  {0.000697920040693134, 0.0137490043416619, -0.00686865951865911 } });
  
  depth_sensor.open(depth_stream);
  color_sensor.open(color_stream);
  
  int ind = 0;
  
  rs2::syncer sync(2);
  color_sensor.start(sync);
  depth_sensor.start(sync);
  
  cframe = getColor();
  dframe = getDepth();
  
  while(isalive){
  
  color_sensor.on_video_frame({ cframe.dat.data(), // Frame pixels from capture API
  [](void*) {cframe.dat.clear();}, // Custom deleter (if required)
  CW*CB, CB, // Stride and Bytes-per-pixel
  (double)ind*10.0, RS2_TIMESTAMP_DOMAIN_HARDWARE_CLOCK, ind,
  color_stream });
  
  depth_sensor.on_video_frame({ dframe.dat.data(), // Frame pixels from capture API
  [](void*) {dframe.dat.clear();}, // Custom deleter (if required)
  DW*DB, DB, // Stride and Bytes-per-pixel
  (double)ind*10.0, RS2_TIMESTAMP_DOMAIN_HARDWARE_CLOCK, ind,
  depth_stream });
  
  usleep(100000);  //!!!!!!!!
  
  ind++;
  
  rs2::frameset fset;
  try{
  fset = sync.wait_for_frames();
  }
  catch(rs2::error e){
  std::cout<<e.what()<<std::endl;
  }
  
  std::cout<<"Found "<<fset.size()<<" frames"<<std::endl;
  if(fset.size() == 1){
  std::cout<<"Only Found 1 Frame "<<fset[0].get_profile().stream_type()<<std::endl;
  continue;
  }
  else{
  cframe = getColor();
  dframe = getDepth();
  }
  
  }
  
  return EXIT_SUCCESS;
  
  }
  catch (const rs2::error & e){
  std::cerr << "RealSense error calling " << e.get_failed_function() << "(" << e.get_failed_args() << "):\n " << e.what() << std::endl;
  return EXIT_FAILURE;
  }
  catch (const std::exception& e){
  std::cerr << e.what() << std::endl;
  return EXIT_FAILURE;
  }

  Note that adding usleep(1E5) or processing the frames makes the syncer fail.

  Solution:

  Don't use the syncer. Use a frame queue with a custom bundler as a processing block. Replace this:

  rs2::syncer sync(2);
  color_sensor.start(sync);
  depth_sensor.start(sync);

  with this:

   
   //Prepare frame-set getter (the frame queue!)
   rs2::frame_queue q(2);
   std::vector<rs2::frame> bundle;
  
    rs2::processing_block bundler([&](rs2::frame f, rs2::frame_source& src) {
      bundle.push_back(f);     //add frame to vector
      if (bundle.size() == 2){ //or however many when full
          auto fs = src.allocate_composite_frame(bundle);
          src.frame_ready(fs);
          bundle.clear();
       }
    });
    bundler.start(q); //when enough frames are loaded, the queue is ready
  
    depth_sensor.start([&](rs2::frame f){
      bundler.invoke(f);  //add frame to bundler
    });
    color_sensor.start([&](rs2::frame f){
      bundler.invoke(f);
    });

  Then it stops caring about the timestamp syncing and processing time and just constructs the frameset for you so you can use them.

   

  0

  Comment actions Permalink

Please sign in to leave a comment.