Unable to properly run 300 FPS
Hi. I am trying to capture the motion of a fast-moving option at 300 FPS. I have seemingly been able to properly configure the mode using:
config->enable_stream(RS2_STREAM_DEPTH, 0, width, height, RS2_FORMAT_Z16, 300);
(Running on OS MAC on Macbook Air, C++)
but when I look at the time between time stamps it is rarely ~3,333 microseconds as it should be. I read the post https://github.com/IntelRealSense/librealsense/issues/6578 and tried some of the solutions there such as adjusting the exposure manually but that did not seem to make a difference. When I look at the time between time-stamps alongside outputting the frame counter metadata a weird behavior emerges. The first 9 frames the camera takes are all at 300 FPS (as they should be) and then on the 10th frame the FPS dramatically slows down (to ~100FPS). Weird thing is that it's ALWAYS the first 9 frames that are successfully at 300 FPS. Shortly afterward the FPS increases and then steadies where about every third frame is at 300 FPS and the other two are at 150 FPS. Example output data can be seen below:
Left Column: Time elapsed since previous received frame (useconds) | Right Column: Frame counter metadata
3328 | 1
3328 | 2
3328 | 3
3392 | 4
3328 | 5
3328 | 6
3328 | 7
3392 | 8
3328 | 9
26752 | 17
10048 | 20
6720 | 22
6656 | 24
3328 | 25
6720 | 27
6720 | 29
3328 | 30
6656 | 32
6720 | 34
6656 | 36
6720 | 38
3328 | 39
6720 | 41
It seems clear that the actual hardware is running at 300 FPS as the missing frames from the frame counter correspond to the time elapsed since the previous frame.
Things I have tried so far:
- Testing if its a frame queue issue (I've tried to increase the frame queue size and also tried it without a queue and the same issue persisted)
- Turned off auto-exposure priority, turned off auto-exposure, manually adjusted exposure as low as possible
Any suggestions would be extremely appreciated and if there's any additional information I could provide that would be helpful let me know. Thanks so much.
-
Hi Jbmayor4 When you turned off Auto Exposure Priority, may I confirm please that you kept Auto Exposure enabled? AE should be on when AE Priority is off in order to enforce a constant FPS with that method.
Also, are you using 848x100 resolution with the 300 FPS mode in your stream width and height variables? This is the only resolution available in this mode, as the shortened vertical height centered on the middle of the image is what allows the stream to run at 300 FPS without introducing slow-down.
-
Hi MartyG. Thanks for the lightning-quick response. As you requested I just tried running it with Auto exposure enabled but AE priority off. I want to confirm that I did this correctly though as the only code I could find related to turning off Auto Exposure Priority related to the rgb camera and not the depth sensor, and when I tried it for the depth sensor it threw an error. Code below for turning of AE priority as you asked:
auto prof = pipe.start(*config, q);
auto sensor = prof.get_device().query_sensors()[1];
sensor.set_option(RS2_OPTION_AUTO_EXPOSURE_PRIORITY, 0.f);
This did not seem to make any difference.
I also want to confirm that I used the other method to enforce constant FPS correctly. I disabled Auto-exposure and manually adjusted. Code Below:
auto prof = pipe.start(*config, q);
auto sensor = prof.get_device().first<rs2::depth_sensor>();
sensor.set_option(RS2_OPTION_ENABLE_AUTO_EXPOSURE, 0.f);
sensor.set_option(RS2_OPTION_EXPOSURE, 1); (I tried multiple exposure values but left in the minimum possible value as I thought would ensure 300FPS)
Also to answer your last question I am running it at 848x100 resolution. I want to emphasize again that the program successfully runs at 300 FPS for the first 9 frames and then stops running at 300 FPS and then resumes to about ~150 a couple of frames later (example in my first post). This really makes me think that there is some firmware trigger or software trigger I'm not disabling that kicks in on the 10th frame. Once again thanks for your help and quick response.
-
It is correct that Auto Exposure Priority is an RGB control. In the RealSense Viewer, an on-off toggle for it can be found under the RGB section of the options side-panel.
Whilst it is possible to change the frame queue size from the default value, it is not recommended, as a value may be set that breaks the streaming.
The link below explains how, when setting manual exposure, FPS can drop if the exposure value exceeds the "FPS period". So you have to be careful to choose a maximum exposure value appropriate to the FPS being used.
https://github.com/IntelRealSense/librealsense/issues/1957#issuecomment-400715579
If you have the budget for more than one camera, Intel's recently introduced External Synchronization (genlock) feature allows you to chain cameras together in a trigger sequence to multiply FPS. For example, if you have three cameras running at 90 FPS and 848x480 then you can capture at 270 FPS (3 x 90 FPS) and 848x480. 3 cameras at 300 FPS and 848x100, meanwhile, would give an FPS of 900 FPS at 848x100.
https://dev.intelrealsense.com/docs/external-synchronization-of-intel-realsense-depth-cameras
Please sign in to leave a comment.
Comments
3 comments