AMG8833 IR Thermal Sensor with RealSense Cameras
Hi everyone
I've been working on a side project involving fusion of thermal imaging and depth sensing, and I thought I'd share what I've found and ask for your input.
I recently picked up the AMG8833 IR thermal camera breakout module, it gives an 8×8 IR thermal array over I²C, with a temperature range of 0°C to 80°C and selectable refresh rates (1 fps / 10 fps).
My goal is to combine its thermal readings with a RealSense depth camera (e.g. D435 or D455) to create richer perception: e.g. identify heat-emitting objects in a 3D scene, enhance object segmentation, or detect human presence more reliably (even in low-light).
Aligning the thermal frame to the depth frame requires timestamping and calibration between thermal sensor and depth camera coordinate systems. The module has ±2.5 °C error margin in some cases, so small temperature differences are hard to detect robustly. T the thermal module’s FOV is about 60° x 60° (per its specs) which may or may not align with the RealSense depth FOV.
Interface & power: the AMG8833 uses I²C and low power, so interfacing (say on an ESP32/Arduino) is manageable. But making sure the RealSense SDK + thermal sensor share synchronized clocks is a chore.
RealSense depth runs at tens of frames per second, while this thermal module is up to 10 fps. How do you deal with mismatched frame rates in fusion? What use cases do you think benefit most from combining thermal + depth (besides simple human detection)? If I upgrade to a higher resolution thermal module (e.g. 32×24), would the extra data be worth it when fusing with RealSense?
Thanks in advance for any thoughts, pointers, or experiences. I’d love to see how others have handled multispectral / multimodal sensor fusion in this community.
MerczDJ
-
Hi Mercio Jonova There have been a couple of past cases where users of FLIR thermal imaging cameras have attempted to align them to the depth of RealSense cameras, so that may be a helpful reference for you if you have not seen it already.
https://github.com/IntelRealSense/librealsense/discussions/11392
The minimum depth FPS of a D435 is 6 FPS and the minimum of D455 is 5 FPS. So using one of these would get you a frame rate that is closer to your thermal sensor's 10 FPS than the default 30 FPS of RealSense cameras.
If you need to match exactly to 10 FPS then it is possible to use programming code to create a custom frame rate for the RealSense camera by only using every "nth" frame. For example, setting a speed of 30 FPS but only using every 3rd frame to simulate 10 FPS.
https://github.com/IntelRealSense/librealsense/issues/3169
In regard to use cases for thermal / depth combination, they could include observing the movement of non-human creatures such as insects or mammals, or analysis of the current integrity of materials on buildings or architectural constructs such as bridges. Some also use thermal and depth sensing for ghost hunting but that is not a field we have experience in supporting!
My knowledge of thermal cameras is insufficient to state whether a higher resolution thermal module would be beneficial when combining the thermal data with RealSense data.
Please sign in to leave a comment.
Comments
1 comment