Abstract
Fire-related incidents continue to be reported as a leading cause of life and property destruction. Automated fire detection and localization (AFDL) systems have grown in importance with the evolution of applied robotics, especially because use of robots in disaster situations can lead to avoidance of human fatality. The importance of AFDL on resource-constrained devices has further grown, as most unmanned vehicles (drones or ground vehicles) are battery operated with limited computational capacity, the disaster situations cannot guarantee uninterrupted communication with high-end resources in the cloud, and yet faster response time is a prime necessity. Traditional computer vision–based techniques require hand-engineered features on a case-by-case basis. Deep Learning–based classifiers perform well for fire/no-fire classification due to the availability of large datasets for training; however, a dearth of good fire localization datasets renders the localization performance below par. We have tried to address both problems with a multi-task learned cascaded model that triggers localization workflow only if the presence of fire is detected, through a strong classifier trained on available large fire datasets. This presents only fire images to a relatively weaker localization model, reducing false positives, false negatives, and thereby improving overall AFDL accuracy. The multi-task learning (MTL) approach for end-to-end training of a stitched classifier and object localizer model on diverse datasets enabled us to build a strong fire classifier and feature extractor. It also resulted in a single unified model, capable of running on “on-board” compute infrastructure without compromising on accuracy.
To achieve the target inference rate for the AFDL deployment, we have investigated the effect of quantization and compression due to hardware acceleration on an MTL model. This article presents an approach to automate the hardware-software co-design to find the optimum parameter partitioning for a given MTL problem, especially when some parts of the model are hardware accelerated. We present combined evaluation results showing that our methodology and the corresponding AFDL model strikes a balance between the frames inferred per second and several accuracy metrics. We report fire localization accuracy in terms of mean average precision (object detection), which has not been done earlier for embedded AFDL systems.
- [1] . 2020. Fire Loss in the United States During 2019. Retrieved from https://go.nfpa.org/l/14662/2021-05-24/8hkgts.Google Scholar
- [2] . 2021. Fire Safety in the United States since 1980. Retrieved from https://go.nfpa.org/l/14662/2021-05-24/8hkgts.Google Scholar
- [3] . 2016. Integrated perception with recurrent multi-task neural networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS’16). Curran Associates Inc., 235—243.Google Scholar
- [4] . 2010. A probabilistic approach for vision-based fire detection in videos. IEEE Trans. Circ. Syst. Vid. Technol. 20, 5 (2010), 721–731.Google Scholar
Digital Library
- [5] . 2021. World Fire Statistics Centre of Fire Statistics. Retrieved from https://ctif.org/sites/default/files/2021-06/CTIF_Report26_KOREA.pdf.Google Scholar
- [6] . 2017. Fismo: A compilation of datasets from emergency situations for fire and smoke analysis. In Proceedings of the Brazilian Symposium on Databases (SBBD’17). SBC, Brazil, 213–223.Google Scholar
- [7] . 2009. Fire detection in video sequences using a generic color model. Fire Safe. J. 44, 2 (2009), 147–158.Google Scholar
Cross Ref
- [8] . 2004. An early fire-detection method based on image processing. In Proceedings of the International Conference on Image Processing (ICIP’04), Vol. 3. IEEE, Singapore, 1707–1710.Google Scholar
- [9] . 2019. Multinet++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE Computer Society, USA, 1200–1210.Google Scholar
- [10] . 2015. Bowfire: Detection of fire in still images by integrating pixel color and texture analysis. In Proceedings of the 28th SIBGRAPI conference on graphics, patterns and images. IEEE, 95–102.Google Scholar
Digital Library
- [11] . 2014. Improving fire detection reliability by a combination of videoanalytics. In Proceedings of the International Conference Image Analysis and Recognition. Springer, Cham, 477–484.Google Scholar
Digital Library
- [12] . 2014. Spatio-temporal flame modeling and dynamic texture analysis for automatic video-based fire detection. IEEE Trans. Circ. Syst. Vid. Technol. 25, 2 (2014), 339–351.Google Scholar
Digital Library
- [13] . 2020. Training with quantization noise for extreme model compression. arXiv:2004.07320. Retrieved from https://arxiv.org/abs/2004.07320.Google Scholar
- [14] . 2015. Real-time fire detection for video-surveillance applications using a combination of experts based on color, shape, and motion. IEEE Trans. Circ. Syst. Vid. Technol. 25, 9 (2015), 1545–1556.Google Scholar
Digital Library
- [15] . 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision. IEEE Computer Society, 1440–1448.Google Scholar
Digital Library
- [16] . 2019. A real-time fire detection method from video with multifeature fusion. Comput. Intell. Neurosci. 2019, Article 1939171 (Jan 2019), 17 pages.Google Scholar
Digital Library
- [17] . 2018. Do semantic parts emerge in convolutional neural networks?Int. J. Comput. Vis. 126, 5 (2018), 476–494.Google Scholar
Digital Library
- [18] . 2012. Covariance matrix-based fire and flame detection method in video. Mach. Vis. Appl. 23, 6 (2012), 1103–1113.Google Scholar
Cross Ref
- [19] . 1997. Long short-term memory. Neural Comput. 9, 8 (1997), 1735–1780.Google Scholar
Digital Library
- [20] . 2012. Diagnosing error in object detectors. In Proceedings of the European Conference on Computer Vision. Springer, Cham, 340–353.Google Scholar
Digital Library
- [21] . 2018. Real-time fire detection based on deep convolutional long-recurrent networks and optical flow method. In Proceedings of the 37th Chinese Control Conference (CCC’18). IEEE, 9061–9066.Google Scholar
Cross Ref
- [22] . 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size. ArXiv preprint arXiv:1602.07360 (2016). Google Scholar
Cross Ref
- [23] . 2018. Intel Neural Compute Stick. Retrieved from https://software.intel.com/en-us/neural-compute-stick.Google Scholar
- [24] . 2021. Intel Distribution of OpenVino Toolkit. Retrieved from https://software.intel.com/en-us/openvino-toolkit.Google Scholar
- [25] . 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition. IEEE, 2704–2713.Google Scholar
Cross Ref
- [26] . 2019. FireNet: A specialized lightweight fire & smoke detection model for real-time iot applications. arXiv:1905.11922. Retrieved from https://arxiv.org/abs/1905.11922.Google Scholar
- [27] Arpit Jadon, Akshay Varshney, and Mohammad Samar Ansari. 2020. Low-complexity high-performance deep learning model for real-time low-cost embedded fire detection systems. Procedia Computer Science 171 (2020), 418–426. Third International Conference on Computing and Network Communications (CoCoNet’19).Google Scholar
- [28] . 2019. Fire Detection Based on Convolutional Neural Networks.
Technical Report . University of Minnesota.Google Scholar - [29] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. In Proceedings of the 22nd ACM International Conference on Multimedia (Orlando, Florida, USA) (MM’14). Association for Computing Machinery, New York, NY, USA, 675–678.Google Scholar
- [30] Norman P. Jouppi and Young. 2017. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture (Toronto, ON, Canada) (ISCA’17). Association for Computing Machinery, New York, NY, USA, 1–12.Google Scholar
- [31] Byoungjun Kim and Joonwhoan Lee. 2019. A video-based fire detection using deep learning models. Applied Sciences 9, 14 (2019), 2862.Google Scholar
- [32] . 2017. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 6129–6138.Google Scholar
Cross Ref
- [33] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1 (Lake Tahoe, Nevada) (NIPS’12). Curran Associates Inc., Red Hook, NY, USA, 1097–1105.Google Scholar
- [34] Chetan Kumar, G. Suhas, B. S. Abhishek, Digvijay Gowda KA, and R. Prajwal. 2020. Fire Detection Using Deep Learning. International Journal of Progressive Research in Science and Engineering 1, 5 (2020), 1–5.Google Scholar
- [35] . 2016. Ssd: Single shot multibox detector. In Proceedings of the European Conference on Computer Vision. Springer, Cham, 21–37.Google Scholar
Cross Ref
- [36] . 1906. Story of an eyewitness: The San Francisco earthquake. Collier’s Week. 37 (1906), 22–26.Google Scholar
- [37] . 2015. Hierarchical convolutional features for visual tracking. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, 3074–3082.Google Scholar
Digital Library
- [38] . 2020. The Challenge 2020. Retrieved July 14, 2021 from https://www.mbzirc.com/challenge/2020.Google Scholar
- [39] . 2016. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 3994–4003.Google Scholar
Cross Ref
- [40] Martin Mueller, Peter Karasev, Ivan Kolesov, and Allen Tannenbaum. 2013. Optical flow estimation for flame detection in videos. IEEE Transactions on Image Processing 22, 7 (2013), 2786–2797.Google Scholar
- [41] . 2018. Early fire detection using convolutional neural networks during surveillance for effective disaster management. Neurocomputing 288, C (2018), 30–42.Google Scholar
Digital Library
- [42] . 2018. Efficient deep CNN-based fire detection and localization in video surveillance applications. IEEE Trans. Syst. Man Cybernet.: Syst. 49, 7 (2018), 1419–1434.Google Scholar
Cross Ref
- [43] . 2018. Convolutional neural networks based fire detection in surveillance videos. IEEE Access 6 (2018), 18174–18183.Google Scholar
Cross Ref
- [44] . 2019. Efficient fire detection for uncertain surveillance environment. IEEE Trans. Industr. Inf. 15, 5 (2019), 3113–3122.Google Scholar
Cross Ref
- [45] . 1994. Fire detection in tunnels using an image processing method. In Proceedings of the Vehicle Navigation and Information Systems Conference (VNIS’94). IEEE, 57–62.Google Scholar
Cross Ref
- [46] . 2020. Jetson Nano Developer Kit.Retrieved July 14, 2021 from https://developer.nvidia.com/embedded/jetson-nano-developer-kit.Google Scholar
- [47] . 2018. Accelerating image classification using feature map similarity in convolutional neural networks. Appl. Sci. 9, 1, Article 108 (
12 2018), 108.DOI: Google ScholarCross Ref
- [48] . 2011. Fire and smoke detection using wavelet analysis and disorder characteristics. In Proceedings of the 3rd International Conference on Computer Research and Development, Vol. 3. IEEE, 262–265.Google Scholar
Cross Ref
- [49] . 2018. YOLOv3: An incremental improvement. arXiv:1804.02767. Retrieved from https://arxiv.org/abs/1804.02767.Google Scholar
- [50] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1 (Montreal, Canada) (NIPS’15). MIT Press, Cambridge, MA, USA, 91–99.Google Scholar
- [51] . 2020. Firefighter Fatalities in the US—2019. Retrieved from https://www.nfpa.org//-/media/Files/News-and-Research/Fire-statistics-and-reports/Emergency-responders/osFFF.pdf.Google Scholar
- [52] . 2011. On the use of stereovision to develop a novel instrumentation system to extract geometric fire fronts characteristics. Fire Safe. J. 46, 1–2 (2011), 9–20.Google Scholar
Cross Ref
- [53] . 2017. An overview of multi-task learning in deep neural networks. arXiv:1706.05098. Retrieved from https://arxiv.org/abs/1706.05098.Google Scholar
- [54] Steve Rudz, Khaled Chetehouna, A. Hafiane, Hélène Laurent, and Olivier Séro-Guillaume. 2013. Investigation of a novel image segmentation method dedicated to forest fire applications. Measurement Science and Technology 24, 7, Article 075403 (2013), 8 pages.Google Scholar
- [55] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. MobileNetV2: Inverted Residuals and Linear Bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 4510–4520.Google Scholar
- [56] . 2018. Multi-Task learning as multi-objective optimization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS’18). Curran Associates Inc., Red Hook, NY, USA, 525–536.Google Scholar
Digital Library
- [57] Sudhakar Sengan, Varadarajan Vijayakumar, C. Sathiya Kumar, V. Priya, Logesh Ravi, and V. Subramaniyaswamy. 2020. Unmanned Aerial Vehicle (UAV) based Forest Fire Detection and monitoring for reducing false alarms in forest-fires. Computer Communications 149, C (2020), 1–16.Google Scholar
- [58] . 2021. Emergency analysis: Multitask learning with deep convolutional neural networks for fire emergency scene parsing. In Advances and Trends in Artificial Intelligence. Artificial Intelligence Practices. Springer International Publishing, Cham, 101–112.Google Scholar
- [59] . 2017. Deep convolutional neural networks for fire detection in images. In Proceedings of the International Conference on Engineering Applications of Neural Networks. Springer, Cham, 183–193.Google Scholar
Cross Ref
- [60] Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. ArXiv preprint arXiv:1409.1556 (2015). Google Scholar
Cross Ref
- [61] . 2019. MnasNet: Platform-Aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19). IEEE Computer Society, Los Alamitos, CA, 2815–2823.Google Scholar
Cross Ref
- [62] . 2020. Deep learning based wildfire event object detection from 4K aerial images acquired by UAS. AI 1, 2 (2020), 166–179.Google Scholar
Cross Ref
- [63] . 2016. Unmanned aerial vehicle based forest fire monitoring and detection using image processing technique. In Proceedings of the IEEE Chinese Guidance, Navigation and Control Conference (CGNCC’16). IEEE, 1870–1875.Google Scholar
Cross Ref
- [64] . 2016. Deep convolutional neural networks for forest fire detection. In Proceedings of the International Forum on Management, Education and Information Technology Application. Atlantis Press, Netherlands, 568–575.Google Scholar
Cross Ref
Index Terms
Accelerated Fire Detection and Localization at Edge
Recommendations
A fire detection and localisation method based on keyframes and superpixels for large-space buildings
Relevant fire detection and localisation methods suffer from the problems of slow detection speeds, low detection accuracies and low localisation precisions when applied to large-space buildings. To address these problems, a fire detection and ...
L2 regularized deep convolutional neural networks for fire detection
Soft Computing ApplicationsFire calamity is one of the worst adversarial events that can happen to the human race. Fire disaster can happen as a manmade disaster or even naturally, and it may cause environmental, social, and financial damages as well. In order to minimalize the ...
Combining Neural Networks and a Color Classifier for Fire Detection
Intelligent SystemsAbstractDeep learning methods have solved several problems in the computer vision area, mainly for image classification. The use of these methods for fire detection can bring great improvements to security systems and prevent many losses. Effective fire ...






Comments