skip to main content
research-article

Accelerated Fire Detection and Localization at Edge

Published:18 October 2022Publication History
Skip Abstract Section

Abstract

Fire-related incidents continue to be reported as a leading cause of life and property destruction. Automated fire detection and localization (AFDL) systems have grown in importance with the evolution of applied robotics, especially because use of robots in disaster situations can lead to avoidance of human fatality. The importance of AFDL on resource-constrained devices has further grown, as most unmanned vehicles (drones or ground vehicles) are battery operated with limited computational capacity, the disaster situations cannot guarantee uninterrupted communication with high-end resources in the cloud, and yet faster response time is a prime necessity. Traditional computer vision–based techniques require hand-engineered features on a case-by-case basis. Deep Learning–based classifiers perform well for fire/no-fire classification due to the availability of large datasets for training; however, a dearth of good fire localization datasets renders the localization performance below par. We have tried to address both problems with a multi-task learned cascaded model that triggers localization workflow only if the presence of fire is detected, through a strong classifier trained on available large fire datasets. This presents only fire images to a relatively weaker localization model, reducing false positives, false negatives, and thereby improving overall AFDL accuracy. The multi-task learning (MTL) approach for end-to-end training of a stitched classifier and object localizer model on diverse datasets enabled us to build a strong fire classifier and feature extractor. It also resulted in a single unified model, capable of running on “on-board” compute infrastructure without compromising on accuracy.

To achieve the target inference rate for the AFDL deployment, we have investigated the effect of quantization and compression due to hardware acceleration on an MTL model. This article presents an approach to automate the hardware-software co-design to find the optimum parameter partitioning for a given MTL problem, especially when some parts of the model are hardware accelerated. We present combined evaluation results showing that our methodology and the corresponding AFDL model strikes a balance between the frames inferred per second and several accuracy metrics. We report fire localization accuracy in terms of mean average precision (object detection), which has not been done earlier for embedded AFDL systems.

REFERENCES

  1. [1] Ahrens Marty and Evarts Ben. 2020. Fire Loss in the United States During 2019. Retrieved from https://go.nfpa.org/l/14662/2021-05-24/8hkgts.Google ScholarGoogle Scholar
  2. [2] Ahrens Marty and Messerschmidt Birgitte. 2021. Fire Safety in the United States since 1980. Retrieved from https://go.nfpa.org/l/14662/2021-05-24/8hkgts.Google ScholarGoogle Scholar
  3. [3] Bilen Hakan and Vedaldi Andrea. 2016. Integrated perception with recurrent multi-task neural networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS’16). Curran Associates Inc., 235—243.Google ScholarGoogle Scholar
  4. [4] Borges Paulo Vinicius Koerich and Izquierdo Ebroul. 2010. A probabilistic approach for vision-based fire detection in videos. IEEE Trans. Circ. Syst. Vid. Technol. 20, 5 (2010), 721731.Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. [5] Brushlinsky Ahrens and Sokolov Wagner. 2021. World Fire Statistics Centre of Fire Statistics. Retrieved from https://ctif.org/sites/default/files/2021-06/CTIF_Report26_KOREA.pdf.Google ScholarGoogle Scholar
  6. [6] Cazzolato Mirela T., Avalhais L. P., Chino D. Y., Ramos Jonathan S., Souza Jessica A. de, Jr Jose F. Rodrigues, and Traina A. J.. 2017. Fismo: A compilation of datasets from emergency situations for fire and smoke analysis. In Proceedings of the Brazilian Symposium on Databases (SBBD’17). SBC, Brazil, 213223.Google ScholarGoogle Scholar
  7. [7] Celik Turgay and Demirel Hasan. 2009. Fire detection in video sequences using a generic color model. Fire Safe. J. 44, 2 (2009), 147158.Google ScholarGoogle ScholarCross RefCross Ref
  8. [8] Chen Thou-Ho, Wu Ping-Hsueh, and Chiou Yung-Chuen. 2004. An early fire-detection method based on image processing. In Proceedings of the International Conference on Image Processing (ICIP’04), Vol. 3. IEEE, Singapore, 17071710.Google ScholarGoogle Scholar
  9. [9] Chennupati Sumanth, Sistu Ganesh, Yogamani Senthil, and Rawashdeh Samir A.. 2019. Multinet++: Multi-stream feature aggregation and geometric loss strategy for multi-task learning. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE Computer Society, USA, 1200–1210.Google ScholarGoogle Scholar
  10. [10] Chino Daniel Y. T., Avalhais Letricia P. S., Rodrigues Jose F., and Traina Agma J. M.. 2015. Bowfire: Detection of fire in still images by integrating pixel color and texture analysis. In Proceedings of the 28th SIBGRAPI conference on graphics, patterns and images. IEEE, 95102.Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. [11] Lascio Rosario Di, Greco Antonio, Saggese Alessia, and Vento Mario. 2014. Improving fire detection reliability by a combination of videoanalytics. In Proceedings of the International Conference Image Analysis and Recognition. Springer, Cham, 477484.Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. [12] Dimitropoulos Kosmas, Barmpoutis Panagiotis, and Grammalidis Nikos. 2014. Spatio-temporal flame modeling and dynamic texture analysis for automatic video-based fire detection. IEEE Trans. Circ. Syst. Vid. Technol. 25, 2 (2014), 339351.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. [13] Fan Angela, Stock Pierre, Graham Benjamin, Grave Edouard, Gribonval R., Jégou H., and Joulin Armand. 2020. Training with quantization noise for extreme model compression. arXiv:2004.07320. Retrieved from https://arxiv.org/abs/2004.07320.Google ScholarGoogle Scholar
  14. [14] Foggia Pasquale, Saggese Alessia, and Vento Mario. 2015. Real-time fire detection for video-surveillance applications using a combination of experts based on color, shape, and motion. IEEE Trans. Circ. Syst. Vid. Technol. 25, 9 (2015), 15451556.Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. [15] Girshick Ross. 2015. Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision. IEEE Computer Society, 14401448.Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. [16] Gong Faming, Li Chuantao, Gong Wenjuan, Li Xin, Yuan Xiangbing, Ma Yuhui, and Song Tao. 2019. A real-time fire detection method from video with multifeature fusion. Comput. Intell. Neurosci. 2019, Article 1939171 (Jan 2019), 17 pages.Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. [17] Gonzalez-Garcia Abel, Modolo Davide, and Ferrari Vittorio. 2018. Do semantic parts emerge in convolutional neural networks?Int. J. Comput. Vis. 126, 5 (2018), 476494.Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. [18] Habiboğlu Yusuf Hakan, Günay Osman, and Çetin A. Enis. 2012. Covariance matrix-based fire and flame detection method in video. Mach. Vis. Appl. 23, 6 (2012), 11031113.Google ScholarGoogle ScholarCross RefCross Ref
  19. [19] Hochreiter Sepp and Schmidhuber Jürgen. 1997. Long short-term memory. Neural Comput. 9, 8 (1997), 17351780.Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. [20] Hoiem Derek, Chodpathumwan Yodsawalai, and Dai Qieyun. 2012. Diagnosing error in object detectors. In Proceedings of the European Conference on Computer Vision. Springer, Cham, 340353.Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. [21] Hu Chao, Tang Peng, Jin WeiDong, He ZhengWei, and Li Wei. 2018. Real-time fire detection based on deep convolutional long-recurrent networks and optical flow method. In Proceedings of the 37th Chinese Control Conference (CCC’18). IEEE, 90619066.Google ScholarGoogle ScholarCross RefCross Ref
  22. [22] Iandola Forrest N., Moskewicz M., Ashraf Khalid, Han Song, Dally W., and Keutzer K.. 2016. SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size. ArXiv preprint arXiv:1602.07360 (2016). Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Intel. 2018. Intel Neural Compute Stick. Retrieved from https://software.intel.com/en-us/neural-compute-stick.Google ScholarGoogle Scholar
  24. [24] Intel. 2021. Intel Distribution of OpenVino Toolkit. Retrieved from https://software.intel.com/en-us/openvino-toolkit.Google ScholarGoogle Scholar
  25. [25] Jacob Benoit, Kligys Skirmantas, Chen Bo, Zhu Menglong, Tang Matthew, Howard Andrew, Adam Hartwig, and Kalenichenko Dmitry. 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern recognition. IEEE, 27042713.Google ScholarGoogle ScholarCross RefCross Ref
  26. [26] Jadon Arpit, Omama Mohd., Varshney Akshay, Ansari M. S., and Sharma Rishabh. 2019. FireNet: A specialized lightweight fire & smoke detection model for real-time iot applications. arXiv:1905.11922. Retrieved from https://arxiv.org/abs/1905.11922.Google ScholarGoogle Scholar
  27. [27] Arpit Jadon, Akshay Varshney, and Mohammad Samar Ansari. 2020. Low-complexity high-performance deep learning model for real-time low-cost embedded fire detection systems. Procedia Computer Science 171 (2020), 418–426. Third International Conference on Computing and Network Communications (CoCoNet’19).Google ScholarGoogle Scholar
  28. [28] Park Sung jae. 2019. Fire Detection Based on Convolutional Neural Networks. Technical Report. University of Minnesota.Google ScholarGoogle Scholar
  29. [29] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional Architecture for Fast Feature Embedding. In Proceedings of the 22nd ACM International Conference on Multimedia (Orlando, Florida, USA) (MM’14). Association for Computing Machinery, New York, NY, USA, 675–678.Google ScholarGoogle Scholar
  30. [30] Norman P. Jouppi and Young. 2017. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture (Toronto, ON, Canada) (ISCA’17). Association for Computing Machinery, New York, NY, USA, 1–12.Google ScholarGoogle Scholar
  31. [31] Byoungjun Kim and Joonwhoan Lee. 2019. A video-based fire detection using deep learning models. Applied Sciences 9, 14 (2019), 2862.Google ScholarGoogle Scholar
  32. [32] Kokkinos Iasonas. 2017. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 61296138.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1 (Lake Tahoe, Nevada) (NIPS’12). Curran Associates Inc., Red Hook, NY, USA, 1097–1105.Google ScholarGoogle Scholar
  34. [34] Chetan Kumar, G. Suhas, B. S. Abhishek, Digvijay Gowda KA, and R. Prajwal. 2020. Fire Detection Using Deep Learning. International Journal of Progressive Research in Science and Engineering 1, 5 (2020), 1–5.Google ScholarGoogle Scholar
  35. [35] Liu Wei, Anguelov Dragomir, Erhan Dumitru, Szegedy Christian, Reed Scott, Fu Cheng-Yang, and Berg Alexander C.. 2016. Ssd: Single shot multibox detector. In Proceedings of the European Conference on Computer Vision. Springer, Cham, 2137.Google ScholarGoogle ScholarCross RefCross Ref
  36. [36] London Jack. 1906. Story of an eyewitness: The San Francisco earthquake. Collier’s Week. 37 (1906), 2226.Google ScholarGoogle Scholar
  37. [37] Ma Chao, Huang Jia-Bin, Yang Xiaokang, and Yang Ming-Hsuan. 2015. Hierarchical convolutional features for visual tracking. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, 30743082.Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. [38] (MBZIRC) The Mohamed Bin Zayed International Robotics Challenge. 2020. The Challenge 2020. Retrieved July 14, 2021 from https://www.mbzirc.com/challenge/2020.Google ScholarGoogle Scholar
  39. [39] Misra Ishan, Shrivastava Abhinav, Gupta Abhinav, and Hebert Martial. 2016. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 39944003.Google ScholarGoogle ScholarCross RefCross Ref
  40. [40] Martin Mueller, Peter Karasev, Ivan Kolesov, and Allen Tannenbaum. 2013. Optical flow estimation for flame detection in videos. IEEE Transactions on Image Processing 22, 7 (2013), 2786–2797.Google ScholarGoogle Scholar
  41. [41] Muhammad Khan, Ahmad Jamil, and Baik Sung Wook. 2018. Early fire detection using convolutional neural networks during surveillance for effective disaster management. Neurocomputing 288, C (2018), 30–42.Google ScholarGoogle ScholarDigital LibraryDigital Library
  42. [42] Muhammad Khan, Ahmad Jamil, Lv Zhihan, Bellavista Paolo, Yang Po, and Baik Sung Wook. 2018. Efficient deep CNN-based fire detection and localization in video surveillance applications. IEEE Trans. Syst. Man Cybernet.: Syst. 49, 7 (2018), 14191434.Google ScholarGoogle ScholarCross RefCross Ref
  43. [43] Muhammad Khan, Ahmad Jamil, Mehmood Irfan, Rho Seungmin, and Baik Sung Wook. 2018. Convolutional neural networks based fire detection in surveillance videos. IEEE Access 6 (2018), 1817418183.Google ScholarGoogle ScholarCross RefCross Ref
  44. [44] Muhammad Khan, Khan Salman, Elhoseny Mohamed, Ahmed Syed Hassan, and Baik Sung Wook. 2019. Efficient fire detection for uncertain surveillance environment. IEEE Trans. Industr. Inf. 15, 5 (2019), 31133122.Google ScholarGoogle ScholarCross RefCross Ref
  45. [45] Noda S. and Ueda K.. 1994. Fire detection in tunnels using an image processing method. In Proceedings of the Vehicle Navigation and Information Systems Conference (VNIS’94). IEEE, 5762.Google ScholarGoogle ScholarCross RefCross Ref
  46. [46] NVIDIA. 2020. Jetson Nano Developer Kit.Retrieved July 14, 2021 from https://developer.nvidia.com/embedded/jetson-nano-developer-kit.Google ScholarGoogle Scholar
  47. [47] Park Keunyoung and Kim Doo-Hyun. 2018. Accelerating image classification using feature map similarity in convolutional neural networks. Appl. Sci. 9, 1, Article 108 (12 2018), 108. DOI:Google ScholarGoogle ScholarCross RefCross Ref
  48. [48] Rafiee Ali, Dianat Reza, Jamshidi Mehregan, Tavakoli Reza, and Abbaspour Sara. 2011. Fire and smoke detection using wavelet analysis and disorder characteristics. In Proceedings of the 3rd International Conference on Computer Research and Development, Vol. 3. IEEE, 262265.Google ScholarGoogle ScholarCross RefCross Ref
  49. [49] Redmon Joseph and Farhadi Ali. 2018. YOLOv3: An incremental improvement. arXiv:1804.02767. Retrieved from https://arxiv.org/abs/1804.02767.Google ScholarGoogle Scholar
  50. [50] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1 (Montreal, Canada) (NIPS’15). MIT Press, Cambridge, MA, USA, 91–99.Google ScholarGoogle Scholar
  51. [51] Fahy Jay T. Petrillo, Rita F. and Molis Joseph L.. 2020. Firefighter Fatalities in the US—2019. Retrieved from https://www.nfpa.org//-/media/Files/News-and-Research/Fire-statistics-and-reports/Emergency-responders/osFFF.pdf.Google ScholarGoogle Scholar
  52. [52] Rossi Lucile, Akhloufi Moulay, and Tison Yves. 2011. On the use of stereovision to develop a novel instrumentation system to extract geometric fire fronts characteristics. Fire Safe. J. 46, 1–2 (2011), 920.Google ScholarGoogle ScholarCross RefCross Ref
  53. [53] Ruder Sebastian. 2017. An overview of multi-task learning in deep neural networks. arXiv:1706.05098. Retrieved from https://arxiv.org/abs/1706.05098.Google ScholarGoogle Scholar
  54. [54] Steve Rudz, Khaled Chetehouna, A. Hafiane, Hélène Laurent, and Olivier Séro-Guillaume. 2013. Investigation of a novel image segmentation method dedicated to forest fire applications. Measurement Science and Technology 24, 7, Article 075403 (2013), 8 pages.Google ScholarGoogle Scholar
  55. [55] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. MobileNetV2: Inverted Residuals and Linear Bottlenecks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, 4510–4520.Google ScholarGoogle Scholar
  56. [56] Sener Ozan and Koltun Vladlen. 2018. Multi-Task learning as multi-objective optimization. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (NIPS’18). Curran Associates Inc., Red Hook, NY, USA, 525536.Google ScholarGoogle ScholarDigital LibraryDigital Library
  57. [57] Sudhakar Sengan, Varadarajan Vijayakumar, C. Sathiya Kumar, V. Priya, Logesh Ravi, and V. Subramaniyaswamy. 2020. Unmanned Aerial Vehicle (UAV) based Forest Fire Detection and monitoring for reducing false alarms in forest-fires. Computer Communications 149, C (2020), 1–16.Google ScholarGoogle Scholar
  58. [58] Sharma Jivitesh, Granmo Ole-Christoffer, and Goodwin Morten. 2021. Emergency analysis: Multitask learning with deep convolutional neural networks for fire emergency scene parsing. In Advances and Trends in Artificial Intelligence. Artificial Intelligence Practices. Springer International Publishing, Cham, 101112.Google ScholarGoogle Scholar
  59. [59] Sharma Jivitesh, Granmo Ole-Christoffer, Goodwin Morten, and Fidje Jahn Thomas. 2017. Deep convolutional neural networks for fire detection in images. In Proceedings of the International Conference on Engineering Applications of Neural Networks. Springer, Cham, 183193.Google ScholarGoogle ScholarCross RefCross Ref
  60. [60] Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. ArXiv preprint arXiv:1409.1556 (2015). Google ScholarGoogle ScholarCross RefCross Ref
  61. [61] Tan M., Chen B., Pang R., Vasudevan V., Sandler M., Howard A., and Le Q. V.. 2019. MnasNet: Platform-Aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR’19). IEEE Computer Society, Los Alamitos, CA, 28152823.Google ScholarGoogle ScholarCross RefCross Ref
  62. [62] Tang Ziyang, Liu Xiang, Chen Hanlin, Hupy Joseph, and Yang Baijian. 2020. Deep learning based wildfire event object detection from 4K aerial images acquired by UAS. AI 1, 2 (2020), 166179.Google ScholarGoogle ScholarCross RefCross Ref
  63. [63] Yuan Chi, Ghamry Khaled A., Liu Zhixiang, and Zhang Youmin. 2016. Unmanned aerial vehicle based forest fire monitoring and detection using image processing technique. In Proceedings of the IEEE Chinese Guidance, Navigation and Control Conference (CGNCC’16). IEEE, 18701875.Google ScholarGoogle ScholarCross RefCross Ref
  64. [64] Zhang Qingjie, Xu Jiaolong, Xu Liang, and Guo Haifeng. 2016. Deep convolutional neural networks for forest fire detection. In Proceedings of the International Forum on Management, Education and Information Technology Application. Atlantis Press, Netherlands, 568575.Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Accelerated Fire Detection and Localization at Edge

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image ACM Transactions on Embedded Computing Systems
          ACM Transactions on Embedded Computing Systems  Volume 21, Issue 6
          November 2022
          498 pages
          ISSN:1539-9087
          EISSN:1558-3465
          DOI:10.1145/3561948
          • Editor:
          • Tulika Mitra
          Issue’s Table of Contents

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 18 October 2022
          • Online AM: 26 January 2022
          • Accepted: 30 December 2021
          • Revised: 24 November 2021
          • Received: 15 July 2021
          Published in tecs Volume 21, Issue 6

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article
          • Refereed

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Full Text

        View this article in Full Text.

        View Full Text

        HTML Format

        View this article in HTML Format .

        View HTML Format
        About Cookies On This Site

        We use cookies to ensure that we give you the best experience on our website.

        Learn more

        Got it!