From Object Detection to Room Categorization in Robotics

ABSTRACT

This article deals with the problem of room categorization, i.e. the classification of a room as being a bathroom, kitchen, living-room, bedroom, etc., by an autonomous robot operating in home environments. For that, we propose a room categorization system based on a Bayesian probabilistic framework that combines object detections and its semantics. For detecting objects we resort to a state-of-the-art CNN, Mask R-CNN, while the meaning or semantics of those detections is provided by an ontology. Such an ontology encodes the relations between object and room categories, that is, in which room types the different object categories are typically found (toilets in bathrooms, microwaves in kitchens, etc.). The Bayesian framework is in charge of fusing both sources of information and providing a probability distribution over the set of categories the room can belong to. The proposed system has been evaluated in houses from the Robot@Home dataset, validating its effectiveness under real-world conditions.

References

1. D. Chaves, J. R. Ruiz-Sarmiento, N. Petkov, and J. Gonzalez-Jimenez. 2019. Integration of CNN into a Robotic Architecture to Build Semantic Maps of Indoor Environments. In Advances in Computational Intelligence. 313--324.
2. Pablo Espinace, Thomas Kollar, Alvaro Soto, and Nicholas Roy. 2010. Indoor scene recognition through object detection. In Robotics and Automation (ICRA), 2010 IEEE International Conference on. IEEE, 1406--1413.
3. Cipriano Galindo, Juan-Antonio Fernández-Madrigal, and Javier González-Jiménez. 2008. Multihierarchical Interactive Task Planning. Application to Mobile Robotics. IEEE Transactions on Systems, Man, and Cybernetics, part B 38, 3 (2008), 785--798.
4. Javier González-Jiménez, Cipriano Galindo, and JR Ruiz-Sarmiento. 2012. Technical improvements of the Giraff telepresence robot based on users' evaluation. In 2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication. IEEE, 827--832.
5. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask R-CNN. arXiv e-prints, Article arXiv:1703.06870 (Mar 2017). arXiv:cs.CV/1703.06870
6. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision. 740--755.
7. Javier Monroy, J. R. Ruiz-Sarmiento, Francisco-Angel Moreno, Francisco Melendez-Fernandez, Cipriano Galindo, and Javier Gonzalez-Jimenez. 2018. A Semantic-Based Gas Source Localization with a Mobile Robot Combining Vision and Chemical Sensing. Sensors 18, 12 (2018).
8. Andrzej Pronobis and Patric Jensfelt. 2011. Hierarchical Multi-Modal Place Categorization.. In European Conference on Mobile Robots (ECMR). 159--164.
9. Andrzej Pronobis, O Martínez Mozos, Barbara Caputo, and Patric Jensfelt. 2009. Multi-modal semantic place classification. The International Journal of Robotics Research (2009).
10. Ariadna Quattoni and Antonio Torralba. 2009. Recognizing indoor scenes. In Computer Vision and Pattern Recognition, 2009. IEEE Conference on. 413--420.
11. Joseph Redmon and Ali Farhadi. 2018. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018).
12. S. Ren, K. He, R. Girshick, and J. Sun. 2017. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 6 (June 2017), 1137--1149.
13. J. G. Rogers and H. I. Christensen. 2012. A conditional random field model for place and object classification. In Robotics and Automation (ICRA), 2012 IEEE International Conference on. 1766--1772.
14. J. R. Ruiz-Sarmiento, Cipriano Galindo, and Javier González-Jiménez. 2015. Joint Categorization of Objects and Rooms for Mobile Robots. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
15. J. R. Ruiz-Sarmiento, Cipriano Galindo, and Javier González-Jiménez. 2015. OLT: A Toolkit for Object Labeling Applied to Robotic RGB-D Datasets. In European Conference on Mobile Robots.
16. J. R. Ruiz-Sarmiento, Cipriano Galindo, and Javier González-Jiménez. 2017. Building Multiversal Semantic Maps for Mobile Robot Operation. Knowledge-Based Systems 119 (2017), 257--272.
17. J. R. Ruiz-Sarmiento, Cipriano Galindo, and Javier González-Jiménez. 2017. Robot@Home, a Robotic Dataset for Semantic Mapping of Home Environments. The International Journal of Robotics Research 36, 2 (2017), 131--141.
18. J. R. Ruiz-Sarmiento, Cipriano Galindo, Javier Monroy, Francisco-Angel Moreno, and Javier Gonzalez-Jimenez. 2019. Ontology-based conditional random fields for object recognition. International Journal of Knowledge-Based Systems 168 (mar 2019), 100--108.
19. P. UrÅąiÄη, R. Mandeljc, A. Leonardis, and M. Kristan. 2016. Part-based room categorization for household service robots. In 2016 IEEE International Conference on Robotics and Automation (ICRA). 2287--2294.
20. Mike Uschold and Michael Gruninger. 1996. Ontologies: principles, methods and applications. The Knowledge Engineering Review 11 (1996), 93--136. Issue 02.
21. David ZuÃśiga-NoÃńl, J. R. Ruiz-Sarmiento, Ruben Gomez-Ojeda, and Javier Gonzalez-Jimenez. 2019. Automatic Multi-Sensor Extrinsic Calibration For Mobile Robots. IEEE Robotics and Automation Letters 4, 3 (jul 2019), 2862--2869.
22. David ZuÃśiga-NoÃńl, J. R. Ruiz-Sarmiento, and Javier Gonzalez-Jimenez. 2019. Intrinsic Calibration of Depth Cameras for Mobile Robots using a Radial Laser Scanner. CAIP. Lecture Notes in Computer Science, Vol. 11678. 659--671.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Knowledge representation and reasoning

      • Cognitive robotics

      • Ontology engineering

      • Probabilistic reasoning