skip to main content
research-article

Augmented Reality Supported Modeling of Industrial Systems to Infer Software Configuration

Published:19 June 2018Publication History
Skip Abstract Section

Abstract

This paper proposes and evaluates an approach for building models of installed industrial Cyber-Physical Systems using augmented reality on smartphones. It proposes a visual language for annotating devices, containers, flows of liquids and networking connections in augmented reality. Compared to related work, it provides a more lightweight and flexible approach for building 3D models of industrial systems. The models are further used to automatically infer software configuration of controllable industrial products. This addresses a common problem of error-prone and time-consuming configuration of industrial systems in the current practice. The proposed approach is evaluated in a study with 16 domain experts. The study participants are involved in creating a model of an industrial system for water treatment. Their comments show that the approach can enable a less error-prone configuration for more complex systems. Opportunities for improvement in usability and reflections on the potential of the approach are discussed.

References

  1. Apple. ARKit: Augmented Reality for iOS. https://developer.apple.com/arkit/.Google ScholarGoogle Scholar
  2. Esteban Arroyo, Mario Hoernicke, Pablo Rodr'ıguez, and Alexander Fay. Automatic derivation of qualitative plant simulation models from legacy piping and instrumentation diagrams. Computers & Chemical Engineering, 92(Supplement C):112 -- 132, 2016.Google ScholarGoogle Scholar
  3. Franz Auinger, Markus Vorderwinkler, and Georg Buchtela. Interface driven domain-independent modeling architecture for soft-commissioning and reality in the loop. In Proceedings of the 31st Conference on Winter Simulation: Simulation--a Bridge to the Future - Volume 1, WSC '99, pages 798--805, New York, NY, USA, 1999. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Bharathan Balaji, Arka Bhattacharya, Gabriel Fierro, Jingkun Gao, Joshua Gluck, Dezhi Hong, Aslak Johansen, Jason Koh, Joern Ploennigs, Yuvraj Agarwal, Mario Berges, David Culler, Rajesh Gupta, Mikkel Baun Kjærgaard, Mani Srivastava, and Kamin Whitehouse. Brick: Towards a unified metadata schema for buildings. In Proceedings of the 3rd ACM International Conference on Systems for Energy-Efficient Built Environments, BuildSys '16, pages 41--50, New York, NY, USA, 2016. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Jan Bosch. Software Product Line Engineering, pages 3--24. Springer Berlin Heidelberg, Berlin, Heidelberg, 2013.Google ScholarGoogle Scholar
  6. B. Brumitt, J. Krumm, B. Meyers, and S. Shafer. Ubiquitous computing and the role of geometry. IEEE Personal Communications, 7(5):41--43, Oct 2000.Google ScholarGoogle ScholarCross RefCross Ref
  7. Ulrik Eklund and Jan Bosch. Architecture for embedded open software ecosystems. Journal of Systems and Software, 92(Supplement C):128 -- 142, 2014.Google ScholarGoogle Scholar
  8. Barrett Ens, Fraser Anderson, Tovi Grossman, Michelle Annett, Pourang Irani, and George W. Fitzmaurice. Ivy: Exploring spatially situated visual programming for authoring and understanding intelligent environments. In Proceedings of the 43rd Graphics Interface Conference 2017, Edmonton, Alberta, Canada, May 16--19, 2017, pages 156--162, 2017. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Markus Funk, Thomas Kosch, Romina Kettner, Oliver Korn, and Albrecht Schmidt. motioneap: An overview of 4 years of combining industrial assembly with augmented reality for industry 4.0.Google ScholarGoogle Scholar
  10. Markus Funk, Albrecht Schmidt, and Lars Erik Holmquist. Antonius: A mobile search engine for the physical world. In Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, UbiComp '13 Adjunct, pages 179--182, New York, NY, USA, 2013. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. J. Gausemeier, J. Fruend, and C. Matysczok. Ar-planning tool: Designing flexible manufacturing systems with augmented reality. In Proceedings of the Workshop on Virtual Environments 2002, EGVE '02, pages 19--25, Aire-la-Ville, Switzerland, Switzerland, 2002. Eurographics Association. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. A. Gyrard, C. Bonnet, K. Boudaoud, and M. Serrano. Lov4iot: A second life for ontology-based domain knowledge to build semantic web of things applications. In 2016 IEEE 4th International Conference on Future Internet of Things and Cloud (FiCloud), pages 254--261, Aug 2016.Google ScholarGoogle ScholarCross RefCross Ref
  13. Valentin Heun, Shunichi Kasahara, and Pattie Maes. Smarter objects: Using ar technology to program physical objects and their interactions. In CHI '13 Extended Abstracts on Human Factors in Computing Systems, CHI EA '13, pages 961--966, New York, NY, USA, 2013. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Peter Hoffmann, Reimar Schumann, Talal M A Maksoud, and Giuliano C. Premier. Virtual commissioning of manufacturing systems a review and new approaches for simplification, pages 175--181. 2010.Google ScholarGoogle Scholar
  15. Alf J. Isaksson, Iiro Harjunkoski, and Guido Sand. The impact of digitalization on the future of control and operations. 2016.Google ScholarGoogle Scholar
  16. S. Jiang, S. K. Ong, and A. Y. C. Nee. An ar-based hybrid approach for facility layout planning and evaluation for existing shop floors. The Int'l Journal of Advanced Manufacturing Technology, 72(1):457--473, Apr 2014.Google ScholarGoogle ScholarCross RefCross Ref
  17. Manabu Kano and Morimasa Ogawa. The state of the art in chemical process control in japan: Good practice and questionnaire survey. Journal of Process Control, 20(9):969 -- 982, 2010. ADCHEM 2009 Special Issue.Google ScholarGoogle ScholarCross RefCross Ref
  18. G. Kortuem and F. Kawsar. Market-based user innovation in the internet of things. In 2010 Internet of Things (IOT), pages 1--8, Nov 2010.Google ScholarGoogle ScholarCross RefCross Ref
  19. Joohyuk Lee, Hyojoo Son, Changmin Kim, and Changwan Kim. Skeleton-based 3d reconstruction of as-built pipelines from laser-scan data. Automation in Construction, 35(Supplement C):199 -- 207, 2013.Google ScholarGoogle Scholar
  20. Jingbo Liu, Oscar Kin-Chung Au, Hongbo Fu, and Chiew-Lan Tai. Two-finger gestures for 6dof manipulation of 3d objects. Comput. Graph. Forum, 31(7pt1):2047--2055, September 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Jiakang Lu and Kamin Whitehouse. Smart Blueprints: Automatically Generated Maps of Homes and the Devices Within Them, pages 125--142. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. S. Mayer, N. Inhelder, R. Verborgh, R. Van de Walle, and F. Mattern. Configuration of smart environments made simple: Combining visual modeling with semantic metadata and reasoning. In 2014 Int'l Conference on the Internet of Things (IOT), pages 61--66, Oct 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. Simon Mayer, Dominic Plangger, Florian Michahelles, and Simon Rothfuss. Ubermanufacturing: A goal-driven collaborative industrial manufacturing marketplace. In Proceedings of the 6th International Conference on the Internet of Things, IoT'16, pages 111--119, New York, NY, USA, 2016. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. Simon Mayer, Andreas Tschofen, Anind K. Dey, and Friedemann Mattern. User interfaces for smart things -- a generative approach with semantic interaction descriptions. ACM Trans. Comput.-Hum. Interact., 21(2):12:1--12:25, February 2014. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. B. Nuernberger, K. C. Lien, T. Höllerer, and M. Turk. Interpreting 2d gesture annotations in 3d augmented reality. In 2016 IEEE Symposium on 3D User Interfaces (3DUI), pages 149--158, March 2016.Google ScholarGoogle ScholarCross RefCross Ref
  26. S. K. Ong, M. L. Yuan, and A. Y. C. Nee. Augmented reality applications in manufacturing: a survey. Int'l Journal of Production Research, 46(10):2707--2742, 2008.Google ScholarGoogle ScholarCross RefCross Ref
  27. Michael E. Porter and James E. Heppelmann. How smart, connected products are transforming companies. Harvard Business Review, 93(10):53--71, #oct# 2015.Google ScholarGoogle Scholar
  28. Tahir Rabbani and Frank Van Den Heuvel. 3d industrial reconstruction by fitting csg models to a combination of images and point clouds. Int'l Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS), 35(B5):2, 2004.Google ScholarGoogle Scholar
  29. Roland Rosen, Georg von Wichert, George Lo, and Kurt D. Bettenhausen. About the importance of autonomy and digital twins for the future of manufacturing. IFAC-PapersOnLine, 48(3):567 -- 572, 2015. 15th IFAC Symposium onInformation Control Problems inManufacturing.Google ScholarGoogle ScholarCross RefCross Ref
  30. Timothy I. Salsbury. A survey of control technologies in the building automation industry. In Proc IFAC World Congress, 2005.Google ScholarGoogle ScholarCross RefCross Ref
  31. Timo Soininen, Juha Tiihonen, Tomi M"annistö, and Reijo Sulonen. Towards a general ontology of configuration. Artif. Intell. Eng. Des. Anal. Manuf., 12(4):357--372, September 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Hyojoo Son, Changmin Kim, and Changwan Kim. 3d reconstruction of as-built industrial instrumentation models from laser-scan data and a 3d cad database based on prior knowledge. Automation in Construction, 49(Part B):193 -- 200, 2015. 30th ISARC Special Issue.Google ScholarGoogle Scholar
  33. K. Thramboulidis, P. Bochalis, and J. Bouloumpasis. A framework for mde of iot-based manufacturing cyber-physical systems. In Proceedings of the 7th International Conference on the Internet of Things, IoT'17, New York, NY, USA, 2017. ACM. Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. Peter Tolmie, Andy Crabtree, Stefan Egglestone, Jan Humble, Chris Greenhalgh, and Tom Rodden. Digital plumbing: The mundane work of deploying ubicomp in the home. Personal Ubiquitous Comput., 14(3):181--196, April 2010. Google ScholarGoogle ScholarDigital LibraryDigital Library
  35. Matúv s Tomlein and Kaj Grønbæk. A visual programming approach based on domain ontologies for configuring industrial iot installations. In Proceedings of the 7th Int'l Conference on the Internet of Things, IoT'17. ACM, 2017. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Matúš Tomlein and Kaj Grønbæk. Building models of installations to recommend applications in iot software ecosystems. In 2016 IEEE 4th International Conference on Future Internet of Things and Cloud (FiCloud), pages 9--16, Aug 2016.Google ScholarGoogle ScholarCross RefCross Ref
  37. Julien Valentin, Vibhav Vineet, Ming-Ming Cheng, David Kim, Jamie Shotton, Pushmeet Kohli, Matthias Nießner, Antonio Criminisi, Shahram Izadi, and Philip Torr. Semanticpaint: Interactive 3d labeling and learning at your fingertips. ACM Trans. Graph., 34(5):154:1--154:17, November 2015. Google ScholarGoogle ScholarDigital LibraryDigital Library
  38. G. Vanderhulst, K. Luyten, and K. Coninx. Pervasive maps: Explore and interact with pervasive environments. In 2010 IEEE International Conference on Pervasive Computing and Communications (PerCom), pages 227--234, March 2010.Google ScholarGoogle ScholarCross RefCross Ref
  39. George Vosselman and Johan W. H. Tangelder. 3D reconstruction of industrial installations by constrained fitting of CAD models to images, pages 285--292. Springer Berlin Heidelberg, Berlin, Heidelberg, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  40. W3C. Resource Description Framework (RDF). https://www.w3.org/RDF/.Google ScholarGoogle Scholar

Index Terms

  1. Augmented Reality Supported Modeling of Industrial Systems to Infer Software Configuration

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!