Abstract

This paper proposes and evaluates an approach for building models of installed industrial Cyber-Physical Systems using augmented reality on smartphones. It proposes a visual language for annotating devices, containers, flows of liquids and networking connections in augmented reality. Compared to related work, it provides a more lightweight and flexible approach for building 3D models of industrial systems. The models are further used to automatically infer software configuration of controllable industrial products. This addresses a common problem of error-prone and time-consuming configuration of industrial systems in the current practice. The proposed approach is evaluated in a study with 16 domain experts. The study participants are involved in creating a model of an industrial system for water treatment. Their comments show that the approach can enable a less error-prone configuration for more complex systems. Opportunities for improvement in usability and reflections on the potential of the approach are discussed.
- Apple. ARKit: Augmented Reality for iOS. https://developer.apple.com/arkit/.Google Scholar
- Esteban Arroyo, Mario Hoernicke, Pablo Rodr'ıguez, and Alexander Fay. Automatic derivation of qualitative plant simulation models from legacy piping and instrumentation diagrams. Computers & Chemical Engineering, 92(Supplement C):112 -- 132, 2016.Google Scholar
- Franz Auinger, Markus Vorderwinkler, and Georg Buchtela. Interface driven domain-independent modeling architecture for soft-commissioning and reality in the loop. In Proceedings of the 31st Conference on Winter Simulation: Simulation--a Bridge to the Future - Volume 1, WSC '99, pages 798--805, New York, NY, USA, 1999. ACM. Google Scholar
Digital Library
- Bharathan Balaji, Arka Bhattacharya, Gabriel Fierro, Jingkun Gao, Joshua Gluck, Dezhi Hong, Aslak Johansen, Jason Koh, Joern Ploennigs, Yuvraj Agarwal, Mario Berges, David Culler, Rajesh Gupta, Mikkel Baun Kjærgaard, Mani Srivastava, and Kamin Whitehouse. Brick: Towards a unified metadata schema for buildings. In Proceedings of the 3rd ACM International Conference on Systems for Energy-Efficient Built Environments, BuildSys '16, pages 41--50, New York, NY, USA, 2016. ACM. Google Scholar
Digital Library
- Jan Bosch. Software Product Line Engineering, pages 3--24. Springer Berlin Heidelberg, Berlin, Heidelberg, 2013.Google Scholar
- B. Brumitt, J. Krumm, B. Meyers, and S. Shafer. Ubiquitous computing and the role of geometry. IEEE Personal Communications, 7(5):41--43, Oct 2000.Google Scholar
Cross Ref
- Ulrik Eklund and Jan Bosch. Architecture for embedded open software ecosystems. Journal of Systems and Software, 92(Supplement C):128 -- 142, 2014.Google Scholar
- Barrett Ens, Fraser Anderson, Tovi Grossman, Michelle Annett, Pourang Irani, and George W. Fitzmaurice. Ivy: Exploring spatially situated visual programming for authoring and understanding intelligent environments. In Proceedings of the 43rd Graphics Interface Conference 2017, Edmonton, Alberta, Canada, May 16--19, 2017, pages 156--162, 2017. Google Scholar
Digital Library
- Markus Funk, Thomas Kosch, Romina Kettner, Oliver Korn, and Albrecht Schmidt. motioneap: An overview of 4 years of combining industrial assembly with augmented reality for industry 4.0.Google Scholar
- Markus Funk, Albrecht Schmidt, and Lars Erik Holmquist. Antonius: A mobile search engine for the physical world. In Proceedings of the 2013 ACM Conference on Pervasive and Ubiquitous Computing Adjunct Publication, UbiComp '13 Adjunct, pages 179--182, New York, NY, USA, 2013. ACM. Google Scholar
Digital Library
- J. Gausemeier, J. Fruend, and C. Matysczok. Ar-planning tool: Designing flexible manufacturing systems with augmented reality. In Proceedings of the Workshop on Virtual Environments 2002, EGVE '02, pages 19--25, Aire-la-Ville, Switzerland, Switzerland, 2002. Eurographics Association. Google Scholar
Digital Library
- A. Gyrard, C. Bonnet, K. Boudaoud, and M. Serrano. Lov4iot: A second life for ontology-based domain knowledge to build semantic web of things applications. In 2016 IEEE 4th International Conference on Future Internet of Things and Cloud (FiCloud), pages 254--261, Aug 2016.Google Scholar
Cross Ref
- Valentin Heun, Shunichi Kasahara, and Pattie Maes. Smarter objects: Using ar technology to program physical objects and their interactions. In CHI '13 Extended Abstracts on Human Factors in Computing Systems, CHI EA '13, pages 961--966, New York, NY, USA, 2013. ACM. Google Scholar
Digital Library
- Peter Hoffmann, Reimar Schumann, Talal M A Maksoud, and Giuliano C. Premier. Virtual commissioning of manufacturing systems a review and new approaches for simplification, pages 175--181. 2010.Google Scholar
- Alf J. Isaksson, Iiro Harjunkoski, and Guido Sand. The impact of digitalization on the future of control and operations. 2016.Google Scholar
- S. Jiang, S. K. Ong, and A. Y. C. Nee. An ar-based hybrid approach for facility layout planning and evaluation for existing shop floors. The Int'l Journal of Advanced Manufacturing Technology, 72(1):457--473, Apr 2014.Google Scholar
Cross Ref
- Manabu Kano and Morimasa Ogawa. The state of the art in chemical process control in japan: Good practice and questionnaire survey. Journal of Process Control, 20(9):969 -- 982, 2010. ADCHEM 2009 Special Issue.Google Scholar
Cross Ref
- G. Kortuem and F. Kawsar. Market-based user innovation in the internet of things. In 2010 Internet of Things (IOT), pages 1--8, Nov 2010.Google Scholar
Cross Ref
- Joohyuk Lee, Hyojoo Son, Changmin Kim, and Changwan Kim. Skeleton-based 3d reconstruction of as-built pipelines from laser-scan data. Automation in Construction, 35(Supplement C):199 -- 207, 2013.Google Scholar
- Jingbo Liu, Oscar Kin-Chung Au, Hongbo Fu, and Chiew-Lan Tai. Two-finger gestures for 6dof manipulation of 3d objects. Comput. Graph. Forum, 31(7pt1):2047--2055, September 2012. Google Scholar
Digital Library
- Jiakang Lu and Kamin Whitehouse. Smart Blueprints: Automatically Generated Maps of Homes and the Devices Within Them, pages 125--142. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. Google Scholar
Digital Library
- S. Mayer, N. Inhelder, R. Verborgh, R. Van de Walle, and F. Mattern. Configuration of smart environments made simple: Combining visual modeling with semantic metadata and reasoning. In 2014 Int'l Conference on the Internet of Things (IOT), pages 61--66, Oct 2014. Google Scholar
Digital Library
- Simon Mayer, Dominic Plangger, Florian Michahelles, and Simon Rothfuss. Ubermanufacturing: A goal-driven collaborative industrial manufacturing marketplace. In Proceedings of the 6th International Conference on the Internet of Things, IoT'16, pages 111--119, New York, NY, USA, 2016. ACM. Google Scholar
Digital Library
- Simon Mayer, Andreas Tschofen, Anind K. Dey, and Friedemann Mattern. User interfaces for smart things -- a generative approach with semantic interaction descriptions. ACM Trans. Comput.-Hum. Interact., 21(2):12:1--12:25, February 2014. Google Scholar
Digital Library
- B. Nuernberger, K. C. Lien, T. Höllerer, and M. Turk. Interpreting 2d gesture annotations in 3d augmented reality. In 2016 IEEE Symposium on 3D User Interfaces (3DUI), pages 149--158, March 2016.Google Scholar
Cross Ref
- S. K. Ong, M. L. Yuan, and A. Y. C. Nee. Augmented reality applications in manufacturing: a survey. Int'l Journal of Production Research, 46(10):2707--2742, 2008.Google Scholar
Cross Ref
- Michael E. Porter and James E. Heppelmann. How smart, connected products are transforming companies. Harvard Business Review, 93(10):53--71, #oct# 2015.Google Scholar
- Tahir Rabbani and Frank Van Den Heuvel. 3d industrial reconstruction by fitting csg models to a combination of images and point clouds. Int'l Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences (ISPRS), 35(B5):2, 2004.Google Scholar
- Roland Rosen, Georg von Wichert, George Lo, and Kurt D. Bettenhausen. About the importance of autonomy and digital twins for the future of manufacturing. IFAC-PapersOnLine, 48(3):567 -- 572, 2015. 15th IFAC Symposium onInformation Control Problems inManufacturing.Google Scholar
Cross Ref
- Timothy I. Salsbury. A survey of control technologies in the building automation industry. In Proc IFAC World Congress, 2005.Google Scholar
Cross Ref
- Timo Soininen, Juha Tiihonen, Tomi M"annistö, and Reijo Sulonen. Towards a general ontology of configuration. Artif. Intell. Eng. Des. Anal. Manuf., 12(4):357--372, September 1998. Google Scholar
Digital Library
- Hyojoo Son, Changmin Kim, and Changwan Kim. 3d reconstruction of as-built industrial instrumentation models from laser-scan data and a 3d cad database based on prior knowledge. Automation in Construction, 49(Part B):193 -- 200, 2015. 30th ISARC Special Issue.Google Scholar
- K. Thramboulidis, P. Bochalis, and J. Bouloumpasis. A framework for mde of iot-based manufacturing cyber-physical systems. In Proceedings of the 7th International Conference on the Internet of Things, IoT'17, New York, NY, USA, 2017. ACM. Google Scholar
Digital Library
- Peter Tolmie, Andy Crabtree, Stefan Egglestone, Jan Humble, Chris Greenhalgh, and Tom Rodden. Digital plumbing: The mundane work of deploying ubicomp in the home. Personal Ubiquitous Comput., 14(3):181--196, April 2010. Google Scholar
Digital Library
- Matúv s Tomlein and Kaj Grønbæk. A visual programming approach based on domain ontologies for configuring industrial iot installations. In Proceedings of the 7th Int'l Conference on the Internet of Things, IoT'17. ACM, 2017. Google Scholar
Digital Library
- Matúš Tomlein and Kaj Grønbæk. Building models of installations to recommend applications in iot software ecosystems. In 2016 IEEE 4th International Conference on Future Internet of Things and Cloud (FiCloud), pages 9--16, Aug 2016.Google Scholar
Cross Ref
- Julien Valentin, Vibhav Vineet, Ming-Ming Cheng, David Kim, Jamie Shotton, Pushmeet Kohli, Matthias Nießner, Antonio Criminisi, Shahram Izadi, and Philip Torr. Semanticpaint: Interactive 3d labeling and learning at your fingertips. ACM Trans. Graph., 34(5):154:1--154:17, November 2015. Google Scholar
Digital Library
- G. Vanderhulst, K. Luyten, and K. Coninx. Pervasive maps: Explore and interact with pervasive environments. In 2010 IEEE International Conference on Pervasive Computing and Communications (PerCom), pages 227--234, March 2010.Google Scholar
Cross Ref
- George Vosselman and Johan W. H. Tangelder. 3D reconstruction of industrial installations by constrained fitting of CAD models to images, pages 285--292. Springer Berlin Heidelberg, Berlin, Heidelberg, 2000. Google Scholar
Digital Library
- W3C. Resource Description Framework (RDF). https://www.w3.org/RDF/.Google Scholar
Index Terms
Augmented Reality Supported Modeling of Industrial Systems to Infer Software Configuration
Recommendations
Collaborative Augmented Reality: Multi-user Interaction in Urban Simulation
IVIC '09: Proceedings of the 1st International Visual Informatics Conference on Visual Informatics: Bridging Research and PracticeAugmented reality (AR) environment allows user or multi-user to interact with 2D and 3D data. AR simply can provide a collaborative interactive AR environment for urban simulation, where users can interact naturally and intuitively. AR collaboration ...
A survey of industrial augmented reality
Highlights- State of the art review of Augmented Reality in industrial environments.
- ...
AbstractThis article aims to evaluate the impact of Augmented Reality (AR) applicability and usefulness on real industrial processes by employing a systematic literature review (SLR). The SLR was performed in five digital libraries to identify ...
Haptics in Augmented Reality
ICMCS '99: Proceedings of the IEEE International Conference on Multimedia Computing and Systems - Volume 2An augmented reality system merges synthetic sensory information into a user's perception of a three-dimensional environment. An important performance goal for an augmented reality system is that the user perceives a single seamless environment. In most ...






Comments