Editorial Notes
The authors have requested minor, non-substantive changes to the VoR and, in accordance with ACM policies, a Corrected VoR was published on April 5, 2023. For reference purposes the VoR may still be accessed via the Supplemental Material section on this page.
Abstract
Many scientific publications report on computational results based on code and data, but even when code and data are published, the main text is usually provided in a separate, traditional format such as PDF. Since code, data, and text are not linked on a deep level, it is difficult for readers and reviewers to understand and retrace how the authors achieved a specific result that is reported in the main text, e.g. a figure, table, or number. In addition, to make use of new the opportunities afforded by data and code availability, such as re-running analyses with changed parameters, considerable effort is required. In order to overcome this issue and to enable more interactive publications that support scientists in more deeply exploring the reported results, we present the concept, implementation, and initial evaluation of bindings. A binding describes which data subsets, code lines, and parameters produce a specific result that is reported in the main text (e.g. a figure or number). Based on a prototypical implementation of these bindings, we propose a toolkit for authors to easily create interactive figures by connecting specific UI widgets (e.g. a slider) to parameters. In addition to inspecting code and data, readers can then manipulate the parameter and see how the results change. We evaluated the approach by applying it to a set of existing articles. The results provide initial evidence that the concept is feasible and applicable to many papers with moderate effort.
Supplemental Material
Available for Download
Version of Record for "Creating Interactive Scientific Publications using Bindings" by Konkol et al., Proceedings of the ACM on Human-Computer Interaction, Vol 3, Issue EICS, article 16
- Muhammad Adnan, Mike Just, and Lynne Baillie. 2016. Investigating Time Series Visualisations to Improve the User Experience. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, 5444--5455. Google Scholar
Digital Library
- JJ Allaire, Yihui Xie, Jonathan McPherson, Javier Luraschi, Kevin Ushey, Aron Atkins, Hadley Wickham, Joe Cheng, and Winston Chang. 2018. rmarkdown: Dynamic Documents for R. https://CRAN.R-project.org/package=rmarkdown R package version 1.10. Accessed: 2019-02--21.Google Scholar
- Giuliano Antoniol, Gerardo Canfora, Gerardo Casazza, Andrea De Lucia, and Ettore Merlo. 2002. Recovering traceability links between code and documentation. IEEE transactions on software engineering 28, 10 (2002), 970--983. Google Scholar
Digital Library
- Alberto Bacchelli, Michele Lanza, and Romain Robbes. 2010. Linking e-mails and source code artifacts. In Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering-Volume 1. ACM, 375--384. Google Scholar
Digital Library
- Nick Barnes. 2010. Publish your computer code: it is good enough. Nature News 467, 7317 (2010), 753--753.Google Scholar
Cross Ref
- Winston Chang, Joe Cheng, JJ Allaire, Yihui Xie, and Jonathan McPherson. 2018. shiny: Web Application Framework for R. https://CRAN.R-project.org/package=shiny R package version 1.1.0. Accessed: 2019-02--21.Google Scholar
- Fernando Chirigati, Rémi Rampin, Dennis Shasha, and Juliana Freire. 2016. Reprozip: Computational reproducibility with ease. In Proceedings of the 2016 International Conference on Management of Data. ACM, 2085--2088.Google Scholar
Digital Library
- Julien Colomb and Björn Brembs. 2014. Sub-strains of Drosophila Canton-S differ markedly in their locomotor behavior. F1000Research 3 (2014).Google Scholar
- Benjamin M. Delory, Mao Li, Christopher N. Topp, and Guillaume Lobet. 2018. archiDART v3. 0: A new data analysis pipeline allowing the topological analysis of plant root systems. F1000Research 7 (2018).Google Scholar
- Francesco Dottori, Rui Figueiredo, Mario LV Martina, Daniela Molinari, A. Scorzini, et al. 2016. INSYDE: a synthetic, probabilistic flood damage model based on explicit cost analysis. Natural Hazards and Earth System Sciences 16 (2016), 2577--2591.Google Scholar
Cross Ref
- Pierre Dragicevic, Yvonne Jansen, Abhraneel Sarma, Matthew Kay, and Fanny Chevalier. 2019. Increasing the Transparency of Research Papers with Explorable Multiverse Analyses. In CHI 2019 - The ACM CHI Conference on Human Factors in Computing Systems. ACM.Google Scholar
- Bastian Drees, Angelina Kraft, and Thomas Koprucki. 2018. Reproducible and comprehensible research results through persistently linked and visualized numerical simulation data. Optical and Quantum Electronics 50, 2 (2018), 59.Google Scholar
Cross Ref
- Steve M Easterbrook. 2014. Open code for open science? Nature Geoscience 7, 11 (2014), 779.Google Scholar
Cross Ref
- Florian Echtler and Maximilian Häußler. 2018. Open Source, Open Science, and the Replication Crisis in HCI. In Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, alt02. Google Scholar
Digital Library
- Gabriel Fils, Zhihao Yuan, Tanu Malik, et al. 2017. Sciunits: Reusable Research Objects. In e-Science (e-Science), 2017 IEEE 13th International Conference on. IEEE, 374--383.Google Scholar
- Robert Gentleman and Duncan Temple Lang. 2007. Statistical analyses and reproducible research. Journal of Computational and Graphical Statistics 16, 1 (2007), 1--23.Google Scholar
Cross Ref
- Virginia Gewin. 2016. Data sharing: An open mind on open data. Nature 529, 7584 (2016), 117--119.Google Scholar
- Yolanda Gil, Varun Ratnakar, and Daniel Garijo. 2015. OntoSoft: Capturing scientific software metadata. In Proceedings of the 8th International Conference on Knowledge Capture. ACM, 32. Google Scholar
Digital Library
- Jeremy Goecks, Anton Nekrutenko, and James Taylor. 2010. Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences. Genome biology 11, 8 (2010), R86.Google Scholar
- Steven N. Goodman, Daniele Fanelli, and John P. A. Ioannidis. 2016. What does research reproducibility mean? Science translational medicine 8, 341 (2016), 341ps12--341ps12.Google Scholar
- Miriam Greis, Hendrik Schuff, Marius Kleiner, Niels Henze, and Albrecht Schmidt. 2017. Input Controls for Entering Uncertain Data: Probability Distribution Sliders. Proceedings of the ACM on Human-Computer Interaction 1, 1 (2017), 3. Google Scholar
Digital Library
- Stevan Harnad, Tim Brody, Francois Vallieres, Les Carr, Steve Hitchcock, Yves Gingras, Charles Oppenheim, Heinrich Stamerjohanns, and Eberhard R. Hilf. 2004. The access/impact problem and the green and gold roads to open access. Serials review 30, 4 (2004), 310--314.Google Scholar
- Anett Hoppe, Jascha Hagen, Helge Holzmann, Günter Kniesel, and Ralph Ewerth. 2018. An Analytics Tool for Exploring Scientific Software and Related Publications. In Digital Libraries for Open Knowledge, Eva Méndez, Fabio Crestani, Cristina Ribeiro, Gabriel David, and João Correia Lopes (Eds.). Springer International Publishing, Cham, 299--303.Google Scholar
- Tomi Kauppinen and Giovana Mira de Espindola. 2011. Linked Open Science-Communicating, Sharing and Evaluating Data, Methods and Results for Executable Papers.. In ICCS. 726--731.Google Scholar
- Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian E. Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B. Hamrick, Jason Grout, Sylvain Corlay, et al. 2016. Jupyter Notebooks-a publishing format for reproducible computational workflows.. In ELPUB. 87--90.Google Scholar
- Donald Ervin Knuth. 1984. Literate programming. Comput. J. 27, 2 (1984), 97--111. Google Scholar
Digital Library
- Markus Konkol and Christian Kray. 2018. In-depth examination of spatio-temporal figures in open reproducible research. Cartography and Geographic Information Science (2018).Google Scholar
- Markus Konkol, Christian Kray, and Max Pfeiffer. 2018. Computational reproducibility in geoscientific papers: Insights from a series of studies with geoscientists and a reproduction study. International Journal of Geographical Information Science (2018), 1--22.Google Scholar
- Friedrich Leisch. 2002. Sweave: Dynamic generation of statistical reports using literate data analysis. In Compstat. Springer, 575--580.Google Scholar
- Jon Mease. 2018. Bringing ipywidgets Support to plotly. py. (2018).Google Scholar
- Daniel Nüst, Carlos Granell, Barbara Hofer, Markus Konkol, Frank O. Ostermann, Rusne Sileryte, and Valentina Cerutti. 2018. Reproducible research and GIScience: an evaluation using AGILE conference papers. PeerJ 6 (2018), e5072.Google Scholar
Cross Ref
- Daniel Nüst, Markus Konkol, Edzer Pebesma, Christian Kray, Marc Schutzeichel, Holger Przibytzin, and Jörg Lorenz. 2017. Opening the publication process with executable research compendia. D-Lib Magazine 23, 1/2 (2017).Google Scholar
Cross Ref
- Jeroen Ooms. 2018. opencpu. https://CRAN.R-project.org/package=opencpu Accessed: 2019-02--21.Google Scholar
- Frank O. Ostermann and Carlos Granell. 2017. Advancing science with VGI: Reproducibility and replicability of recent studies using VGI. Transactions in GIS 21, 2 (2017), 224--237.Google Scholar
Cross Ref
- Jeffrey M. Perkel. 2018. Data visualization tools drive interactivity and reproducibility in online publishing. Nature 554, 7690 (2018), 133--134.Google Scholar
- Quan Pham, Tanu Malik, and Ian Foster. 2013. Using Provenance for Repeatability. TaPP 13 (2013), 2. Google Scholar
Digital Library
- Quan Pham, Tanu Malik, Ian Foster, Roberto Di Lauro, and Raffaele Montella. 2012. SOLE: linking research papers with science objects. In International Provenance and Annotation Workshop. Springer, 203--208. Google Scholar
Digital Library
- R Core Team. 2018. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org Accessed: 2019-02--21.Google Scholar
- Richard J Radke, Srinivas Andra, Omar Al-Kofahi, and Badrinath Roysam. 2005. Image change detection algorithms: a systematic survey. IEEE transactions on image processing 14, 3 (2005), 294--307. Google Scholar
Digital Library
- Thomas Rathfux, Roman Popp, and Hermann Kaindl. 2016. Adding custom widgets to model-driven GUI generation. In Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems. ACM, 16--26. Google Scholar
Digital Library
- Geir Kjetil Sandve, Anton Nekrutenko, James Taylor, and Eivind Hovig. 2013. Ten simple rules for reproducible computational research. PLoS computational biology 9, 10 (2013), e1003285.Google Scholar
- Carson Sievert, Chris Parmer, Toby Hocking, Scott Chamberlain, Karthik Ram, Marianne Corvellec, and Pedro Despouy. 2017. plotly: Create Interactive Web Graphics via 'plotly. js'. R package version 4.7. 1.Google Scholar
- Dalmeet Singh Chawla. 2015. 'Living figures' make their debut. Nature News 521, 7550 (2015), 112.Google Scholar
Cross Ref
- Stefan Steiniger and Erwan Bocher. 2009. An overview on current free and open source desktop GIS developments. International Journal of Geographical Information Science 23, 10 (2009), 1345--1370.Google Scholar
Cross Ref
- Victoria Stodden, Matthew S. Krafczyk, and Adhithya Bhaskar. 2018. Enabling the Verification of Computational Results: An Empirical Evaluation of Computational Reproducibility. In Proceedings of the First International Workshop on Practical Reproducible Evaluation of Computer Systems. ACM, 3.Google Scholar
Digital Library
- Victoria Stodden, Marcia McNutt, David H. Bailey, Ewa Deelman, Yolanda Gil, Brooks Hanson, Michael A. Heroux, John P. A. Ioannidis, and Michela Taufer. 2016. Enhancing reproducibility for computational methods. Science 354, 6317 (2016), 1240--1241.Google Scholar
- Trestle Technology, LLC. 2018. plumber: An API Generator for R. https://CRAN.R-project.org/package=plumber R package version 0.4.6. Accessed: 2019-02--21.Google Scholar
- Ramnath Vaidyanathan, Yihui Xie, JJ Allaire, Joe Cheng, and Kenton Russell. 2016. htmlwidgets: HTML Widgets for R. https://CRAN.R-project.org/package=htmlwidgets R package version 1.3. Accessed: 2019-02--21.Google Scholar
- Bret Victor. 2018. Explorable Explanations. http://worrydream.com/#!/ExplorableExplanations. Accessed: 2019-02--21.Google Scholar
- Chat Wacharamanotham, Krishna Subramanian, Sarah Theres Völkel, and Jan Borchers. 2015. Statsplorer: Guiding novices in statistical analysis. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2693--2702. Google Scholar
Digital Library
- L.J. Welty, L.V. Rasmussen, A.S. Baldridge, and E. Whitley. 2016. StatTag. Galter Health Sciences Library, Chicago, Illinois, United States.Google Scholar
- Mark D. Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E. Bourne, et al. 2016. The FAIR Guiding Principles for scientific data management and stewardship. Scientific data 3 (2016).Google Scholar
- Kanit Wongsuphasawat, Dominik Moritz, Arvind Satyanarayan, and Jeffrey Heer. 2018. Vega-Lite - A Grammar of Interactive Graphics. https://vega.github.io/vega-lite/. Accessed: 2019-02--21.Google Scholar
Index Terms
Creating Interactive Scientific Publications using Bindings
Recommendations
Acknowledgments in scientific publications: Presence in Spanish science and text patterns across disciplines
The acknowledgments in scientific publications are an important feature in the scholarly communication process. This research analyzes funding acknowledgment presence in scientific publications and introduces a novel approach for discovering text ...
Who support open access publishing? Gender, discipline, seniority and other factors associated with academics' OA practice
This paper presents the findings from a survey study of UK academics and their publishing behaviour. The aim of this study is to investigate academics' attitudes towards and practice of open access (OA) publishing. The results are based on a survey ...
Nobel Citation Effects on Scientific Publications: A Case Study in Physics
AbstractThe Nobel Prize has been a prominent research topic in Scientometrics. This study aims to explore Nobel citation effects, which refer to changes in the cumulative citation counts of publications after authors have won the Nobel Prize. ...






Comments