skip to main content
research-article

Big Data Processing on Volunteer Computing

Authors Info & Claims
Published:22 July 2021Publication History
Skip Abstract Section

Abstract

In order to calculate the node big data contained in complex networks and realize the efficient calculation of complex networks, based on voluntary computing, taking ICE middleware as the communication medium, the loose coupling distributed framework DCBV based on voluntary computing is proposed. Then, the Master, Worker, and MiddleWare layers in the framework, and the development structure of a DCBV framework are designed. The task allocation and recovery strategy, message passing and communication mode, and fault tolerance processing are discussed. Finally, to calculate and verify parameters such as the average shortest path of the framework and shorten calculation time, an improved accurate shortest path algorithm, the N-SPFA algorithm, is proposed. Under different datasets, the node calculation and performance of the N-SPFA algorithm are explored. The algorithm is compared with four approximate shortest-path algorithms: Combined Link and Attribute (CLA), Lexicographic Breadth First Search (LBFS), Approximate algorithm of shortest path length based on center distance of area division (CDZ), and Hub Vertex of area and Core Expressway (HEA-CE). The results show that when the number of CPU threads is 4, the computation time of the DCBV framework is the shortest (514.63 ms). As the number of CPU cores increases, the overall computation time of the framework decreases gradually. For every 2 additional CPU cores, the number of tasks increases by 1. When the number of Worker nodes is 8 and the number of nodes is 1, the computation time of the framework is the shortest (210,979 ms), and the IO statistics data increase with the increase of Worker nodes. When the datasets are Undirected01 and Undirected02, the computation time of the N-SPFA algorithm is the shortest, which is 4520 ms and 7324 ms, respectively. However, the calculation time in the ca-condmat_undirected dataset is 175,292 ms, and the performance is slightly worse. Overall, however, the performance of the N-SPFA and SPFA algorithms is good. Therefore, the two algorithms are combined. For networks with less complexity, the computational scale coefficient of the SPFA algorithm can be set to 0.06, and for general networks, 0.2. When compared with other algorithms in different datasets, the pretreatment time, average query time, and overall query time of N-SPFA algorithm are the shortest, being 49.67 ms, 5.12 ms, and 94,720 ms, respectively. The accuracy (1.0087) and error rate (0.024) are also the best. In conclusion, voluntary computing can be applied to the processing of big data, which has a good reference significance for the distributed analysis of large-scale complex networks.

References

  1. Z. Chen, J. Wu, Y. Xia, and X. Zhang. 2017. Robustness of interdependent power grids and communication networks: A complex network perspective. IEEE Transactions on Circuits and Systems II: Express Briefs 65, 1 (2017), 115—119.Google ScholarGoogle ScholarCross RefCross Ref
  2. W. Guan, X. Wen, L. Wang, Z. Lu, and Y. Shen. 2018. A service-oriented deployment policy of end-to-end network slicing based on complex network theory. IEEE Access 6 (2018), 19691–19701.Google ScholarGoogle ScholarCross RefCross Ref
  3. B. Liu, Z. Li, X. Chen, Y. Huang, and X. Liu. 2017. Recognition and vulnerability analysis of key nodes in power grid based on complex network centrality. IEEE Transactions on Circuits and Systems II: Express Briefs 65, 3 (2017), 346–350.Google ScholarGoogle ScholarCross RefCross Ref
  4. X. Wei, S. Gao, T. Huang, E. Bompard, R. Pi, and T. Wang. 2018. Complex network-based cascading faults graph for the analysis of transmission network vulnerability. IEEE Transactions on Industrial Informatics 15, 3 (2018), 1265–1276.Google ScholarGoogle ScholarCross RefCross Ref
  5. J. V. Milanović, and W. Zhu. 2017. Modeling of interconnected critical infrastructure systems using complex network theory. IEEE Transactions on Smart Grid 9, 5 (2017), 4637–4648.Google ScholarGoogle ScholarCross RefCross Ref
  6. N. Martin, P. Frasca, and C. Canudas-de-Wit. 2018. Large-scale network reduction towards scale-free structure. IEEE Transactions on Network Science and Engineering 6, 4 (2018), 711–723.Google ScholarGoogle ScholarCross RefCross Ref
  7. M. Herlihy. 2019. Blockchains from a distributed computing perspective. Communications of the ACM 62, 2 (2019), 78–85. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. V. Kremen, B. H. Brinkmann, I. Kim, and H. Guragain et al. 2018. Integrating brain implants with local and distributed computing devices: a next generation epilepsy management system. IEEE Journal of Translational Engineering in Health and Medicine 6 (2018), 1–12.Google ScholarGoogle ScholarCross RefCross Ref
  9. K. Yang, Y. Shi, and Z. Ding. 2019. Data shuffling in wireless distributed computing via low-rank optimization. IEEE Transactions on Signal Processing 67, 12 (2019), 3087–3099. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. J. Du, L. Zhao, J. Feng, and X. Chu. 2018. Computation offloading and resource allocation in mixed fog/cloud computing systems with min-max fairness guarantee. IEEE Transactions on Communications 66, 4 (2018), 1594–1608.Google ScholarGoogle ScholarCross RefCross Ref
  11. A. J. Ferrer, J. M. Marquès, and J. Jorba. 2019. Towards the decentralised cloud: Survey on approaches and challenges for mobile, ad hoc, and edge computing. ACM Computing Surveys (CSUR) 51, 6 (2019), 1–36. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. H. J. Newberg, S. Shelton, J. Weiss, J. Bauer et al. 2019. Reconstructing the orphan stream progenitor with milkyway@ home volunteer computing. AAS 233 (2019), 129–06.Google ScholarGoogle Scholar
  13. P. Jenviriyakul, G. Chalumporn, T. Achalakul, F. Costa, K. Akkarajitsakul. 2019. ALICE Connex: A volunteer computing platform for the time-of-flight calibration of the ALICE experiment. An opportunistic use of CPU cycles on Android devices. Future Generation Computer Systems 94 (2019), 510–523.Google ScholarGoogle ScholarCross RefCross Ref
  14. G. Zhang, F. Shen, N. Chen, P. Zhu, X. Dai, and Y. Yang. 2018. DOTS: delay-optimal task scheduling among voluntary nodes in fog networks. IEEE Internet of Things Journal 6, 2 (2018), 3533–3544.Google ScholarGoogle ScholarCross RefCross Ref
  15. S. S. Parkhomenko and T. M. Ledeneva. 2019. Scheduling in volunteer computing networks, based on neural network prediction of the job execution time. International Journal of Parallel, Emergent and Distributed Systems 34, 4 (2019), 430–447.Google ScholarGoogle ScholarCross RefCross Ref
  16. J. Á. Morell, A. Camero, and E. Alba. 2019. JSDoop and tensorflow. js: Volunteer distributed web browser-based neural network training. IEEE Access 7 (2019), 158671–158684.Google ScholarGoogle ScholarCross RefCross Ref
  17. E. Kijsipongse, A. Piyatumrong, and U. Suriya. 2018. A hybrid GPU cluster and volunteer computing platform for scalable deep learning. The Journal of Supercomputing 74, 7 (2018), 3236–3263. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. T. M. Mengistu and D. Che. 2019. Survey and taxonomy of volunteer computing. ACM Computing Surveys (CSUR) 52, 3 (2019), 1–35. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. M. C. Kasapbaşi. 2019. A new chaotic image steganography technique based on Huffman compression of Turkish texts and fractal encryption with post-quantum security. IEEE Access 7 (2019), 148495–148510.Google ScholarGoogle ScholarCross RefCross Ref
  20. J. Jin, J. Luo, Y. Li, and R. Xiong. 2018. COAST: A cooperative storage framework for mobile transparent computing using device-to-device data sharing. IEEE Network 32, 1 (2018), 133–139.Google ScholarGoogle ScholarCross RefCross Ref
  21. R. Mickey, J. Long, P. S. Dalyander, N. Plant, and D. Thompson. 2018. A framework for modeling scenario-based barrier island storm impacts. Coastal Engineering 138 (2018), 98–112.Google ScholarGoogle ScholarCross RefCross Ref
  22. Y. Mo, S. Dasgupta, and J. Beal. 2019. Robustness of the Adaptive Bellman–Ford algorithm: Global stability and ultimate bounds. IEEE Transactions on Automatic Control 64, 10 (2019), 4121–4136.Google ScholarGoogle ScholarCross RefCross Ref
  23. I. Szczesniak, A. Jajszczyk, and B. Wozna-Szczesniak. 2019. Generic Dijkstra for optical networks. IEEE/OSA Journal of Optical Communications and Networking 11, 11 (2019), 568–577.Google ScholarGoogle ScholarCross RefCross Ref
  24. Y. Qiao, J. Yang, Q. Zhang, J. Xi, and L. Kong. 2019. Multi-UAV cooperative patrol task planning novel method based on improved PFIH algorithm. IEEE Access 7 (2019), 167621–167628.Google ScholarGoogle ScholarCross RefCross Ref
  25. J. Mu, X. Liu, and X. Yi. 2019. Simplified energy-balanced alternative-aware routing algorithm for wireless body area networks. IEEE Access 7 (2019), 108295–108303.Google ScholarGoogle ScholarCross RefCross Ref
  26. M. S. Reza, F. Sadeque, M. M. Hossain, A. M. Ghias, and V. G. Agelidis. 2019. Three-phase PLL for grid-connected power converters under both amplitude and phase unbalanced conditions. IEEE Transactions on Industrial Electronics 66, 11 (2019), 8881–8891.Google ScholarGoogle ScholarCross RefCross Ref
  27. H. Ge, R. Xia, H. Sun, Y. Yang, and M. Huang. 2019. Construction and simulation of rear-end conflicts recognition model based on improved TTC algorithm. IEEE Access 7 (2019), 134763–134771.Google ScholarGoogle ScholarCross RefCross Ref
  28. I. Rocha, J. Janssen, and N. Kalyaniwalla. 2018. Recovering the structure of random linear graphs. Linear Algebra and its Applications 557 (2018), 234–264.Google ScholarGoogle Scholar
  29. S. Ludwig, B. Le Teurnier, G. Pedrini, X. Peng, and W. Osten. 2019. Image reconstruction and enhancement by deconvolution in scatter-plate microscopy. Optics Express 27, 16 (2019), 23049–23058.Google ScholarGoogle ScholarCross RefCross Ref
  30. M. A. Shaheen, H. M. Hasanien, S. F. Mekhamer, and H. E. Talaat. 2019. Optimal power flow of power systems including distributed generation units using sunflower optimization algorithm. IEEE Access 7 (2019), 109289–109300.Google ScholarGoogle ScholarCross RefCross Ref
  31. A. Rydzewski and P. Czarnul. 2017. A distributed system for conducting chess games in parallel. Procedia Computer Science 119 (2017), 22–29. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. J. Park, H. Byun, and J. R. Lee. 2016. Bio-inspired load-balancing framework for loosely coupled heterogeneous server systems. IEEE Transactions on Computers 65, 11 (2016), 3280–3292. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. P. Yadav, I. Charalampidis, J. Cohen, J. Darlington, and F. Grey. 2018. A collaborative citizen science platform for real-time volunteer computing and games. IEEE Transactions on Computational Social Systems 5, 1 (2018), 9–19.Google ScholarGoogle ScholarCross RefCross Ref
  34. J. Panadero, J. de Armas, X. Serra, and J. M. Marquès. 2018. Multi criteria biased randomized method for resource allocation in distributed systems: Application in a volunteer computing system. Future Generation Computer Systems 82 (2018), 29–40.Google ScholarGoogle ScholarCross RefCross Ref
  35. Q. Lin, H. Song, X. Gui, X. Wang, and S. Su. 2018. A shortest path routing algorithm for unmanned aerial systems based on grid position. Journal of Network and Computer Applications 103 (2018), 215–224. Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. Z. Liu, L. Mou, and Z. Yang. 2015. Scheme for partitioning and load restoration of power grid containing distributed generation after blackout. Power System Protection and Control 43, 22 (2015), 55–61.Google ScholarGoogle Scholar

Index Terms

  1. Big Data Processing on Volunteer Computing

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in

          Full Access

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader

          HTML Format

          View this article in HTML Format .

          View HTML Format
          About Cookies On This Site

          We use cookies to ensure that we give you the best experience on our website.

          Learn more

          Got it!