Abstract
In order to calculate the node big data contained in complex networks and realize the efficient calculation of complex networks, based on voluntary computing, taking ICE middleware as the communication medium, the loose coupling distributed framework DCBV based on voluntary computing is proposed. Then, the Master, Worker, and MiddleWare layers in the framework, and the development structure of a DCBV framework are designed. The task allocation and recovery strategy, message passing and communication mode, and fault tolerance processing are discussed. Finally, to calculate and verify parameters such as the average shortest path of the framework and shorten calculation time, an improved accurate shortest path algorithm, the N-SPFA algorithm, is proposed. Under different datasets, the node calculation and performance of the N-SPFA algorithm are explored. The algorithm is compared with four approximate shortest-path algorithms: Combined Link and Attribute (CLA), Lexicographic Breadth First Search (LBFS), Approximate algorithm of shortest path length based on center distance of area division (CDZ), and Hub Vertex of area and Core Expressway (HEA-CE). The results show that when the number of CPU threads is 4, the computation time of the DCBV framework is the shortest (514.63 ms). As the number of CPU cores increases, the overall computation time of the framework decreases gradually. For every 2 additional CPU cores, the number of tasks increases by 1. When the number of Worker nodes is 8 and the number of nodes is 1, the computation time of the framework is the shortest (210,979 ms), and the IO statistics data increase with the increase of Worker nodes. When the datasets are Undirected01 and Undirected02, the computation time of the N-SPFA algorithm is the shortest, which is 4520 ms and 7324 ms, respectively. However, the calculation time in the ca-condmat_undirected dataset is 175,292 ms, and the performance is slightly worse. Overall, however, the performance of the N-SPFA and SPFA algorithms is good. Therefore, the two algorithms are combined. For networks with less complexity, the computational scale coefficient of the SPFA algorithm can be set to 0.06, and for general networks, 0.2. When compared with other algorithms in different datasets, the pretreatment time, average query time, and overall query time of N-SPFA algorithm are the shortest, being 49.67 ms, 5.12 ms, and 94,720 ms, respectively. The accuracy (1.0087) and error rate (0.024) are also the best. In conclusion, voluntary computing can be applied to the processing of big data, which has a good reference significance for the distributed analysis of large-scale complex networks.
- Z. Chen, J. Wu, Y. Xia, and X. Zhang. 2017. Robustness of interdependent power grids and communication networks: A complex network perspective. IEEE Transactions on Circuits and Systems II: Express Briefs 65, 1 (2017), 115—119.Google Scholar
Cross Ref
- W. Guan, X. Wen, L. Wang, Z. Lu, and Y. Shen. 2018. A service-oriented deployment policy of end-to-end network slicing based on complex network theory. IEEE Access 6 (2018), 19691–19701.Google Scholar
Cross Ref
- B. Liu, Z. Li, X. Chen, Y. Huang, and X. Liu. 2017. Recognition and vulnerability analysis of key nodes in power grid based on complex network centrality. IEEE Transactions on Circuits and Systems II: Express Briefs 65, 3 (2017), 346–350.Google Scholar
Cross Ref
- X. Wei, S. Gao, T. Huang, E. Bompard, R. Pi, and T. Wang. 2018. Complex network-based cascading faults graph for the analysis of transmission network vulnerability. IEEE Transactions on Industrial Informatics 15, 3 (2018), 1265–1276.Google Scholar
Cross Ref
- J. V. Milanović, and W. Zhu. 2017. Modeling of interconnected critical infrastructure systems using complex network theory. IEEE Transactions on Smart Grid 9, 5 (2017), 4637–4648.Google Scholar
Cross Ref
- N. Martin, P. Frasca, and C. Canudas-de-Wit. 2018. Large-scale network reduction towards scale-free structure. IEEE Transactions on Network Science and Engineering 6, 4 (2018), 711–723.Google Scholar
Cross Ref
- M. Herlihy. 2019. Blockchains from a distributed computing perspective. Communications of the ACM 62, 2 (2019), 78–85. Google Scholar
Digital Library
- V. Kremen, B. H. Brinkmann, I. Kim, and H. Guragain et al. 2018. Integrating brain implants with local and distributed computing devices: a next generation epilepsy management system. IEEE Journal of Translational Engineering in Health and Medicine 6 (2018), 1–12.Google Scholar
Cross Ref
- K. Yang, Y. Shi, and Z. Ding. 2019. Data shuffling in wireless distributed computing via low-rank optimization. IEEE Transactions on Signal Processing 67, 12 (2019), 3087–3099. Google Scholar
Digital Library
- J. Du, L. Zhao, J. Feng, and X. Chu. 2018. Computation offloading and resource allocation in mixed fog/cloud computing systems with min-max fairness guarantee. IEEE Transactions on Communications 66, 4 (2018), 1594–1608.Google Scholar
Cross Ref
- A. J. Ferrer, J. M. Marquès, and J. Jorba. 2019. Towards the decentralised cloud: Survey on approaches and challenges for mobile, ad hoc, and edge computing. ACM Computing Surveys (CSUR) 51, 6 (2019), 1–36. Google Scholar
Digital Library
- H. J. Newberg, S. Shelton, J. Weiss, J. Bauer et al. 2019. Reconstructing the orphan stream progenitor with milkyway@ home volunteer computing. AAS 233 (2019), 129–06.Google Scholar
- P. Jenviriyakul, G. Chalumporn, T. Achalakul, F. Costa, K. Akkarajitsakul. 2019. ALICE Connex: A volunteer computing platform for the time-of-flight calibration of the ALICE experiment. An opportunistic use of CPU cycles on Android devices. Future Generation Computer Systems 94 (2019), 510–523.Google Scholar
Cross Ref
- G. Zhang, F. Shen, N. Chen, P. Zhu, X. Dai, and Y. Yang. 2018. DOTS: delay-optimal task scheduling among voluntary nodes in fog networks. IEEE Internet of Things Journal 6, 2 (2018), 3533–3544.Google Scholar
Cross Ref
- S. S. Parkhomenko and T. M. Ledeneva. 2019. Scheduling in volunteer computing networks, based on neural network prediction of the job execution time. International Journal of Parallel, Emergent and Distributed Systems 34, 4 (2019), 430–447.Google Scholar
Cross Ref
- J. Á. Morell, A. Camero, and E. Alba. 2019. JSDoop and tensorflow. js: Volunteer distributed web browser-based neural network training. IEEE Access 7 (2019), 158671–158684.Google Scholar
Cross Ref
- E. Kijsipongse, A. Piyatumrong, and U. Suriya. 2018. A hybrid GPU cluster and volunteer computing platform for scalable deep learning. The Journal of Supercomputing 74, 7 (2018), 3236–3263. Google Scholar
Digital Library
- T. M. Mengistu and D. Che. 2019. Survey and taxonomy of volunteer computing. ACM Computing Surveys (CSUR) 52, 3 (2019), 1–35. Google Scholar
Digital Library
- M. C. Kasapbaşi. 2019. A new chaotic image steganography technique based on Huffman compression of Turkish texts and fractal encryption with post-quantum security. IEEE Access 7 (2019), 148495–148510.Google Scholar
Cross Ref
- J. Jin, J. Luo, Y. Li, and R. Xiong. 2018. COAST: A cooperative storage framework for mobile transparent computing using device-to-device data sharing. IEEE Network 32, 1 (2018), 133–139.Google Scholar
Cross Ref
- R. Mickey, J. Long, P. S. Dalyander, N. Plant, and D. Thompson. 2018. A framework for modeling scenario-based barrier island storm impacts. Coastal Engineering 138 (2018), 98–112.Google Scholar
Cross Ref
- Y. Mo, S. Dasgupta, and J. Beal. 2019. Robustness of the Adaptive Bellman–Ford algorithm: Global stability and ultimate bounds. IEEE Transactions on Automatic Control 64, 10 (2019), 4121–4136.Google Scholar
Cross Ref
- I. Szczesniak, A. Jajszczyk, and B. Wozna-Szczesniak. 2019. Generic Dijkstra for optical networks. IEEE/OSA Journal of Optical Communications and Networking 11, 11 (2019), 568–577.Google Scholar
Cross Ref
- Y. Qiao, J. Yang, Q. Zhang, J. Xi, and L. Kong. 2019. Multi-UAV cooperative patrol task planning novel method based on improved PFIH algorithm. IEEE Access 7 (2019), 167621–167628.Google Scholar
Cross Ref
- J. Mu, X. Liu, and X. Yi. 2019. Simplified energy-balanced alternative-aware routing algorithm for wireless body area networks. IEEE Access 7 (2019), 108295–108303.Google Scholar
Cross Ref
- M. S. Reza, F. Sadeque, M. M. Hossain, A. M. Ghias, and V. G. Agelidis. 2019. Three-phase PLL for grid-connected power converters under both amplitude and phase unbalanced conditions. IEEE Transactions on Industrial Electronics 66, 11 (2019), 8881–8891.Google Scholar
Cross Ref
- H. Ge, R. Xia, H. Sun, Y. Yang, and M. Huang. 2019. Construction and simulation of rear-end conflicts recognition model based on improved TTC algorithm. IEEE Access 7 (2019), 134763–134771.Google Scholar
Cross Ref
- I. Rocha, J. Janssen, and N. Kalyaniwalla. 2018. Recovering the structure of random linear graphs. Linear Algebra and its Applications 557 (2018), 234–264.Google Scholar
- S. Ludwig, B. Le Teurnier, G. Pedrini, X. Peng, and W. Osten. 2019. Image reconstruction and enhancement by deconvolution in scatter-plate microscopy. Optics Express 27, 16 (2019), 23049–23058.Google Scholar
Cross Ref
- M. A. Shaheen, H. M. Hasanien, S. F. Mekhamer, and H. E. Talaat. 2019. Optimal power flow of power systems including distributed generation units using sunflower optimization algorithm. IEEE Access 7 (2019), 109289–109300.Google Scholar
Cross Ref
- A. Rydzewski and P. Czarnul. 2017. A distributed system for conducting chess games in parallel. Procedia Computer Science 119 (2017), 22–29. Google Scholar
Digital Library
- J. Park, H. Byun, and J. R. Lee. 2016. Bio-inspired load-balancing framework for loosely coupled heterogeneous server systems. IEEE Transactions on Computers 65, 11 (2016), 3280–3292. Google Scholar
Digital Library
- P. Yadav, I. Charalampidis, J. Cohen, J. Darlington, and F. Grey. 2018. A collaborative citizen science platform for real-time volunteer computing and games. IEEE Transactions on Computational Social Systems 5, 1 (2018), 9–19.Google Scholar
Cross Ref
- J. Panadero, J. de Armas, X. Serra, and J. M. Marquès. 2018. Multi criteria biased randomized method for resource allocation in distributed systems: Application in a volunteer computing system. Future Generation Computer Systems 82 (2018), 29–40.Google Scholar
Cross Ref
- Q. Lin, H. Song, X. Gui, X. Wang, and S. Su. 2018. A shortest path routing algorithm for unmanned aerial systems based on grid position. Journal of Network and Computer Applications 103 (2018), 215–224. Google Scholar
Digital Library
- Z. Liu, L. Mou, and Z. Yang. 2015. Scheme for partitioning and load restoration of power grid containing distributed generation after blackout. Power System Protection and Control 43, 22 (2015), 55–61.Google Scholar
Index Terms
Big Data Processing on Volunteer Computing
Recommendations
Accelerating the [email protected] volunteer computing project with GPUs
PPAM'09: Proceedings of the 8th international conference on Parallel processing and applied mathematics: Part IGeneral-Purpose computing on Graphics Processing Units (GPGPU) is an emerging field of research which allows software developers to utilize the significant amount of computing resources GPUs provide for a wider range of applications. While traditional ...
Scaling up MapReduce-based Big Data Processing on Multi-GPU systems
MapReduce is a popular data-parallel processing model encompassed with recent advances in computing technology and has been widely exploited for large-scale data analysis. The high demand on MapReduce has stimulated the investigation of MapReduce ...
Bridging the Semantic Gaps of GPU Acceleration for Scale-out CNN-based Big Data Processing: Think Big, See Small
PACT '16: Proceedings of the 2016 International Conference on Parallel Architectures and CompilationConvolutional Neural Networks (CNNs) have substantially advanced the state-of-the-art accuracies of object recognition, which is the core function of a myriad of modern multimedia processing techniques such as image/video processing, speech recognition, ...






Comments