Comparative Analysis of Energy-Efficient Scheduling Algorithms for Big Data Applications
ABSTRACT:
Nowadays, big data analytics has been widely applied in addressing the growing cybercrime threats. However, energy consumption is explosive increasing with the fast growth of big data processing in anti-cybercrime. In this paper, an energy-efficient framework for big data applications is proposed to reduce energy consumption while satisfying deadline constrains. First, the problem of energy-efficient tasks scheduling of a single Spark job is modeled as an Integer Program. We design an energy-efficient tasks scheduling algorithm to minimize the energy consumption for big data application in Spark. To avoid SLA violations for execution time, we propose an optimal task scheduling algorithm with deadline constrains by trading off execution time and energy consumption. Experiments on a Spark cluster are performed to determine the energy consumption and execution time for several workloads from the HiBench benchmark suite. Our algorithms consume less energy on average than FIFO and FAIR under deadlines. The optimal algorithm is able to find near optimal tasks schedules to trade off energy consumed and response time benefit in small shuffle partitions.
Existing system:
The existing studies on Spark scheduling focused on minimizing the time between the arrival and the completion time of a big data application [6] [8] [9] [10]. To increase performance and reduce costs, Muhammed et al. [6] proposed a resource allocation framework of Spark to provide fine-grained resource allocation for any type of big data applications. Resource waste occurs while a big data application runs in all the nodes in a Spark cluster
Proposed System:
We design tasks scheduling algorithms that optimize the energy efficiency of running big data application in Spark clusters, while satisfying the deadline constrains. In this algorithm, based on an energy consumption model for Spark, a strategy table for the relationship between task and executor is designed to reduce the energy consumption for spark applications. Compared with original scheduling algorithms of Spark FIFO and FAIR, Our algorithm can satisfy the SLA by trading off the execution time and energy consumption. It also effectively reduces the total energy consumption of Spark application under deadline constrains.
CONCLUSION:
Spark as a unified engine for big data processing has been widely used in anti-cybercrime. While big data applications are executed on large Spark clusters, energy consumption is a critical concern in data centers. Energy-Efficient tasks scheduling under deadline constraint has become a key problem for Spark. In this paper, we design two energy-efficient tasks scheduling algorithms in Spark for big data applications. Our objective is to understand the tradeoff between energy consumption and execution time for several different kinds of workload in benchmarks such as Sort, TeraSort, and PageRank. This work provides some insight on the relationship between energy consumption and SLA guaranteeing.
REFERENCES:
[1] P. Dhaka and R. Johari, “CRIB: Cyber crime investigation, data archival and analysis using big data tool,” in Computing, Communication and Automation (ICCCA), 2016 International Conference on, Noida, India, 2016, pp. 117–121.
[2] O. M. Adedayo, “Big data and digital forensics,” in Cybercrime and Computer Forensic (ICCCF), IEEE International Conference on, Vancouver, BC, Canada, 2016, pp. 1–7.
[3] A. Shalaginov, J. W. Johnsen, and K. Franke, “Cyber crime investigations in the era of big data,” in Big Data (Big Data), 2017 IEEE International Conference on, Boston, MA, USA, 2017, pp. 3672–3676.
[4] M. Zaharia, R. S.Xin, and P. Wendell et al., “Apache spark: a unified engine for big data processing,” Communications of the ACM, vol. 59, no. 11, pp. 56–65, Nov. 2016, DOI. 10.1145/2934664.
[5] M. Zaharia, M. Chowdhury, and T. Das et al., “Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing,” in Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation, San jose, CA, USA, 2012, pp. 2–2.
[6] M. T. Islam, S. Karunasekera, and R. Buyya, “dSpark: Deadline-based Resource Allocation for Big Data Applications in Apache Spark,” in eScience (e-Science), 2017 IEEE 13th International Conference on, Auckland, New Zealand, 2017, pp. 89–98.
[7] A. Z. Zhang, “Scheduler module explain,” in Spark internals: design and implement principle of Spark core, 1th ed. Beijing, China: China Machine Press, 2015, ch. 4, sec. 3, pp. 57–72.
[8] J. Chen, K. Li, and Z. Tang et al., “A parallel random forest algorithm for big data in a Spark cloud computing environment,” IEEE Transactions on Parallel and Distributed Systems, vol. 28, no. 4, pp. 919–933, Apr. 2017, DOI. 10.1109/TPDS.2016.2603511.
[9] H. Yang, X. Liu, and S. Chen et al., “Improving Spark performance with MPTE in heterogeneous environments,” in Audio, Language and Image Processing (ICALIP), 2016 International Conference on, Shanghai, China, 2016, pp. 28–33.
[10] A. Gounaris, G. Kougka, and R. Tous et al., “Dynamic configuration of partitioning in spark applications,” IEEE Transactions on Parallel and Distributed Systems, vol. 28, no. 7, pp. 1891–1904, Jul. 2017, DOI. 10.1109/TPDS.2017.2647939.
[11] J. Huang, Y. Zhou, and Q. Duan et al., “Semantic Web Service Composition in Big Data Environment,” in GLOBECOM 2017 – 2017 IEEE Global Communications Conference, Singapore, Singapore, 2017, pp. 1-7.
[12] J. Huang, J. Zou, and C. Xing, “Competitions a Service Providers in Cloud Computing: A New Economic Model,” IEEE Transactions on Network and Service Management, PP. 1–12, Apr. 2018, DOI. 10.1109/TNSM.2018.2825358.
[13] J. Huang, Q. Duan, and S. Guo et al., “Converged NetworkCloud Service Composition with End-to-End Performance Guarantee,” IEEE Transactions on Cloud Computing, pp. 1–15, Oct. 2015, DOI. 10.1109/TCC.2015.2491939.
[14] R. Buyya, C. S. Yeo, and S. Venugopal et al., “Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility,” uture Generation computer systems, vol. 25, no. 6, pp. 599-616, Jun. 2009, DOI. 10.1016/j.future.2008.12.001.
[15] S. Srikantaiah, A. Kansal, and F. Zhao, “Energy Aware Consolidation for Cloud Computing,” in Proceedings of the 2008 conference on Power aware computing and systems, San Diego, CA, USA, 2008, pp. 10-10.