Evaluation of Multiple Apache Spark Applications using Kubernetes as a Cluster manager on Google Cloud

作     者：M. Jayanthi K. Ram Mohan Rao Vuppala Sukanya 

作者机构：Assistant Professor Dept of Computer Science and Informatics Mahatma Gandhi University Nalgonda Professor Dept of Information Technology Vasavi College of Engineering Hyderabad Assistant Professor Dept of CSE OU Hyderabad 

出 版 物：《Procedia Computer Science》 

年 卷 期：2025年第252卷

页      面：576-582页

主　　题：Homogenous Apache Spark Kubernetes Google Kubernetes Engine Containerization 

摘      要：Big data processing frameworks demands for scalable and efficient cluster management. Apache Spark has emerged as prominent big data processing framework providing high-speed data processing and analytics capabilities for multiple applications. This paper explores the integration of Kubernetes as a cluster manager for Apache Spark applications leveraging its containerization capabilities to improve resource utilization and simplify deployment. In this paper the challenges of deploying spark applications on traditional cluster managers and showcase the advantages of adopting Kubernetes are analysed. The experimental evaluation demonstrates the benefits of Kubernetes as a cluster manager for Apache Spark framework. To execute the multiple Apache Spark applications on Kubernetes a homogenous cluster on Google Cloud is created by History bucket and service account. Finally multiple applications are executed on Google Kubernetes Engine. Output can be shown as the number of executor pods created with the performance metrics can be viewed. In conclusion, this paper compares the performance metrics such as job execution time and resource utilization with the different cluster.