Spark Developer

• *Role Overview**

We are looking for a skilled **Spark Developer** with 4–6 years of experience in building, optimizing, and maintaining large\-scale data processing pipelines. The ideal candidate must have strong hands\-on experience in **Apache Spark**, **Spark performance optimization**, **AWS cloud services**, **SQL**, and **Hive**.

This role requires someone who can work with large datasets, design scalable ETL pipelines, tune Spark jobs for performance, and collaborate with data engineering and analytics teams.

• *Key Responsibilities**

• Design, develop, and maintain scalable data pipelines using **Apache Spark**.
• Perform **Spark job optimization**, including memory tuning, partitioning, caching, broadcast joins, shuffle optimization, and query performance improvement.
• Build and manage ETL workflows for large\-scale structured and semi\-structured data.
• Work extensively with **AWS services** such as S3, EMR, Glue, Athena, Redshift, Lambda, IAM, and CloudWatch.
• Write complex and optimized **SQL queries** for data extraction, transformation, and validation.
• Develop and optimize **Hive queries**, tables, partitions, and data models.
• Analyze performance bottlenecks in Spark applications and recommend improvements.
• Work with data stored in formats such as Parquet, ORC, Avro, JSON, and CSV.
• Collaborate with business analysts, data architects, and QA teams to understand requirements and deliver high\-quality data solutions.
• Implement data quality checks, monitoring, logging, and error\-handling mechanisms.
• Participate in code reviews, deployment activities, and production support.

• *Mandatory Skills**

• Strong hands\-on experience in **Apache Spark** using PySpark, Scala, or Java.
• Must\-have expertise in **Spark optimization and performance tuning**.
• Strong experience with **AWS cloud\-based data engineering services**.
• Proficiency in **SQL** and query optimization.
• Hands\-on experience with **Hive**, HiveQL, partitioning, bucketing, and performance tuning.
• Good understanding of distributed computing concepts.
• Experience working with large datasets and big data processing frameworks.
• Strong debugging, analytical, and problem\-solving skills.

Pay: ₹600,000\.00 \- ₹900,000\.00 per year

Benefits

• Flexible schedule
• Provident Fund

Work Location: In person