Design, develop, test, deploy and maintain large\-scale data pipelines using Airflow to extract insights from various sources such as Kafka and ETL processes.
Collaborate with cross\-functional teams to identify business requirements and design scalable solutions for data processing and storage on AWS Glue.
Develop high\-quality code in Python using PySpark, Spark SQL, and Kubernetes to ensure efficient data processing and deployment.
Troubleshoot complex issues related to data quality, performance optimization, and system reliability.
*Job Requirements :**
4\-7 years of experience in Data Engineering with expertise in Airflow, AWS Glue, ETL/Kafka/Data Bricks/Spark.
Strong understanding of big data technologies including Hadoop ecosystem (HDFS) and NoSQL databases like Apache Cassandra or MongoDB.
Proficiency in writing efficient code in Python using PySpark/Scala/Kotlin programming languages.
Pay: ₹800,000\.00 \- ₹1,600,000\.00 per year
Benefits
Health insurance
Leave encashment
Life insurance
Provident Fund
Work Location: Hybrid remote in Bengaluru, Karnataka