Senior Spark engineer - Python (Remote)

• *Job Description – Senior Spark Engineer (Python** )

Location: Bangalore

Employment Type: Full\-Time

Experience Required: 10\+ Years

• *About the Role**

We are looking for highly experienced **Spark experts** with deep expertise in **Spark internals and advanced Python programming skills** . This is a senior engineering role for professionals who have built and optimized large\-scale distributed data processing systems and can work hands\-on with complex Spark architectures and performance tuning.

This is an urgent hiring requirement.

• *Key Responsibilities**

Design, develop, and optimize **large\-scale distributed data processing applications using Apache Spark**

Work extensively on **Spark internals, execution engine optimization, memory management, partitioning, and performance tuning**

Develop scalable and production\-grade data pipelines using **Python and Spark**

Analyze and resolve performance bottlenecks in distributed systems

Collaborate with engineering and **architecture teams to build high\-performance data platforms**

Review, refactor, and improve existing Spark codebases for scalability and efficiency

Drive best practices around **distributed computing, coding standards, and system reliability**

• *Required Skills \& Experience**

10\+ years of overall software engineering experience

Expert\-level knowledge of Apache Spark internals

Strong hands\-on expertise in PySpark and advanced Python programming

Deep understanding of **distributed systems and big data processing**

Strong experience with **Spark optimization, DAG execution, shuffling, caching, serialization, and cluster tuning**

Experience working with **large\-scale data processing environments**

• *Strong debugging, problem\-solving, and system design skills**

Excellent communication and collaboration abilities

• *Mandatory Requirement**

Candidates must be willing to share previously written Spark code for technical evaluation and review as part of the assessment process.

• *Preferred Qualifications**

Experience with **Hadoop ecosystem and cloud platforms**

Exposure to **distributed computing frameworks and data engineering best practices**

Prior experience in **performance\-critical data platforms** is highly preferred