Senior Data Lead - Azure Data Factory (ADF), Databricks

• *Job Summary**
• --------------

• *Key Responsibilities**

• Design, develop, and maintain scalable data pipelines using **Databricks**.
• Build and optimize ETL/ELT processes for structured and unstructured data.
• Develop solutions using **PySpark, Spark SQL, Delta Lake, and Databricks Workflows**.
• Implement data ingestion frameworks from various source systems including APIs, databases, cloud storage, and streaming platforms.
• Design and maintain **Bronze, Silver, and Gold** data layers following Medallion Architecture.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Integrate Databricks solutions with cloud platforms such as Azure, AWS, or GCP.
• Collaborate with business analysts, data scientists, architects, and stakeholders to gather requirements.
• Ensure data quality, governance, security, and compliance standards.
• Monitor and troubleshoot production data pipelines and workflows.
• Participate in code reviews and implement data engineering best practices.

• *Key Responsibilities**
• -----------------------

• *Required Qualifications**

• 7\+ years of experience in Data Engineering and Data Warehousing.
• 4\+ years of hands\-on experience with Databricks.
• Strong expertise in:
• + PySpark

+ Spark SQL

+ Python

+ SQL

+ Delta Lake

• Experience building large\-scale batch and streaming data pipelines.
• Strong understanding of data modeling and data warehousing concepts.
• Experience with cloud platforms (Azure, AWS, or GCP).
• Knowledge of CI/CD and DevOps practices.

• *Skill Requirements**
• ---------------------

• *Technical Skills**

• *Databricks \& Big Data**

• Databricks Lakehouse Platform
• Apache Spark
• PySpark
• Spark SQL
• Delta Lake
• Databricks Workflows
• Unity Catalog

• *Programming**

• Python
• SQL
• Scala (Preferred)

• *Cloud Platforms**

• Azure Databricks
• Azure Data Factory (ADF)
• AWS EMR / Glue
• Google Cloud Platform (GCP)

• *Data Storage**

• Azure Data Lake Storage (ADLS)
• Amazon S3
• Snowflake
• Synapse Analytics
• SQL Server
• Oracle

• *Data Integration**

• Kafka
• Event Hub
• REST APIs
• CDC Implementation

• *DevOps \& Monitoring**

• Git
• Azure DevOps
• Jenkins
• Terraform
• Monitoring and Logging Tools

• *Must Have Skill**
• ------------------

Azure Data Factory (ADF),Azure Databricks,Snowflake

• *Good to Have Skill**
• ---------------------

• *Other Requirements**
• ---------------------

• *Preferred Qualifications**

• Experience with Azure Data Factory (ADF), Synapse Analytics, or Snowflake.
• Knowledge of Data Governance and Unity Catalog.
• Experience building data lakes and lakehouse architectures.
• Exposure to machine learning workflows and MLOps.
• Databricks Certification:
• + Databricks Certified Data Engineer Associate

+ Databricks Certified Data Engineer Professional

Experience working in Agile/Scrum environments.