Lead Cloud Data Engineer

Job Description: We are looking for an accomplished Lead Data Engineer to drive the architectural design, development, and optimization of our enterprise\-scale data ecosystem. In this senior role, you will spearhead the build\-out of high ‑ performance data pipelines, enforce rigorous data quality and validation frameworks, and ensure end ‑ to ‑ end reliability, scalability, and integrity of data flows across the organization. You will play a pivotal role in shaping our data engineering strategy, enabling advanced analytics and mission ‑ critical, data ‑ driven decision\-making within our insurance\-focused business domains.

• Responsibilities: Lead the design and implementation of scalable data architectures on Azure, leveraging Databricks, Delta Lake, and related Azure data services.
• Build and optimize high ‑ volume ETL/ELT pipelines using PySpark, SQL, Databricks Workflows, and Delta Live Tables.
• Define and enforce best practices for data engineering across notebooks, jobs, CI/CD, Unity Catalog, security, and workspace governance.
• Integrate and orchestrate data pipelines using Azure Data Factory, Azure Synapse pipelines, or Azure Databricks Workflows.
• Drive performance tuning for Spark jobs, cluster configurations, and data storage layers to balance speed and cost efficiency.
• Implement robust data quality and validation frameworks using tools like Great Expectations, DLT expectations, or custom PySpark checks.
• Ensure compliance, governance, and lineage tracking through Unity Catalog, Purview integration, and RBAC/ABAC policies.
• Architect end ‑ to ‑ end data solutions supporting analytics, ML, actuarial models, and business\-critical reporting workloads.
• Evaluate new Databricks features, MLflow enhancements, Photon execution, serverless compute, and recommend adoption strategies.
• Familiar with working on Agile methodologies \- scrum, sprint planning, backlog refinement etc.

• Qualifications: 8\-12 years experience on Data Engineering role working with Databricks \& Azure Cloud technologies.
• Bachelor’s degree in computer science, Information Technology, or related field.
• Strong proficiency in PySpark, Python, SQL.
• Strong experience in data modeling, ETL/ELT pipeline development, and automation
• Hands\-on experience with performance tuning of data pipelines and workflows
• Proficient in working on Azure cloud components Azure Data Factory, Azure DataBricks, Azure Data Lake etc.
• Experience with data modeling, ETL processes, Delta Lake and data warehousing.
• Experience on Delta Live Tables, Autoloader \& Unity Catalog.
• Preferred \- Knowledge of the insurance industry and its data requirements.
• Strong analytical skills with the ability to collect, organize, analyze, and disseminate significant amounts of information with attention to detail and accuracy.
• Excellent communication and problem\-solving skills to work effectively with diverse teams
• Excellent problem\-solving skills and ability to work under tight deadlines.