The ideal candidate should be able to handle the complete architecture, Solutioning and handle client calls effectively.
Drive end\-to\-end technical solutions, ensuring performance, scalability, and reliability.
Establish coding standards, best practices, and development guidelines.
Conduct code reviews, provide technical mentorship, and resolve complex technical issues.
Architect and implement large\-scale data ingestion, transformation, and storage layers.
Design Lakehouse architecture using Delta Lake, Unity Catalog, Medallion model.
Ensure data quality, lineage, and governance across data assets.
Collaborate with cloud architects to integrate platform services across Azure and AWS.
Develop optimized Spark jobs (PySpark/Scala) with strong performance tuning practices.
Implement ETL/ELT pipelines using Databricks notebooks, Workflows, and Glue Jobs.
Integrate data from multiple sources into ADLS / S3 with appropriate security and metadata management.
Build CI/CD pipelines using Azure DevOps / GitHub Actions.
Define IAM roles, managed identities, ACLs, and storage permissions for secure operations.
Work with Terraform/CloudFormation for infrastructure automation (preferred).
Monitor cloud resource utilization and optimize cost.
Interact with stakeholders, analysts, architects, and business teams to translate requirements into technical designs.
Present architectural recommendations and solution proposals to leadership.
Coordinate with DevOps and Cloud Engineering for deployment and operational readiness.
10\+ years of experience in Data Engineering, Big Data, or Cloud Data Platforms
8\+ years of hands\-on experience with Databricks (PySpark/Scala, Delta Lake, Lakehouse architecture)
8\+ years of experience with Databricks (RBAC, ACLs, hierarchical namespaces)
6\+ years of experience designing data ingestion and transformation pipelines using Spark\-based frameworks.
4\+ years of experience in AWS Glue (Glue ETL jobs, Workflows, Crawlers, Glue Catalog).
4\+ years of experience in athena, spark and python scripts.
Experience with version control systems with GitHub.
Exposure to GitHub pipelines.
Knowledge of performance optimization techniques.
Support the overall development and unit testing, system testing, and deployment of the solution
Ensure smooth handover to operations and support teams.
Ensure proper integration with existing systems, data sources, and third\-party services.
Sen. Mobile App Tester
Testvox · Mumbai
GenAI / AI-ML Engineer
Premier IT Solutions · Ghaziabad
Network SME (Subject Matter Expert)
TalentNest Solutions · Mumbai