Software Engineer

Job ID

66753

Category

Enterprise Technology

Location

India

Work Type

On\-site

• *Description** As a GCP Data Engineer, you will be instrumental in modernizing our data infrastructure by building robust, automated ETL/ELT pipelines and leveraging Google Cloud Platform's powerful suite of big data tools. Collaborating closely with Enterprise Architects, you will spearhead the migration from legacy Hadoop/CDP systems to BigQuery. Your daily focus will involve architecting historical and incremental data loads, orchestrating tasks with Apache Airflow, and creating scalable, data\-driven solutions that empower the organization with clean, accessible, and organized data.

• *Responsibilities**

• **Pipeline Development:** Develop and maintain efficient ETL/ELT pipelines, implementing and automating tasks using Apache Airflow.
• **Data Architecture:** Architect historical and incremental data loads, continuously evaluating and refining the data architecture to ensure optimal performance.
• **Data Warehousing:** Support, organize, and optimize data within Google BigQuery data warehouses.
• **System Migration:** Work alongside Enterprise Architects to lead and deliver the migration of data from legacy Hadoop/Cloudera Data Platform (CDP) systems to BigQuery.
• **Enterprise Solutions:** Design and build end\-to\-end, GCP data\-driven solutions for enterprise data warehouses and data lakes.

• *Must Have:**

• **Certification:** Professional GCP Data Engineer Certification or equivalent.
• **Development Experience:** 2\+ years of coding experience in Java/Python and Infrastructure as Code using Terraform.
• **GCP Expertise:** 2\+ years of experience working in GCP\-based Big Data deployments (both Batch and Real\-Time), specifically leveraging BigQuery, Bigtable, Google Cloud Storage, Pub/Sub, Data Fusion, Dataflow, Dataproc, and Airflow.
• **Data Design:** Proven history of designing and delivering comprehensive data lake and data warehousing solutions.
• **Data Processing:** Strong hands\-on experience in extracting, loading, transforming (ETL/ELT), cleaning, and validating data, as well as designing pipelines and architectures for data processing.
• **Database Skills:** Proficiency in at least one SQL language.
• **Big Data Tools:** Practical experience working with Spark services.
• **Methodologies:** Experience working within Agile and Lean methodologies.

• *Good to Have (Preferred):**

• **Data Visualization:** Experience using visualization tools such as Qlik or Looker Studio.
• **Migration Experience:** Proven experience in migrating legacy systems (preferably Big Data Cloudera Data Platform) into GCP technologies.
• **Large\-Scale Systems:** Experience working with either a MapReduce or an MPP (Massively Parallel Processing) system at any size or scale.