Senior Data Engineer

Role Summary

NTT DATA is seeking a highly experienced Senior Data Engineer / Technical Lead for a large\-scale banking data modernization program involving migration of approximately 2 PB of historical data from Cloudera Hadoop/Hive to Apache Iceberg.

The role requires a strong hands\-on engineer who can lead the engineering team while continuing to design, develop, troubleshoot and optimize the solution. The candidate will own technical execution of the migration framework, Spark/PySpark engineering, Iceberg implementation, data modelling/schema mapping, Cyber/TaaS integration, migration controls, reconciliation, performance tuning and technical delivery of migration waves.

This is not a coordination\-only Technical Lead role. The candidate is expected to contribute to code, conduct design/code reviews, troubleshoot complex production issues and mentor Data Engineers.

Experience

• *10\+ years of Data Engineering experience, including:**

• 5\+ years hands\-on Apache Spark/PySpark.
• Strong Hadoop/Hive experience.
• Experience leading Data Engineering teams.
• Large\-scale data migration/data modernization experience.
• Strong data modelling and source\-to\-target mapping experience.

Multi\-hundred\-TB/PB\-scale migration experience is strongly preferred.

Key Responsibilities

Technical Leadership

• Lead a team of Data Engineers through assessment, development, migration, validation and production execution.
• Translate architecture and migration standards into detailed engineering designs and reusable implementation patterns.
• Own technical estimates, engineering backlog, work allocation and technical delivery.
• Conduct design and code reviews and enforce engineering standards.
• Mentor Data Engineers and provide hands\-on technical guidance.
• Act as the primary engineering escalation point for complex migration and production issues.
• Partner with Data Architect, Technical PM, security, infrastructure and PNC technical teams.
• Lead root\-cause analysis and ensure fixes are incorporated into reusable framework patterns rather than implemented as one\-off solutions.

Hadoop\-to\-Iceberg Migration

• Lead technical execution of approximately 2 PB of Hadoop/Hive historical\-data migration to Apache Iceberg.
• Harden and extend the reusable, configuration\-driven migration framework developed during the POC.
• Analyze Hive/HDFS metadata, tables, partitions, file layouts and migration complexity.
• Implement migration strategies including bulk/CTAS, partition\-wise and chunk\-wise processing.
• Design execution approaches for small, medium, large and very\-large tables.
• Optimize parallel migration across tables and partitions.
• Implement checkpointing, restart/recovery, idempotency and safe retries.
• Lead technical execution and troubleshooting of migration waves.

Data Modelling Schema Engineering

• Analyze existing Hive data models and translate them into approved Apache Iceberg physical models.
• Develop and review conceptual/logical/physical data models where required, with primary emphasis on physical modelling for migration.
• Perform source\-to\-target data mapping and schema/data\-type mapping.
• Define and implement table structures consistent with approved Iceberg standards.
• Analyze:

o entities and relationships,

o primary/foreign\-key relationships,

o data types,

o partition keys,

o business keys,

o reference/cross\-reference structures.

• Identify schema incompatibilities and recommend remediation.
• Work with the Data Architect on partitioning, file\-layout and model\-design decisions.
• Ensure migrated models preserve required business relationships and downstream compatibility.
• Support future Bronze/Silver/Gold modelling requirements where applicable.

• *Important:** enterprise modelling standards remain owned by the Data Architect; the Senior DE/Technical Lead is responsible for detailed modelling and implementation within those standards.

Spark/PySpark Engineering Performance

• Develop and review production\-grade Python/PySpark components.
• Tune Spark jobs for PB\-scale migration.
• Diagnose Spark stages, executors, shuffle, skew, memory and I/O bottlenecks.
• Optimize table/partition/chunk parallelism.
• Tune workloads to meet migration\-throughput objectives.
• Identify bottlenecks across:

Hadoop Read → Spark → TaaS/Tokenization → Iceberg Write → Reconciliation

• Coordinate technical remediation with infrastructure/storage/network teams where bottlenecks are outside the migration framework.

Apache Iceberg Engineering

• Implement approved Iceberg table standards.
• Develop/configure Iceberg tables and Spark→Iceberg processing.
• Implement:

o partitioning,

o schema evolution,

o snapshots,

o file sizing,

o compaction,

o metadata maintenance,

o small\-file remediation.

• Troubleshoot Iceberg catalog/write/commit issues.
• Optimize Iceberg table performance and storage characteristics.

Cyber\-3 / Tokenization

• Implement PNC\-approved TaaS/Protegrity tokenization and retokenization patterns.
• Integrate tokenization into Spark migration pipelines.
• Implement batching, retry, timeout and recovery handling.
• Troubleshoot tokenization failures/performance bottlenecks.
• Measure impact of tokenization on end\-to\-end migration throughput.
• Support Cyber/tokenization validation based on PNC\-defined rules.

Migration Controls Reconciliation

* Lead implementation of PNC Data Movement Controls, including

o checkpointing,

o idempotency,

o control totals,

o restart/recovery,

o retry,

o audit logging,

o exception handling.

• Build reusable source\-to\-target reconciliation capabilities covering:

o row counts,

o schema validation,

o partition completeness,

o control totals,

o aggregates,

o hash/checksum where appropriate,

o null/duplicate validation,

o tokenization validation.

• Isolate discrepancies at table/partition level.
• Enable selective reprocessing rather than full\-table reruns.
• Produce technical evidence required for migration acceptance.

Mandatory Skills

• *Strong hands\-on expertise in:**

• Python
• PySpark
• Apache Spark / Spark SQL
• Hadoop / HDFS
• Hive / HiveQL / Hive Metastore
• SQL
• Apache Iceberg
• Data modelling
• Source\-to\-target mapping
• Parquet
• Linux
• Git
• Large\-volume ETL/data migration
• Data validation/reconciliation
• Spark performance tuning
• Distributed\-system troubleshooting.

Data Modelling

• *Strong experience with:**

• logical and physical data modelling,
• source\-to\-target mapping,
• schema/data\-type mapping,
• entity relationships,
• keys and referential relationships,
• dimensional/relational modelling concepts,
• large\-scale analytical data models.

Technical Leadership

• *Demonstrated experience:**

• leading Data Engineering teams,
• technical estimation/planning,
• detailed solution design,
• code/design reviews,
• mentoring engineers,
• production troubleshooting,
• root\-cause analysis,
• communicating technical issues with architects/customer teams.

Preferred Skills

• Apache Polaris/Nessie/Glue.
• OpenShift/Kubernetes/Spark Operator.
• S3\-compatible object storage.
• NetApp StorageGRID/ONTAP.
• TaaS/Protegrity or equivalent tokenization technology.
• Trino.
• PostgreSQL.
• Control\-M/Airflow.
• CI/CD for Spark/Python applications.
• Banking/financial\-services experience.
• PB\-scale data migration.
• Bronze/Silver/Gold modelling and implementation.
• HiveQL→Spark SQL modernization.