Spark Developer

• *Position**: Spark Developer

• *Requirement**: Create the parquet file Then read the parquet file and read in Ab\-initio

• *Experience**: 6\+ Years

• *Working Hours**: 3–4 hours per day (Between 7 PM IST to 1 AM IST) or (6 AM IST To 11 AM IST) \| Monday to Friday

• *Working Days:** Monday to Friday

• *Location**: Remote

• *Job Description / Task Overview**

We are looking for a Spark Developer with hands\-on experience in big data processing to implement data pipelines involving **Parquet file creation and consumption**, along with integration into **Ab Initio workflows**.

• *Key Responsibilities**

• *1\. Create Parquet File using Apache Spark**

• Develop Spark jobs (using PySpark/Scala) to ingest data from source systems (CSV, JSON, database, etc.)
• Transform and cleanse data as per business rules
• Write processed data into **Parquet format** with optimized schema and partitioning
• Ensure efficient storage using compression (Snappy preferred)

• *2\. Read Parquet File using Apache Spark**

• Load Parquet datasets into Spark DataFrames
• Perform transformations, aggregations, or validations
• Optimize read performance using partition pruning and predicate pushdown

• *3\. Read Parquet File in Ab Initio**

• Configure Ab Initio graph to ingest Parquet files generated by Spark
• Use appropriate components such as:
• Reformat
• Input File / Dataset component (via external table or conversion layer)
• Convert Parquet data into Ab Initio compatible format (e.g., DML structure)

• *Approaches:**

• Use **Parquet\-to\-CSV/Avro conversion** before ingestion
• OR leverage **custom scripts / connectors** to read Parquet directly
• Define **DML schema** matching Parquet structure

• *Required Skills**

• Strong experience in **Apache Spark (PySpark/Scala)**
• Hands\-on with **Parquet file format**
• Knowledge of **data partitioning and compression techniques**
• Experience with **Ab Initio (GDE, Co\>Operating System)**
• Understanding of **ETL pipelines and data engineering best practices**

• *Good to Have**

• Experience with **Hive / HDFS / Data Lake architectures**
• Familiarity with **Avro / ORC formats**
• Performance tuning of Spark jobs

• *Expected Outcome**

• Successfully create optimized Parquet files using Spark
• Efficiently read and process Parquet data in Spark
• Seamlessly integrate and consume Parquet data within Ab Initio workflows

Job Types: Part\-time, Freelance

Contract length: 6 months

Pay: From ₹30,000\.00 per month

Benefits

• Work from home

Experience

• Spark: 6 years (Required)
• Parquet processing: 5 years (Required)
• Ab\-initio: 5 years (Required)

Work Location: Remote